Internationalization under Windows

[Christophe]

Most of the problems we had with internationalization of the Windows interface have been worked out. We now have a system which integrates reasonably well with the internationalization of the core and of the linux interfaces.

I'd now like to collect feedback from users of the various currently supported languages. Especialy users of non latin alphabets.

A binary is available here. Simply unpack it in your current Celestia directory, no file will be overwriten, then start celestia-i18n.exe.

If you're using a localized version of Windows, the corresponding language for Celestia should be loaded. Otherwise you can force another language by setting the LANGUAGE environment variable to the two letter code of the desired language. For example, to start Celestia in Spanish, start a command line and enter:

Code: Select all

> set LANGUAGE=es
> celestia-i18n.exe

The list of available translations and their current status is available on http://celestia.teyssier.org/i18n.

If your language uses characters not present in the WGL4 character set, you'll need to use a font texture different from the default one. If you don't have one, just post in this thread and I'll make one for you (you'll need to provide a link to a suitable TTF font).

For CJK languages, input through IME is supposed to work, but I haven't been able to test it. Feedback would be very welcome.

A new option has been added to the option dialog to switch from latin to localized constellation names.

[Sui Ota]

I appreciate the effort for internationalization of Celestia!
This is a great implement for non-ASCII languages, indeed.
Translation into Japanese using this was succeeded, but some bugs were found.

1. Garbled characters on the Solar System Browser

This is Solar System Browser in Japanese. Mercury, Venus and Earth are translated correctly, but other translated planets(Mars, Jupiter, ...) are garbled(however, their satellites are translated correctly).

2. Garbled textbox
Bookmarks window get garbled.

At first, selected object's name is shown in the textbox. If it is translated one, the string is garbled.
It seems that Celestia mistakes UTF-8 for ANSI in the textbox.

3. Problem on input support for non-ASCII characters

I checked typing non-ASCII characters on "Target Name:" method using .txf files including Japanese characters.
Through Japanese IME, I could type Hiragana, Katakana, and Kanji (Chinese character), and succeeded to
select objects by their translated name.

But some characters don't shown in it though they are included in .txf file.
(Punctuation marks, parenthesis, hyphens, mathematical symbols, etc. of Japanese. Some of them are in WGL4. These characters can shown through .cel or .celx scripts.)

About Chinese, input was succeeded through Chinese IME. Punctuation marks and some specific characters
are not shown, either.
(Characters which is not exist in Japanese are shown as "?", but this may be OS problem, not related to Celestia.)

For reference, about Greek and Russian, as long as environment variable of language is changed into each one ("el" or "ru"), input and selection of object are succeeded through Greek or Russian IME, respectively.

4. Strange strings in Eclipse Finder
When eclipse is not occurred, strange messages appear:

This happens on all languages which is available in Celestia except English.

[t00fri]

Christophe,

as you know, personally I am not a i18n fan . So how can I avoid permanently that upon startup of CVS-Celestia the canvas text is in (bad) German and the Menues etc in English? I just would love to run Celestia in plain English, for EVER . Can I set my i18n preferences in celestia.cfg?

Thanks,

Bye Fridger

[Christophe]

Aren't you using an English version of Windows?

Anyway, for the time being the solution is to set LANGUAGE=en before starting Celestia.

Or to get the menu in broken German as well, launch po/translate_resources.pl to generate the resource dlls.

I just realized that your problem was probably occuring in Linux rather than Windows (?). Setting LANGUAGE should work there too.

[ElChristou]

...is in (bad) German...

Aouch... here something should be done... I'm not saying you should do the work (you have much more to do) but if a German user is reading those lines it would be cool to fix the traduction once for all... anyone?

[t00fri]

Aren't you using an English version of Windows?

Anyway, for the time being the solution is to set LANGUAGE=en before starting Celestia.

Or to get the menu in broken German as well, launch po/translate_resources.pl to generate the resource dlls.

I just realized that your problem was probably occuring in Linux rather than Windows (?). Setting LANGUAGE should work there too.

Christophe,

no I meant Windows. In Linux the LANGUAGE variable is long set in order to prevent any internationalization attempts on my system . That means if I set LANGUAGE=en once for all via the Control Panel-> System, I should be back safely in a purely English environment?

Bye Fridger

[Christophe]

That means if I set LANGUAGE=en once for all via the Control Panel-> System, I should be back safely in a purely English environment?

Well yes, and that will also 'fix' any other application using gettext for internationalization.

That being said, I wonder how you end up with German in your Windows version of Celestia, by default you don't have a locale directory, hence no catalog file.

[t00fri]

That being said, I wonder how you end up with German in your Windows version of Celestia, by default you don't have a locale directory, hence no catalog file.

That was the real issue I actually was trying to find out from you .

I have definitely never set any LANGUAGE variable under Windows. I continuously update from CVS on my XP notebook and get since some time this deplorable mixed German/English Celestia display that I never wanted. I checked that LANGUAGE is neither set under Windows nor in Cygwin.

While I do have the English version of Windows installed, I notice that recently some commercial updates of software produce German menues etc. Well, in the Windows Control Panel ->"Regional and Language Options" Germany is set as location and for the display options of date, etc. I have no idea whether there is a cross-connection with gettext.

Bye Fridger

[Christophe]

For all its magic, gettext just can't make out the German translation out of the blue. You have to have catalogs files installed somewhere. Look for files named celestia.mo

[Christophe]

Thanks a lot for this very nice report Sui.

1. Garbled characters on the Solar System Browser
This is Solar System Browser in Japanese. Mercury, Venus and Earth are translated correctly, but other translated planets(Mars, Jupiter, ...) are garbled(however, their satellites are translated correctly).

I'm not sure what's happening here. I've made a change, but I'm not sure it fixes it, let me know.

2. Garbled textbox
Bookmarks window get garbled.
At first, selected object's name is shown in the textbox. If it is translated one, the string is garbled.
It seems that Celestia mistakes UTF-8 for ANSI in the textbox.

Yep, I forgot to switch to ANSI here.

3. Problem on input support for non-ASCII characters
I checked typing non-ASCII characters on "Target Name:" method using .txf files including Japanese characters.
Through Japanese IME, I could type Hiragana, Katakana, and Kanji (Chinese character), and succeeded to
select objects by their translated name.
But some characters don't shown in it though they are included in .txf file.
(Punctuation marks, parenthesis, hyphens, mathematical symbols, etc. of Japanese. Some of them are in WGL4. These characters can shown through .cel or .celx scripts.)

Ok, I'll need more traces here. I've made a modification to trace the input in the console. Turn the console on with ~

For each character entered you get a line which looks like this:

Code: Select all

Char input: (ANSI) 233 - UTF8 -> 195 169 [?©]

For IME input you get 'IME input' instead of 'Char input'. It gives you the values of the one or two bytes of the ANSI code, as well as the bytes of the UTF-8 translation, and then the corresponding glyph between square brakets.

I've also made a modification so that characters not found in the txf file are displayed as '?' instead of nothing.

4. Strange strings in Eclipse Finder
When eclipse is not occurred, strange messages appear:
This happens on all languages which is available in Celestia except English.

This looks like an older bug, strange things happen in 1.4.1 too. It shouldn't be too hard to fix.

The binary package has been updated.

[Sui Ota]

I tried new binary:

1. Garbled characters on the Solar System Browser

OK, this bug had gone.

2. Garbled textbox

The garble still exists...

Ok, I'll need more traces here. I've made a modification to trace the input in the console. Turn the console on with ~

For each character entered you get a line which looks like this:

Code: Select all

Char input: (ANSI) 233 - UTF8 -> 195 169 [?©]

For IME input you get 'IME input' instead of 'Char input'. It gives you the values of the one or two bytes of the ANSI code, as well as the bytes of the UTF-8 translation, and then the corresponding glyph between square brakets.

I've also made a modification so that characters not found in the txf file are displayed as '?' instead of nothing.

I typed same sentence as above. Here is the result.
And the output sentence was also same as the former post.

Another some characters through IME...

4. Strange strings in Eclipse Finder
When eclipse is not occurred, strange messages appear:
This happens on all languages which is available in Celestia except English.

This looks like an older bug, strange things happen in 1.4.1 too. It shouldn't be too hard to fix.

OK. This seems to be something what is not related internationalization.
But shown string is a part of each .po file...

[Christophe]

Hum, I was expecting an input or transcoding problem, but in fact it's also an older and known problem.

When a character is entered we check wether this is an alphanumeric character with:

Code: Select all

(iswalpha(wc) || iswdigit(wc) || iswpunct(c) || c == ' ')

The isw... functions are supposed to be locale aware, I suppose they are somewhat broken for CJK on Windows. There is also a known problem with these functions on MacOSX.

We'll have to find another solution to filter out non alphanumeric characters, maybe simply: !iswcntrl()

I'll try this tonight.

[Christophe]

I've updated the binary package, all bugs should be fixed now.

[Sui Ota]

I've updated the binary package, all bugs should be fixed now.

With this, the bugs had gone!
Strings are input or displayed correctly.
Onece again, thank you for your great work.

[Christophe]

That's great, thanks a lot for the tests and feed-back Sui.

One more thing, could you send me the font textures you're using? I think we need to distribute Celestia with fonts for all supported languages. You can send it to celestia-i18n at teyssier.org.

[Sui Ota]

That's great, thanks a lot for the tests and feed-back Sui.

One more thing, could you send me the font textures you're using? I think we need to distribute Celestia with fonts for all supported languages. You can send it to celestia-i18n at teyssier.org.

OK. I sent it.

-------------------------------------------------------------------------------------------------------------------

I found another bug:

1.
For translated-name objects, I cannot go to selected object by inputting neither translated name nor original one in Navigation -> Goto Object. Meanwhile, I can do with other (not translated) objects.
(Simply ASCII-Unicode problem? This does not occur in other language.)

2.
For translated-name objects, I cannot select object by translated name; can by only original one in Navigation -> Select Object. It's not good unable to use translated name

3.
About non-alphanumeric name of a object...

I tried to add objects with non-alphanumeric name. As I search them in Star browser or Star System Browser, their name is garbled. (With this sample, about Eclipse Finder, too.)


Each object's name is consist of WGL4 characters in this sample.

Body:

Code: Select all

# This is for test. Parameters are from Metis.
"??u215B??u215C??u215D??u215E" "Sol/Earth"
{
   Texture       "asteroid.jpg"
   Mesh      "asteroid.cms"
   Radius      21.5

   EllipticalOrbit
   {
   Period      0.294780
   SemiMajorAxis   127979
   Eccentricity   0.0012
   Inclination   0.019
   AscendingNode   263.953   #
   ArgOfPericenter 274.588   # J2000.0
   MeanAnomaly   7.649   #
   }

   RotationOffset      366

   Albedo 0.061
}


Star:

Code: Select all

"??u215B??u215C??u215D??u215E"
{
   RA 2.34669
   Dec 15.6456
   Distance 5.3697
   SpectralType "G2V"
   AbsMag 4.12587
}


And when these objects is in cel://URL, Celestia can't go to the object; no object is selected, followed(Sync orbit, Lock, Chase), or tracked. This doesn't happen always; depending on the name.

These have already existed in 1.4.1 official release, I checked.

[Christophe]

Ok, fixing 1 and 2 shouldn't be too difficult.

About 3, do these characters exist in the Japanese ANSI table? If they don't then there is no way to display them in the Windows dialogs.

The only solution would be to compile Celestia as a Unicode windows application, but that would be a lot of work.

[t00fri]

Christophe,

there was indeed a celestia.mo file in the locale/de/LC_MESSAGES directory. God knows how it went there...

Just for the fun of it: What steps are exactly necessary as of today to fully internationalize the CVS code under Windows?? There are many patches of yours floating around and it is entirely unclear to me what has been committed already and what not.

I used the latest GnuWin32 gettext (libintl.lib, libintl.h) for compilation.

If I run perl (from Cygwin)

perl translate_resources.pl
Unknown encoding 'CP1256' at translate_resources.pl line 73

I get this error. Since I am a real 'virgin' as to i18n this is it for now...

Who generates these celestia.mo files in various locale dirs? I am using VC2003.net as Win32 compiler.

Bye Fridger

[Christophe]

Just for the fun of it: What steps are exactly necessary as of today to fully internationalize the CVS code under Windows?? There are many patches of yours floating around and it is entirely unclear to me what has been committed already and what not.

I used the latest GnuWin32 gettext (libintl.lib, libintl.h) for compilation.

Well, for know the easiest is to run my binary. I'll probably commit the missing bits this week-end. Once that's done, no other step than running translate_resources.pl will be needed.

If I run perl (from Cygwin)

perl translate_resources.pl
Unknown encoding 'CP1256' at translate_resources.pl line 73

I get this error. Since I am a real 'virgin' as to i18n this is it for now...

That's a bit strange, which version of perl do you have? I use both the cygwin (5.8.7) and activestate (5.8.8) versions successfuly.

Who generates these celestia.mo files in various locale dirs? I am using VC2003.net as Win32 compiler.

translate_resources.pl does everything:
- translating the resource file (src/celestia/rc/celestia.rc -> src/celestia/rc/celestia_XX.rc)
- compiling them (src/celestia/rc/celestia_XX.rc -- rc.exe -> src/celestia/rc/celestia_XX.res -- link.exe -> locale/resXXX.dll)
- compiling the catalogs (po[2]/XX.po -- msgfmt.exe -> locale/XX/LC_MESSAGES/celestia[_constellations].mo)

The thing is that Windows only accepts ANSI character tables in rc files, hence the need to convert the UTF-8 strings comming from the po files to the language default ANSI coding when translating celestia.rc. The %lang hash gives you the Windows language Id and ANSI codepage for each ISO language code.

The Windows language Id is not really used anywhere, I could just have used the ISO code, but at some point I thought that having it would be useful. Each po file includes a translation for "WinLangID" which is the actual langID (eg "407" in de.po), this translation is then used to dynamicaly load the dll matching the gettext catalog file :

Code: Select all

    char res[255];
    sprintf(res, "locale\\res%s.dll", _("WinLangID"));
    if ((hRes = LoadLibrary(res)) == NULL) {
        cout << "Couldn't load localized resources: "<< res << "\n";
        hRes = hInstance;
    }


[Sui Ota]

About 3, do these characters exist in the Japanese ANSI table? If they don't then there is no way to display them in the Windows dialogs.

No. But using characters exist in the Japanese ANSI table, similar matter occurs. I used WGL4 characters for simple explanation.

The only solution would be to compile Celestia as a Unicode windows application, but that would be a lot of work.

OK. Then, this problem seems to be ought to put aside until the time comes.


Thank you. I'll post here when I find another bug related to internationalization.