On Tue, Apr 26, 2011 at 9:27 PM, Artyom <artyomtnk@yahoo.com> wrote:
From: Mathias Gaunard <mathias.gaunard@ens-lyon.org>
On 26/04/2011 11:17, Sebastian Redl wrote:
GCC has options to control both the source (-finput-charset) and the execution character set (-fexec-charset). They both default to UTF-8. However, MSVC is more complicated. It will try to auto-detect the source character set, but while it can detect UTF-16, it will treat everything else as the system narrow encoding (usually a Windows-xxxx codepage) unless the file starts with a UTF-8-encoded BOM. The worse problem is that, except for a very new, poorly documented, and probably experimental pragma, there is *no way* to change MSVC's execution character set away from the system narrow encoding.
A long time ago, I asked Vladimir Prus to help me add an option to Boost.Build that would allow to automatically prepend the BOM to source files when using MSVC, but unfortunately he was never able to help me do this.
The problem even if the source is UTF-8 with BOM "שלום" would be encoded according to locale's 8bit codepage like 1255 or 936 and not UTF-8 string (codepage 65001).
It is rather stupid, but this is how MSVC works or understands the place of UTF-8 in this world.
Unicode and Visual Studio is just broken...
The real obstacle for localization is not a software which wasn't programmed for a localization in mind. We can replace hard coded text.(whether the source code is provided or not) It's rather tedious but straight forward task. Most program don't need a runtime language switch anyway. Hard coded text is all right. The real obstacle is ASCII. If ASCII is used instead of UTF-8, UTF-16, or UTF-32, we have to use ASCII compatible encoding. In Windows and for Japanese, it's CP932(Microsoft variant of Shift-JIS). In that case, we can't translate a program simply replacing the text. Because Windows can't tell which encoding it is(it can be anything, we can't detect it heuristically), we have to explicitly specify it. For example, an argument for every call of CreateFont API's fdwCharSet parameter must be modified to SHIFTJIS_CHARSET. In Windows, software should use UTF-16. If a locale library expect ASCII input, even though it support wchar_t output, I wonder how many people actually use it. By using ASCII input, this library encourage to use ASCII. Another obstacle for real world localization.
Artyom _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-- Ryou Ezoe