On 26/06/2015 10:24, Sebastian Redl wrote:

...

On 6/25/2015 7:12 AM, Beman Dawes wrote:

...

It looks to me like it is high time to change the Boost guideline for C++ source file encoding to 7-bit ASCII without BOM or UTF-8 with BOM, and to change the inspect program accordingly.

Comments?

BOM is evil. The Microsoft compiler will treat files without a BOM as encoded in its local codepage, with no way to override. If you want MSVC to read

On 26.06.2015 00:15, Paul Mensonides wrote: the source as UTF-8, you need a BOM.

They have no plans to change this either, see https://connect.microsoft.com/VisualStudio/Feedback/Details/888437 . The bug is closed as wontfix.

"Unfortunately, we currently have no plans to implement the support of UTF-8 files without byte order marks."

Thus, we need a BOM in our source files if they contain UTF-8. That's just a sad fact.

This is a real issue too - I've had bug reports in the past from non-English users who were unable to compile Boost source that had accidently acquired something other than a 7-bit character. John.

On 26.06.2015 13:04, Niall Douglas wrote:

On 26 Jun 2015 at 11:24, Sebastian Redl wrote:

...

Thus, we need a BOM in our source files if they contain UTF-8. That's just a sad fact. This suggests, I suppose, that the Boost lint script will need to error out on any source files without a BOM right?

Only if they contain characters outside the basic source set, I suppose. Sebastian

On Fri, Jun 26, 2015 at 5:24 AM, Sebastian Redl < sebastian.redl@getdesigned.at> wrote:

On 26.06.2015 00:15, Paul Mensonides wrote:

...

On 6/25/2015 7:12 AM, Beman Dawes wrote:

It looks to me like it is high time to change the Boost guideline for C++

...

source file encoding to 7-bit ASCII without BOM or UTF-8 with BOM, and to change the inspect program accordingly.

Comments?

BOM is evil.

The Microsoft compiler will treat files without a BOM as encoded in its local codepage, with no way to override. If you want MSVC to read the source as UTF-8, you need a BOM.

They have no plans to change this either, see https://connect.microsoft.com/VisualStudio/Feedback/Details/888437 . The bug is closed as wontfix.

"Unfortunately, we currently have no plans to implement the support of UTF-8 files without byte order marks."

Thus, we need a BOM in our source files if they contain UTF-8. That's just a sad fact.

It isn't just Microsoft. My first draft of N3463, Portable Program Source Files http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3463.html specified UTF-8 without BOM, but when a draft was circulated several non-Microsoft compiler writers from the committee's core working group explained that to them the BOM was essential. The scenario they were concerned with was environments in Asia where the default encoding is commonly not UTF-8 and most files that go into a translation unit are encoded in that default encoding, but one file is encoded in UTF-8 without a BOM. The compiler needs to be able to identify that file as UTF-8 without all files be UTF-8 encoded, and the compiler writers believe that is not possible 100% of the time without a BOM. Some compilers or IDEs, including Visual Studio do have an opt-in option "Auto-detect UTF-8 encoding without signature", but Boost can't count on such an option being turned on. I wasn't present in core when N3463 was discussed, but the unofficial feedback I got was that CWG saw no need to explicitly state an environmental feature as a requirement that users forced required all compilers to support anyhow. --Beman

On 26 June 2015 at 01:15, Andrey Semashev wrote:

On 25.06.2015 17:12, Beman Dawes wrote:

...

It looks to me like it is high time to change the Boost guideline for C++ source file encoding to 7-bit ASCII without BOM or UTF-8 with BOM, and to change the inspect program accordingly.

Comments?

Why not just always assume UTF-8, whether there is BOM or not? I don't think UTF-8 BOM makes much sense, and I don't think editors commonly insert one.

Also, http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf says: "Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature. " Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

On 06/25/2015 04:15 PM, Andrey Semashev wrote:

On 25.06.2015 17:12, Beman Dawes wrote:

...

It looks to me like it is high time to change the Boost guideline for C++ source file encoding to 7-bit ASCII without BOM or UTF-8 with BOM, and to change the inspect program accordingly.

Comments?

Why not just always assume UTF-8, whether there is BOM or not? I don't think UTF-8 BOM makes much sense, and I don't think editors commonly insert one.

Also, while we're at it, are tabs still banned?

Tabs are still banned. -- Michael Caisse ciere consulting ciere.com