In article <cl268e$okv$1@sea.gmane.org>, "Robert Ramey" <ramey@rrsd.com> wrote:

...

I think you should spend a little more time investigating the following:

a) The "vault" files section has code by A Barbati which addresses issue related to unicode. b) Ron Garcia contributed codecvt facets for unicode that have been incorporated into boost are currently used by two boost libraries (serialization and program options.) c) asni library functions exist for converting strings and characters to/from wstrings/wchar s in accordance with the currently selected locale. Not all libraries implement these functions however.

So its not clear to me what exactly needs to be done here - other than fixing up some older stdandard libraries. I don't think that's what you had in mind.

There is a lot Unicode work to be done in the standard C++ library and boost. C++ currently has no Unicode-aware string abstraction, and this is a big

for anyone who has to deal with Unicode strings in C++ code. std::string is poorly suited for any Unicode-savvy work, for many reasons -- mainly having to do with the fact that std::string and STL and boost algorithms using std::string::iterator don't know how to handle strings in accordance with

"Miro Jurisic" <macdev@meeroh.org> wrote in message news:macdev-320EB0.01505419102004@sea.gmane.org... problem the

Unicode spec.

Hmmm - it would never occur to me to use std::string for characters wider than 8 bits. My studied this issue in some detail and concluded that one uses unicode or othe 2 ro 4 byte encoding, the simplest and most natural way is to use std::wstring (a synonym for std::basic_string<wchar_t>. At this point the only issues would be a) implementations which are not based on basic_string (I don't know if there are any of these around) b) input/output to other encoding such as utf-8 or ? - this is handled by codecvt facets. I believe that STL and boost algorithms that handle std::string can (or should) be able to handle any std::basic_string<?> . That is my basis for the view that unicode shouldn't be a big issue. Of course if one want's to handle unicode as std::string containing - say UTF-8 encoding of unicode characters - then that would be a separate issue. I don't think anyone would want to do that. I'm willing to be convinced I'm wrong about this - but I just don't see it yet. Robert Ramey

In article <cl2d2p$7a3$1@sea.gmane.org>, Vladimir Prus <ghost@cs.msu.su> wrote:

This was discussed extensively before. For example, Miro has pointed out that even plain "find" is not suitable for unicode strings because some characters can be represeted with several wchar_t values.

Then, there's an issue of proper collation. Given that Unicode can contain accents and various other "marks", it is not obvious that string::operator< will always to the right thing.

Thanks, it's comforting to know /someone/ is paying attention :-) meeroh

Robert Ramey wrote:

"Vladimir Prus" <ghost@cs.msu.su> wrote in message news:cl2d2p$7a3$1@sea.gmane.org...

...

This was discussed extensively before. For example, Miro has pointed out that even plain "find" is not suitable for unicode strings because some characters can be represeted with several wchar_t values.

Then, there's an issue of proper collation. Given that Unicode can contain accents and various other "marks", it is not obvious that string::operator< will always to the right thing.

My reference (Stroustrup, The C++ Programming language) shows the locale class containing a function

template<class Ch, class Tr, class A> // compare strings using this locale bool operator()(const basic_string<Ch, Tr, A> & const basic_string<Ch, Tr,

...

& ) const;

So I always presumed that there was a "unicode" locale that implemented this as well all other required information. Now that I think about it I realize that it was only a presumption that I never really checked. Now I wonder what facitlities do most libraries do provide for unicode facets. I know there are ansi functions for translating between multi-byte and wide character strings. I've used these functions and they did what I expected them to do. I presumed they worked in accordance with the currently selected locale and its related facets. If the basic_string<wchar_t>::operator<(...) isn't doing "the right thing" wouldn't it be just a bug in the implementation of the standard library rather than a candidate for a boost library?

The use of 'wchar_t' is purely implementation defined as what it means, other than the very little said about it in the C++ standard in relation to 'char'. It need have nothing to do with any of the Unicode encodings, or it may represent a particular Unicode encoding. This is purely up to the implementation. So doing the "right thing" is purely up to the implementer although, of course, the implementer will tell you what the wchar_t represents for that implementation.

Robert Ramey wrote:

Basically my reservations about the utility of a unicode library stem from the following:

a) the standard library has std:::basic_string<T> where T is any type char, wchar_t or whatever.

Yes. The problem with unicode is that it is not really possible to represent a character as an atomic value. A single glyph could in extreme cases be made up of 3 (or even more) 32 bit code units (UTF-32), and therefore defining a good T, is nigh on impossible.

b) all algorithms that use std::string are (or should be) applicable to std::basic_string<T> regardless of the actual type of T (more or less) c) character encodings can be classified into two types - single element types like unicode (UCS-2, UCS-4) and ascii, and multi element types like JIS, and others.

As i said, Unicode is not fixed width. Not in any encoding scheme. Therefore it is very difficult to teach the basic_string class to correctly handle unicode strings.

d) there exist ansi functions which translate strings from one type to an other based on information in the current locale. This information is dependent on the particular encoding. e) There is nothing particularly special about unicode in this scheme. Its just one more encoding scheme among many. Therefore making a special unicode library would be unnecessarily specific. Any efforts so spent would be better invested in generic encoding/decoding algorithms and/or setting up locale facts for specific encodings UTF-8, UTF-16, etc.

The reason for focusing on Unicode is that is has become the de facto standard for character representation. It is supported by most OSes and many programming languages. This is not likely to change. As for other encoding schemes. I actually had support for other encodings (like UCS, Shift JIS etc.) in the back of my mind when I wrote the implementation I described earlier. That is why the string class is called encoded_string, and not unicode_string. If the interface of the encoding_traits class is made general enough, it should be a piece of cake to add support for additional encoding schemes at a later date.

Robert Ramey wrote:

"Edward Diener" <eddielee@tropicsoft.com> wrote in message news:cl3umm$pkp$1@sea.gmane.org...

...

Robert Ramey wrote:

...

"Vladimir Prus" <ghost@cs.msu.su> wrote in message news:cl2d2p$7a3$1@sea.gmane.org...

...

This was discussed extensively before. For example, Miro has pointed out that even plain "find" is not suitable for unicode strings because some characters can be represeted with several wchar_t values.

Then, there's an issue of proper collation. Given that Unicode can contain accents and various other "marks", it is not obvious that string::operator< will always to the right thing.

My reference (Stroustrup, The C++ Programming language) shows the locale class containing a function

template<class Ch, class Tr, class A> // compare strings using this locale bool operator()(const basic_string<Ch, Tr, A> & const basic_string<Ch, Tr,

...

& ) const;

So I always presumed that there was a "unicode" locale that implemented this as well all other required information. Now that I think about it I realize that it was only a presumption that I never really checked. Now I wonder what facitlities do most libraries do provide for unicode facets. I know there are ansi functions for translating between multi-byte and wide character strings. I've used these functions and they did what I expected them to do. I presumed they worked in accordance with the currently selected locale and its related facets. If the basic_string<wchar_t>::operator<(...) isn't doing "the right thing" wouldn't it be just a bug in the implementation of the standard library rather than a candidate for a boost library?

The use of 'wchar_t' is purely implementation defined as what it means, other than the very little said about it in the C++ standard in relation to 'char'. It need have nothing to do with any of the Unicode encodings, or it may represent a particular Unicode encoding. This is purely up to the implementation. So doing the "right thing" is purely up to the implementer

OK I can buy that

...

although, of course, the implementer will tell you what the wchar_t represents for that implementation.

OK - are there standard library implementations which use other than unicode (or variants there of) for wchar_t encodings?

I do not know if there is or not. My point being that wchar_t is not a Unicode character by definition. That is why I believe that any new character types, either as future built-in characters, or as a C++ class, are needed to support Unicode encodings. As soon as one says that wchar_t should be changed depending on that locale/facet in order to support a Unicode encoding, one is doing the wrong thing and the reason why is that wchar_t is C++'s idea of an implementation defined wide character only.

Basically my reservations about the utility of a unicode library stem from the following:

a) the standard library has std:::basic_string<T> where T is any type char, wchar_t or whatever.

Agreed

b) all algorithms that use std::string are (or should be) applicable to std::basic_string<T> regardless of the actual type of T (more or less)

The standard algorithms work using iterators, and treat std::basic_string<T> as a container. That is fine but it doesn't produce results that treat a string as a meaningful collection of characters which represent a character encoding. For that the std::basic_string<T> member functions are needed and should be used.

c) character encodings can be classified into two types - single element types like unicode (UCS-2, UCS-4) and ascii, and multi element types like JIS, and others.

OK, at least in the present.

d) there exist ansi functions which translate strings from one type to an other based on information in the current locale. This information is dependent on the particular encoding.

OK, I see where you are going and you may be right. You want to continue using 'char' and 'wchar_t' but use only the locale to define their encoding and functionality. That sounds possible except for one issue of which I can think. It is that 'char' and 'wchar_t' do not encompass all the possible popular encoding sizes for fixed size encodings. Right now we have 8, 16, and 32 bits. Perhaps we need a new C++ basic character type, maybe 'lwchar_t', and the rule sizeof(char) <= sizeof(wchar_t) <= sizeof(lwchar_t). This would give us a better fighting chance to represent all the most popular encodings at least as far as fixed size character sizes are concerned. Along with your suggestion, thought would then have to be given to what a std::basic_string<T> really means beyond the use of narrow characters. Right now it is implementation defined, but in the future how and where do we specify character encodings via locales ?

e) There is nothing particularly special about unicode in this scheme. Its just one more encoding scheme among many. Therefore making a special unicode library would be unnecessarily specific. Any efforts so spent would be better invested in generic encoding/decoding algorithms and/or setting up locale facts for specific encodings UTF-8, UTF-16, etc.

You have made a good point but see above. In your scheme, I would still want to have the C++ standard library have all current functionality regarding characters and strings be templated on built-in character types everywhere. There would probably need to be a review of current and future functionality to determine in which situations locales, with their character encoding information, need to be passed along with a string.

e) There is nothing particularly special about unicode in this scheme. Its just one more encoding scheme among many. Therefore making a special unicode library would be unnecessarily specific. Any efforts so spent would be better invested in generic encoding/decoding algorithms and/or setting up locale facts for specific encodings UTF-8, UTF-16, etc.

Robert, there is a lot more to Unicode than an encoding form: Character properties. Collation. Bidirectional character handling. Character shaping algorithms. And lots more I've forgotten: do take a look at ICU, I don't like the C++ design much, but it's a *huge* and technically very competent library. John.

...

Hi. I am in the process of planning a library for handling unicode strings in C++ ... .... functionality which does not, ie exceptions and file names and locale message files, and assume that only narrow characters exist in its usage. I am still amazed that programmers from countries which would normally use wide characters as Unicode encodings, such as the Japanese, have not made more of an issue with this, but perhaps they are so used to their far more difficult DBCS roots that pursuing wide characters everywhere, much less a real Unicode encoding, is a minor issue with them.

For Japanese strings I use either Shift-JIS or UTF-8 encodings, both of which work fine with std::string for most tasks. Matching sub-strings could give a false match however. When talking about Unicode does that include UTF-8? Darren

Rogier van Dalen wrote:

An assumption I think is wrong is that wchar_t would be suitable for Unicode. Correct me if I'm wrong, but IIRC wchar_t has 16 bits on Microsoft compilers, for example. The utf8_codecvt_facet implementation will on these compilers cut off any codepoints over 0xFFFF. (U+1D12C will come out as U+D12C.)

This is because the Windows NT ABI is hardwired for 16-bit wide characters. I beleive that means the wide characters are actually UTF-16 characters that use "surrogate pairs." Regardless of whether this is a good thing or not, Windows compilers need to follow suit as the underlying implementation of their wide characters is in Windows, not in the compiler. It might be possible for a compiler to provide their own Unicode implementation, and map that to Windows' wide characters, but in the user-visible situations where the two implementations disagreed, there might be suprising results that might make the compiler-provided implementation unusable. Aaron W. LaFramboise

----- Original Message ----- From: "Rogier van Dalen" <rogiervd@gmail.com>

I've recently started on the first draft of a Unicode library.

Interesting. Is there a discussion going about this library that I have missed, or haven't you posted anything about it yet? I'd hate to start something like this, if there is already being made an effort on the subject.

An assumption I think is wrong is that wchar_t would be suitable for Unicode. Correct me if I'm wrong, but IIRC wchar_t has 16 bits on Microsoft compilers, for example. The utf8_codecvt_facet implementation will on these compilers cut off any codepoints over 0xFFFF. (U+1D12C will come out as U+D12C.)

I agree. The "unicode is wide strings" assumption is wrong in my opinion, and I would stribe to provide a correct implementation based on the Unicode standard if I were to go ahead with this.

I think a definition of unicode::code as uint32_t would be much better. Problem is, codecvt is only implemented for wchar_t and char, so it's not possible to make a Unicode codecvt without manually adding (dummy) implementations of codecvt<unicode::code,char,mbstate_t> to the std namespace. I guess this is the reason that Ron Garcia just used wchar_t.

I don't really feel locking the code unit size to 32bits is a good solution either as strings would then become unneccesarily large. In a test implementation I have recently made, I templated the entire encoding scheme (using an encoding_traits class) and made a common interface for strings that lets you iterate over the code points it controls, no matter what the underlying encoding is. (I will post another message with more details of this library.) This does of course make for problems with other parts of the standard, but solutions to these problems is what I want my thesis to be all about.

About Unicode strings: I suggest having a codepoint_string, with the string of code units as a template parameter. Its interface should work with 21 (32) bits values, while internally these are converted to UTF-8, UTF-16, or remain UTF-32. template <class CodeUnitString> class codepoint_string { CodeUnitString code_units; // ... };

The real unicode::string would be the character string, which uses a base character with its combining marks for its interface. template <class CodePointString> class string { CodePointString codepoints; // ... };

So unicode::string<unicode::codepoint_string<std::string> > would be a UTF8-encoded string that is manipulated using its characters.

unicode::string should take care of correctly searching for a character string, rather than a codepoint string.

Thanks. I will take that into consideration. I'm glad to hear any design/implementation ideas since I want this library to be useable for the largest amount of people possible.

operator< has never done "the right thing" anyway: it does not make a difference between uppercase and lowercase, for example. Probably, locales should be used for collation. The Unicode collation algorithm is pretty well specified.

Yes. I hope to be able to add support for the collation algorithm to enable proper, locale specific collation.

In article <e094f9eb041019032718d58d04@mail.gmail.com>, Rogier van Dalen <rogiervd@gmail.com> wrote:

An assumption I think is wrong is that wchar_t would be suitable for Unicode.

wchar_t is to be avoided at all costs. Its size differs from compiler to compiler, and even depends on compiler settings (2 or 4 bytes). Encoding of wchar_t strings is ill-defined and also varies from system to system (usually UCS-2 or UCS-4, but there is no guarantee it's a Unicode encoding).

So unicode::string<unicode::codepoint_string<std::string> > would be a UTF8-encoded string that is manipulated using its characters.

Encoded characters or abstract characters? (See section 2.4 of Unicode standard for definitions) meeroh

In article <e094f9eb04102006096b92c870@mail.gmail.com>, Rogier van Dalen <rogiervd@gmail.com> wrote:

...

...

So unicode::string<unicode::codepoint string<std::string> > would be a UTF8-encoded string that is manipulated using its characters.

Encoded characters or abstract characters? (See section 2.4 of Unicode standard for definitions)

I mean a base character with its combining characters. I don't think this is the same as "abstract character", is it?

That is an abstract character, yes.

My plan was to decompose all characters in unicode::string. This makes manipulation of diacritics easier. Correct me if I'm wrong, but your example of finding "ü" in a string would come down to finding the codepoint sequence "U+0075 U+0308" and checking whether it is not followed by another combining character, pretty trivial still.

You have to not only decompose them but put them in a canonical decomposed order in order for that to work. meeroh

On Wed, 20 Oct 2004 12:20:22 -0400, Miro Jurisic <macdev@meeroh.org> wrote:

In article <e094f9eb04102006096b92c870@mail.gmail.com>, Rogier van Dalen <rogiervd@gmail.com> wrote:

...

My plan was to decompose all characters in unicode::string. This makes manipulation of diacritics easier. Correct me if I'm wrong, but your example of finding "ü" in a string would come down to finding the codepoint sequence "U+0075 U+0308" and checking whether it is not followed by another combining character, pretty trivial still.

You have to not only decompose them but put them in a canonical decomposed order in order for that to work.

Yes, of course. I left it out thinking it was trivial (which it may be; you'd need a small part of the Unicode Database though). Regards, Rogier

On Wed, 20 Oct 2004 10:44:04 +0400, Vladimir Prus <ghost@cs.msu.su> wrote:

Rogier van Dalen wrote:

...

So unicode::string<unicode::codepoint_string<std::string> > would be a UTF8-encoded string that is manipulated using its characters.

After reading your post, I don't understand what exactly the two levels of template parameters give you.

Even though I think that a Unicode library should by default work on base characters with their combining characters, sometimes you may need to manipulate codepoints directly.

And, even if they are needed, it's very important that unicode::string with different template parameters are always convertible between each other. Otherwise, two libraries with different types won't be able to interoperate.

Of course. This should be fairly trivial though. Regards, Rogier

In article <cl3hl9$g4e$1@sea.gmane.org>, "Erik Wien" <wien@start.no> wrote:

The basic idea I have been working around, is to make a nencoded_string class templated on unicode encoding types (i.e. UTF-8, UTF-16). This is made possible through a encoding_traits class which contains all nececcary implementation details for working on strings of code units.

I generally agree with this design approach, but I don't think that code point iterators alone are sufficient. Iteration over encoded characters and abstract characters would be needed for some algorithms to function sensibly. For example, the simple task of: find(begin, end, "ü") needs to use abstract characters in order to be able to find precomposed and decomposed versions of ü.

You could use the encoded_string class like this:

// Constructor converts the ASCII string to UTF-16. encoded_string<utf16> some_string("Hello World"); // Run some standard algorithm on the string: std::for_each(some_string.begin(), some_string.end(), do_some_operation);

Again, taking this example, you let's say that do_some_operation performs canonicalization to some Unicode canonical form; you can't do this by iterating over code points.

I am aware that this implementation will be less that ideal for integration with the current c++ standard, but it's issues like that I would like to get deeper into during the develpoment.

You should explain what problems with integration you foresee. meeroh

In article <cl3nps$4d8$1@sea.gmane.org>, "Erik Wien" <wien@start.no> wrote:

Hi. Thanks for the feedback!

My pleasure :-)

"Miro Jurisic" <macdev@meeroh.org> wrote in message news:macdev-BACD3C.13585519102004@sea.gmane.org...

...

I generally agree with this design approach, but I don't think that code point iterators alone are sufficient.

Neither do I as the matter a fact, but this is as far as I have come right now. :) There would probably be different types of iterators (or iterator wrappers) made available to enable iterations over everything from code units to code points/abstract characters.

Yes, I agree.

...

Iteration over encoded characters and abstract characters would be needed for some algorithms to function sensibly. For example, the simple task of:

find(begin, end, "ü")

needs to use abstract characters in order to be able to find precomposed and decomposed versions of ü.

True... And this is a point where implemtation would be less than trivial.

Yeah, that's how far I got before I decided that I didn't have the time to deal with the problem given my current schedule.

...

Again, taking this example, you let's say that do_some_operation performs canonicalization to some Unicode canonical form; you can't do this by iterating over code points.

Nope. A code unit iterator would be needed for things like that.

I am pretty sure you mean abstract character here, not code unit. My understanding of the Unicode terminology is that the decomposed version of ü consists of one abstract character (ü) two encoded characters (u, š) two UTF-32 code units (0x00000075 0x00000308) two UTF-16 code units (0x0075 0x0308) three UTF-8 code units (0x75 0xCC 0x88) but perhaps I have it backwards...

The implementation described here would not pose too much of a problem, I was thinking more of the problems that arise when you take things like collation and locales into consideration. From what i understand there is a real issue in enabling proper unicode support in the standard classes like locale, ctype and collate, as they assume things that do not neccesarily apply to a unicode representation of text. A failiure to enable good support in those classes (at least locale and ctype), would also make the iostream support break, and things start to snowball. I could very well be wrong on this (Actually, I hope I am! :) ), as I haven't had the time to read up on all issues concerning this. But again, this is one of many problems I hope running this project will help reveal.

I don't know enough about locales to comment on this, unfortunately. meeroh

Erik Wien wrote:

Ultimately I feel that the operation of normalization (which involves canonical decomposition) of unicode strings should be hidden from the user completely and be performed automatically by the library where that is needed. (Like on a call to the == operator.)

It appears that there are two schools of thought when it comes to string design. One approach treats a string purely as a sequential container of values. The other tries to represent "string values" as a coherent whole. It doesn't help that in the simple case where the value_type is char the two approaches result in mostly identical semantics. My opinion is that the std::char_traits<> experiment failed and conclusively demonstrated that the "string as a value" approach is a dead end, and that practical string libraries must treat a string as a sequential container, vector<char>, vector<char16_t> and vector<char32_t> in our case. The interpretation of that sequence of integers as a concrete string value representation needs to be done by algorithms. In other words, I believe that string::operator== should always perform the per-element comparison std::equal( lhs.begin(), lhs.end(), rhs.begin() ) that is specified in the Container requirements table. If I want to test whether two sequences of char16_t's, interpreted as UTF16 Unicode strings, would represent the same string in a printed form, I should be given a dedicated function that does just that - or an equivalent. Similarly, if I want to normalize a sequence of chars that are actually UTF8, I'd call the appropriate 'normalize' function/algorithm. But I may be wrong. :-)

In article <cl4bs4$prn$1@sea.gmane.org>, "Erik Wien" <wien@start.no> wrote:

Peter Dimov wrote:

...

In other words, I believe that string::operator== should always perform the per-element comparison std::equal( lhs.begin(), lhs.end(), rhs.begin() ) that is specified in the Container requirements table.

If I want to test whether two sequences of char16_t's, interpreted as UTF16 Unicode strings, would represent the same string in a printed form, I should be given a dedicated function that does just that - or an equivalent. Similarly, if I want to normalize a sequence of chars that are actually UTF8, I'd call the appropriate 'normalize' function/algorithm.

Though I see where you are coming from, I don't agree with you on that. In my opinion a good unicode library should hide as much as possible of the complexity of the actual character representation from the user. If we were to require the user to know that a direct binary comparison of strings is not the same as a actual textual comparison, we loose some of the simplicity of the library. Most users that use such a library would not know that the character ö can be represented as both 'oš' and 'ö', and that as a consequence of that, calling == on to strings could result in the behaviour "ö" != "ö". By removing the need for such knowledge by the user, we reduce the learning curve considerably, which is one of the main reasons for abstracting this functionality anyway.

I completely agree with Erik on this. For anything except for US english, the interface of basic_string grafted on top of a sequence of UTF code points produces wrong results for most people most of the time. Unicode is hard enough as it is, we don't need to expose it via an interface whose default behavior violates the principle of least surprise most of the time. meeroh

In article <00da01c4b6a3$114a1a10$0600a8c0@pdimov>, "Peter Dimov" <pdimov@mmltd.net> wrote:

Miro Jurisic wrote:

...

I completely agree with Erik on this. For anything except for US english, the interface of basic_string grafted on top of a sequence of UTF code points produces wrong results for most people most of the time.

How did basic_string even enter the discussion?

You said string::operator== which I interpreted to mean basic_string::operator== rather than fictitious_unicode_string::operator==. meeroh

"Peter Dimov" <pdimov@mmltd.net> wrote in message news:00d501c4b6a2$f541bda0

That may be so. But I don't see how the user can be isolated from the binary representation if he needs to pick one of utf8_string, utf16_string, ucs2_string, ucs4_string to store his strings. Perhaps I misunderstand your idea. Can you post a sketch of your spec? How many string classes do you have? What encoding do they use? What do begin(), end(), size() return? Are the iterators random access? Bidirectional? Constant? How can the user obtain the underlying element sequence to persist it somewhere or to pass it to an external library?

First you need to understand that what I have so far, is just a preliminary test implementation for my own amusement. I anticipate a lot of things will change if I go forward with this project. Right now i have a single encoded_string class that has two template parameters, namely encoding and encoding_traits. encoding_traits is a class where all encoding specific implementation is kept, and this class is used to setup the encoded_string class to correctly represent strings in the given encoding. begin() and end() return a code unit iterator that has the same interface and value_type ++, no matter what the underlying encoding is. That is, you only see code points when iterating over a string, not the underlying code unit sequence. The iterators used are bidirectional, not random access (impossible on UTF-8 and UTF-16) and they are as of now not constant. It IS possible to assign a code unit to a UTF-8 encoded string through an iterator, even if the resulting code unit sequence would be longer than the one the iterator is pointing to. The underlying container is automatically resized to make room for the new sequence. (This is of course slow!)

At 03:46 PM 10/20/2004, Peter Dimov wrote:

...

The iterators used are bidirectional, not random access (impossible on UTF-8 and UTF-16) and they are as of now not constant. It IS possible to assign a code unit to a UTF-8 encoded string through an iterator, even if the resulting code unit sequence would be longer than the one the iterator is pointing to. The underlying container is automatically resized to make room for the new sequence. (This is of course slow!)

This is another basic_string mistake that effectively rules out efficient

reference counting. ;-) Just make the iterators constant. The functionality can be obtained with explicit erase/insert/replace members.

There are additional advantages to a constant iterator and explicit erase/insert/replace design; various caching and disk-residence implementations become both possible and efficient. --Beman

Eric Niebler wrote:

Erik Wien wrote:

...

The iterators used are bidirectional, not random access (impossible on UTF-8 and UTF-16)

No. Andrei Alexandrescu explained a scheme to me whereby a UTF-16 encoded string can have a random-access iterator, and I think it should. The basic idea is you keep a plain array of 16-bit integers which are the 16-bit characters and the first 16 bits of surrogate pairs. Then you have a data structure which maps from string offsets to the second 16 bits of surrogate pairs. Random access involves a simple index and a map look-up. Sequential access requires no map look-up. And since surrogate pairs are very rare, the map will almost always be empty and the look-up is skipped.

Nice! But this seems to make c_str O(N) operation. If I need to speak to a library in the common extern "C" language of interoperability, and that library happens to need UTF-16 encoded wchar_t const [], which by coincidence has the same representation as char16_t const [], I won't be very happy if The C++ string seems to ignore this common scenario.

On Wed, 20 Oct 2004 12:48:31 -0700, Eric Niebler <eric@boost-consulting.com> wrote:

I think the default should be UTF-16 encoding, and that the iterator should use a scheme like this to be random access. Rationale: there are string algorithms that benefit from random access (Boyer-Moore comes to mind).

Correct me if I'm wrong. From what I gather from a Google search, Boyer-Moore is a fast string search algorithm. Why not use the algorithm on the code units rather than codepoints? UTF-8 and UTF-16 are both not stateful, specifically to allow optimisations such as this (as well as error recovery). As was pointed out earlier in this thread, searching for Unicode characters takes looking at combining characters as well. I think this will go for many, if not all, algorithms that you can think of: either they can be made to work with code units, or they must work on abstract characters, which means a variable-width encoding anyway. (See the Unicode Standard 4, Section 2.5 for a similar argument for UTF-16 over UTF-32, even though the latter is fixed-width.) I'm ready to be proven wrong; however, at this moment at least I believe that any effort to make UTF-16 randomly accessible is not useful. Regards, Rogier

On Thu, 21 Oct 2004 12:10:28 -0700, Eric Niebler <eric@boost-consulting.com> wrote:

Rogier van Dalen wrote:

...

On Wed, 20 Oct 2004 12:48:31 -0700, Eric Niebler <eric@boost-consulting.com> wrote:

...

I think the default should be UTF-16 encoding, and that the iterator should use a scheme like this to be random access. Rationale: there are string algorithms that benefit from random access (Boyer-Moore comes to mind).

Correct me if I'm wrong. From what I gather from a Google search, Boyer-Moore is a fast string search algorithm. Why not use the algorithm on the code units rather than codepoints? UTF-8 and UTF-16 are both not stateful, specifically to allow optimisations such as this (as well as error recovery).

Searching a Unicode string for a particular bit pattern is not particularly meaningful because the same string can be represented with different bit patterns. Have I misinterpreted what you are suggesting?

If the strings are not normalised, that is correct. However, if the strings are normalised (to the same form), then the codepoint patter will be the same; and so will the code unit pattern (through the way UTF-8 and UTF-16 are specified). If you search a Unicode string for a substring using code units, you *will* find all matches, plus some bogus matches. For example, if you're looking for "can" you will match "cañ" too, in normalisation form D. To get rid of these bogus matches, all you have to do is check that no combining marks are following, and otherwise, proceed to the next match. So Boyer-Moore may be used on the code units, as long as it is checked that a match includes full abstract characters only. Hope this makes clearer what I'm trying to say. Rogier

Peter Dimov wrote:

...

Ultimately I feel that the operation of normalization (which involves canonical decomposition) of unicode strings should be hidden from the user completely and be performed automatically by the library where that is needed. (Like on a call to the == operator.)

It appears that there are two schools of thought when it comes to string design. One approach treats a string purely as a sequential container of values. The other tries to represent "string values" as a coherent whole. It doesn't help that in the simple case where the value_type is char the two approaches result in mostly identical semantics.

My opinion is that the std::char_traits<> experiment failed

I agree to that.

and conclusively demonstrated that the "string as a value" approach is a dead end,

How was it demonstrated? There are two separate questions. First, is how many operations are methods of 'string' and how many are external. Contrary to what Exception C++ says, I believe many methods in string is OK. As an example, QString presents huge but consistent interface, while in standard C++ we have string, boost::format, boost::tokenizer and boost::string_algo , and simply it's too many separate docs to look at. Second question is if operator==, operator< or 'find' should operate on vector<char_XX> or on abstract characters, using Unicode rules, or there should be two versions. I don't really understand why 'unicode-unaware' semantic is ever needed, so we should have only 'unicode-aware' one.

But I may be wrong. :-)

Me too. - Volodya

Peter Dimov wrote:

Vladimir Prus wrote:

...

Second question is if operator==, operator< or 'find' should operate on vector<char_XX> or on abstract characters, using Unicode rules, or there should be two versions. I don't really understand why 'unicode-unaware' semantic is ever needed, so we should have only 'unicode-aware' one.

Look at 21.3/2: "The class template basic_string conforms to the requirements of a Sequence, as specified in (23.1.1).

Additionally, because the iterators supported by basic_string are random access iterators (24.1.5), basic_string conforms to the the requirements of a Reversible Container, as specified in (23.1)."

Now look at Table 65, Container requirements, operator==:

"== is an equivalence relation.

a.size()==b.size() && equal(a.begin(), a.end(), b.begin())"

Yes, I know that.

The question is now, what do begin(), end() and size() return for our hypothetical string16?

First two return some hypothetical iterator. Its operator* will return either "unicode_character" (base character + all accents) or "unicode_character_ref" which will refer back to storage and extract components on demand. The performance of both version is unclear. Further, with "unicode_character_ref", the interator won't be lvalue iterator (or random access iterator in standard terms).

I maintain that the library design is much cleaner if begin(), end() and size() are random access iterators over the underlying _storage_, not over the codepoint representation or abstract character representation.

But those methods allow to work directly on storage, and I don't know if that's ever needed, or more common that working on character level. And after all, unicode_string can have 'storage()' method that gives vector<char_16>&, if direct storage manipulation is desired.

Codepoint iterators and abstract character iterators would still be provided, but they would be constant bidirectional with char32_t as the value_type.

Codepoint and abstract character operations would be provided by algorithms, taking an iterator range.

The user should remember and honor the encoding (UTF-16, UCS-2, other) of a particular container of char16_t, not the container itself.

So, vector<char16_t> could be either UTF-16 or UCS-2? I think that's a bad idea. If a library accepts unicode string, then its interface can either: - use 'unicode_string' - use 'unicode_string<some_encoding>' - use 'vector<char16_t>' and have a comment that the string is UTF8. I think the first option is best, and the last is too easy to misuse. - Volodya

Peter Dimov wrote:

Vladimir Prus wrote:

...

If a library accepts unicode string, then its interface can either: - use 'unicode_string' - use 'unicode_string<some_encoding>' - use 'vector<char16_t>' and have a comment that the string is UTF8.

I think the first option is best, and the last is too easy to misuse.

Yes.

So let's see if I understand your position correctly.

A single string class shall be used to store Unicode strings, i.e. logical sequences of Unicode abstract characters.

This string shall be stored in one chosen encoding, for example UTF-8. The user does not have direct access to the underlying storage, however, so it might be regarded as an implementation detail.

An invariant of the string is that it is always in one chosen normalized form. Iteration over the string gives back a sequence of char32_t abstract characters. Comparisons are defined in terms of these sequences.

Is this a fair summary?

Such a one-size-fits-all unicode_string is guaranteed to be inefficient for some applications. If it is always stored in a decomposed form, an XML library probably wouldn't want to use it, because it requires a composed form. And making the encoding an implementation detail makes it inefficient to use in situations where binary compatibility matters (serialization, for example). Also, it is impossible to store an abstract unicode character in char32_t because there may be N zero-width combining characters associated with it. Perhaps having a one-size-fits-all unicode_string might be a nice default, as long as users who care about encoding and canonical form have other types (template + policies?) with knobs they can twiddle. -- Eric Niebler Boost Consulting www.boost-consulting.com

Erik Wien wrote:

"Eric Niebler" <eric@boost-consulting.com> wrote in message

...

Such a one-size-fits-all unicode_string is guaranteed to be inefficient for some applications.

Yes... That's why I would like the encoding to be templated. Allowing the programmer to choose the encoding best suited for his/her needs.

It's good to have one string class for library interoperability reasons. Otherwise library A would demand utf8_string, library B would demand utf16_string, and library C would demand utf32_string. No matter which one you choose, you'll pay a price. (This doesn't change even if you spell utf8_string as string<utf8>.)

Erik Wien wrote:

...

It's good to have one string class for library interoperability reasons. Otherwise library A would demand utf8_string, library B would demand utf16_string, and library C would demand utf32_string. No matter which one you choose, you'll pay a price. (This doesn't change even if you spell utf8_string as string<utf8>.)

That is true. Though the strings of different encodings should be assignable to each other, libraries taking references to encoded_strings would need some conversion to be done.

We have a similar problem today with basic_string<char> and basic_string<wchar_t>, and I think it could also be solved in a way that is very similar to what is done in the <string> header.

Just to clarify: the string and wstring in the standard have a huge problem: you can't convert string to wstring in any way: there's just no appropriate converting constructor.

If we typedef a unicode_string or something as encoded_string<utf16>, and promote that as THE string class, most users would use that as their primary string representation, and simply be oblivious to the underlying encoding. (A good thing.)

That would still make it easy for a user to use some different encoding without good reason.

Advanced user could (just like we do today with basic_string) choose to support multiple encodings by templating their own functions on encoding as well.

Oh well. I just hope nobody will ever make an implementation of XML parser + XML Schema + XPath + XQuery + SOAP + HTML renderer which is fully templated on string type, unless the same person speeds up gcc by 10 times previously. - Volodya

On Fri, 22 Oct 2004 02:43:19 +0200, Erik Wien <wien@start.no> wrote:

"Vladimir Prus" <ghost@cs.msu.su> wrote in message news:cl7kha$tk4

...

Just to clarify: the string and wstring in the standard have a huge problem: you can't convert string to wstring in any way: there's just no appropriate converting constructor.

Correct, and that should be provided if we were to make a unicode_string templated in encoding. (Not that we neccesarily will.)

I think that if you don't know the encoding (and normalisation form) at compile time, iterators will have to be really slow. I hope you will try and find out during your research how important that is. I guess things like regular expressions would be slow, unless you provide algorithms with a possibility to switch to versions tailored to the encoding at run-time, similar to the visitor pattern boost::variant provides.

...

That would still make it easy for a user to use some different encoding without good reason.

It would be possible yes, not even difficult, but I don't think that means people will actually do that. People usually use std::string today, not basic_string<whatever>, because that is "the string class". I think a similar thing would happen with a unicode_string typedef. Especially if there is no difference in the interface between the different template versions. Then you'd have to know what the differences between the different encodings are (be an andvanced user, and aware of the drawbacks), to actually bother using anything other that UTF-16.

I fully agree. Rogier

"Rogier van Dalen" <rogiervd@gmail.com> wrote in message

...

I think the best solution is to store the string in the form it was originally recieved (decomposed or not), and instead provide composition functions or even iterator wrappers that compose on the fly. That would allow for composed strings to be used if needed (like in a XML library, but not imposing that requirement on all other users.

I don't think I can agree on that. If you do a lot of input/output, this might yield a better performance, but even in reading XML, you probably need to compare strings a lot, and if they are not normalised, this will really take a lot of processing. Correct me if I'm wrong, but a simple comparison of two non-normalized Unicode strings would take looking up the characters in the Unicode Character Database, decomposing every single character, gathering base characters and combining marks, and ordering the marks, then comparing them. And this must be done for every character. I don't have any numbers, of course, but I have this feeling it is going to be really really slow.

You are quite correct... It is slow. And that is why I am hesitant to make decomposition something that will happen every time you assign something to a string. What this really boils down to, is what kind of usage pattern is the most common? The library should be written to provide the best performance on the operations most people do.

"Ben Hutchings" <ben.hutchings@businesswebsoftware.com> wrote in message

...

You are quite correct... It is slow. And that is why I am hesitant to make decomposition something that will happen every time you assign something to a string.

What this really boils down to, is what kind of usage pattern is the most common? The library should be written to provide the best performance on the operations most people do.

How about this:

- when initialising or assigning to a string, you can opt to normalise one way or the other, or not at all - normalised strings are flagged as such - normalisation on comparison can be skipped if both strings are flagged as being normalised the same way - normalisation on assignment can be skipped if the right-hand side is flagged as being normalised appropriately

That is not a bad idea... I think... ;) Providing a "normalization scheme" as a setting in the string would be possible. I'll take that into consideration.

Such a one-size-fits-all unicode_string is guaranteed to be inefficient for some applications. If it is always stored in a decomposed form, an XML library probably wouldn't want to use it, because it requires a composed form. And making the encoding an implementation detail makes it inefficient to use in situations where binary compatibility matters (serialization, for example).

This is a good point. I should think however that a codecvt facet should be responsible for serialization rather than the unicode string. Furthermore, IMO, the invariants of any unicode string should be checked when reading from a file anyway. This should happen on two levels: the UTF-8 or UTF-16 encoding must be correct, and no dangling combining characters or combining characters on control characters should occur; furthermore, the normalisation form should probably be checked as well. So I'm not sure whether using Normalisation Form C rather than D will give you any big performance gains - you may need less memory though.

Also, it is impossible to store an abstract unicode character in char32_t because there may be N zero-width combining characters associated with it.

I'm not sure what you mean here, but if you mean that one abstract character would be one codepoint: that's not true, I'm sorry to say. Especially languages for which there was no encoding before Unicode, and funny scientists like mathematicians or linguists (I count myself among the latter) will use abstract characters that have not been encoded as precomposed characters in Unicode. Nor will they be; the precomposed forms are there for backwards compatibility, mainly. Note that adding a combining mark to a precomposed character takes decomposing it and recomposing it, so that might be pretty slow.

Perhaps having a one-size-fits-all unicode_string might be a nice default, as long as users who care about encoding and canonical form have other types (template + policies?) with knobs they can twiddle.

I do agree with that; and also I seem to remember from the discussion back in April that some people felt they needed to iterate over codepoints too. So please allow me to propose an altered version of my earlier proposal, taking in various suggestions from this thread. namespace unicode { // ***** Level 1: code units ***** // The code unit sequence is not explicitly specified, but it // could be std::string, or SGI rope<char16_t>, or whatever. // I think it would be reasonable to require replace, find, // find_first_of and similar. // ***** Level 2: codepoints ***** // The codepoint sequence is templatised on the code unit // sequence. // Depending on CodeUnits::value_type the encoding will // be UTF-8, UTF-16, or UTF-32. template <class CodeUnits> class codepoint_string { CodeUnits _code_units; public: // ... // A user is not allowed to change the code unit // sequence, but it may be copied, or serialised. const CodeUnits & code_units(); // The iterator is a bidirectional iterator. // This is cheap to implement on any correct Unicode- // encoded string since the iterator is not stateful. typedef ... iterator; // A size() member function is not included; // count() may be nice though. }; // ***** Level three: characters ***** // Normalisation policies struct normalisation_form_c {}; struct normalisation_form_d {}; // Input policies struct as_utf8 {}; struct as_latin1 {}; struct as_utf16 {}; // etcetera // Error checking policies struct throw_on_encoding_error {}; struct workaround_encoding_error {}; // An abstract Unicode character // I have not given this guys' interface much though yet. template <class NormalisationForm> class character { char32_t _base; std::vector<char32_t> _marks; public: character (char32_t base); character & operator = (char32_t base); const char32_t & base() const; void add_mark (char32_t mark); // An iterator to iterate over the combining marks. // It is a const_iterator because we wouldn't want to // allow introducing non-marks in the list of marks. typedef std::vector<char32_t>::const_iterator mark_iterator; mark_iterator mark_begin() const; mark_iterator mark_end() const; // .... }; // The actual Unicode string template <class CodeUnits, class NormalisationForm, class ErrorChecking> class string { codepoints<CodeUnits> _codepoints; public: // Initialise with a utf8 string; normalise and check for errors string (const CodeUnits &, as_utf8_tag); template <class CodeUnits2, class NormalisationForm2, class ErrorChecking2> string (const string <CodeUnits2, NormalisationForm2, ErrorChecking2> &); // .... const codepoints<CodeUnits> & codepoints(); const CodeUnits & code_units(); // Another bidirectional iterator, this one iterates // over abstract characters. class iterator { public: // Returns an object with an interface equal to // unicode::character, but it changes the string. character_ref operator *() const; // ... }; }; } // namespace unicode // ***** That was all ***** Mutating operations on unicode::string may require O(n) time where n is the length of the code unit sequence, depending on CodeUnit's properties. That's why using an SGI rope would make sense. Some default template parameters for unicode::string should be thought of. Regards, Rogier

On Wed, 20 Oct 2004 23:05:08 +0300, Peter Dimov <pdimov@mmltd.net> wrote:

Rogier van Dalen wrote:

...

// The actual Unicode string template <class CodeUnits, class NormalisationForm, class ErrorChecking> class string

By using ErrorChecking as a template parameter, you are encoding it as part of the string type, but this is not necessary, because there is no difference between values of strings with different ErrorChecking policies (ErrorChecking does not change the invariant). You should just provide different member functions for the two ErrorChecking behaviors, or pass the ErrorChecking parameter to the member functions that require it.

I hadn't yet looked at it this way, but you are right from a theoretical point of view at least. To get more to practical matters, what do you think this should do: unicode::string s = ...; s += 0xDC01; // An isolated surrogate, which is nonsense ? Should it throw, or convert the isolated surrogate to U+FFFD REPLACEMENT CHARACTER (Unicode standard 4 Section 2.7), or something else? And what should the member function with the opposite behaviour be called? Rogier

"Rogier van Dalen" <rogiervd@gmail.com> wrote in message

I hadn't yet looked at it this way, but you are right from a theoretical point of view at least. To get more to practical matters, what do you think this should do:

unicode::string s = ...; s += 0xDC01; // An isolated surrogate, which is nonsense

? Should it throw, or convert the isolated surrogate to U+FFFD REPLACEMENT CHARACTER (Unicode standard 4 Section 2.7), or something else? And what should the member function with the opposite behaviour be called?

The best solution would be to never append single code units, but instead code points. The += operator would determine how many code units is required for the given code point.

In article <41780922.9070406@boost-consulting.com>, "Eric Niebler" <eric@boost-consulting.com> wrote:

Erik Wien wrote:

...

"Rogier van Dalen" <rogiervd@gmail.com> wrote in message

...

I hadn't yet looked at it this way, but you are right from a theoretical point of view at least. To get more to practical matters, what do you think this should do:

unicode::string s = ...; s += 0xDC01; // An isolated surrogate, which is nonsense

? Should it throw, or convert the isolated surrogate to U+FFFD REPLACEMENT CHARACTER (Unicode standard 4 Section 2.7), or something else? And what should the member function with the opposite behaviour be called?

The best solution would be to never append single code units, but instead code points. The += operator would determine how many code units is required for the given code point.

I disagree. The user should be allowed to twiddle as many bits as she pleases, even permitted to create an invalid UTF string. However, operations that interpret the string as a whole (comparison, canonicalization, etc.) should detect invalid strings and throw. The reason is that people will need to manipulate strings at the bit level, and intermediate states may be invalid, but that the final state may be valid. We shouldn't do too much nannying during these intermediate states.

I am not sure I buy this. I think that if you want to have unchecked Unicode data, you should use a vector<char*_t>. Unicode strings have well-defined invariants with respect to canonicalization and well-formedness, and I think that the a Unicode string abstraction should enforce those invariants. Having intermediate states that are invalid and a final state that is valid is not a feature, it's a bug. It's a silent failure that I want to know about. meeroh

Erik Wien wrote:

"Miro Jurisic" <macdev@meeroh.org> wrote in message news:macdev-

...

I am not sure I buy this. I think that if you want to have unchecked Unicode data, you should use a vector<char*_t>. Unicode strings have well-defined invariants with respect to canonicalization and well-formedness, and I think that the a Unicode string abstraction should enforce those invariants.

Having intermediate states that are invalid and a final state that is valid is not a feature, it's a bug. It's a silent failure that I want to know about.

Amen. ;)

No fair bringing religion into this. ;-) I'll repeat what I said before -- this would be an unfortunate design, and you'll hear about it from your users. If you force people to do their bit twiddling in vector<char*_t>, then you impose an extra allocation and a copy to get it into a unicode::string, and most people won't bother. -- Eric Niebler Boost Consulting www.boost-consulting.com

Rogier van Dalen wrote:

On Thu, 21 Oct 2004 16:02:34 -0700, Eric Niebler <eric@boost-consulting.com> wrote:

...

No fair bringing religion into this. ;-) I'll repeat what I said before -- this would be an unfortunate design, and you'll hear about it from your users. If you force people to do their bit twiddling in vector<char*_t>, then you impose an extra allocation and a copy to get it into a unicode::string, and most people won't bother.

What frequent use cases do you see where people will want to change bits rather than work with characters?

utf8_string str; str.reserve(some_big_number); ifstream file("utf8.txt"); istreambuf_iterator<char8_t> begin(file), end; std::copy( begin, end, back_inserter(str) ); This can't throw if you want people to use your string class. -- Eric Niebler Boost Consulting www.boost-consulting.com

Eric Niebler wrote:

utf8_string str; str.reserve(some_big_number);

ifstream file("utf8.txt"); istreambuf_iterator<char8_t> begin(file), end;

std::copy( begin, end, back_inserter(str) );

This can't throw if you want people to use your string class.

utf8_string str( begin, end );

From: Rogier van Dalen <rogiervd@gmail.com>

<eric@boost-consulting.com> wrote:

...

-- this would be an unfortunate design, and you'll hear about it from your users. If you force people to do their bit twiddling in vector<char*_t>, then you impose an extra allocation and a copy to get it into a unicode::string, and most people won't bother.

What frequent use cases do you see where people will want to change bits rather than work with characters?

It is easy to build a safe interface atop a fast one, but you can't do the reverse. Perhaps the normal Unicode string, the safe one, can be implemented using the lower level one. Then, as someone mentioned, you can use the likes of c_str() to get a copy of the low level string. (You'll want to be able to convert from and assign from the low level string, too.) -- Rob Stewart stewart@sig.com Software Engineer http://www.sig.com Susquehanna International Group, LLP using std::disclaimer;

"Erik Wien" <wien@start.no> writes:

"Miro Jurisic" <macdev@meeroh.org> wrote in message news:macdev-

...

I am not sure I buy this. I think that if you want to have unchecked Unicode data, you should use a vector<char*_t>. Unicode strings have well-defined invariants with respect to canonicalization and well-formedness, and I think that the a Unicode string abstraction should enforce those invariants.

Having intermediate states that are invalid and a final state that is valid is not a feature, it's a bug. It's a silent failure that I want to know about.

Amen. ;)

How is this different from the situation with filesystem::path, where eager checking has turned out to be painful for a broad spectrum of users? -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

"Peter Dimov" <pdimov@mmltd.net> writes:

...

How is this different from the situation with filesystem::path, where eager checking has turned out to be painful for a broad spectrum of users?

"We have a valid Unicode string" is a pretty sensible invariant.

How is that more sensible than "we have a valid and portable path?" -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

In article <uis92u4hg.fsf@boost-consulting.com>, David Abrahams <dave@boost-consulting.com> wrote:

"Erik Wien" <wien@start.no> writes:

...

"Miro Jurisic" <macdev@meeroh.org> wrote in message news:macdev-

...

...

I am not sure I buy this. I think that if you want to have unchecked Unicode data, you should use a vector<char*_t>. Unicode strings have well-defined invariants with respect to canonicalization and well-formedness, and I think that the a Unicode string abstraction should enforce those invariants.

Having intermediate states that are invalid and a final state that is valid is not a feature, it's a bug. It's a silent failure that I want to know about.

Amen. ;)

How is this different from the situation with filesystem::path, where eager checking has turned out to be painful for a broad spectrum of users?

I am not familiar with the problem in boost::fs so I can't comment on that specifically, but I can comment in a more general sense. I think we can safely agree that: 1. There is an invariant that is valuable in some problem domains 2. There may be some problem domains in which it is valuable to sidestep that invariant boost::fs, as far as I understand it, ran into the problem that it was impossible to sidestep the invariant. My position is not that we should prohibit people from manipulating Unicode strings in a manner that does not maintain well-formedness. My position is that we should permit use of Unicode strings that guarantees well-formedness. I haven't taken the time to figure out how to do that (if I had the time, I would not be discussing how Erik could do this, I'd be writing a proposal myself :-) ). I come from a problem domain in which an abstraction which guarantees well-formedness has more value, but I am not looking for an answer that satisfies me and nobody else. meeroh

At 01:10 PM 10/22/2004, Miro Jurisic wrote:

boost::fs, as far as I understand it, ran into the problem that it was impossible to sidestep the invariant.

No, rather than the error check was on by default. Some people want it off as the default. As far as Unicode strings are concerned, the question is a little different. Is it well defined behavior to create a string that does not meet the Unicode invariants? If so, can ordinary operations break invariants, or is such dangerous activity restricted to "experts only" functions? --Beman

On Sat, 23 Oct 2004 10:56:55 +0100, John Maddock <john@johnmaddock.co.uk> wrote:

...

No, rather than the error check was on by default. Some people want it off as the default.

As far as Unicode strings are concerned, the question is a little different. Is it well defined behavior to create a string that does not meet the Unicode invariants? If so, can ordinary operations break invariants, or is such dangerous activity restricted to "experts only" functions?

For what it's worth: the Unicode standard *requires* conforming implementations to neither accept nor generate ill-formed Unicode sequences.

Ref: Unicode Chapter3 C12 and C12a.

It does explicitly allow processing ill-formed sequences, or, at least, "talk[ing] about" concatenating two ill-formed code unit sequences to form a valid one, though. (Ch. 3, D30e.) Rogier

Beman Dawes wrote:

At 01:10 PM 10/22/2004, Miro Jurisic wrote:

...

boost::fs, as far as I understand it, ran into the problem that it was impossible to sidestep the invariant.

No, rather than the error check was on by default. Some people want it off as the default.

I interpret it a little differently: the "error" check provided no value to users; in fact, it "provided" a negative value, which is why most prefer "off by default". This in itself does not prove that all possible portability checks do not provide value to users, just that this particular check is a net loss. By saying that "some people want it off as default" you are discounting the results of the experiment, attributing them to personal preference. ;-)

Beman Dawes wrote:

At 08:45 AM 10/25/2004, Peter Dimov wrote:

...

Beman Dawes wrote:

...

At 01:10 PM 10/22/2004, Miro Jurisic wrote:

...

boost::fs, as far as I understand it, ran into the problem that it was impossible to sidestep the invariant.

No, rather than the error check was on by default. Some people want it off as the default.

I interpret it a little differently: the "error" check provided no value to users;

That's true for some users, but not all users.

Certainly. Think of my statement as integrated over all possible users. Each user has his own unique needs, but it's still possible to form meaningful sentences that describe the general case.

...

in fact, it "provided" a negative value, which is why most prefer "off by default".

Only for the users who don't want to perform the check. For those that do want the check, it has a positive value.

I'm afraid I haven't been able to express my point well. My point was that if you focus on The Check, you'll only conclude that most users don't want The Check by default. This doesn't mean that most users don't want A Check by default; it just may not be The Check. There is a spectrum of possibilities between Strictest Check and No Check which may contain the optimal default that provides just the right amount of portability checking.

You could attribute the preference to the degree of portability required by the user's application's but I think it is more than that. Some people would rather be safe by default, while others would rather error checks only be applied if explicitly invoked. That seems like a personal preference to me.

I think that the main problem is that the current checks do not always signal errors. Errors are never a matter of personal preference, they are always part of a specification or dictated by external requirements. Tolerating checks that typically produce a significant amount of false positives can be a matter of personal preference, of course. But it also depends on how such checks are implemented. I tolerate a relatively high degree of false positives in compiler warnings, but I'd rather not see a similar rate in compiler errors. I certainly wouldn't mind a filesystem _warning_ for nonportable paths, if we find a suitable delivery mechanism.

Beman Dawes wrote:

At 02:10 PM 10/25/2004, Peter Dimov wrote:

...

... I certainly wouldn't mind a filesystem _warning_ for nonportable paths, if we find a suitable delivery mechanism.

Interesting! A fresh perspective.

Let's see what can be done within the current design. A adaptor function could intercept what otherwise would be errors and turn them into warnings: bool warn_nonportable_name( std::string const & name )

{ if ( !fs::portable_name( name ) ) boost::issue_warning( "Warning, path contains non-portable name: " + p ); return true; }

Because the adaptor always returns true, an error exception is never thrown. This function could become the new default, or could be explicitly coded like any name_check function.

[...]

Does that capture what you had in mind?

Almost. The current validation scheme (with the above addition) is (IIUC): 1. Parse path according to grammar, throw on parse error; 2. Check every path element with checker(e), throw on false; Warning checker calls issue_warning on nonportable paths. What I had in mind was more like: 1. Parse path according to grammar, throw on parse error; 2. Call checker(e) for every path element; if false, call fs_nonportable_path_element Where: void (*fs_nonportable_path_element)( std::string const & path, std::string const & element ) = __fsnp_default; (as an aside, the exceptions really need their own types, because these aren't filesystem errors, i.e. no underlying filesystem error code is associated with them.) The general idea of this approach is that if the user wants an exception on nonportable path elements, the name checker can simply throw. If the user wants the current behavior, he can install an fs_nonportable_path_element that throws. Otherwise, he can install an appropriate handler that logs or ignores the nonportable path. It might even be possible to get rid of the default name checker "replaceable global constant", if the original default is now good enough and nobody needs to change it.

On Thu, 21 Oct 2004 20:31:24 +0200, Erik Wien <wien@start.no> wrote:

The best solution would be to never append single code units, but instead code points. The += operator would determine how many code units is required for the given code point.

I fully agree with you on that; I was considering what should happen if the user appended something invalid (e.g., an isolated surrogate). Sorry for any confusion caused. I made a second mistake in mixing up the two levels in an unclear way. I very much like Peter's suggestion of using free functions converting invalid values to valid ones. Using that I suggest: unicode::codepoint_string should throw when an invalid codepoint is appended to it (e.g., an isolated surrogate). unicode::correct_codepoint() should convert an invalid codepoint into U+FFFD, and could be used to "safely" insert codepoints. char32_t correct_codepoint (char32_t); unicode::string should take a unicode::character for appending. A unicode::character object may be constructed with a single codepoint, which will be its base character. If this codepoint is invalid, it should throw. If the codepoint is a combining mark, it should also throw. unicode::correct() should convert an invalid codepoint into U+FFFD, and if it is input a combining mark, it should use U+0020 SPACE as a base character. character correct (char32_t); Regards, Rogier

On Fri, 22 Oct 2004 12:46:00 -0400 (EDT), Rob Stewart <stewart@sig.com> wrote:

From: Rogier van Dalen <rogiervd@gmail.com>

...

unicode::string should take a unicode::character for appending. A unicode::character object may be constructed with a single codepoint, which will be its base character. If this codepoint is invalid, it should throw. If the codepoint is a combining mark, it should also throw. unicode::correct() should convert an invalid codepoint into U+FFFD, and if it is input a combining mark, it should use U+0020 SPACE as a base character.

Why not have unicode::character's ctor invoke unicode::correct()?

unicode::correct() replaces every encoding error in the input by a replacement character. This loses information and it is not recoverable. The combining character bit is only slightly better. When I proposed a policy I called it workaround_encoding_error; maybe we need a better name than "correct". I agree with Peter Dimov, however, that the default should be to throw rather than to throw away information and pretend nothing happened. Regards, Rogier

David Abrahams wrote:

Is this going to be illegal for most fs, then?

std::copy( std::istream_iterator<char>(f), std::istream_iterator<char>(), std::back_inserter(my_utf8_string));

I think it pretty much has to work.

my_utf8_string.append( std::istream_iterator<char>(f), std::istream_iterator<char>() ); is the proper spelling. Or, if you prefer, my_utf8_string.insert( my_utf8_string.end(), std::istream_iterator<char>(f), std::istream_iterator<char>() ); Character by character push_back loops just aren't cool.

David Abrahams wrote:

"Peter Dimov" <pdimov@mmltd.net> writes:

...

David Abrahams wrote:

...

Is this going to be illegal for most fs, then? std::copy( std::istream_iterator<char>(f), std::istream_iterator<char>(), std::back_inserter(my_utf8_string)); I think it pretty much has to work.

my_utf8_string.append( std::istream_iterator<char>(f), std::istream_iterator<char>() );

is the proper spelling. Or, if you prefer,

my_utf8_string.insert( my_utf8_string.end(), std::istream_iterator<char>(f), std::istream_iterator<char>() );

Character by character push_back loops just aren't cool.

Okay, but should we make this well-known and understood idiom one that compiles and fails at runtime?

The well-known idiom is evil, but I see your point. A checking utf8_string probably shouldn't have push_back( char ). However, how did an utf8_string enter the discussion? Wasn't our unicode string supposed to hold UTF-16 in a clever way that permitted random access, and expose char32_t as its value_type?

In article <uoeiuu4n6.fsf@boost-consulting.com>, David Abrahams <dave@boost-consulting.com> wrote:

Is this going to be illegal for most fs, then?

std::copy( std::istream_iterator<char>(f), std::istream_iterator<char>(), std::back_inserter(my_utf8_string));

I think it pretty much has to work.

I think this is a red herring. A UTF-8 string is not a sequence of chars, nor is every char convertible to a UTF-8 char, so why should this work any more than vector<void*> my_vector; std::copy( std::istream_iterator<char>(f), std::istream_iterator<char>(), std::back_inserter(my_vector)); ? OTOH, A UTF-8 string is a sequence of Unicode chars, so what should work is: std::copy( std::istream_iterator<unicode_char>(f), std::istream_iterator<unicode_char>(), std::back_inserter(my_utf8_string)); with the semantics that every time you hit the iterator, an entire unicode character is read off the stream and appended to the string. meeroh

Eric Niebler wrote:

...

A single string class shall be used to store Unicode strings, i.e. logical sequences of Unicode abstract characters.

This string shall be stored in one chosen encoding, for example UTF-8. The user does not have direct access to the underlying storage, however, so it might be regarded as an implementation detail.

An invariant of the string is that it is always in one chosen normalized form. Iteration over the string gives back a sequence of char32_t abstract characters. Comparisons are defined in terms of these sequences.

Is this a fair summary?

Such a one-size-fits-all unicode_string is guaranteed to be inefficient for some applications. If it is always stored in a decomposed form, an XML library probably wouldn't want to use it, because it requires a composed form. And making the encoding an implementation detail makes it inefficient to use in situations where binary compatibility matters (serialization, for example).

This seems right, but there's a catch. Configurable encoding would help if all components of your application you the same encoding. Say XML parser wants composed form, so you use unicode_string<utf16, composed>. Now another part of your application (library written by somebody else) uses different encoding, and you have to convert the data on the interface. If there's only one encoding, you need to do conversion for code which really, really needs other encoding. If there are several encoding, then different libraries will use different encoding based on educated guesses about data, and you'll be converting everywhere.

Also, it is impossible to store an abstract unicode character in char32_t because there may be N zero-width combining characters associated with it.

Perhaps having a one-size-fits-all unicode_string might be a nice default, as long as users who care about encoding and canonical form have other types (template + policies?) with knobs they can twiddle.

Maybe, I just wish there was some efficient mechanism to prevent users who did not read the entire Unicode standard 10 times and so know what there's doing to touch the knobs ;-) - Volodya

In article <001301c4b635$bda16750$6501a8c0@pdimov2>, "Peter Dimov" <pdimov@mmltd.net> wrote:

My opinion is that the std::char_traits<> experiment failed and conclusively demonstrated that the "string as a value" approach is a dead end, and that practical string libraries must treat a string as a sequential container, vector<char>, vector<char16_t> and vector<char32_t> in our case.

The interpretation of that sequence of integers as a concrete string value representation needs to be done by algorithms.

There is no dispute that the rep of the string needs to be a container. (Though I do not agree that it's obvious that it should be a vector.) However, the basic_string interface grafted on top of a container of Unicode code units will produce bogus Unicode strings. This is why I strongly believe that basic_string is not a suitable container for Unicode strings. A separate container which does not provide convenient and completely incorrect member functions (such as find and assign) should be used. Consider this; pretend that - c and d are characters - C and D are the same character with an umlaut - C and D do not have precomposed code units in Unicode basic_string<char16_t> s("Cc"); // pretend assign and find use iterator ranges, for simplicity s.assign(s.find("c"), "d"); This will result in "Dc", which is completely wrong IMNSHO, and there should not be a simple interface that allows you to shoot yourself in the foot so thoroughly. It is not strings-as-containers that I am opposed to, but the deceptive simplicity of basic_string member functions. meeroh

Erik Wien wrote:

...

Ultimately I feel that the operation of normalization (which involves canonical decomposition) of unicode strings should be hidden from the user completely and be performed automatically by the library where that is needed. (Like on a call to the == operator.)

It appears that there are two schools of thought when it comes to string design. One approach treats a string purely as a sequential container of values. The other tries to represent "string values" as a coherent whole. It doesn't help that in the simple case where the value_type is char the two approaches result in mostly identical semantics.

My opinion is that the std::char_traits<> experiment failed and conclusively demonstrated that the "string as a value" approach is a dead end, and that practical string libraries must treat a string as a sequential container, vector<char>, vector<char16_t> and vector<char32_t> in our case.

The interpretation of that sequence of integers as a concrete string value representation needs to be done by algorithms.

In other words, I believe that string::operator== should always perform the per-element comparison std::equal( lhs.begin(), lhs.end(), rhs.begin() ) that is specified in the Container requirements table.

If I want to test whether two sequences of char16_t's, interpreted as UTF16 Unicode strings, would represent the same string in a printed form, I should be given a dedicated function that does just that - or an equivalent. Similarly, if I want to normalize a sequence of chars that are actually UTF8, I'd call the appropriate 'normalize' function/algorithm.

Right, and there are several different Normalised forms so we have to be able to choose the algorithm that does the right thing for what we want here. Can I make one other plea here: *please* lets not get too stuck on string class representations; we can have iterator sequences as well (these may well be part of a string, or they may be part of a memory mapped file, or some other smart iterator - like the Unicode encoding transformation iterators I've just been writing), and operations / algorithms on iterators are more important too me than YASC (Yet Another String Class) :-) John.

Eric Niebler wrote:

Erik Wien wrote:

...

Ultimately I feel that the operation of normalization (which involves canonical decomposition) of unicode strings should be hidden from the user completely and be performed automatically by the library where that is needed. (Like on a call to the == operator.) I think that solution would be satisfactory for most users as the normalization process is somewhat intricate and really not something users should be forced to understand.

Are we at all on the same page now?

No. "Normalization" doesn't always mean canonical decomposition. There are several canonical forms, some of which *require* the use of composite characters. In fact, the XML standard requires such a canonical form. A Unicode library cannot hide the issue of canonicalization from the user, because users will care which canonical form is being used.

Why? If I want to compare two string, I don't really care which normalized form is used. - Volodya

Peter Dimov wrote:

Vladimir Prus wrote:

...

Eric Niebler wrote:

...

No. "Normalization" doesn't always mean canonical decomposition. There are several canonical forms, some of which *require* the use of composite characters. In fact, the XML standard requires such a canonical form. A Unicode library cannot hide the issue of canonicalization from the user, because users will care which canonical form is being used.

Why? If I want to compare two string, I don't really care which normalized form is used.

But if you need a particular normalized form for other purposes (to store it into a database, perhaps), you have no way to obtain it from operator==.

Yes. But it's possible to have standalone "normalization" function, and still use default normalized representation for the string class. - Volodya

Peter Dimov wrote:

...

Yes. But it's possible to have standalone "normalization" function, and still use default normalized representation for the string class.

Thereby assuming that all users need to pay for normalization (twice) on every comparison?

Noway, of course.

Or maybe you are arguing that the string should always be kept in a particular normalized form?

That's what I meant. - Volodya

"Rogier van Dalen" <rogiervd@gmail.com> wrote in message news:e094f9eb0410200629617a4e01@mail.gmail.com...

On Wed, 20 Oct 2004 15:51:21 +0300, Peter Dimov <pdimov@mmltd.net> wrote:

...

Vladimir Prus wrote:

...

Peter Dimov wrote:

...

But if you need a particular normalized form for other purposes (to store it into a database, perhaps), you have no way to obtain it from operator==.

Yes. But it's possible to have standalone "normalization" function, and still use default normalized representation for the string class.

Thereby assuming that all users need to pay for normalization (twice) on every comparison?

Or maybe you are arguing that the string should always be kept in a particular normalized form?

That seems to be the only way of keeping comparison, search, etcetera, implementable in terms of char_traits<> functions --- and so, the only way of getting performance similar to std::basic_string<>'s.

Note that normalisation of any kind requires access to the Unicode Character Database, which may take some time, especially if the relevant parts happen not to be in the processor cache.

Comparing any Unicode data in different or unknown normalisation forms will therefore by definition be slow.

True.. So what we basically need to determine, is what is most critical? Fast comparing of strings (Strings always represented in a given NF), or fast genereal string handling (NF determined when needed)

Rob Stewart wrote:

What if the class had the option, at least, to hold multiple forms, creating each on demand? Then, the operations you invoke would simply request the particular form they require. If that form is not currently available, it is generated.

The classic problem is that logically const operations are now physically non-const (as I was surprised to learn from Howard Hinnant, this is the case with std::locale.) The implicit thread safety contract ("basic thread safety") says that unless stated otherwise, several logically const operations can be performed concurrently. This would now require a mutex lock.

On Wed, 20 Oct 2004 20:18:27 +0200, Erik Wien <wien@start.no> wrote:

"Rogier van Dalen" <rogiervd@gmail.com> wrote in message news:e094f9eb0410200629617a4e01@mail.gmail.com...

...

Comparing any Unicode data in different or unknown normalisation forms will therefore by definition be slow.

True.. So what we basically need to determine, is what is most critical? Fast comparing of strings (Strings always represented in a given NF), or fast genereal string handling (NF determined when needed)

I'm not quite sure what you mean. Do you propose to check whether a string is valid when reading it? And do you propose to make sure it is in some normalization form? Or will you leave it in any form it is in? What use cases do you envision where only "string handling" that does not need normalization is used? I have not been able to think of any. Regards, Rogier

Erik Wien wrote:

The basic idea I have been working around, is to make a nencoded_string class templated on unicode encoding types (i.e. UTF-8, UTF-16). This is made possible through a encoding_traits class which contains all nececcary implementation details for working on strings of code units.

The outline of the encoding traits class looks something like this:

template<typename encoding> struct encoding_traits { // Type definitions for code_units etc. // Is the encoding fixed width? (allows a good deal of iterator optimizations) // Algoritms for iterating forwards and backwards over code units. // Function for converting a series of code units to a unicode code point. // Any other operations that are encoding specific. }

Why do you need the traits, at compile-time? - Why would the user want to change the encoding? Especially between UTF-16 and UTF-32? - Why would the user want to specify encoding at compile time? Are there performance benefits to that? Basically, if we agree that UTF-32 is not needed, then UTF-16 is the only encoding which does not require complex handling. Maybe, for other encodings using virtual functions in character iterator is OK? And if iterators have abstract characters" as value_type, maybe the overhead if that is much large that virtual function call even for UTF-16. (As a side note, discussion about templated vs. non-templated interface seems a reasonable addition to a thethis. It's sure thing that if anybody wrote such a thethis in our lab, he would be asked to justify such a global decisions). - What if the user wants to specify encoding at run time? For example, XML files specify encoding explicitly. I'd want to use ascii/UTF-8 encoding if XML document is 8-bit, and UTF-16 when it's Unicode. - Volodya

Erik Wien wrote:

...

- Why would the user want to change the encoding? Especially between UTF-16 and UTF-32?

Well... Different people have different needs. If you are mostly using ASCII characters, and require small size, UTF-8 would fit your bill. If you need the best general performance on most operations, use UTF-16. If you need fast iteration over code points and size doesn't matter, use UTF-32.

Ok, since everybody agreed characters outside 16 bits are very rare, UTF-32 seems to never be needed. As for UTF-8 vs. UTF-16: yes, the need for choice seems present. However, UTF-16 string class would be better than no string class at all, and extra genericity will cost you development time.

...

- Why would the user want to specify encoding at compile time? Are there performance benefits to that? Basically, if we agree that UTF-32 is not needed, then UTF-16 is the only encoding which does not require complex handling. Maybe, for other encodings using virtual functions in character iterator is OK? And if iterators have abstract characters" as value_type, maybe the overhead if that is much large that virtual function call even for UTF-16.

Though I haven't confirmed this by testing, I would assume templating the encoding and thus specifying it at compile time would result in better performance since you don't have the overhead of virtual function calls. (Polymorphy would probably be needed if templates were scrapped.)

It would. The question is by how much.

Avoiding virtual calls also enables the compiler to optimize (inline) more thouroughly, something that is very benificial in this case because of the amount of different small, specialized functions that are needed in string manipulation.

This is a bit abstract. Virtual function is a inlining barrier, but it would be placed only for character access. On both sides of the barrier, compiler can freely optimize everything.

...

- What if the user wants to specify encoding at run time? For example, XML files specify encoding explicitly. I'd want to use ascii/UTF-8 encoding if XML document is 8-bit, and UTF-16 when it's Unicode.

That is one problem with the templating of encoding. You would have to ether template all file scanning functions in the XML parser on encoding as well, of you would need to do some run-time checks and use the correct template depending on the encoding used in the file. This is of course not ideal, but only where encoding is something that is specified upon run-time. What the most common scenario is, is something that needs to be determined before a final design is decided on.

Another possibility is that you can decide if UTF8 of UTF16 should be used dynamically -- just counting the number of non-ascii characters. That would mean that only really advanced users need make the decision themself. I think I'm starting to like Peter's idea that advanced users need vector<char_xxx> together with a set of algorithms. - Volodya

Mathew Robertson wrote:

...

...

...

- Why would the user want to change the encoding? Especially between UTF-16 and UTF-32?

Well... Different people have different needs. If you are mostly using ASCII characters, and require small size, UTF-8 would fit your bill. If you need the best general performance on most operations, use UTF-16. If you need fast iteration over code points and size doesn't matter, use UTF-32.

Ok, since everybody agreed characters outside 16 bits are very rare, UTF-32 seems to never be needed. As for UTF-8 vs. UTF-16: yes, the need for choice seems present. However, UTF-16 string class would be better than no string class at all, and extra genericity will cost you development time.

<rant> umm... so your saying that no one will ever need more than 640K RAM? Just because YOU dont need more than 16bits, doesn't meen that I dont need more than 16bits. </rant>

The main question of a Unicode library should _always_ be, can the library represent every character that can be drawn; things like iteraters, algorithms, etc are nice-to-haves -> the representation of the written language is the first priority, everything else is secondary.

Also, the Unicode standard will evolve over time to include more characters from many more characters sets that you or I may never use but someone else might; who knows, maybe the ASCII character set will get a 27th character one day... A library shouldn't preclude the use of these new characters, just because we thought "no one will ever need more than 16bits"... So, how about we dont make the same mistakes as we made in the past...

Do you realize that "nobody needs UTF-32" is not the same that "nobody needs character which can't be represented in 16 bits"? UTF-16 can represent all Unicode characters.

Whatever desision finially gets chosen will come down to one of two choices: a) variable length string format, eg: UTF8, or something similar b) fix width format with so many bits that humans are unlikely to use all the address space at any time in the next 50/100 years, eg UTF-32, or similar

FWIW: my personal preference would be to go for a variable with encoding -> so that we never have to solve this problem again... although this makes concepts like text-reflow quite a bit harder to implement.

What's "text-reflow", BTW? - Volodya

yes I do realise... the origonal statement was "...everybody agreed characters outside 16 bits are very rare, UTF-32 seems to never be needed." UTF-16 can indeed represent every Unicode character, but that is not what was written. It must have been obvious to the poster of the original statement.

One point that hasn't been mentioned so far is that, word sizes on most modern CPU's are 32bits wide. From a performance POV, the word-alignment may be a suitable justification for offsetting the increased storage requirements of a 32bit unit. Of course from a performance POV you surely don't want to waste twice (four times) as much memory in the cache either. Performance is always a tradeoff.

Cheers, Michael

"John Maddock" <john@johnmaddock.co.uk> wrote in message

Interesting: funnily enough I've just started experimenting with Unicode support for Boost.Regex (based initially on top of ICU, but it could equally sit on top of Boost.Unicode or whatever). The first thing I had to do was write a bunch of iterators for interconverting between encoding forms (I needed Bidirectional Iterators which code conversion facets don't/can't provide). So I guess we're all on a similar page here, can your encoding converters proved efficient iterator-based interconversion?

Well.. "efficient" is probably not the word I would use ;), yet that is. The way it is implemented right now, the value_type of a encoded_string iterator of any encoding is 32bit. (A unicode code-point.) So when iterating over any encoding, the external interface always looks as a vector of code points. Consequently you can use iterators from one string (UTF-8) to initialize another string (UTF-16) and the conversion between the two encodings would happen automatically. I'm guessing this is something similar to what you have. I also have a rather hackish implementation that can provide non-const (assignable) code point iterators on any encoding. This involves a lot of trickery with iterators changing the size the container they are iterating over, and proxy classes as a reference_type in the iterator. (something that is not allowed (yet) in standard C++, but is in boost) As you can imagine, this implementation is everything but efficient. Kinda neat though! ;)

Are you aware of the Unicode Technical Report (TR) being prepared by the C committee? See http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1040.pdf

While there is no guarantee, it is a definite possibility that the C++ committee will use this C TR as part of Unicode support in C++. Because this TR is so limited in scope (it just has a few conversion functions), that probably won't cause problems for a C++ Unicode library, but you should still be aware of the data types introduced.

No, I was not aware of that. Thanks for the link. There doesn't seem to be any problems associated with a possible unicode library, and that C library being included into the C++ standard. It basically does a lot of the low-level unicode stuff that I am currently doing by hand, so it would probably be more of a blessing than a curse if it got included into standard C++.

"Miro Jurisic" <macdev@meeroh.org> wrote in message news:macdev-

...

I think it would be illuminating to see a short summary of the classes and functions such a library is expected to provide

I agree; I think that the feedback given so far covers most of the important perspectives, and I think we should not fool ourselves into thinking every one of them can be accommodated immediately. A library that addresses some of them designed to allow further work to address additional applications would be a huge step from where we are today, so I suggest that we (Erik :-) ) pick a set of objectives manageable for his thesis timeline, and submit a preliminary proposal.

It's coming. :) I will have to write a description of what I want to achieve with this for my college to look at by this weekend, and I will try to take everything said here into consideration when writing that. I will try to post a translation of the description here (Or preferably in a new thread) once it's handed in. As for preliminary proposals and the like. You will probably not see too much of the kind before the project (hopefully) gets accepted and I am able to work full time on it. Right now I am buried in other work that needs attention, so I'm not able to dedicate myself completely to this just yet. I will try to use any spare moment though!

"Andy Little" <andy@servocomm.freeserve.co.uk> wrote in message news:clbvg3$f4o$1@sea.gmane.org...

"Erik Wien" <wien@start.no> wrote in message news:cl1tqh$qp$1@sea.gmane.org...

...

Hi. I am in the process of planning a library for handling unicode

strings

...

in C++, and would like to probe the interest in the boost community for something like that. I read through the unicode dicussion that was up back in april, and from what I could gather there was some amount of interest, but no one felt comfortable taking on the task as of yet.

I forgot to say ... What ever scheme you come up with, this is an essential C+ library and needs to be done by somebody, sometime soon. :-) ....... As a potential user... The other problem that immediately springs to my mind is , how do I compose a unicode string in my C++ source code. I really dont want to be dealing with some unicode_character<TheEncode>(0x78,99,'c',-1) ; style. The unicode charts I have seen are mind boggling, when taken in the raw. However I always want to know the (common) encoding(s) for a particular character. For that I would , I guess, need a naming convention for common characters eg the functionality of http://www.unicode.org/charts/charindex.html eg: I would prefer to do (similar to HTML character entities): // whatever... target unicode string encoded_string str; // set of common characters in required encoding typedef unicode_common_character_set<TheEncode> uni_chars; // eg "um" where u == 'micro' symbol etc. uni_chars my_str[] = {uni_chars::quot, uni_chars::micro, uni_chars::m, uni_chars::quot}; str = my_str; As a user thats probably the bit I am most interested in rather than the implementation details. regards Andy Little

"Beman Dawes" <bdawes@acm.org> wrote in message news:6.0.3.0.2.20041023085844.028a7018@mailhost.esva.net...

At 07:08 AM 10/23/2004, Andy Little wrote:

...

The other problem that immediately springs to my mind is , how do I

compose

...

a unicode string in my C++ source code. I really dont want to be dealing with some unicode_character<TheEncode>(0x78,99,'c',-1) ; style.

The C TR provides additional string literals and character constants. See below.

--Beman

5.1 String literals and character constants notations

[snip detail] Ok I understand that long term the C/C++ language should support Unicode and I am all in favour. The other question is whether there is a need for a C++ library which provides definitive encodings ( Not restricted to Unicode) that can be implemented in the Current C++ language. I could certainly make use of it today. I would assume that such a library would be extremely useful in deciding how to implement language support in C++. I would guess that to be generic the characters could use named indirection in some cases, similar to the html scheme... typedef typename default_charset<GUI>:: type charset_type; encoded_string<charset_type> str = "length " ; str += charset_type::micro ; // much nicer ... str += "m"; etc regards Andy Little