Andrey Semashev <andysem@mail.ru> writes: [snip]
I may not support character combining from several code points if it is not used or uncommon in languages A, B and C. Moreover, many precombined characters exist in Unicode as a single code point.
They do; it may be largely for compatibility reasons, I think. I don't think it is a very good idea to attempt to provide partial support for Unicode by supporting only single-code-point grapheme clusters. Furthermore, I don't see that it would be a huge gain.
[snip]
That will just require duplicating the tables and algorithms required to process the text correctly.
What algorithms do you mean and why would they need duplication?
Examples of such algorithms are string collation, comparison, line breaking, word wrapping, and hyphenation.
Why would these algorithms need duplication? If we have all locale-specific traits and tools, such as collation tables, character checking functions like isspace, isalnum, etc. along with new ones that might be needed for Unicode, encapsulated into locale classes, the essence of algorithms should be independent form the text encoding.
Using standard data tables, and a single algorithm that merely accesses the locale-specific data tables, you can provide these algorithms for UTF-16 (and other Unicode encodings) for essentially all locales. This is done by libraries like IBM ICU. Providing them in addition for other encodings, however, would require separate data tables and separate implementations.
Besides, comparison is not the only operation on strings. I expect iterating over a string or operator[] complexity to rise significantly once we assume that the underlying string has variable-length chars.
The complexity remains the same if operator[] indexes over encoded units, or you are iterating over the encoded units. Clearly, if you want an iterator that converts from the existing encoding, which might be UTF-8 or UTF-16, to UTF-32, then there will be greater complexity. As stated previously, however, it is not clear why this is likely to be a frequently useful operation.
What do you mean by encoded units?
By encoded units I mean e.g. a single byte with utf-8, or a 16-bit quantity with utf-16, as opposed to a code point.
What I was saying, if we have a UTF-8 encoded string that contains both latin and national characters that encode to several octets, it becomes a non-trivial task to extract i-th character (not octet) from the string. Same problem with iteration - iterator has to analyze the character it points to to adjust its internal pointer to the beginning of the next character. The same thing will happen with true UTF-16 and UTF-32 support. As an example of the need in such functionality, it is widely used in various text parsers.
I'm still not sure I quite see it. I would think that the most common case in parsing text is to read it in order from the beginning. I suppose in some cases you might be parsing something where you know a field is aligned to e.g. the 20th character, but such formats tend to assume very simple encodings anyway, because they don't make much sense if you are to support complicated accents and such. -- Jeremy Maitin-Shepard