On Fri, 11 Feb 2011 00:07:58 +0100 Mathias Gaunard <mathias.gaunard@ens-lyon.org> wrote:
There is no need for any reasoning: look at the code of your code point iterator. It uses a pointer and indexes, and is therefore not a generic iterator adaptor.
It wasn't meant to be generic. It was meant to be exactly what it is: an iterator specific to the UTF type where it's defined. For that purpose, it's designed exactly as it should be, IMHO.
It was already stated on this list that the ability to deal with arbitrary ranges is more valuable; I am merely stating that your iterator could work with any iterator with virtually no change.
Either I missed that statement, or I don't recognize it in this context, because I'm not sure what you mean.
I could make it fully generic, but it wouldn't be nearly as efficient that way. I chose to do the extra work to make it efficient.
Your code never uses the fact that the iterator is a pointer or that memory is stored contiguously.
How would you suggest that it use that information? I think we have different definitions of generic. As I said in my last message, I define fully generic as working with any UTF-encoded string (not just UTF-8). That would be possible, but would almost certainly be less processor-efficient than having iterators customized to each type.
I also don't think that for example your utf-8 iterating strategy is very fast. Your utf-8 decoding itself seems to have lots of repetition and unnecessary tests and memory accesses...
Not that it is easy to make that kind of thing fast anyway.
There's always room for further optimization, at the cost of more programmer time and more code.
It could be a bidirectional iterator, as it has all of the abilities of one. And it could be a random access iterator, as it has all but one of the requirements for that (and in many cases has all of them). Given that choice, I chose to make it a random access iterator.
It has constant-time distance, which isn't very useful and adds unnecessary overhead to your iterator.
I disagree. I find it extremely useful to have a true random access iterator for some strings (most, in UTF-16, and arguably most in many cases of UTF-8 too), and an emulated one for the rest. And for that, the overhead isn't unnecessary.
I'm not a fan of returning a reference in operator* as well.
No choice in that, I ran into at least one STL algorithm under GCC that wouldn't compile if it wasn't a reference, even when it was only being read. I don't remember which one, but it was something important and commonly-used enough that breaking it was not an option.
I believe the standard containers indeed require it to be a reference, but I'm not aware of any problems with any implementation.
Would you mind telling me what libstdc++ algorithm relies on this?
If I recalled which one it was, I would have put it in the original message.
They give me a way to prevent my iterator code from walking off the beginning or end of the underlying string.
The only other way to do it would be to store a pointer to the string object in every iterator, or a pair of iterators or pointers to the underlying type, which I considered worse.
You seek the next "first" character of a code-unit sequence. That indeed causes problems when you reach the end (unless you put a 0 at the end of your buffer, which isn't such a bad idea). UTF-8 and 16, however, don't require this. You can deduce how many code units you need to consume from the first code unit. You do that in your decoders.
And I didn't want to duplicate that code in the iterators as well, or separate it out and possibly add the overhead of another function call to the decoder functions.
As an important side benefit, they also provide an efficient way to calculate the difference in code points for operator-, which I feel is important.
And the size in code-points is (supposed to be) stored at all times.
What's the point of this? The size in code points is not a very useful thing in general.
On the contrary, keeping track of the length of the string is *very* useful. The alternative is to calculate it on the fly, whenever someone asks for it. If you want that, you can always go back to using C-style strings.
All the UTF types were very carefully designed so that there's no chance of invalid data in them, barring extraordinary measures to deliberately corrupt it.
That's a good thing. However, you could use this opportunity to make decoding much faster, since you don't need to check for correctness anymore.
As I said, there's always room for further optimization.
Anything else? :-)
You would probably encounter problems on platforms where int is 8 or 16 bits.
I haven't seen a platform where an int is 16 bits since DOS, which I stopped coding for in the late nineties. And I've never seen one where it's eight bits. Do you know of any modern platform -- as in one that uses Unicode, and could usefully use this library -- where that's the case? -- Chad Nelson Oak Circle Software, Inc. * * *