----Original Message---- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Janek Kozicki Sent: 18 July 2006 14:21 To: boost@lists.boost.org Subject: Re: [boost] [tokenizer] is exception unaware

Sebastian Redl said: (by the date of Tue, 18 Jul 2006 14:06:59 +0200)

...

Janek Kozicki wrote:

...

boost::tokenizer<>::iterator pos=tok.begin(); - is this IMO bad design intentional?

Boost.Tokenizer is written as to model an iterator. Have you even seen an iterator throw an exception on reaching the end of the sequence?

The rationale for container iterators is that iterating over containers should not be slowed down by exceptions. Those iterators don't even check for the iterator's validity! That depends on the implementation of the standard library. Some standard libraries /do/ check for the iterator's validity.

And trying to access beyond the container is not detected by the program, but simply causes segmentation fault. That's good for usual containers.

I'd say it causes undefined behaviour, and an assertion failure, a segmentation fault, and silently returning the wrong value are all acceptable (although the first two are preferred).

...

So yes, the design is probably intentional.

However tokenizer::iterator is not a typical container nor typical iterator. It is even unusual that it checks for the iterator's validity (because trying to access beyond the container causes assert(valid_) to fail ).

The rationale from typical container iterators makes no sense when applied to tokenizer::iterator, because iterating over tokenizer items is not fast - it requires parsing of the content.

There are two solutions to correct this:

- remove assert(valid_) check from tokenizer, so it follows more precisely the standard, and causes segmantation fault on failure.

I don't see how a segmentation fault follows the standard any more than an assertion failure.

- do opposite, and accept the fact that tokenizer is not a typical container, and replace assert_fail with throw.

-- Martin Bonner Martin.Bonner@Pitechnology.com Pi Technology, Milton Hall, Ely Road, Milton, Cambridge, CB4 6WZ, ENGLAND Tel: +44 (0)1223 203894