24 Apr
2005
24 Apr
'05
4:54 p.m.
"Beman Dawes" <bdawes@acm.org> wrote in message news:6.0.3.0.2.20050422110501.042b2a20@mailhost.esva.net...
|I'd add a tokenizer to the bunch. Not specially sexy but |massively helpful --the typical impulse of a beginner is to |do the tokenizing stuff by hand, many a time resulting |in off-by-one errors, mishandling of unexpected separators Tokenizer was rejected some years ago on the grounds of being too slow. I think the regex part contains faster tokenization algorithms and the remaning can probably be found in the string library. -Thorsten