28 Feb
2010
28 Feb
'10
4:17 p.m.
It is also very useful as a simple and lightweight replacement for string interning: sometimes in text processing you need to represent unique words or phrases with a short numeric id, for example for fast comparison or simple compression, both in memory and on disk.
I don't think that a `tokenmap` will be suitable for string interning because a `tokenmap` essentially needs to choose the generated hashes (tokens). With string interning, the hash of the string should serve as the token. You probably want an unordered map.
In practice I need a way to efficiently convert a value to a key.
`tokenmap`s do not allow this.