Re: [Boost-users] Fastest way of serializing a huge vector of ints into a human readable string

Hi Roland, I'm not sure if karma speeds up the process at all. Also it runs afaik on an iterator interface, so you'd have the option to control what happens underneath.

Jens Weller wrote:

...

Hm, maybe boost::spirit::karma could do you a favor here, but I think it will not gain much perfomance, if at all.

Maybe you could think about a functor which does it on your own. Do the numbers repeat often, then you could also build a table to only convert once to string.

Also parellizing with boost Thread would be an option.

Hi Jens,

hmm. Documentation about boost::spirit::karma seems to be well hidden. Even a google search like

karma site:www.boost.org/doc/libs/1_38_0/libs/spirit

did not reveal any hits. Can you give me a link?

Karma seems not to be documented yet well enough in the docs. But there are some examples which are shipped with boost: boost/libs/spirit/examples/karma/ f.e. http://www.boost.org/doc/libs/1_38_0/libs/spirit/example/karma/quick_start1....

Writing an own functor might be an option, I guess. The numbers do not repeat often, though. Quite the opposite: It is for certain that every number appears just once. And it could be anything within a 64bit range.

hm, I ask my self, how much time the conversiont number->string takes. Once you remove all the copying of tempstrings etc. you probably already end up at a pretty fast way.

Parallelization certainly could be helpful in some scenarios, but I a have a multi-threaded application already. So splitting up the serialization might increase speed for the serialization itself, while slowing down the total process due to increased context switches. I therefore want to have the single threaded operation as efficient as possible.

Well, ofcourse you'd have to do the optimisation first. Just using more processors doesn't make it faster in the processing. regards Jens Weller

...

-------- Original-Nachricht --------

...

Datum: Fri, 27 Feb 2009 18:46:07 +0100 Von: Roland Bock An: boost-users@lists.boost.org Betreff: [Boost-users] Fastest way of serializing a huge vector of ints into a human readable string

...

Hi,

I have a program which produces a vector of integers (several million entries). I need to write that into a human-readable string of space separated numbers.

I wonder, what would be the fastest way?

My first attempt was

stringstream resultStream; copy(integers.begin(), integers.end(), ostream_iterator<int>(resultStream, " ")); string result = resultStream.str();

But that requires copying the string.

Thus I was wondering if I could use boost to write to the target string directly?

boost::iostreams allow me to do the following:

string result; boost::iostreams::filtering_ostream out(boost::iostreams::back_inserter(result)); copy(integers.begin(), integers.end(), ostream_iterator<uint64>(out, " "));

This is faster by almost a factor of 2.

Any ideas how to increase speed even more?

Thanks in advance,

Roland _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

-- Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger01