Compress to buffer

older
Boost.Heap (i.e fibonacci_heap)...

Kenneth Adam Miller

14 Jan 2014 14 Jan '14

2:20 p.m.

I have an object for which operator << is defined, myobj. I want to compress into a buffer, and I know that this can be done with boost. I would prefer a buffer that dynamically resizes, but that only allocates new space, preferably treating discontiguous space brought on by new allocations as though the whole buffer was one piece; I don't want to find that it's copying data at all. So vector won't work as a buffer; there's no need for it to be contiguous, just as fast as possible. I've been trying to do this with filtering_istreambuf by pushing a bzip2_compressor (any compression filter will work). So far, it's giving me a lot of trouble. I did find an example where a string was passed to a function, but they went through a process where they first moved the string into an stringstream. I don't want to settle for that extra copy either. What would be nice is to be able to do something like compressor << myobj compressor.buf(); //returns compressed myobj output

Attachments:

attachment.html (text/html — 1.1 KB)

Show replies by date

Kenneth Adam Miller

14 Jan 14 Jan

3:09 p.m.

So, I found that I could use a array_sink and pass it the location and size of an array to use. But now I need to know, when bytes are written in, how much is written into it in order that I can get that exact data out. I also need to make sure that, in the case of something like not enough space occurring, I can still get the rest of the data compressed. Is there anything that I can pass to an array_sink initializer other than a char * pointer that might help me accomplish this? On Tue, Jan 14, 2014 at 8:20 AM, Kenneth Adam Miller < kennethadammiller@gmail.com> wrote:

...

I have an object for which operator << is defined, myobj.

I want to compress into a buffer, and I know that this can be done with boost. I would prefer a buffer that dynamically resizes, but that only allocates new space, preferably treating discontiguous space brought on by new allocations as though the whole buffer was one piece; I don't want to find that it's copying data at all. So vector won't work as a buffer; there's no need for it to be contiguous, just as fast as possible.

I've been trying to do this with filtering_istreambuf by pushing a bzip2_compressor (any compression filter will work). So far, it's giving me a lot of trouble. I did find an example where a string was passed to a function, but they went through a process where they first moved the string into an stringstream. I don't want to settle for that extra copy either.

What would be nice is to be able to do something like

compressor << myobj

compressor.buf(); //returns compressed myobj output

Kenneth Adam Miller

3:59 p.m.

I figured out how to do it: filtering_ostream fos; fos.push(back_insert_device<string>(x)); On Tue, Jan 14, 2014 at 9:09 AM, Kenneth Adam Miller < kennethadammiller@gmail.com> wrote:

...

So, I found that I could use a array_sink and pass it the location and size of an array to use. But now I need to know, when bytes are written in, how much is written into it in order that I can get that exact data out. I also need to make sure that, in the case of something like not enough space occurring, I can still get the rest of the data compressed.

Is there anything that I can pass to an array_sink initializer other than a char * pointer that might help me accomplish this?

On Tue, Jan 14, 2014 at 8:20 AM, Kenneth Adam Miller < kennethadammiller@gmail.com> wrote:

...
I have an object for which operator << is defined, myobj.

I want to compress into a buffer, and I know that this can be done with boost. I would prefer a buffer that dynamically resizes, but that only allocates new space, preferably treating discontiguous space brought on by new allocations as though the whole buffer was one piece; I don't want to find that it's copying data at all. So vector won't work as a buffer; there's no need for it to be contiguous, just as fast as possible.

I've been trying to do this with filtering_istreambuf by pushing a bzip2_compressor (any compression filter will work). So far, it's giving me a lot of trouble. I did find an example where a string was passed to a function, but they went through a process where they first moved the string into an stringstream. I don't want to settle for that extra copy either.

What would be nice is to be able to do something like

compressor << myobj

compressor.buf(); //returns compressed myobj output

Krzysztof Czainski

4:08 p.m.

2014/1/14 Kenneth Adam Miller

...

I figured out how to do it: filtering_ostream fos; fos.push(back_insert_device<string>(x));

[...]

...

On Tue, Jan 14, 2014 at 8:20 AM, Kenneth Adam Miller <

...
kennethadammiller@gmail.com> wrote:

...
I have an object for which operator << is defined, myobj.

I want to compress into a buffer, and I know that this can be done with boost. I would prefer a buffer that dynamically resizes, but that only allocates new space, preferably treating discontiguous space brought on by new allocations as though the whole buffer was one piece; I don't want to find that it's copying data at all. So vector won't work as a buffer; there's no need for it to be contiguous, just as fast as possible.

Hi,

Did you satisfy your "no copying" requirement? How? Was the requirement based only on performance concerns? Regards, Kris

Kenneth Adam Miller

4:15 p.m.

Pretty much on performance concerns. I know that there's at least going to be one copy performed while doing the compression, from uncompressed to compressed. Here's how I do it: filtering_ostream *fos = new filtering_ostream(); fos->push(bzip2_compressor()); string *x = acquireStringFromPool(); //This is just a blocking pointer return that reaches into a list of string *, each that are allocated with new string(30000,0); (it's multithreaded, ok lol :) ) fos->push(boost::iostreams::back_insert_device<string>(x)); //This is what I was searching for all along. then later, when I want to write to fos I do, *fos << *doc; //I go straight from container to compression. Maybe my specifications that "I don't want to find that it's copying at all" were a bit weird, because obviously it has to move the data right? I'm just saying that most of the examples I would see would be something like compress(string x) { stringstream ss(x); //unnecessary initialization in my case, couldn't find an example without this //something similar to what I did... } On Tue, Jan 14, 2014 at 10:08 AM, Krzysztof Czainski <1czajnik@gmail.com>wrote:

...

2014/1/14 Kenneth Adam Miller

...
I figured out how to do it: filtering_ostream fos; fos.push(back_insert_device<string>(x));

[...]

...
On Tue, Jan 14, 2014 at 8:20 AM, Kenneth Adam Miller <

...
kennethadammiller@gmail.com> wrote:

...
I have an object for which operator << is defined, myobj.

I want to compress into a buffer, and I know that this can be done with boost. I would prefer a buffer that dynamically resizes, but that only allocates new space, preferably treating discontiguous space brought on by new allocations as though the whole buffer was one piece; I don't want to find that it's copying data at all. So vector won't work as a buffer; there's no need for it to be contiguous, just as fast as possible.

Hi,

Did you satisfy your "no copying" requirement? How?

Was the requirement based only on performance concerns?

Regards, Kris

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Kenneth Adam Miller

4:16 p.m.

By the way, I'm working on a master's thesis, so I frequently skip sleep. Sometimes after a lack of sleep, getting across precisely what is needed/understood can take some an iteration or two :) On Tue, Jan 14, 2014 at 10:15 AM, Kenneth Adam Miller < kennethadammiller@gmail.com> wrote:

...

Pretty much on performance concerns. I know that there's at least going to be one copy performed while doing the compression, from uncompressed to compressed. Here's how I do it:

filtering_ostream *fos = new filtering_ostream(); fos->push(bzip2_compressor()); string *x = acquireStringFromPool(); //This is just a blocking pointer return that reaches into a list of string *, each that are allocated with new string(30000,0); (it's multithreaded, ok lol :) ) fos->push(boost::iostreams::back_insert_device<string>(x)); //This is what I was searching for all along.

then later, when I want to write to fos I do,

*fos << *doc; //I go straight from container to compression.

Maybe my specifications that "I don't want to find that it's copying at all" were a bit weird, because obviously it has to move the data right? I'm just saying that most of the examples I would see would be something like

compress(string x) { stringstream ss(x); //unnecessary initialization in my case, couldn't find an example without this //something similar to what I did... }

On Tue, Jan 14, 2014 at 10:08 AM, Krzysztof Czainski <1czajnik@gmail.com>wrote:

...
2014/1/14 Kenneth Adam Miller

...
I figured out how to do it: filtering_ostream fos; fos.push(back_insert_device<string>(x));

[...]

...
On Tue, Jan 14, 2014 at 8:20 AM, Kenneth Adam Miller <

...
kennethadammiller@gmail.com> wrote:

...
I have an object for which operator << is defined, myobj.

I want to compress into a buffer, and I know that this can be done with boost. I would prefer a buffer that dynamically resizes, but that only allocates new space, preferably treating discontiguous space brought on by new allocations as though the whole buffer was one piece; I don't want to find that it's copying data at all. So vector won't work as a buffer; there's no need for it to be contiguous, just as fast as possible.

Hi,

Did you satisfy your "no copying" requirement? How?

Was the requirement based only on performance concerns?

Regards, Kris

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Krzysztof Czainski

6:16 p.m.

2014/1/14 Kenneth Adam Miller

...

By the way, I'm working on a master's thesis, so I frequently skip sleep. Sometimes after a lack of sleep, getting across precisely what is needed/understood can take some an iteration or two :)

Good luck with your thesis ;-) Btw, I think top-posting is inappropriate on this list. On Tue, Jan 14, 2014 at 10:15 AM, Kenneth Adam Miller <

...

kennethadammiller@gmail.com> wrote:

...
Pretty much on performance concerns. I know that there's at least going to be one copy performed while doing the compression, from uncompressed to compressed. Here's how I do it:

filtering_ostream *fos = new filtering_ostream(); fos->push(bzip2_compressor()); string *x = acquireStringFromPool(); //This is just a blocking pointer return that reaches into a list of string *, each that are allocated with new string(30000,0); (it's multithreaded, ok lol :) ) fos->push(boost::iostreams::back_insert_device<string>(x)); //This is what I was searching for all along.

Doesn't this append your 30k string? If a preallocated chunk of memory is ok for you, check out vector::reserve() ;-) then later, when I want to write to fos I do,

...

...
*fos << *doc; //I go straight from container to compression.

Maybe my specifications that "I don't want to find that it's copying at all" were a bit weird, because obviously it has to move the data right? I'm just saying that most of the examples I would see would be something like

compress(string x) { stringstream ss(x); //unnecessary initialization in my case, couldn't find an example without this //something similar to what I did... }

OK, so did you measure performance of your solution compared to the above example? And then to some version with std::vector with reserved memory? My suggestions have nothing to do with compression and streams. But you might be optimizing prematurely here. HTH, Kris

Kenneth Adam Miller

7:24 p.m.

Ah ok. I apologize for posting too fast. I will be sure to go further in exhausting all resources before I post in the future. However, I did find some of the descriptions of the iostreams parts and how they compose (especially examples, that's what I was really looking for) vague, IMHO. No, I haven't measured performance. And I understand well your concern with inappropriate attention to performance. The general advice is make it work first, then make it work faster; I agree with this advice strongly, but I think you might agree with me if I explain my context. If I see a particular solution that I believe will more likely guarantee lower latency due to unnecessary reallocations and re-initializations (as when a vector expands and must make all memory that it contains contiguous), I will try to eliminate it in the process of making it work the first time. I'm writing a pintool, and you just kind of have to understand what that is in order to know where I'm coming from. Intel's PIN tool suite is a dynamic binary instrumentation framework designed to allow programmers to specify callbacks at different levels of granularity, as well as write their own instrumentation functions. So if you wanted to gather data about every image load, or instruction execution, or routine call, on the fly you could do it. Anyway, the context in which I am using this compression utility is a particularly important one for speed. A set of dynamic analysis routines steadily generate data from the target program that is being analyzed, and the point where I'm asking about now has to do with where, once the analysis threads have generated enough data, they report a handle to their buffer to a pool of compression threads; the compression threads drop the data they've been handed directly into a fresh buffer (ideally) that goes through a compressor. The problem is, because I'm instrumenting at instruction level granularity, my analysis code, between target application threads has to synchronize a lamport clock (now I'm really getting out of hand with my explanation lol); so, the whole program might produce a 400mb xml file, and these buffers get flushed at the frequency of accumulating about 30kb. So that's a lot of flushing! Those unnecessary allocation copies would, in the worst case, result in a 2-3x slowdown of the entire program due to the fact that it's literally pausing to repeat something that's unnecessary. But actually, if this appends my 30k string, I didn't know it. The back_insert_device<string> object wasn't something that I understood very well from the description at http://www.boost.org/doc/libs/1_45_0/libs/iostreams/doc/classes/back_inserte...

...

From what I had thought though, if my string was empty with 0's, and I appended to it, then I would end up with precisely my compressed data. The idea of acquireStringFromPool is that it returns a large string that is always cleared out for the filtering_ostream to fill up as though it were a device. Is that incorrect? How can I get that functionality?

Perhaps, however, that reserve approach would be good. I apologize the email was so long lol. On Tue, Jan 14, 2014 at 12:16 PM, Krzysztof Czainski <1czajnik@gmail.com>wrote:

...

2014/1/14 Kenneth Adam Miller

...
By the way, I'm working on a master's thesis, so I frequently skip sleep. Sometimes after a lack of sleep, getting across precisely what is needed/understood can take some an iteration or two :)

Good luck with your thesis ;-)

Btw, I think top-posting is inappropriate on this list.

On Tue, Jan 14, 2014 at 10:15 AM, Kenneth Adam Miller <

...
kennethadammiller@gmail.com> wrote:

...
Pretty much on performance concerns. I know that there's at least going to be one copy performed while doing the compression, from uncompressed to compressed. Here's how I do it:

filtering_ostream *fos = new filtering_ostream(); fos->push(bzip2_compressor()); string *x = acquireStringFromPool(); //This is just a blocking pointer return that reaches into a list of string *, each that are allocated with new string(30000,0); (it's multithreaded, ok lol :) ) fos->push(boost::iostreams::back_insert_device<string>(x)); //This is what I was searching for all along.

Doesn't this append your 30k string?

If a preallocated chunk of memory is ok for you, check out vector::reserve() ;-)

then later, when I want to write to fos I do,

...
...
*fos << *doc; //I go straight from container to compression.

Maybe my specifications that "I don't want to find that it's copying at all" were a bit weird, because obviously it has to move the data right? I'm just saying that most of the examples I would see would be something like

compress(string x) { stringstream ss(x); //unnecessary initialization in my case, couldn't find an example without this //something similar to what I did... }

OK, so did you measure performance of your solution compared to the above example? And then to some version with std::vector with reserved memory?

My suggestions have nothing to do with compression and streams. But you might be optimizing prematurely here.

HTH, Kris

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Krzysztof Czainski

7:55 p.m.

2014/1/14 Kenneth Adam Miller

...

Ah ok. I apologize for posting too fast. I will be sure to go further in exhausting all resources before I post in the future. However, I did find some of the descriptions of the iostreams parts and how they compose (especially examples, that's what I was really looking for) vague, IMHO.

No, I haven't measured performance.

[...]

...

Those unnecessary allocation copies would, in the worst case, result in a 2-3x slowdown of the entire program

How do you know?

...

due to the fact that it's literally pausing to repeat something that's unnecessary.

But actually, if this appends my 30k string, I didn't know it. The back_insert_device<string> object wasn't something that I understood very well from the description at http://www.boost.org/doc/libs/1_45_0/libs/iostreams/doc/classes/back_inserte...

From what I had thought though, if my string was empty with 0's,

Right here's your mistake. string(30000,0) creates a non empty string. It's a string of 30k nulls.

...

and I appended to it, then I would end up with precisely my compressed data. The idea of acquireStringFromPool is that it returns a large string that is always cleared out for the filtering_ostream to fill up as though it were a device. Is that incorrect? How can I get that functionality?

Like I said, you want std::vector::reserve(). Perhaps, however, that reserve approach would be good. I apologize the

...

email was so long lol.

Btw, you're still top-posting. The idea is to leave the (minimal) context above your reply message ;-) Regards, Kris

Kenneth Adam Miller

7:58 p.m.

Thank you. On Tue, Jan 14, 2014 at 1:55 PM, Krzysztof Czainski <1czajnik@gmail.com>wrote:

...

2014/1/14 Kenneth Adam Miller

...
Ah ok. I apologize for posting too fast. I will be sure to go further in exhausting all resources before I post in the future. However, I did find some of the descriptions of the iostreams parts and how they compose (especially examples, that's what I was really looking for) vague, IMHO.

No, I haven't measured performance.

[...]

...
Those unnecessary allocation copies would, in the worst case, result in a 2-3x slowdown of the entire program

How do you know?

...
due to the fact that it's literally pausing to repeat something that's unnecessary.

But actually, if this appends my 30k string, I didn't know it. The back_insert_device<string> object wasn't something that I understood very well from the description at http://www.boost.org/doc/libs/1_45_0/libs/iostreams/doc/classes/back_inserte...

From what I had thought though, if my string was empty with 0's,

Right here's your mistake. string(30000,0) creates a non empty string. It's a string of 30k nulls.

...
and I appended to it, then I would end up with precisely my compressed data. The idea of acquireStringFromPool is that it returns a large string that is always cleared out for the filtering_ostream to fill up as though it were a device. Is that incorrect? How can I get that functionality?

Like I said, you want std::vector::reserve().

Perhaps, however, that reserve approach would be good. I apologize the

...
email was so long lol.

Btw, you're still top-posting. The idea is to leave the (minimal) context above your reply message ;-)

By top posting you meant just the position of my text. I thought you meant it to be me posting when I could have more fully investigated the issue. I will follow this convention.

...

Regards, Kris

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Kenneth Adam Miller

15 Jan 15 Jan

1:46 a.m.

On Tue, Jan 14, 2014 at 1:58 PM, Kenneth Adam Miller < kennethadammiller@gmail.com> wrote:

...

Thank you.

On Tue, Jan 14, 2014 at 1:55 PM, Krzysztof Czainski <1czajnik@gmail.com>wrote:

...
2014/1/14 Kenneth Adam Miller

...
Ah ok. I apologize for posting too fast. I will be sure to go further in exhausting all resources before I post in the future. However, I did find some of the descriptions of the iostreams parts and how they compose (especially examples, that's what I was really looking for) vague, IMHO.

No, I haven't measured performance.

[...]

...
Those unnecessary allocation copies would, in the worst case, result in a 2-3x slowdown of the entire program

How do you know?

...
due to the fact that it's literally pausing to repeat something that's unnecessary.

But actually, if this appends my 30k string, I didn't know it. The back_insert_device<string> object wasn't something that I understood very well from the description at http://www.boost.org/doc/libs/1_45_0/libs/iostreams/doc/classes/back_inserte...

From what I had thought though, if my string was empty with 0's,

Right here's your mistake. string(30000,0) creates a non empty string. It's a string of 30k nulls.

...
and I appended to it, then I would end up with precisely my compressed data. The idea of acquireStringFromPool is that it returns a large string that is always cleared out for the filtering_ostream to fill up as though it were a device. Is that incorrect? How can I get that functionality?

Like I said, you want std::vector::reserve().

Ok, so supposing I just use a vector; do I use vector<byte>? Is it more efficient to use vector< word_size_of_computer > since any write to it would obviously be faster or will the compiler catch this? Also, when I grab all of the bytes from vector, does it automatically detect when the end is? Like, one of the purposes of having 0's all through the string was so that it was always terminated. In the case of vector, if I push_back 3 times, and then I output, will I get 3 bytes?

...

...
Perhaps, however, that reserve approach would be good. I apologize the

...
email was so long lol.

Btw, you're still top-posting. The idea is to leave the (minimal) context above your reply message ;-)

By top posting you meant just the position of my text. I thought you meant it to be me posting when I could have more fully investigated the issue. I will follow this convention.

...
Regards, Kris

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Gavin Lambert

5:14 a.m.

On 15/01/2014 14:46, Quoth Kenneth Adam Miller:

...

From what I had thought though, if my string was empty with 0's,

Right here's your mistake. string(30000,0) creates a non empty string. It's a string of 30k nulls.

[...]

...

Ok, so supposing I just use a vector; do I use vector<byte>? Is it more efficient to use vector< word_size_of_computer > since any write to it would obviously be faster or will the compiler catch this? Also, when I grab all of the bytes from vector, does it automatically detect when the end is? Like, one of the purposes of having 0's all through the string was so that it was always terminated. In the case of vector, if I push_back 3 times, and then I output, will I get 3 bytes?

You can still use a string if you want, but you're not initialising the pool correctly. std::string *p1 = new std::string(30000, 0); This declaration creates a string that is 30k characters long, where all characters happen to be 0. (It is legal for a string to contain embedded nulls.) If you appended to this string, you would have a 30k+1 character string, and it *might* do a memory allocation. This is obviously not what you want. std::string *p2 = new std::string(); p2->reserve(30000); This declaration creates an empty string with an initial capacity of at least 30k characters; if you append a char to it then the string will be 1 char long. It should not allocate any further memory unless you try to append more characters than that. (Though test it, as the STL is permitted to ignore this.) Also string will automatically ensure that it is always null-terminated when viewed from .c_str(), so you normally don't need to worry about filling with zeroes or anything like that, and you should not try to append a zero yourself unless you actually want that in the string data. But since you're talking about compressed data here, it will no longer be printable characters and could potentially contain a 0 as actual data. Either vector or string can cope with this just fine (vector is typically more efficient at appending, but less efficient at copying) but either way you'll need to use the .size() to determine how much data is actually in there instead of relying on any termination.

Kenneth Adam Miller

5:20 a.m.

ah that's a really good explanation, thanks so much. After staring at code for hours, I start to question things like the laws of simply being. On Tue, Jan 14, 2014 at 11:14 PM, Gavin Lambert wrote:

...

On 15/01/2014 14:46, Quoth Kenneth Adam Miller:

...
From what I had thought though, if my string was empty with 0's,

Right here's your mistake. string(30000,0) creates a non empty string. It's a string of 30k nulls.

[...]

...
Ok, so supposing I just use a vector; do I use vector<byte>? Is it more efficient to use vector< word_size_of_computer > since any write to it would obviously be faster or will the compiler catch this? Also, when I grab all of the bytes from vector, does it automatically detect when the end is? Like, one of the purposes of having 0's all through the string was so that it was always terminated. In the case of vector, if I push_back 3 times, and then I output, will I get 3 bytes?

You can still use a string if you want, but you're not initialising the pool correctly.

std::string *p1 = new std::string(30000, 0);

This declaration creates a string that is 30k characters long, where all characters happen to be 0. (It is legal for a string to contain embedded nulls.) If you appended to this string, you would have a 30k+1 character string, and it *might* do a memory allocation. This is obviously not what you want.

std::string *p2 = new std::string(); p2->reserve(30000);

This declaration creates an empty string with an initial capacity of at least 30k characters; if you append a char to it then the string will be 1 char long. It should not allocate any further memory unless you try to append more characters than that. (Though test it, as the STL is permitted to ignore this.)

Also string will automatically ensure that it is always null-terminated when viewed from .c_str(), so you normally don't need to worry about filling with zeroes or anything like that, and you should not try to append a zero yourself unless you actually want that in the string data.

But since you're talking about compressed data here, it will no longer be printable characters and could potentially contain a 0 as actual data. Either vector or string can cope with this just fine (vector is typically more efficient at appending, but less efficient at copying) but either way you'll need to use the .size() to determine how much data is actually in there instead of relying on any termination.

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Kenneth Adam Miller

16 Jan 16 Jan

4:08 a.m.

On Tue, Jan 14, 2014 at 7:46 PM, Kenneth Adam Miller < kennethadammiller@gmail.com> wrote:

...

On Tue, Jan 14, 2014 at 1:58 PM, Kenneth Adam Miller < kennethadammiller@gmail.com> wrote:

...
Thank you.

On Tue, Jan 14, 2014 at 1:55 PM, Krzysztof Czainski <1czajnik@gmail.com>wrote:

...
2014/1/14 Kenneth Adam Miller

...
Ah ok. I apologize for posting too fast. I will be sure to go further in exhausting all resources before I post in the future. However, I did find some of the descriptions of the iostreams parts and how they compose (especially examples, that's what I was really looking for) vague, IMHO.

No, I haven't measured performance.

[...]

...
Those unnecessary allocation copies would, in the worst case, result in a 2-3x slowdown of the entire program

How do you know?

...
due to the fact that it's literally pausing to repeat something that's unnecessary.

But actually, if this appends my 30k string, I didn't know it. The back_insert_device<string> object wasn't something that I understood very well from the description at http://www.boost.org/doc/libs/1_45_0/libs/iostreams/doc/classes/back_inserte...

From what I had thought though, if my string was empty with 0's,

Right here's your mistake. string(30000,0) creates a non empty string. It's a string of 30k nulls.

...
and I appended to it, then I would end up with precisely my compressed data. The idea of acquireStringFromPool is that it returns a large string that is always cleared out for the filtering_ostream to fill up as though it were a device. Is that incorrect? How can I get that functionality?

Like I said, you want std::vector::reserve().

Ok, so supposing I just use a vector; do I use vector<byte>? Is it more efficient to use vector< word_size_of_computer > since any write to it would obviously be faster or will the compiler catch this? Also, when I grab all of the bytes from vector, does it automatically detect when the end is? Like, one of the purposes of having 0's all through the string was so that it was always terminated. In the case of vector, if I push_back 3 times, and then I output, will I get 3 bytes?

Also, supposing I go with something like vector or something, how do I create a device to make the filtering stream complete? I had: fos->push(boost::iostreams::back_insert_device<string>(*x)); now I need to do: fos->push(boost::iostreams::back_insert_device>(*x)); but it complains with "error: no matching function for call to" construct it, and then says that it wants one of a ton of different valid things, but it template expands all of them so it's a bunch of text...

...

...
...
Perhaps, however, that reserve approach would be good. I apologize the

...
email was so long lol.

Btw, you're still top-posting. The idea is to leave the (minimal) context above your reply message ;-)

By top posting you meant just the position of my text. I thought you meant it to be me posting when I could have more fully investigated the issue. I will follow this convention.

...
Regards, Kris

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Gavin Lambert

10:03 p.m.

On 16/01/2014 17:08, Quoth Kenneth Adam Miller:

...

Also, supposing I go with something like vector or something,

Why would you? Unless you can guarantee that you're always going to get a multiple of 8 bytes after compression, that seems like asking for trouble.

...

how do I create a device to make the filtering stream complete? I had: fos->push(boost::iostreams::back_insert_device<string>(*x)); now I need to do: fos->push(boost::iostreams::back_insert_device>(*x)); but it complains with "error: no matching function for call to" construct it, and then says that it wants one of a ton of different valid things, but it template expands all of them so it's a bunch of text...

Your ostream is templated on char, therefore it produces chars. You can't just insert a char into a vector of int64s.

Kenneth Adam Miller

10:27 p.m.

Ah ok. Well I didn't know, what I thought was, if the word size of the machine was 8 bytes, then each clock cycle move would result in 8 bytes being moved. I wasn't sure though that's why I asked. I know it's silly to consider this level of optimization, but I question things extremely hard lol. Thanks, the list of errors the compiler was giving was just overwhelming. On Thu, Jan 16, 2014 at 4:03 PM, Gavin Lambert wrote:

...

On 16/01/2014 17:08, Quoth Kenneth Adam Miller:

Also, supposing I go with something like vector or something,

...
Why would you? Unless you can guarantee that you're always going to get a multiple of 8 bytes after compression, that seems like asking for trouble.

how do I create a device to make the filtering stream complete? I had:

...
fos->push(boost::iostreams::back_insert_device<string>(*x)); now I need to do: fos->push(boost::iostreams::back_insert_device>(*x)); but it complains with "error: no matching function for call to" construct it, and then says that it wants one of a ton of different valid things, but it template expands all of them so it's a bunch of text...

Your ostream is templated on char, therefore it produces chars. You can't just insert a char into a vector of int64s.

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Gavin Lambert

10:48 p.m.

On 17/01/2014 11:27, Quoth Kenneth Adam Miller:

...

Ah ok. Well I didn't know, what I thought was, if the word size of the machine was 8 bytes, then each clock cycle move would result in 8 bytes being moved. I wasn't sure though that's why I asked. I know it's silly to consider this level of optimization, but I question things extremely hard lol. Thanks, the list of errors the compiler was giving was just overwhelming.

You can get that speed benefit if you use a string and you make a copy of the entire string -- the STL usually optimises this to a memcpy, which will do an intelligent data blit. Vector typically doesn't do that, it'll make an element-by-element copy, although the compiler *might* be smart enough to optimise that away to a memcpy-like blit. But neither is likely to be any better than the other for insertion. If your source is producing chars then you're just going to waste more time trying to merge and combine writes than it would to just do byte writes in the first place. And a vector of 64-bit values doesn't have any way to express a number of bytes less than a multiple of 64 bits, so you would be forced to have padding.

Kenneth Adam Miller

10:58 p.m.

Ok. Well all my producer does is append. Vector sounds good for the ability to do random access fast like an array, and you said that vector has good append speed. On Thu, Jan 16, 2014 at 4:48 PM, Gavin Lambert wrote:

...

On 17/01/2014 11:27, Quoth Kenneth Adam Miller:

Ah ok. Well I didn't know, what I thought was, if the word size of the

...
machine was 8 bytes, then each clock cycle move would result in 8 bytes being moved. I wasn't sure though that's why I asked. I know it's silly to consider this level of optimization, but I question things extremely hard lol. Thanks, the list of errors the compiler was giving was just overwhelming.

You can get that speed benefit if you use a string and you make a copy of the entire string -- the STL usually optimises this to a memcpy, which will do an intelligent data blit.

Vector typically doesn't do that, it'll make an element-by-element copy, although the compiler *might* be smart enough to optimise that away to a memcpy-like blit.

But neither is likely to be any better than the other for insertion. If your source is producing chars then you're just going to waste more time trying to merge and combine writes than it would to just do byte writes in the first place. And a vector of 64-bit values doesn't have any way to express a number of bytes less than a multiple of 64 bits, so you would be forced to have padding.

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Gavin Lambert

11:12 p.m.

On 17/01/2014 11:58, Quoth Kenneth Adam Miller:

...

Ok. Well all my producer does is append. Vector sounds good for the ability to do random access fast like an array, and you said that vector has good append speed.

Random access should be the same speed for both. Appending will be slightly faster for the vector than for the string, because the string also has to write a new terminator. But if you can get the data to be appended as a complete sequence (either a pair of iterators or a start pointer and length), then string typically wins again, as you can use a single .append() instead of multiple .push_back()s. But as always, these are just approximate guidelines; to be really sure, you need to test and measure.

3954

Age (days ago)

3956

Last active (days ago)

List overview

Download

18 comments

3 participants

participants (3)

Gavin Lambert
Kenneth Adam Miller
Krzysztof Czainski