Hi there, I did a little performance test of io_new vs io2. The code is here. http://pastebin.com/eCeFZuMS On my machine io_new runs in 1.15sec for 2658 jpegs. IO2 run in 1.005sec. All jpegs are the same image but they are a different file. This way I'm sure I don't have to convert the image into a different color space. I did that since io2 doesn't support any_image, I believe. Thanks, Christian
Hi there, I did a little update to include more files. Now I have 26580 jpeg files distributed over 10 folders. I also tried to reduce the number of image memory allocation by reusing the same image over and over again. The new code can be found here: http://pastebin.com/B8R1MuCs I run each backend three times. Here are the results: IO2: 1. 30.975 sec 2. 29.894 sec 3. 25.595 sec io_new: 1. 33.086 sec 2. 28.223 sec 3. 28.867 sec So far, I have changed any code inside my backend ( io_new ). The next step is to profile. Regards, Christian On Tue, Dec 14, 2010 at 3:26 PM, Christian Henning <chhenning@gmail.com> wrote:
Hi there, I did a little performance test of io_new vs io2. The code is here. http://pastebin.com/eCeFZuMS
On my machine io_new runs in 1.15sec for 2658 jpegs. IO2 run in 1.005sec.
All jpegs are the same image but they are a different file. This way I'm sure I don't have to convert the image into a different color space. I did that since io2 doesn't support any_image, I believe.
Thanks, Christian
On 15/12/10 23:46, Christian Henning wrote:
Hi there, I did a little update to include more files. Now I have 26580 jpeg files distributed over 10 folders. I also tried to reduce the number of image memory allocation by reusing the same image over and over again. The new code can be found here: http://pastebin.com/B8R1MuCs
I run each backend three times. Here are the results:
IO2: 1. 30.975 sec 2. 29.894 sec 3. 25.595 sec
io_new: 1. 33.086 sec 2. 28.223 sec 3. 28.867 sec
So far, I have changed any code inside my backend ( io_new ). The next step is to profile.
It looks very good. I'm going to run this benchmark on Linux over the weekend. In the meantime, I'm caching your test program at my Git repo; https://github.com/mloskot/workshop/tree/master/benchmarking/tiling/gil/io_n... Cheers, -- Mateusz Loskot, http://mateusz.loskot.net Charter Member of OSGeo, http://osgeo.org Member of ACCU, http://accu.org
"Christian Henning" <chhenning@gmail.com> wrote in message news:AANLkTimTSE6YRJDKqG120cyn9hosC9QaQBTQf40JLdDF@mail.gmail.com...
Hi there, I did a little update to include more files. Now I have 26580 jpeg files distributed over 10 folders. I also tried to reduce the number of image memory allocation by reusing the same image over and over again.
But you still used boost::filesystem which is also not the most lightweight thing in the Universe (for example, simply adding its cpp files to my test project added ~150kB to the release binary...also between the "if ( fs::is_directory( dir_path ) )" line and the first read_image() call I counted 16 memory allocations and 6 memory allocations between each subsequent call to read_image()...)... Your test description is also missing relevant information, such as the test machine's hardware and software configuration, third party library versions and compiler and linker options used to build them and the test project... But, as argued to Lubomir, this 'obfuscation' or 'damping' of results (i.e. the comparison between different backend wrappers) with filesystem access is IMO moot...
All jpegs are the same image but they are a different file. This way I'm sure I don't have to convert the image into a different color space. I did that since io2 doesn't support any_image, I believe.
I did add any_image support at the beginning but did not test it in a long time so I don't know whether it still works...but, why would you need any_image even if you did use different images with different formats...the backend/backend wrapper can do the conversion (e.g. your read_and_convert_image())... -- "What Huxley teaches is that in the age of advanced technology, spiritual devastation is more likely to come from an enemy with a smiling face than from one whose countenance exudes suspicion and hate." Neil Postman
Hi Domagoj, good to hear back from you. Hope all is well. On Sun, Jan 16, 2011 at 8:58 AM, Domagoj Saric <dsaritz@gmail.com> wrote:
"Christian Henning" <chhenning@gmail.com> wrote in message news:AANLkTimTSE6YRJDKqG120cyn9hosC9QaQBTQf40JLdDF@mail.gmail.com...
Hi there, I did a little update to include more files. Now I have 26580 jpeg files distributed over 10 folders. I also tried to reduce the number of image memory allocation by reusing the same image over and over again.
But you still used boost::filesystem which is also not the most lightweight thing in the Universe (for example, simply adding its cpp files to my test project added ~150kB to the release binary...also between the "if ( fs::is_directory( dir_path ) )" line and the first read_image() call I counted 16 memory allocations and 6 memory allocations between each subsequent call to read_image()...)...
Maybe so, but I use filesystem in both tests. Meaning the overhead is prevalent in my IO and your IO. Also the code is platform independent. How do you count these memory allocations? I did some changes in io_new. No strings anymore, for example.
Your test description is also missing relevant information, such as the test machine's hardware and software configuration, third party library versions and compiler and linker options used to build them and the test project...
Sorry, my bad. I use a pretty powerful box with 2 Intel Xeon E5630 and 12GB of RAM. For compiling I used VS2010 on Windows 7. For boost I'm using some trunk version for the 1.46 release.
But, as argued to Lubomir, this 'obfuscation' or 'damping' of results (i.e. the comparison between different backend wrappers) with filesystem access is IMO moot...
There are several things that I still need to make better use of libjpg.
All jpegs are the same image but they are a different file. This way I'm sure I don't have to convert the image into a different color space. I did that since io2 doesn't support any_image, I believe.
I did add any_image support at the beginning but did not test it in a long time so I don't know whether it still works...but, why would you need any_image even if you did use different images with different formats...the backend/backend wrapper can do the conversion (e.g. your read_and_convert_image())...
I should get the latest from your project I suppose. Regards, Christian
"Christian Henning" <chhenning@gmail.com> wrote in message news:AANLkTi=xCtCvgjnW4XqMXux==0Mdjnm3zL8_PzXVdMQG@mail.gmail.com...
Hi Domagoj, good to hear back from you. Hope all is well.
Hi, thanks, well other than a little insomnia all is well ;-D
On Sun, Jan 16, 2011 at 8:58 AM, Domagoj Saric <dsaritz@gmail.com> wrote:
"Christian Henning" <chhenning@gmail.com> wrote in message news:AANLkTimTSE6YRJDKqG120cyn9hosC9QaQBTQf40JLdDF@mail.gmail.com...
Hi there, I did a little update to include more files. Now I have 26580 jpeg files distributed over 10 folders. I also tried to reduce the number of image memory allocation by reusing the same image over and over again.
But you still used boost::filesystem which is also not the most lightweight thing in the Universe (for example, simply adding its cpp files to my test project added ~150kB to the release binary...also between the "if ( fs::is_directory( dir_path ) )" line and the first read_image() call I counted 16 memory allocations and 6 memory allocations between each subsequent call to read_image()...)...
Maybe so, but I use filesystem in both tests. Meaning the overhead is prevalent in my IO and your IO.
? That exactly is the problem...by adding constant overhead to all benchmarked cases you diminish/'dampen' the resulting difference percentage...(otherwise you could have left the image allocation inside the main loop...)
How do you count these memory allocations?
A breakpoint in ::operator new() ?
I did some changes in io_new. No strings anymore, for example.
The allocations I mentioned were (only the ones) done in/by boost::filesystem...
Your test description is also missing relevant information, such as the test machine's hardware and software configuration, third party library versions and compiler and linker options used to build them and the test project...
Sorry, my bad. I use a pretty powerful box with 2 Intel Xeon E5630 and 12GB of RAM. For compiling I used VS2010 on Windows 7. For boost I'm using some trunk version for the 1.46 release.
3rd party lib versions, build options, image(s) used (e.g. as I've described in the post to Mateusz)... But, the bottom line is why are we still wasting time with this (comparing io_new and io2) when the difference (at least regarding efficiency) is obvious... If we are going to make a 'merge' as was previously speculated (although I don't actually see how is that possible considering the substantial difference in interfaces and the fact that io_new has already been accepted based on its present interface) it makes no sense to fix what is deficient in io_new if there is a better counterpart in io2 and vice verse... -- "What Huxley teaches is that in the age of advanced technology, spiritual devastation is more likely to come from an enemy with a smiling face than from one whose countenance exudes suspicion and hate." Neil Postman
participants (3)
-
Christian Henning -
Domagoj Saric -
Mateusz Loskot