Domagoj Saric wrote:
"Phil Endecott" <spam_from_boost_dev@chezphil.org> wrote in message news:1291751283459@dmwebmail.dmwebmail.chezphil.org...
Having said that, I suspect that even with lots of extensions GIL might not really be what I need. Here's a typical problem that I will have to address in the next few weeks: I have a 1e12 pixel image - a map - which is supplied as a few thousand 5000x5000 TIFF tiles; I have to join them together, tweak the colours, draw some lines on top, and re-chop it into a few million 256x256 PNG tiles. The issue of course is that that image can't be held in RAM. I will do this with my own wrappers around libtiff and libpng, in rows or chunks.
Hi Phil, if you are willing to try (and can work/test with MSVC on Windows) I think io2 should already be able to help you here (partial ROI based access to big images) at least with the WIC backend...
I'm doing this on Linux. Wikipedia tells me WIC is "Windows Imaging Component".
Example code demonstrating a possible (skeleton) solution: http://codepad.org/WD7CpIJ8 ...
I don't really follow what that code is doing, and it's not obvious to me what its memory footprint will be. In contrast, I think I can write something that's not much longer and more obviously correct by using the sort of simple wrappers around the libraries that I have been proposing and explicitly managing the tiled input and output: class TiledReadImage { typedef shared_ptr<ReadTiff> readtiff_ptr; readtiff_ptr images[1400]; // One row of input tiles int rownum; // in pixels void open_next_row() { for (int c = 0; c < 1400; ++c) { images[c] = new ReadTiff(input_tile_filename(c,rownum/5000); } } public: TiledReadImage(): rownum(0) {} read_row(pixel_t* data) { if (rownum%5000 == 0) { open_next_row(); } for (int c = 0; c < 1400; ++c) { images[c]->read_row(data+5000*c); } ++rownum; } }; // Something similar for TiledWriteImage TiledReadImage i; TiledWriteImage o; pixel_t data[700000]; for (int row=0; row<1300000; ++row) { i.read_row(data); o.write_row(data); } Anyone can look at that and see that it keeps 1400 input files open (bad) and needs 700 kwords of buffer memory (good). If you wanted to have fewer files open you could code explicitly something else that had one input file open (good) and buffered complete tiles (probably 2 x 5000 x 700,000 = 7 GBytes) (bad).
ps. unfortunately I do not have access to such a huge image to test whether WIC can actually handle such monster images...
If you're interested in experimenting, I suggest making random input tiles or just replicating them.
On the grounds that this is better than what is currently there, and on the assumption that Domagoj Saric is not imminently going to post something else that would supersede this, I believe that it should be accepted.
If we've already waited so long, why rush if some of us agree that there are still things that need 'polishing'...
To get the ball moving on GIL extensions, and because this is better than what GIL currently has. Past experience suggests that the existence of this io extension will not prevent other similar and incompatible things (i.e. yours) from being accepted in the future.
You said you already did some LibXXX wrappers of your own...if this is so...why not join the effort even if only temporary to make sure you get what you want...Christian also seems open for cooperation...
My wrappers have all been written to implement some subset of the functionality that I needed at the time. For example I have never implemented any of the error handling stuff, and in some cases I have only read or write but not both. Regards, Phil.