I remember looking at GStreamer, and IIRC it gave me the impression that one could not build a C++ - only GStreamer app. I could be wrong, though - perhaps you can share your experience on this?
From what I have done so far, it does not seem possible to be 100% C++. On their mail lists, they say that it is possible, so that may just be due to my lack of knowledge. I'm still a little new at c/c++. What I have done is to use the standard GStreamer C code as a wrapper for my C++ objects. So I end up with a C++ class that really does all the work. What you are doing, seems very similar to GStreamer to me. Although it is billed as a multimedia framework, it does not have to be audio/video data. It will work with any mime type. It is just that what is already there is geared towards. It would probably be worth taking a look at, if you are seeking some ideas. The learning curve is a little steep, but you can dive in pretty quickly with the basics.