[Serialization] serializing handles to pointers, and serializing as garbage collection

I'm a bit new to boost and am trying to determine if I can coax the existing boost serialization system to work for a fairly standard asset loading system. Assets in my domain of game programming are things such as bitmaps, model geometry, audio data, shaders, and various chunks of meta data attached to them. The quantity of such data should always be assumed to vastly exceed the ability to load it all at once, which means getting sneaky when dealing with persistence. From my point of view the boost::serialization code is missing a lot of important features, so I am kind of surprised that they are not there. A) Serializing for persistence: Assets can reference other assets via pointers, but serializing for persistence should only store names of other assets, and query a resource manager to help convert the pointer into a name (or vice versa). A model object might reference via a pointer a shader and a texture in addition to its geometric data. When I serialize the model for persistence I want to store the geometric data, but when it comes to the shader and textures I only want to store their names. I could manually do this in each method's serialize methods. However the serialization system should also be able to do this via type traits, so that I don't have to hack up the serialize methods. The type trait method for this is preferable, since the method bodies of serialize could then be written as the originally would want to be, so that completely different kinds of archivers can operate on them, especially ones that depend on actually being able to walk into the pointers. Its the archive's call how to walk the data really. B) Serializing for garbage collection: A full set of raw assets for an engine can easily push into the terabyte range, which means you never actually work with all of the data at once. To work you need to be able to load what you need as you go, but to keep your working set down you need to evict unused assets as much as possible. Another type trait could be used to build a list of reachable objects containing the type trait from an array of root objects which are serialized. You can then compare this list to the full list of what the resource manager has currently loaded, and either delete or save the modified but now unreachable nodes. This system has the advantage of being useful for other purposes, such as making making an asset browser, without having to also implement a full blown system for reflection and introspection. C) Serializing for counting memory: You can also make the archive simply count the memory overhead of your objects. It needs the ability to count memory without counting the memory twice, and it also needs to be able to either do this for a single object (i.e. don't follow reference to other assets), or for a hierarchy (i.e. 'how much memory does using this asset require, factoring in its dependencies'). So my real question boils down to : I can't really be the first person to want it to be able to do these things right? Am I crazy for wanting to coax the boost serialization code into working this way?

I looked at the email and really didn't understand what you wanted the
library to do that it doesn't already do. I could have starting a back and
forth trying to figure this out, but I think it would better just to
summarize what the library does do.
The library saves and restores any data structures which can be represented
in C++.
Soooooooo. If you have some dataset which cannot fit into memory all at
once, its not really representable as a C++ data structure. What you would
have is a C++ data structure for handling data too large to fit all at once
in memory. Presumably you already have some sort of infrastructure for
handling this data (or plan to build one). If what you are asking is
whether serialization library provides such an infrastructure, the answer
would be No - it does not.
Howeever, it might still be helpful. Suppose you have some sort set up some
sort of code for paging, on-demand load or what ever implemented in C++
which pointed to or managed some sort of off-line storage, the C++
implemenation could save its state and restore it which seems to me would be
what you want.
Its also possible that the serialization library might be used to save and
restore pieces at a time.
Sooooo, I don't think the serialization library addresses what you want, but
it may (or may not) be helpful in achieving your goal.
Robert Ramey
"Sean Cavanaugh"
participants (2)
-
Robert Ramey
-
Sean Cavanaugh