Adobe Source Libraries, GIL, and Boost

Lubomir pointed me at the GIL thread on the list - and I wanted to make a few comments about how I see the relationship between Boost and ASL. The charter for Boost is to vet libraries for possible inclusion in the standard. The charter for ASL is to explore the use of generic and declarative techniques to develop large applications. Our goal is to replace 85% of the imperative code being developed with small declarative constructs. Certainly there is going to be some overlap between the two project and ASL makes _heavy_ use of Boost and borrows ideas from Boost quite freely. For parts of ASL, inclusion into Boost makes perfect sense - and we try to actively defer to Boost when a Boost library can meet our needs. Boost certainly has a significantly larger user base than ASL and far more active developers (anyone interested in helping out ASL???). I _really_ wish that I had enough time/resources on my team to take some parts of ASL and contribute them to Boost. However, my teams focus currently needs to stay on the higher level libraries (which I don't think are ready for Boost yet) and we simply can't afford the time to go through the submission process. But I'm 100% supportive of Lubomir's contribution of GIL (Lubomir is not on my team, and I'm thrilled that he can take the time for this submission to Boost - just I'm I'm thrilled that we were able to release it as part of ASL). GIL is a very significant piece of work. I will also support anyone who would like to borrow portions of ASL to contribute them to Boost. Some libraries that I think would be interesting to get into Boost with some comments about what I see as still needing to be done. --- Forest <http://opensource.adobe.com/group__asl__tutorials__forest.html> I've started to map the concepts from forest into the terminology of BGL <http://opensource.adobe.com/wiki/index.php/ Edge_Interface_For_Forest_Iterators>. There is also some work here about generic algorithms for node based iterators (see the undocumented code for reverse_append here <http:// opensource.adobe.com/reverse_8hpp-source.html>. With a bit more polish I think this could be a _really_ nice collections of node based algorithms and concepts, which could be applied to a single linked list and other pre-traversed structures such as forest. There are also some forest specific algorithms I'd like to see such as the rotate necessary for balancing AVL trees (any forest is a binary tree, and vice versa - see Knuth). Range Based Algorithms (using the boost range library) <http:// opensource.adobe.com/group__standard__extensions.html> I've just started busting these out into separate headers (along the boost style - for example adobe/algorithm/reverse.hpp) - for inclusion in Boost I'd like to see this complete. This would be pretty simple to incorporate into Boost (and they are quite handy!). CMath C99 compatibility <http://opensource.adobe.com/group__cmath.html> There are also some rounding modes which were missing - This seems like a pretty straight forward "grab". ZUIDs <http://opensource.adobe.com/classadobe_1_1zuid__t.html> I'd like to see some review of the platform dependent code which gathers entropy to ensure we're getting enough entropy. A short tutorial would be good. ZUIDs would be a good addition to the serialization library. md5 and SHA code <http://opensource.adobe.com/classadobe_1_1md5__t.html> <http://opensource.adobe.com/group__adobe__sha.html> Review and a short tutorial ONCE <http://opensource.adobe.com/group__adobe__once.html> The STATIC_INSTANCE stuff might be able to be removed - this was done for the Mac where static initializer on DLLs were getting executed on a thread after load so other functions in the compilation unit could be executed concurrently. That may have been fixed (I haven't tracked the issue) - this library would be a good addition to Boost threads. ThreadID <http://opensource.adobe.com/group__asl__thread__id.html> Another good addition to Boost threads. Final <http://opensource.adobe.com/group__adobe__final.html> I don't use inheritance much but this would be a nice addition to Boost utilities. Counter <http://opensource.adobe.com/classadobe_1_1counter__t.html> If boost would surface the lightweight mutex used in shared_ptr - this would go away. (This is a heavyweight one - as I'm counting on Boost providing a lightweight one in the future :-) ). value_t <http://opensource.adobe.com/classadobe_1_1value__t.html> I think there are some nice ideas here which could be stolen for boost::any - value_t also captures equality comparison, makes use of a variant to avoid heap allocations for some common types (I'm not sure but I think boost any does two heap allocations, and value_t never does more than one). regular_object <http://opensource.adobe.com/ classadobe_1_1regular__object.html> This is a generalization of value_t (and boost any) - the idea is to be able to parameters an "any" object by concept. Mat Marcus has been doing a fair amount of work with this library. Needs better docs and tutorial. dictionary_t, array_t, name_t <http://opensource.adobe.com/ group__asl__tutorials__dictionary__t.html> ASL using these heavily to communicate with the runtime languages - could be used as part of the Python bindings (we're currently looking at doing Lua bindings with them. dictionary_t needs a closed hash implementation - there is one in the /future/ directory in ASL inspired by the Lua table structure, I haven't had a chance to compare it in depth to the Google hash implementations. Copy-On-Write <http://opensource.adobe.com/ classadobe_1_1copy__on__write.html> This is a fairly polished library - I've done lots of experimenting with COW (I prefer the term copy-on-modify - but COM confuses everyone...) over the last 10 years or so. This is a nice, simple, and efficient implementations (but it does need a better counter - see above). XML Parser <http://opensource.adobe.com/ classadobe_1_1xml__parser__t.html> This is a subset of an XML parser developed for xstrings. In our upcoming .18 release it has been expanded to support more uses by Eric Berdahl and Foster Brereton. It is designed to be a very efficient parser which pareses in place, and allows for preorder and postorder handling of elements (allowing for inplace substitutions). Needs a fair amount of work still to be a general XML parser - but I'd like to see that happen. --- Wow - that's not a complete list (much longer than I set out to write - I forget how large of a library we've created!). If you're not aware of ASL - it's all released under the MIT license (making it fairly straight forward to get through a code audit). I'd be happy to help out anyone who has the time to prepare some of these libraries (or any other library from ASL) for inclusion in Boost - ASL would also benefit from such an effort. Sean

Hi Sean -- A couple thoughts and pointers below: Sean Parent wrote:
I _really_ wish that I had enough time/resources on my team to take some parts of ASL and contribute them to Boost. However, my teams focus currently needs to stay on the higher level libraries (which I don't think are ready for Boost yet) and we simply can't afford the time to go through the submission process.
But I'm 100% supportive of Lubomir's contribution of GIL (Lubomir is not on my team, and I'm thrilled that he can take the time for this submission to Boost - just I'm I'm thrilled that we were able to release it as part of ASL). GIL is a very significant piece of work.
I'll just say a public thanks to yourself, Adobe, and the folks behind ASL for dipping their toe in the open source water. I think it will take several years to see the full effect, but it's very exciting to me to see you guys getting a chance to bring these ideas out into the world.
I will also support anyone who would like to borrow portions of ASL to contribute them to Boost.
This is really great -- it seems there is much common activity with other things happening in Boost (see below).
Some libraries that I think would be interesting to get into Boost with some comments about what I see as still needing to be done.
Forest <http://opensource.adobe.com/group__asl__tutorials__forest.html> .....
We have an active SOC project to build a generic tree... https://boost-consulting.com:8443/trac/soc/wiki/tree https://boost-consulting.com:8443/trac/soc/wiki/tree/design
Range Based Algorithms (using the boost range library) <http:// opensource.adobe.com/group__standard__extensions.html>
Eric Niebler has written some range algorithms in the vault -- be nice to see if he's covered all the ones you'all have: http://www.boost-consulting.com/vault/index.php?&direction=0&order=&directory=Algorithms
ZUIDs <http://opensource.adobe.com/classadobe_1_1zuid__t.html>
Hmm, I guess I would think zuid should be its own little library. There is already a GUID in the vault which again might benefit from collaboration here: http://www.boost-consulting.com/vault/index.php -- guid_v3.zip
md5 and SHA code <http://opensource.adobe.com/classadobe_1_1md5__t.html> <http://opensource.adobe.com/group__adobe__sha.html> Review and a short tutorial
I wonder if these would be logical additions to boost.hash which is new in 1.34? http://engineering.meta-comm.com/resources/cs-win32_metacomm/doc/html/hash.h...
ONCE <http://opensource.adobe.com/group__adobe__once.html>
ThreadID <http://opensource.adobe.com/group__asl__thread__id.html> Another good addition to Boost threads.
I don't know if we've really found an active maintainer for Boost.Threads....library abandonment is starting to become a much bigger issue for Boost as time goes on :-(
Final <http://opensource.adobe.com/group__adobe__final.html> I don't use inheritance much but this would be a nice addition to Boost utilities.
Yep.
Counter <http://opensource.adobe.com/classadobe_1_1counter__t.html> If boost would surface the lightweight mutex used in shared_ptr - this would go away.
I'll let Peter take this one...
value_t <http://opensource.adobe.com/classadobe_1_1value__t.html>
Agree, we'll have to push Kevlin to be more active ;-)
regular_object <http://opensource.adobe.com/ classadobe_1_1regular__object.html> This is a generalization of value_t (and boost any) - the idea is to be able to parameters an "any" object by concept. Mat Marcus has been doing a fair amount of work with this library. Needs better docs and tutorial.
Interesting. Don't know of any current Boost work here.
dictionary_t, array_t, name_t <http://opensource.adobe.com/ group__asl__tutorials__dictionary__t.html>
Don't know of any current Boost work here.
Copy-On-Write <http://opensource.adobe.com/ classadobe_1_1copy__on__write.html>
I guess the ever-languishing policy_pointer is the closest Boost work.
XML Parser <http://opensource.adobe.com/ classadobe_1_1xml__parser__t.html>
There is a Boost xml, but I haven't seen much Boost activity here. Property tree does some too, but full XML support is hard enough that it will be hard to do in Boost. http://www.boost-consulting.com/vault/index.php?direction=0&order=&directory=Progamming%20Interfaces
Wow - that's not a complete list (much longer than I set out to write - I forget how large of a library we've created!). If you're not aware of ASL - it's all released under the MIT license (making it fairly straight forward to get through a code audit). I'd be happy to help out anyone who has the time to prepare some of these libraries (or any other library from ASL) for inclusion in Boost - ASL would also benefit from such an effort.
Hopefully people will make you very busy :-) Jeff

Jeff Garland wrote:
md5 and SHA code <http://opensource.adobe.com/classadobe_1_1md5__t.html> <http://opensource.adobe.com/group__adobe__sha.html> Review and a short tutorial
I wonder if these would be logical additions to boost.hash which is new in 1.34?
Not really, it's only concerned with providing an implementation std::tr1::hash - ie. for TR1 style unordered associative containers. Which is why it's officially called 'Boost.Functional/Hash', although I know that name's unpopular - 'TR1 Hash' might be better. MD5 and SHA aren't at all appropriate for that, are used in a very different manner and produce much larger hash values - so it doesn't even make sense to provide a common interface. Boost also has a CRC library, but again, that's intended for another use altogether. I suppose the different libraries could be loosely grouped together under a Boost.Hash moniker, but for organisation purposes more than anything. Daniel

Daniel James wrote:
Jeff Garland wrote:
md5 and SHA code <http://opensource.adobe.com/classadobe_1_1md5__t.html> <http://opensource.adobe.com/group__adobe__sha.html> Review and a short tutorial I wonder if these would be logical additions to boost.hash which is new in 1.34?
Not really, it's only concerned with providing an implementation std::tr1::hash - ie. for TR1 style unordered associative containers. Which is why it's officially called 'Boost.Functional/Hash', although I know that name's unpopular - 'TR1 Hash' might be better.
Yeah, I don't see what functional has to do with it...
MD5 and SHA aren't at all appropriate for that, are used in a very different manner and produce much larger hash values - so it doesn't even make sense to provide a common interface.
Ok.
Boost also has a CRC library, but again, that's intended for another use altogether.
I suppose the different libraries could be loosely grouped together under a Boost.Hash moniker, but for organisation purposes more than anything.
Perhaps. Largely the intent of my email was partially to encourage folks to think about working with Sean et. al. on synergy with other Boost libraries. In this case, maybe it's in name only... Jeff

Jeff Garland wrote:
Daniel James wrote:
Not really, it's only concerned with providing an implementation std::tr1::hash - ie. for TR1 style unordered associative containers. Which is why it's officially called 'Boost.Functional/Hash', although I know that name's unpopular - 'TR1 Hash' might be better.
Yeah, I don't see what functional has to do with it...
Well, Boost.Functional is concerned with enhancing the functional header and TR1 hash lives in the functional header. Although I suppose it has a few other boost libraries for neighbours there. Also, it perhaps doesn't deserve a place at the 'top level' as for most people it's just an implementation detail. I'm not that bothered about the name, I don't think it was my idea. But I'd like to put of any reorganisation until after the migration to subversion - just so I can track renames.
Perhaps. Largely the intent of my email was partially to encourage folks to think about working with Sean et. al. on synergy with other Boost libraries. In this case, maybe it's in name only...
Sorry if I sounded negative. I think it would be great to have an SHA and MD5 library in Boost, But right now, I haven't got the energy to get much done. Daniel

Daniel James wrote:
Perhaps. Largely the intent of my email was partially to encourage folks to think about working with Sean et. al. on synergy with other Boost libraries. In this case, maybe it's in name only...
Sorry if I sounded negative. I think it would be great to have an SHA and MD5 library in Boost, But right now, I haven't got the energy to get much done.
Fair enough, but I hope this means you are working on unsorted_* ? What's left to do there? Jeff

Jeff Garland wrote:
Fair enough, but I hope this means you are working on unsorted_* ? What's left to do there?
Yes, I've working on it for the last few months, but very slowly. Before it's review ready, the documentation needs a little more work and I've got a few small issues with the standard that I need to check up on. The tests could also be improved a bit and the implementation would benefit from some refactoring. But I don't think that's required for the review.

Sean Parent wrote:
ONCE <http://opensource.adobe.com/group__adobe__once.html> The STATIC_INSTANCE stuff might be able to be removed - this was done for the Mac where static initializer on DLLs were getting executed on a thread after load so other functions in the compilation unit could be executed concurrently. That may have been fixed (I haven't tracked the issue) - this library would be a good addition to Boost threads.
adobe::call_once should really be folded into boost::call_once, the rest would require more thought.
Counter <http://opensource.adobe.com/classadobe_1_1counter__t.html> If boost would surface the lightweight mutex used in shared_ptr - this would go away. (This is a heavyweight one - as I'm counting on Boost providing a lightweight one in the future :-) ).
The closest boost equivalent is boost::detail::atomic_count. We really need an atomic operations library to isolate the platform-specific functionality that is currently scattered all over the place... As for lightweight_mutex, it should ideally be subsumed by boost::mutex; there is no real reason for boost::mutex to be heavyweight (and in order to replace lightweight_mutex, it should be header-only because of its "critical piece of infrastructure" status.) One subtlety when specifying a counter type such as counter_t is that its memory synchronization guarantees need to be documented. No msync on increment, full msync on decrement is probably the simplest option that supports a reference-counted pointer and doesn't sacrifice performance too much; a slightly more refined approach is to guarantee acquire semantics on a decrement with new value zero, release semantics otherwise. Copy on write requires even less.

On Sun, 2 Jul 2006 09:52:26 -0700, Sean Parent <sparent@adobe.com> wrote:
[snip] md5 and SHA code <http://opensource.adobe.com/classadobe_1_1md5__t.html> <http://opensource.adobe.com/group__adobe__sha.html> Review and a short tutorial
I had a quick look to this, as I was implementing an md5 accumulator myself. One thing that hit my eyes were all those C-style macros, in particular this one #define ROTATE_LEFT(x, n) (((x) << (n)) | ((x) >> (32-(n)))) I guess you know: if used with n=0 (or n=32) on a 32- bit width integer it invokes undefined behavior.
[...]
Final <http://opensource.adobe.com/group__adobe__final.html> I don't use inheritance much but this would be a nice addition to Boost utilities.
I suppose the reason why this has never been included is that it can't be generalized as we usually like on boost :) template<class T> class final { private: final() { } friend class T; }; I'd add another base class to this so that the user hadn't to remember to write virtual, but that's not the main point. In any case, this does not prevent derivation: it prevents *instantiating* any derived class (so a derived class which would only have static members... :)). It will probably get into boost when "friend class T" will become legal; but again, I suppose, as there's a school of thought saying it is of no use: to make a class non derivable specify that it shouldn't be derived from, in the docs.
[...] I'd be happy to help out anyone who has the time to prepare some of these libraries (or any other library from ASL) for inclusion in Boost - ASL would also benefit from such an effort.
Everyone benefits from good code :) -- [ Gennaro Prota, C++ developer for hire ] [ resume: available on request ]

Gennaro Prota <gennaro_prota@yahoo.com> writes: [...] | template<class T> | class final | { | private: | final() { } | friend class T; | }; | | I'd add another base class to this so that the user hadn't to remember | to write virtual, but that's not the main point. In any case, this | does not prevent derivation: it prevents *instantiating* any derived | class (so a derived class which would only have static members... :)). | | It will probably get into boost when "friend class T" will become | legal; friend T; has been voted into the current draft of C++, i.e. it is now possible to declare a template parameter as a friend. You're right that it is not widely supported yet. -- Gaby
participants (6)
-
Daniel James
-
Gabriel Dos Reis
-
Gennaro Prota
-
Jeff Garland
-
Peter Dimov
-
Sean Parent