You don't need a special case for emptiness if you make them base classes when they're classes (a.k.a. use boost::compressed_pair)
In VC11, we've introduced such special-casing for container allocators/comparators, but we haven't had time to extend it to shared_ptr. I've made a note to myself to do this in the future.
That's what compressed_pair is for; you don't have to keep repeating the same special case over and over ;-)
The downside of having it be a base class is that classes can now be marked final, which means they can't be used as base classes. This approach was used by libc++ for implementing the same optimization that STL mentions VC's STL implements, but I think they end up falling back to storing it if it's empty and final. I suppose what we really want is static if. :)