----- Original Message ----- From: <strasser@uni-bremen.de> To: <boost@lists.boost.org> Sent: Sunday, January 17, 2010 10:30 PM Subject: Re: [boost] Boost library submission (poll for interest)

Zitat von "vicente.botet" <vicente.botet@wanadoo.fr>:

...

...

Yes. Stefan had suggested this in private e-mails earlier, and I'll have to adapt to fit his interfaces. As I've alluded previously, I would like to see it possible to use Boost.Persistence with my library, as several shortcomings of STLdb would be addressed in that way. Part of doing that involved bringing all transaction control under one manager. Stefans library is the most robust in this sense, so I plan to work with and adapt to his interfaces.

I have proposed in a private mail to split its library in two Boost.Transaction and Boost.Persistency. I think this is the right direction.

before I do that, I would like to have a thorough discussion of the design decisions I made, especially the first one of the list, because it deeply affects the way your libraries would have to be set up.

1.

every transactional entity like a trans_map or a shared_loc must be aware of the static type of the transaction manager, which can change by user configuration.

this results from the following:

every access to a transactional entity without explicit "transaction" argument must go through the transaction manager ==> calls through transaction manager must be efficient ==> there can be no dynamic polymorphism ==> the static type of the transaction manager changes by configuration(a transaction manager using Boost.Persistent has another type than one that uses STLdb, which has another type than a transaction manager using both libraries) ==> all transactional entities must be aware of the static type of the transaction manager.

so in my library every transactional entity is a template with the type of the transaction manager as parameter:

basic_transaction<TxMgr> basic_shared_loc<T,TxMgr> basic_loc_multiset<T,TxMgr,...>

and so on. for convenience of the user that only uses one type of transaction manager, there are aliases defined:

transaction, alias for basic_transaction<transaction_manager>, shared_loc<T>, alias for basic_shared_loc<T,transaction_manager> ...

transaction_manager is a typedef that currently is by default a transaction manager that uses Boost.Persistent, but can be changed by defining BOOST_PERSISTENT_CONFIGURATION before including the aliases.

there are several obstacles with this approach: - every type that ought to be used with a configurable transaction manager must be a template. - C++98 has no template aliases. in C++0x shared_loc could be a simple alias of basic_shared_loc:

template<class T> using shared_loc = basic_loc<T,transaction_manager>;

in C++98 however a class "shared_loc" that acts like an alias for "basic_shared_loc" has to be implemented.

this would have to be done for every class like that of your libraries, too.

I don't think the cost of dynamic polymorphism will be high respect to the cost of a transaction system, and even less if it concerns persistent objects. Anyway, I agree with the static approach. I think that once we will have static polymorphism, it would be very easy to add dynamic polymorphism.

2. do your libraries allow access by multiple threads to the SAME transaction?

Not yet, but I would like to explore how parallelism and transactions can work together.

because this requires a mutex lock in the transaction manager for each access, which might be a performance concern? if your libraries only allow one thread to use a transaction, this mutex lock can be removed by changing a template argument of basic_transaction_manager.

what about having two classes, transaction and shared_transaction? The first one is local to a thread and could be used without the mutext, while the second can be shared between threads and need this mutex protection. We can start by the non-shreable transaction.

3. transactions are bound to the current thread when they are created, but transaction managers are bound to the whole application.

my_transaction_manager t1; my_transaction_manager t2;

//t2 is the current transaction manager, even when another thread is started. bind() can be used to bind another transaction manager for all running threads.

creating and binding of transaction managers is not thread-safe.

Sorry. I'm a little bit lost. Why do you need to change the instance of the transaction manager. I thought that there was only a transaction manager. When an application will need to change it?

4. please have a look at the following parts of the documentation:

https://svn.boost.org/svn/boost/sandbox/persistent/libs/persistent/doc/html/...

TransactionManager concept: https://svn.boost.org/svn/boost/sandbox/persistent/libs/persistent/doc/html/...

ResourceManager concept: https://svn.boost.org/svn/boost/sandbox/persistent/libs/persistent/doc/html/...

I'm not sure if the interface of resource_manager is enough to implement a two-phase commit protocol, but we will see this once we talk about the protocol that ensure that the transaction is ACID for all the resources. I see that there are TransactionManager::transaction and a ResourceManager::transaction types. Could you point on the documentation (if already written) how these types are correlated, and when these function are called? struct TransactionManager { typedef unspecified transaction; ... struct ResourceManager { typedef unspecified services; // A MPL Sequence of tags of all Service concepts implemented by this ResourceManager. typedef unspecified tag; // The tag that identifies this ResourceManager. typedef unspecified transaction; // public member functions unspecified begin_root_transaction() ; unspecified begin_nested_transaction(transaction &) ; void commit_transaction(transaction &) ; void rollback_transaction(transaction &) ; }; Best, Vicente

----- Original Message ----- From: "Stefan Strasser" <strasser@uni-bremen.de> To: <boost@lists.boost.org> Sent: Monday, January 18, 2010 1:21 AM Subject: Re: [boost] Boost library submission (poll for interest)

Am Monday 18 January 2010 00:25:54 schrieb vicente.botet:

...

I don't think the cost of dynamic polymorphism will be high respect to the cost of a transaction system, and even less if it concerns persistent

I do, it's a virtual call for each access to a persistent object. note that if only a base class of the transaction manager is known to the "transactional entity" it can also only return a base class of the resource manager, so it also severly limits the interface of the resource managers, since there are no virtual template functions. but since we agree on the result there is no need to discuss this in detail I guess.

what do you think about the basic_* and alias approach? is that feasible for your library?

Yes I think so.

...

...

4. please have a look at the following parts of the documentation:

https://svn.boost.org/svn/boost/sandbox/persistent/libs/persistent/doc/ht ml/persistent/configuring.html#persistent.configuring.defaultconf

TransactionManager concept: https://svn.boost.org/svn/boost/sandbox/persistent/libs/persistent/doc/ht ml/boost/persistent/TransactionManager.html

ResourceManager concept: https://svn.boost.org/svn/boost/sandbox/persistent/libs/persistent/doc/ht ml/boost/persistent/ResourceManager.html

I'm not sure if the interface of resource_manager is enough to implement a two-phase commit protocol, but we will see this once we talk about the protocol that ensure that the transaction is ACID for all the resources.

right, the interface for distributed transactions is missing. there has to be at least a prepare_transaction(), a recover() and a transaction_id with a way to obtain it from a transaction object, so the transaction manager can log the IDs of the resource transactions in its distributed transaction log.

I also need a finish_transaction() but this is a minor issue. I'm quite sure this can be an empty function for your libraries.

Yes, this is the kind of functions I was looking for.

...

I see that there are TransactionManager::transaction and a ResourceManager::transaction types. Could you point on the documentation (if already written) how these types are correlated, and when these function are called?

I don't think it's documented except for the Reference. TransactionManager::transaction is sometimes called the "global transaction", and ResourceManager::transaction the "local transaction" in other systems.

I see, so TransactionManager::transaction is the distributed transaction, and ResourceManager::transaction can be persistent::transaction, stm::transaction, ...

the details are as follows:

the user creating a transaction scope:

transaction tx;

calls TransactionManager::begin_transaction and stores a TransactionManager::transaction. note that no resource manager is called at this point.

Does TransactionManager::begin_transaction makes a transaction or registers the stack allocated one? unspecified TM::begin_transaction() ; what about void TM::begin_transaction(TM::transaction&) ;

accessing a resource, e.g. by calling Locator::operator*, obtains ResourceManager::transaction by calling TransactionManager::resource_transaction(), and uses it to access the object. TransactionManager::resource_transaction() returns the ResourceManager::transaction that is associated with the TransactionManager::transaction. if there is none associated yet, ResourceManager::begin_transaction is called to create one.

OK. I understand the logic.

the point of all this is that if there are multiple resources used by a transaction manager, a local transactions in a resource manager is only started if the resource is actually used in this global transaction. for example most of my transactions may access Boost.STM transactional memory only, and only from time to time I access Boost.STM and Boost.Persistent in the same transaction.

if more than one ResourceManager::transaction's were created in the course of a global transaction, a two-phase-commit has to be performed.

Oh I see here the advantage to creating lazyly the ResourceManager::transaction that are needed.

...

...

3. transactions are bound to the current thread when they are created, but transaction managers are bound to the whole application.

my_transaction_manager t1; my_transaction_manager t2;

//t2 is the current transaction manager, even when another thread is started. bind() can be used to bind another transaction manager for all running threads.

creating and binding of transaction managers is not thread-safe.

Sorry. I'm a little bit lost. Why do you need to change the instance of the transaction manager. I thought that there was only a transaction manager. When an application will need to change it?

yes, one at a time. this might be rarely used, but it is possible to have 2 transaction managers of the same type if the right one is active at the time you access e.g. objects that use it. my (minor) point of this was that transactions are bound to threads, and transaction managers are bound to applications.

I agree with the last point :) I would like to see something in the line of namespace this_transaction { transaction* current(); ... } or namespace current_transaction { ... } Best, Vicente

On Sun, Jan 17, 2010 at 3:22 PM, vicente.botet <vicente.botet@wanadoo.fr> wrote:

...

I'm not relying on the durability of the boost interprocess region.

Why? What are the main problems with this approach?

I haven't figured out how to do it yet without having something similar to the Boost.Persistence locator construct to represent a pointer in shared memory, and permit the tracking of modifications made to portions of non-concrete key/value pairs. There is an overhead to that approach which seems unavoidable. Otherwise the approach requires that all changes to the shared memory region data are postponed until those changes are first determined and written to the log in order to ensure recoverability. Ultimately, another implementation of map is required. When I started STLdb, I chose to dodge these issues in favor of writing serlaized forms of key/value types, and permitting direct use of shared memory for storing some transaction-local storage. The later helps with some aspects of my concurrency model, but also meant the region can't be recoverable.

...

Nor am I trying to serialize the shared memory from directly to disk. Stefan has challenged me to do that, and I'm considering a variation which would do that, but it isn't without it's own issues.

Could you develop?

I am working on this in my spare time, and can't commit to any timeframe. However, in the course of working on a unified transactional layer, if it becomes possible to share some of the Boost.Persistent optimizations which eliminate the need for Serialization of objects in the case of concrete types, I should be able to use those, and that would at least minimize the difference between what exists and the ideal.

I don't understand what K and V are? (Key, Value?)

Yes.

On Sun, Jan 17, 2010 at 4:47 PM, <strasser@uni-bremen.de> wrote:

if you keep using serialization we should also think about unifying this aspect of our libraries. as we've discussed before I have some ADL functions that avoid boost.serialization if possible.

they 1. use std::memcpy() if serialization::is_bitwise_serializable<> yields true 2. use user-supplied functions, described here: https://svn.boost.org/svn/boost/sandbox/persistent/libs/persistent/doc/html/... 3. only if 1 and 2 fails, invokes Boost.Serialization.

so for a trans_map<int,int> for example, there would be no overhead(besides memcpy) and no calls to Boost.Serialization (and no compile-time instantiations!)

I really like this idea. I wonder if this can be achieved by unifying some of this into Boost.Transaction. One thing I have noticed is that there is (probably) tremendous duplication in our libraries around such topics as write-ahead logging, and my concept of a checkpoint is probably very much like the file you are using to contain serialized images of objects, complete with the free list management approach. I'm not against throwing away portions of my code base to better unify. I need time to look through the code you have been sending out. I have been in a very busy work month and have not had time to 'digest' the concepts yet. - Bob

Am Tuesday 19 January 2010 08:47:19 schrieb Rutger ter Borg:

Interesting, when building an abstraction on top of Berkeley DB, here I used

is that layer publicly available?

1) type_traits/has_trivial_copy 2) introduced a has_trivial_array type trait (type that serializes with ptr/size, and can be constructed with a ptr/size), e.g., std::vector, std::string, etc. 3) serialization fallback

I think there is no way around a user-defined trait. a POD containing a pointer has a trivial copy constructor but it is not bitwise serializable.