Boost.Dynamic Config

Hi guys, I feel as part of the Boost community and having worked on multiple platforms that we're missing something to manage reading/writing configuration information for an application thread safely across platforms. The best example I know of that handles this job is Qt's QSettings which writes to the registry on windows, inis on linux, and some sort of xml on Mac OS X. The inspiration for this comes from the fact that I am looking to build web browser plugins that run on multiple platforms. I started by looking at how I could extend boost::program_options, and decided it wasn't the right tool because it was primarily for reading configurations, but not really writing to them. Likewise, it doesn't appear to provide anything in terms of thread safety. The interface should model after several things already in existence such as: QSettings, std::map, and tbb::concurrent_hash_map. It should also take into consideration how systems such as the Windows Registry works, as well as browser specific configuration systems like the one that comes with Mozilla Firefox, and of course file based storage. It should be easily extensible with a simple interface to implement a new backend with minimal work. At a later date, I think it should also be able to support things like boost::property_map or boost::fusion to make it easy to insert and extract objects from the underlying configuration system with automatic type conversion. I'm not sure how to combine it with boost::fusion or boost::property_map yet, so I'm hoping that's where you guys come in. Thanks, Dan Here is a preliminary interface: namespace boost { namespace dynamic_config { template<typename charT> class backend { public: enum operation_performed { none, // e.g. no value changed insert, update }; bool insert(basic_string<charT> key, (int, basic_string<charT>, bool, int64_t, char) value); // These values may be easily implementable with boost::variant, or separately if enable_if is used. bool update(basic_string<charT> key, (see above..) value); // Note this returns operation performed because optional locking will be done around these functions, so combining insert with update replaces a possible race condition over those 2 operations performed separately operation_performed replace(basic_string<charT> key, (...) value); template <T> bool select(basic_string<charT> key, T& value); // returns success if retrieved, this can also use boost::optional otherwise bool delete(basic_string<charT> key, bool cascade); // cascade takes effect if key takes a whole tree with it }; template <typename charT, typename Backend, typename Mutex> class property_config : boost::noncopyable { public: property_config(); property_config(basic_string<charT> organizationName, basic_string<charT> applicationName); // Pretty much required for use in the windows registry and in firefox ~property_config(); size_type erase(basic_string<charT> key); void clear(); template<typename T> bool find(basic_string<charT> key, T & value); template<typename T> bool insert(basic_string<charT> key, T value); template<typename T> bool update(basic_string<charT> key, T value); template<typename T> backend::operation_performed replace(basic_string<charT> key, T value); template<typename T> bool select(basic_string<charT> key, T& value); template<typename T> bool delete(basic_string<charT> key, T& value); }; template <typename charT, typename Backend, typename Mutex> template <typename T> backend::operation_performed property_config<charT, Backend, Mutex>::replace(basic_string<charT> key, T val) { scoped_lock(mutex_); backend_.replace(key,val); } } }

On 12/02/2011 18:51, Dan Weber wrote:
Hi guys,
I feel as part of the Boost community and having worked on multiple platforms that we're missing something to manage reading/writing configuration information for an application thread safely across platforms. The best example I know of that handles this job is Qt's QSettings which writes to the registry on windows, inis on linux, and some sort of xml on Mac OS X.
What about property_tree and program_options?

Hi Mathias, I've found that property_tree and program_options are deficient in solving this problem. For instance, property_tree requires full serialization back and forth for each time you write to the underlying data store, so it could not guarantee that it has the most accurate data all the time. As for program_options, it only works one way that is for reading options. It is also not thread safe. The goal of this is keep live runtime configuration that can be globally modified, both in the application and outside of it while making it easy to support new backends. I've started the initial implementation starting with the registry, and it can be found here: https://github.com/omnisip/boost.dynamic_config git://github.com/omnisip/boost.dynamic_config.git There are several things I'm already aware of what I want to change. For instance, to support backend specific configuration, I want to switch the constructor of basic_property_config to take an argument of backend::configuration_type const &. For backends that support or require those configurations, they can be passed to the constructor, with a default being backend::configuration_type(). So in the event of a simple backend, you could possibly have a backend::configuration_type of none_t. While others like the registry could have custom parameters regarding how it handles HKCU/HKLM or both as well as the organization name and application name components. Additionally, I think backends could be simpler to implement using a facade of sorts that the backend implementation inherits from with a template parameter of itself. This would make it so that the bare bones backend would only need to support things like std::string/std::wstring or string_type as I have it typedef'd. This is beneficial because it simplifies implementation of any sort of file backend at a bare minimum. I've used an approach that takes boost::lexical_cast<> for all non-native types of the backend and converts them to strings in the windows registry backend. There are also some other nifty features of that backend that I've been working on. For instance, the windows registry supports only a couple of integer types, DWORD and QWORD which are uint32_t and uint64_t respectively. The backend implementation takes this into account for any Integral types that come in using std::numeric_limits<Integral>::digits to make a distinction as to how to store it in the registry with the right amount of precision necessary. I know I also need to add some sort of support for iteration of subkeys and iteration of values, but I haven't quite come up with the method I want to use to do that yet. Likewise, I would need to standardize on a notation for searching through subkeys and so forth, e.g. 'dot notation' or '/' notation.. Dan On 02/13/2011 03:44 AM, Mathias Gaunard wrote:
On 12/02/2011 18:51, Dan Weber wrote:
Hi guys,
I feel as part of the Boost community and having worked on multiple platforms that we're missing something to manage reading/writing configuration information for an application thread safely across platforms. The best example I know of that handles this job is Qt's QSettings which writes to the registry on windows, inis on linux, and some sort of xml on Mac OS X.
What about property_tree and program_options?
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Hi, Dan Just a reminder here: Denis Shevchenko has once proposed Boost.Configurator which I think is in the very similar problem domain, but it seems that you want to achieve much more than what he has done so far (it's still in early development and I have no experience with it). Anyway, it's in the SVN sandbox, and the doc is here: http://svn.boost.org/svn/boost/sandbox/configurator/libs/configurator/doc/in... For such a lib, I prefer the name Boost.Configurator to something like Boost.DynamicConfig since 'DynamicXXX' will let me think it's the runtime counterpart of XXX and I don't think this is the case.

TONGARI wrote:
Denis Shevchenko has once proposed Boost.Configurator which I think is in the very similar problem domain, but it seems that you want to achieve much more than what he has done so far (it's still in early development and I have no experience with it).
That library is a direct competitor with Boost.ProgramOptions. It is only for reading a configuration file. This proposal supports multiple backends and includes support for writing runtime changes to the configuration store.
For such a lib, I prefer the name Boost.Configurator to something like Boost.DynamicConfig since 'DynamicXXX' will let me think it's the runtime counterpart of XXX and I don't think this is the case.
I don't care for "DynamicConfig" either. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

Dan Weber wrote:
I feel as part of the Boost community and having worked on multiple platforms that we're missing something to manage reading/writing configuration information for an application thread safely across platforms. The best example I know of that handles this job is Qt's QSettings which writes to the registry on windows, inis on linux, and some sort of xml on Mac OS X.
The idea is interesting, but there are a great many features such a mechanism could include beyond those you've mentioned, though they may not work so well for the various backends. For example, when processing configuration files, you could support include directives or conditional processing. You could support reusable values; IOW, permit reusing a given value in defining the value of various other entries. What about the ability to filter which entries apply? In, again, a configuration file, there could be key elements that identify the application or instance to which they apply. Then, based upon runtime values, entries with those extra key parts would be filtered if they don't match. (Another approach is to require wildcards so all keys have the same structure.) There are various configuration formats that support data structures like dictionaries, lists, etc. These are highly useful for capturing come configuration information. Without direct support, one must create ad hoc string formats that must be parsed by each application requiring them. Why do you want to update the configuration store at runtime? What are the use cases for that behavior? What do you mean by thread safety? Do you anticipate much MT demand for updating configuration information that the application couldn't just synchronize updates rather than embedding that behavior in the configuration class(es)? _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

Hi Rob, [notes inlined below]
The idea is interesting, but there are a great many features such a mechanism could include beyond those you've mentioned, though they may not work so well for the various backends.
There are certainly many possible features, however, I want to keep this as simple to implement as far as new backends.
For example, when processing configuration files, you could support include directives or conditional processing. You could support reusable values; IOW, permit reusing a given value in defining the value of various other entries.
I think it should be noted that I am not particularly interested in targeting configuration files. I'd rather support something along the lines of sqlite databases than flat files since the goal is to make this primarily for the most up to date runtime configuration/program options.
What about the ability to filter which entries apply? In, again, a configuration file, there could be key elements that identify the application or instance to which they apply. Then, based upon runtime values, entries with those extra key parts would be filtered if they don't match. (Another approach is to require wildcards so all keys have the same structure.)
There are various configuration formats that support data structures like dictionaries, lists, etc. These are highly useful for capturing come configuration information. Without direct support, one must create ad hoc string formats that must be parsed by each application requiring them.
Why do you want to update the configuration store at runtime? What are the use cases for that behavior? When I think of configuration files, I think of /etc. You almost never see an application update it's own configuration file at runtime. The goal of runtime configuration is to store user specific settings that otherwise could not be easily stored. For instance, in one of my applications, I'd like to store the position of the volume control, the default layout of the window, and so forth.
What do you mean by thread safety? Do you anticipate much MT demand for updating configuration information that the application couldn't just synchronize updates rather than embedding that behavior in the configuration class(es)?
The value in this for me is a volatile online database. Previously, I have used things like TBB's concurrent_hash_map in a static context to keep data available for all threads to reach and update while the application is running, but this provides no native persistence mechanisms. However, consider the Windows Registry or GConf. These systems can serve the role of quickly retrieving configuration values and dynamically storing new ones while still persisting to disk at some point. Likewise, these backends are natively thread safe. In addition to those, making it easy to add new backends, also makes applications more portable. Interestingly enough, sqlite has become incredibly important for configuration on things like phones because it provides the capabilities I need from my applications that run on desktops and servers. It can fully meet the criteria to operate as a backend. After several people have said this, I will also focus on some name changes that seem suitable. Thanks, Dan
participants (4)
-
Dan Weber
-
Mathias Gaunard
-
Stewart, Robert
-
TONGARI