On Wed, September 6, 2006 7:30 pm, Stefan Seefeld wrote:

Have you followed the discussions around my proposal for an XML API in boost that I implemented on top of libxml2 (http://xmlsoft.org/) ?

I wasn't around for the early discussions, but have caught up on them now. It's an interesting discussion, though I'm not sure how relevant it is. I'll elaborate later.

I think that starting a new implementation from scratch is the wrong way to approach this (rather big) topic.

This in particular since an 'XML library' shouldn't just provide ways to de- and encode XML documents into generic tree structures, but instead needs to provide quite a substantional amount of functionality in order to be considered complete (even if you approach this in a modular way). As an example, imagine querying your DOM-like structure with an XPath expression. Think about all this does involve, from regular expression handling, over XPath pattern matching, http lookup, entity handling, unicode, etc., etc.

This is why I don't think that you should think about such a project one step at a time (e.g. the 'XML reading side of things').

It seems to me that your earlier proposal was mainly about a few API specifications, that were then supposed to be implemented somehow - preferably on top of an existing XML library, in order to avoid reinventing the wheel. This idea certainly has a lot of merit, but it also has some distinct disadvantages. First, an API specification is nice for standardization, but not very usable within the context of Boost. In order to be useful, there must be at least one implementation of the API. Otherwise, the specification is worth nothing to the end user. This implementation must exist within Boost, i.e. it must be completely contained within Boost. Libraries like Regex and Iostreams offer enhanced functionality if certain external libraries are available, but they will work without them, too. Obviously, the Xml library could not work without the external XML implementation if it is just a wrapper around it. This means that, if the library is a wrapper around an external one, the external library (let's for argument's sake assume libxml2, which seems to bring less licensing trouble compared to Xerces, the only other sufficiently complete XML library I can think of) must be distributed with Boost. What does this entail? The library must build as part of Boost. I haven't checked, but I assume libxml2's build system right now is based on automake. That would have to be translated to Boost.Build. As part of this process, configuration macros might need to be translated. This could easily lead to a real fork of the code base. Unless Boost wants to rely on the regression testing done by the authors of libxml2, regression tests, portability tests and everything else must be written and maintained. And last but certainly not least, there's the licensing issue. Boost is working hard to get all code under the Boost license. Would we want an external library under any other license, no matter how permissive, in that code base? Or would the authors of libxml2 permit relicensing of the source? (As a programmer, I'd rather reimplement a library than pursuing such goals. ;) ) Second, the recommendation focused on a DOM-style API. As at least two people [1][2] pointed out, DOM-style APIs are not as universally useful as other APIs. That said, I do intend to provide a DOM-syle API, but only after having completed the event-based API and thought long and hard about what a DOM-style API means in C++. Still, this is one of the main reasons why I asked for real-world use cases. My own uses of XML have usually been satisfied by SAX, although I would have preferred a pull-style API. I'd love to hear how other people use XML. I know that two Boost-internal uses could work with a pull API very well: Property Tree's XML reader and the Serialization XML archive. To sum up, I do believe we should reinvent the wheel here. But we should create an improved wheel, and I think the Boost community is uniquely suited to create a wheel that works particularly well with C++. To maintain thread integrity, I'll reply to each post individually. [1] http://lists.boost.org/Archives/boost/2005/11/96131.php [2] http://lists.boost.org/Archives/boost/2005/11/96521.php

Sebastian Redl wrote:

I know that two Boost-internal uses could work with a pull API very well: Property Tree's XML reader and the Serialization XML archive.

To sum up, I do believe we should reinvent the wheel here. But we should create an improved wheel, and I think the Boost community is uniquely suited to create a wheel that works particularly well with C++.

If you've looked at serialization of xml you'll see that it uses spirit and an xml grammer derived from a complete one included in the spirit library. If you really don't want to re-invent the wheel, they why not just use the same approach? The spirit library contains a very complete XML grammer already. I don't recall this even having been considered - much less rejected. Creating XML is easy. Handling issues of code conversion (e.g. from locale specific code to UTF-8 or UTF-16) is easily handled with i/o stream facets - some of which are also already available. It is also already "Boost Friendly" and works with all boost platforms. It also does most of the heavy lifting at compile time - very much in line with other boost tools. I can't understand why any other approach would be attractive for users of other boost libraries. To me the whole idea is "re-inventing the wheel" Robert Ramey

"Sebastian Redl" <sebastian.redl@getdesigned.at> writes:

This implementation must exist within Boost, i.e. it must be completely contained within Boost.

Not if you put any stock in past Boost discussions and the intention behind Boost. We've discussed this many times and we've always said that a library whose implementation just happens to be as a C++ wrapper over code available elsewhere could be valuable and appropriate for Boost. -- Dave Abrahams Boost Consulting www.boost-consulting.com

On Thu, September 7, 2006 11:04 am, loufoque wrote:

It could also be possible to make the push and pull parsers more or less independant, so that each one can be as efficient as it can be.

True. Basically, by making a direct push parser implementation, you can avoid the overhead of state saving that a pull parser requires. However, that effectively means duplicating the work, so the library should be written in such a way that the client can easily substitute the push parser that is implemented on top of a pull parser with a direct push parser.

...

- Since it needs to access resources from various sources, typically specified as URLs, it needs a flexible and runtime-switchable input system. - In particular, it should be possible to plug schema resolvers in at runtime, so that program extensions can provide support for, say, the ftp: schema.

That would be the work of another library, that would provide a way to read any kind of resource from an URL, a bit like what PHP has. That kind of library would be very useful too outside of the XML library.

True, and I have no intention of providing such a library within the Xml library. But it is something that needs to be kept in mind when thinking about I/O.

Maybe a more low-level approach like what boost asio provides could be interesting, especially since this models also provides asynchronous I/O.

I'm not sure how useful async I/O is for a parser. Parsing of incomplete data seems more important. If you have that and you want asynchronous parsing events, you can start the async I/O and have a handler that parses the newly received data, posting events to the async queue. The Xml library could provide such a handler, but that would be a very independent feature. The problem with the really low-level approach is the one you mention next.

Since XML needs good Unicode support and the like, maybe there is work to be done in that area first in boost.

Oh, yes, the Unicode problem. It would take an examination of systems in use, but my impression is that most programs use either UTF-16 or UTF-8 as their internal coding. With that in mind, I think it might be best to have the XML library internally support exactly these two encodings (perhaps as two template specializations) and interact with the user only in these two encodings. The transcoding of whatever external character set/encoding is used would then be an issue for the I/O interface. However, such transcoding requires the I/O interface to be sufficiently abstractable to provide it transparently - which is an obstacle for the low-level approach you suggest above.

The ability to parse partial content would be a great plus.

Yes, that seems to be important. However, it should be at the discretion of the user to switch it off, enabling the parser to work with a single lookahead character. (To support partial content, either the parser needs to support extremely complex state saving, or cache content until a complete event has been generated.)

Writing a complete XML solution is a lot of work, especially if you want to support all XML technologies (XMLSchema, RelaxNG, XPath, XLink, XInclude, XPointer...)

It is. However, it is something that, I think, can be done very well in steps, i.e. first release supports only pull parsing, second release adds push parsing, third adds a DOM, fourth another technology, etc. As long as you consider all possible technologies when implementing the basic ones, this ought to be feasible. And yes, it's a lot of work. I'm willing to put a lot of work into it.

Maybe it could be interesting to reuse libxml2, which is under the MIT license, to build something on top of it. Of course first we need to weight the gains behind a new C++ implementation.

See my reply to Stefan Seefeld. I think that within Boost, depending on an external library, no matter what license, is a very bad idea. I also think that an implementation intended from the ground up to work with C++ is a better choice. Thank you for your comments. They've given me some ideas.

On 9/7/06, Nils Springob <nils.springob@nicai-systems.de> wrote:

http://download.nicai-systems.com/xmlpp/xmlpp.tar.gz

the link is broken !!

A few thoughts on adding XML to the boost libraries: I'll start with what my issues are with existing alternatives, since I think that indicates why there's a rationale for a boost XML library, and then move onto a few other points specific to boost. 1) My problems with existing C++ DOM/XML libraries: a) Most are inadequate/incomplete or don't really support DOM (thin DOM-like wrappers around event-based APIs don't really cut it). b) Those that are complete have very annoying and/or intractable dependencies. Examples: Libxml++ is simply a beast to build (lots of GNOME/Glib dependencies, and it produces compiler errors on MSVC that are left to the implementor to fix); Microsoft's XML implementation is pretty nice but it is tied to the whole .NET/Windows environment. 2) Using another library for the under-the-hood technology seems reasonable, considering the amount of effort involved in getting all of the features of XML working. We currently use Expat in our products, simply because it's easy to incorporate, but of course that wouldn't be adequate for many of the recent features of XML. Libxml strikes me as perhaps the lesser of evils, since it has the needed support, isn't hard to build (at least compared to libxml++) and the binary libraries are easy to obtain. 3) I think DOM support is critical. There's ample C++ libraries and wrappers for doing non-DOM XML manipulations, but there doesn't seem to be adequate options for good DOM libraries. It seems to me that if the library implementor doesn't tackle the challenge of making a nice C++ DOM class interface, then I'm left with the impression that boost wouldn't really be adding anything new to the world.

Could you detail a bit what you mean by 'DOM' and 'DOM-like' here ? We all are probably thinking of some tree structure that can be navigated, queried, etc.. However, some have already argued that the DOM API as it exists for Java is inappropriate for C++, or even that the DOM API is already conceptually broken. Thus, I think it might help if we could detail a bit what it should and should not be, and what use cases it should support. I don't know why the Java "Document" interface wouldn't be an appropriate model to work with. It simply defines XML documents as a hierarchy of nodes, individually representing such objects as "elements," content (character) areas, comments, and so forth. The Element objects contain useful methods for inspecting attributes and so forth. The two most common-use cases for a program interacting with XML are: (a) a need to easily extract data from the document, (b) modify and save an existing XML document. I guess I'm not acquainted with the arguments that the DOM model is broken. It's a good model for saving data up to fairly large sizes. Random-access or streamed reading (essentially Expat-type methods) are better if all you want to do is extract data. Are critics of DOM looking for some record-based approach that allows them to lock, modify and save parts of an XML document without the need to load an entire document into memory? If so, I suppose I could see some advantages of that, but it also seems like that sort of functionality could be an extension of the DOM concept rather than an entirely new approach. A good C++ implementation, in my mind, would attempt to utilize a Document class hierarchy, perhaps based directly upon the object hierarchy presented in the Java Document interface. In C++ we can enhance it by using familiar STL containers for a lot of things. For example, Java provides a getElementsByTagName API (gets me a list of all the elements given a particular name). In C++ it would be nice if the equivalent function gave me a hash_multimap<string,Element> so I could work whatever iteration magic I felt like. If the DOM class inherited from some intermediary random-access class, the latter might provide an interface for those unwilling to load the whole document into memory, yet also save people the trouble of dealing with writing the event handling & function-callbacks that's required in a C-style implementation based on Expat.

Jon Radoff wrote :

b) Those that are complete have very annoying and/or intractable dependencies. Examples: Libxml++ is simply a beast to build (lots of GNOME/Glib dependencies, and it produces compiler errors on MSVC that are left to the implementor to fix);

That's because libxml++ wanted to use a special string type for unicode that was available in glibmm (glib C++ wrapper). You can disable this though and handle the utf-8 through std::string, which removes the dependencies, and probably the building problems with older MSVC versions. (current versions of gtkmm/glibmm are known to support MSVC8 only) I think using such a string type is a good idea though, but something available as a self-supporting library would be better.

3) I think DOM support is critical. There's ample C++ libraries and wrappers for doing non-DOM XML manipulations, but there doesn't seem to be adequate options for good DOM libraries. It seems to me that if the library implementor doesn't tackle the challenge of making a nice C++ DOM class interface, then I'm left with the impression that boost wouldn't really be adding anything new to the world.

While DOM is memory consuming, it's still the simplest way I can think of to edit an existing XML document. Moreoever it is the standard API. So of course supporting it is very important.