Matt Gruenke wrote:
> This confused me.  I've always heard event-driven or callback-based 
> interfaces described as "push", since the user's code gets invoked by an 
> external event source.  Do I correctly understand that you're talking 
> about a SAX-like interface (in that it processes the document in-order, 
> and limits visibility to one node at a time) that's "pull" (i.e. user 
> code calls the parser) instead of "push" (i.e. parser calls user 
> provided methods)?
>   
Correct. I said "event-based" because that's what's "pulled" from the
parser each time: an event.
> You mean two types of "pull" interfaces, right?
>   
Yes. "Reader" and "pull" are often used synonymously.
>> 1) The Monolithic Interface
>>     
> I don't see any fundamental reason why such an interface can't support 
> inheritance.
>   
Because the parser object *is* the event object of the later models. If
it were to use inheritance to hide inappropriate methods, it would have
to change type dynamically. That's not possible.
> If I were using an interface that let me iterate over the document, I'd 
> at least want to be able to decide whether to get the next sibling node 
> or the next child node.
You could realize something like that by filtering events. The option to
insert event filters is definitely part of the plan.
> A pull interface, like this, could not only 
> support copying the current node, but could even copy entire subtrees 
> (e.g. copyUntilNextSibling()) - though this operation would require 
> dynamic memory allocation.
>   
Copying in what way? To a writer? To a tree of objects? I don't understand.
>> 2) The Inheritance Interface
>>  
>> Contra: Event objects need to be allocated on the heap.
>>     
> Why does an inheritance-based parser need to store objects on the heap?  
> If the memory is owned by the parser, it can pre-allocate a temporary 
> object for each type of node.  Based on the type of node, it fills the 
> temporary object of the appropriate type, and returns a const ref to 
> that object.
>   
True, that's possible. It voids the advantage of directly storing the
returned objects, though.
> If the caller wants a copy, the copy would only have to be 
> heap-allocated if copied via some virtual function in the node 
> base-class.  For the concrete classes (obtained via dynamic-casting), 
> copy constructors and assignment operators would work just fine.  
>   
Certainly an option.
>> It does not
>> return a reference to the event object, though, but instead a
>> boost::variant of all possible events.
>>     
>
> That could be big, depending on how much text you buffer.  Not only 
> would it waste memory, but memcpy'ing around all of that could waste 
> some of the performance savings gained by avoiding heap allocation.  
> Maybe RVO eliminates some or all of the performance penalty, but it's 
> probably unwise to depend so much on RVO.
>   
It also depends on the type of string used. If it's something like the
proposed const_string, the copy overhead would be small.
> Of course, passing in the result might be the solution - at least to the 
> performance issues.  A parser that allows the user to pass in the result 
> would also facilitate copying subtrees, if your node type has addChild() 
> and addSibling() methods.
>   
I have no node type. I'm not building a DOM tree here - this interface
is more low-level.
>> Independently of the type of interface chosen, another issue is
>> important: the scope of the interface. Should it report all XML events,
>> including those coming from DTD parsing?
>>     
> Why re-invent more than necessary?
I don't understand. A generic XML library needs to cater to all users.
This means that it must be able to report the structure of DTDs to those
clients that want it, like graphical XML construction kits, while still
being easy to use in the case of high-level demands such as an
application reading an XML configuration file.
This means the library either needs one interface that can do both, or
one interface per use case.

And that's the pull parsing alone. The same decision has to be made for
push parsing (although the SAX specification pretty much takes care of
the decisions there) and again for any in-memory object models that the
library wants to support.
> Use DOM and/or pick some other, 
> existing object models (unless you have specific issues which they don't 
> address).
>   
I'm not picking any object models at the moment. I'm working strictly on
the basis of a linear stream of events right now.
>> Should errors be reported as error events, or as
>> exceptions? Should this, too, be a user choice?
>>     
> I think the biggest reason to avoid exceptions would be for the 
> performance impact.
That is one. The other is a sort of consistency: if we're already
handling events, we might as well add errors to those events.
> I don't know whether the difference would be 
> significant, in the case of XML parsing.  However, to get the full 
> performance benefit, I think you'd need to use empty exception 
> specifications - in which case the choice would have to be made at 
> compile-time (at the latest).
>   
Not necessarily. Because of the way the system works, an empty exception
specification does not require that all called methods also have empty
specifications. If only the outermost layer of the parser implementation
uses exceptions (i.e. it translates error events to exceptions if that's
the user's choice) then only that layer cannot have the specification. I
think under these circumstances, the overhead would be indeed completely
negligible.
The other issue here is that no-overhead-without-error exception
handling is possible, so an interface shouldn't really be compromised
because an XML error should still be handled at high performance.
> Perhaps there are some other benefits to using an iostreams style 
> error-handling model, where the parser is treated like a stream.
>   
I don't know what you mean by that.
>> How about warnings:
>> exceptions are inappropriate for them. Should it be possible to disable
>> them completely?
>>     
> What's a warning?
Here are some warnings:
1) A non-validating parser encounters, in the internal document type
subset, entity declarations or attribute-list declarations after a
reference to an external entity. According to XML 1.0, section 5.1, it
MUST NOT process these, but neither does the spec say that is must
signal an error. This is an excellent place to generate a warning.
2) A parser that is not validating or has validation disabled encounters
a reference to an undeclared entity. The requirement that all entities
are declared is a validity constraint, not a well-formedness constraint,
but issuing a warning for this case still makes sense.
3) A validating parser with validation enabled encounters any of the
predefined entities "amp", "lt", "gt", etc. without them being
explicitly declared. Sections 4.1 and 4.6 both mention that valid
documents, for interoperability, SHOULD declare these entities
explicitly, but don't require it. This would be a good place to issue a
warning.

Other warnings that mention bad styles can be come up with. If the
interface doesn't have a way of reporting such warnings, an application
that wants to lint XML files would have to use its own parser interface.
Having the facility included in the interface of Boost.XML increases
flexibility.

There is another thing here: most modern parser libraries operate on the
philosophy of the XML Infoset, not the XML text serialization. This is
very important for a new library, now that a binary serialization of XML
is being formulated. Any parser library thus must provide a way of
plugging a different parser behind the same interface.
And once you have this option, why stop at XML? You could allow any
format that represents the annotated content tree that XML uses (a tree
of nodes, each node either a named element with an unordered set of
key-value pairs and an arbitrary sequence of child nodes, or a text
(leaf) node without name or attributes).
Like HTML. And HTML, with all the quirks that are not valid but still
need to be accepted if there should be any hope of interoperability, is
a great source of warnings of all kinds. For example, mixing alphabetic
and numeric characters in an unquoted attribute value. Not terminating
an entity. That sort of stuff.

Sebastian Redl

Stefan Seefeld wrote:

(And I agree, validation should be handled independently, so users don't have to pay for things they don't use.)

Little detail: the XML specification actually requires that validation is at user discretion. I plan to implement it as a filter. But even then it must always be present: the spec requires quite a few things from it. Sebastian Redl

loufoque wrote:

Sebastian Redl wrote:

...

Once again I'm turning to the list for discussion about a design issue in the XML library. This time I hope to avoid any discussion about the implementation on the library and focus on interface only.

Have you thought about asynchronous parsing? How could that be available?

I have thought about it. A pull interface is not very suited for asynchronous parsing, but it will provide non-blocking parsing (returning a "would-block" event if not enough input is available). The push interface will provide asynchronous parsing by somehow registering with an ASIO io_service. I'll have to take a closer look at ASIO to find out how exactly to realize this, though.

There are of course variations, like the one Matt Gruenke revealed. You could provide the inheritance interface but with the objects actually owned by the parser (making it kind of like the monolithic interface), and use variant to store those objects on the stack.

This idea doesn't look so bad actually, since you have the second solution without its drawbacks and that you only gain the advantages of the first solution (if you provide the appropriate tools to allow copy construction of the referenced objects, that is).

Yes, that sounds like a good solution indeed.

I don't understand, though, if you mean that the parser containing its state is a good thing or not.

Neither. Both modes have advantages and disadvantages.

Examples of how some basic operations could be done with those interfaces would come in handy to compare them for the ones, like me, that don't have much experience with parsing XML.

Yes, good idea. I'll work something up.

Validation is quite costly: a way to prevent it would be nice. And it's not just DTD, there are other validation means.

Like Relax NG and Schema. I know. But as Stefan Seefeld correctly posted, this is not about validation. This is about what to do with errors that come up during validation and/or well-formedness checking.

However, without validation you don't know what the `id' attribute is, which is quite annoying. It seems that's why they introduced xml:id. Browser engines like Gecko don't validate but they know what the id attributes are for each namespace that they handle. Maybe something similar could be done, be it with static data or user input.

I plan to support the xml:id specification, but not store any knowledge of specific namespaces. Of course, there will be a way to feed the validator programmatically, so this could be implemented easily on top of that.

...

Should errors be reported as error events, or as exceptions?

We expect errors to happen, so we shouldn't use exceptions.

Do we? A SOAP server typically expects to receive programmatically generated XML, so it ought to be error-free. On the other hand, an XML-aware editor fully expects errors, because they're guaranteed to be there in incomplete documents.

We could allow them to be toggled on though, for users that don't want to check for such things and are not looking for super efficiency. That's what I think, too. Maybe they should be using a higher level API then though.

Perhaps, but some people might have memory as their main constraint, not speed. They would still want to use a low-level interface, yet not expect errors. Sebastian Redl

Oleg Abrosimov wrote:

Hello Sebastian, let me share my ideas inspired by your post.

First of all, I'm confused why there is the 'event' term is used if one needs to call 'next' method to get the next one? It looks like this is a good word here though, but it should be used more sequentially. In a current installment you can safely omit this word and think of your XMLReader as a

std::copy(std::input_iterator<XMLObject>(std::cin), std::input_iterator<XMLObject>(), MyIteratorAdapterForXMLObject());

where MyIteratorAdapterForXMLObject - is a user defined iterator that do the actual processing.

Of course, this view is simplistic, but hope you catch the point: There is no place for the Event concept!

Right.

But don't get me wrong, please, my point is that Boost.XML should be made around the real Event concept.

Let me rephrase that in order to make sure I really understand what you are getting at here. This is about 'push' vs. 'pull', right ? While the 'next()' approach suggests the user is indeed iterating over input 'tokens' that are getting pulled put of some input stream, you suggest a toplevel 'parse_input' resulting in some underlying reactor to push events on the user. Is that correct ? If so, I disagree. I think it is perfectly fine to let the user keep control over the iteration process. The question is how to manage the fact that the tokens are polymorphic. Can a call to 'next()' be combined with some statically typed handlers that neither force the user to do the downcasting himself, nor does force the base class interface be the union of all the wrapped interfaces. (In fact one could wonder whether there needed to be a common base class at all.) Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...

loufoque wrote:

Stefan Seefeld wrote:

...

What about an interface similar to SAX, where the user provides a set of handlers, one per type, and then the reader calls the appropriate one ?

This is a discussion about the interface for a Pull parser. You're talking about a Push one.

No I'm not; at least, I don't think I am. :-) The user still has control over how the reader advances from one token to the next, since it is the user who calls 'next()' (or however it will be spelled). A push parser would be one where you call a single 'run' method and then the parser dispatches tokens (embedded into events). I don't think the fact that my design involves a dispatch qualifies it as push. Compare this to a visitor to resolve the token's type. Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...

Hi Boris, Boris Kolpackov wrote:

Hi Stefan,

Stefan Seefeld <seefeld@sympatico.ca> writes:

...

What about an interface similar to SAX, where the user provides a set of handlers, one per type, and then the reader calls the appropriate one ? For example:

void handle1(token1 const &); void handle2(token2 const &); ...

typedef reader<handle1, handle2, ...> my_reader; my_reader r(filename); while (r.next()) r.process();

I think there is not much you can do in that while loop except calling process() which then brings the question of why not use the push model (e.g., SAX) since that is what you are essentially emulating.

Are you arguing against the pull model here or against my use of callbacks ?

Also note that you can get this behavior with a normal reader and a visitor but with a modular design as a bonus:

typedef visitor<handle1, handle2, ...> v; reader r (filename); while (node* n = r.next()) v.visit (n);

That's right, but that is inefficient: The reader already does know the type of the token, but in the name of 'modular design' you throw it away only to recover it later with an extra round-robin dispatch through the visitor. What is the advantage of that ? Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...

Boris Kolpackov wrote:

typedef visitor<handle1, handle2, ...> v; reader r (filename); while (node* n = r.next()) v.visit (n);

Why do we have a node* here? Shouldn't it be a boost::variant<Node, ...>& ?