On 15/07/2014 10:15 PM, Mathias Gaunard wrote:

On 15/07/14 08:04, Michael Shepanski wrote:

...

1. If you want some class to be mapped into SQL data, quince lets you write this sort of thing:

struct point { float x; float y; }; QUINCE_MAP_CLASS(point, (x)(y))

extern database db; table<point> points(db, "points");

The macro QUINCE_MAP_CLASS uses a bit of Boost.Preprocessor magic and a bit of template metaprogramming to produce all the statically typed metadata that quince needs, regarding the user's type. There is no code generation step.

There are already macros to do this in Boost Fusion. Why not re-use them?

Could you clarify what you mean by "do this"? QUINCE_MAP_CLASS builds a class with exactly the data members, function members, and bases that quince requires. Also, if quince did use Booost.Fusion, it would only be at one site. So it seemed best to avoid the inter-library dependence. I think I'm following the advice in http://www.boost.org/development/reuse.html.

Boost.Fusion is a tuple manipulation library initially developed for a parser generator EDSL.

I'm not actually doing tuple manipulation here. That's not to say the task couldn't be solved by a detour through tuples, but it /would/ be a detour. Regards, --- Michael

On 17/07/2014 2:13 AM, pfultz2 wrote:

...

Could you clarify what you mean by "do this"? QUINCE_MAP_CLASS builds a class with exactly the data members, function members, and bases that quince requires. Boost.Fusion already provides the ability to map the classes to fusion sequences.

But that is not what QUINCE_MAP_CLASS and QUINCE_MAP_CLASS_WITH_BASES do. Those macros define mapper classes with specific data members, function members, and bases, which are required by the rest of quince. Also the members and bases are exposed to user code, so they have to be defined in exactly the way that those macros define them.

Here is a very simple example how to build an ORM using Boost.Fusion in C++14:

https://gist.github.com/pfultz2/abbea635bdc8a4971424

Of course as a simple example, it doesn't map the types to the database types(it just uses `typeid` for illustration purposes only).

Now this example just demonstrates mapping to the database with fusion. The other half is building queries in C++. This can easily be done by taking advantage that the structs are adapted as associative sequences. So we can use the keys to refer to the fields. So we could build queries something like this:

table<person> p("person"); auto query = from(p) .where(at_key<fields::age>(p) > 60) .select(at_key<fields::name>(p));

Of course, some additional macros would be useful to reduce the boilerplate, but an ORM in C++ should really start with Boost.Fusion.

It sounds like you are proposing a new product, rather than a change to quince. I wish you well.

...

I'm not actually doing tuple manipulation here. That's not to say the task couldn't be solved by a detour through tuples, but it /would/ be a detour. But it would be a 'detour' that would save you from writing mapping macros.

I still need to define my mapper *class*, with its appropriate members and bases, as described here http://quince-lib.com/mapped_data_types/user_defined_class_types.html . When I come back from the tuples detour, how will I get this class defined? I think I will be back to using Boost.Preprocessor magic. By the way, you say "mapping macros" but it's really just one. The implementations of QUINCE_MAP_CLASS and QUINCE_MAP_CLASS_WITH_BASES both invoke one macro (QUINCE_DEFINE_CLASS_MAPPER_WITH_BASES) to do all the hard work. If I were to take the tuples detour, that would be the only site where it happens. Regards, --- Michael

On 17/07/2014 2:38 PM, pfultz2 wrote:

...

When I come back from the tuples detour, how will I get this class defined? There is no need to define this class. Boost.Fusion takes care of it already.

Again, you say "takes care of it" -- but your "it" is not the "it" that I need. :) Let's look from the point of view of a quince user. The user writes some code like this: extern table<point> points; for (const point &p: points.where(points->x < 4.0f)) std::cout << p.y << std::endl; Look at the expression "points->x". "points" is the table. Its operator->() returns a pointer to that table's value mapper. points->x is a mapper for one part of points. In order to make this syntax work, the mapper type has to have a member named x, i.e. a member whose name is the same as the name of the value type's member. This is a requirement, so that my EDSL looks the way I want it to look. The QUINCE_MAP_CLASS macro makes this happen by defining a structure with members named in this way. If you can do *that* with Boost.Fusion then perhaps you can show us the code that does it, and we discuss costs and benefits.

...

If I were to take the tuples detour, that would be the only site where it happens. What do you mean?

I mean, if I were to take the approach of implementing QUINCE_MAP_CLASS and QUINCE_MAP_CLASS_WITH_BASES by (a) using Boost.Fusion to define some kind of tuple, and then (b) using some other means to turn that tuple into the class I need, then I expect it would all be done in one place, _viz._ inside the definition of quince's private macro QUINCE_DEFINE_CLASS_MAPPER_WITH_BASES. So I would be introducing a dependency on Boost.Fusion for the sake of changing one point in my code. --- Michael

On 18/07/2014 2:11 AM, pfultz2 wrote:

...

If you can do *that* with Boost.Fusion then perhaps you can show us the code that does it, and we discuss costs and benefits. Sure. You just access the field by using the key.

I know, but that's not what I asked. I want the syntax to be points->x.

Here is a quick-and-dirty example here(I haven't compiled it yet, but it should communicate the idea):

https://gist.github.com/pfultz2/cfc447c7ccfa8ac2e76f

So then you should be able to write:

table<person> persons;

for(const person& p:persons.where(at_key<fields::age>(persons) == 60)) { ... }

Another idea is to generate the member in with the key, perhaps something like this:

#define QUINCE_GEN_KEY(name, ...) \ struct name : __VA_ARGS__ \ { \ template<class T> \ struct lazy \ { \ T name; \ }; \ };

Then you can use the fusion keys to compose a class that inherits from each lazy member, so you could still support accessing it using the `->` operator. I don't show an example here.

When you say "you can use the fusion keys to compose a class ...", do you mean that the user writes each such class by hand, or can you write something that will make it automatic? If you are telling me that Boost.Fusion can replace QUINCE_MAP_CLASS etc., then your challenge is to make it automatic. I need to see your code for this, before I can assess whether a dependence on Boost.Fusion will buy me a nett reduction in quince code.

However, the first approach I think is sufficient.

...

I mean, if I were to take the approach of implementing QUINCE_MAP_CLASS and QUINCE_MAP_CLASS_WITH_BASES by (a) using Boost.Fusion to define some kind of tuple, and then (b) using some other means to turn that tuple into the class I need, then I expect it would all be done in one place, _viz._ inside the definition of quince's private macro QUINCE_DEFINE_CLASS_MAPPER_WITH_BASES. So I would be introducing a dependency on Boost.Fusion for the sake of changing one point in my code. You wouldn't generate your mapper class using the preprocessor, you can just generate it using C++ template metaprogramming. Perhaps, you would use the preprocessor to generate the keys, but not the mapper or table class(as I show in my little example).

Likewise in quince, most of the work of QUINCE_MAP_CLASS etc. is done without macros, by something called typed_class_mapper_base (which the user never mentions). It's also perhaps worth pointing out that the whole implementation of QUINCE_MAP_CLASS and QUINCE_MAP_CLASS_WITH_BASES is a very small part of quince. On the other hand the mapper classes that they generate are referenced ubiquitously.

Boost.Fusion does a great job of handling product types, which an ORM would need. It seems unnecessary to reinvent the wheel here.

My philosophy is somewhat different. I don't believe that I should accept sacrifices (degraded EDSL, indirect implementation) for the sake of using Boost.Fusion. On the contrary, I think any such decision needs to be justified by benefits, to outweigh the negative of inter-library dependence. Again I cite http://www.boost.org/development/reuse.html . Cheers, --- Michael

On 17/07/14 07:08, Michael Shepanski wrote:

for (const point &p: points.where(points->x < 4.0f)) std::cout << p.y << std::endl;

Look at the expression "points->x". "points" is the table. Its operator->() returns a pointer to that table's value mapper. points->x is a mapper for one part of points.

I see why you need a macro for that. I think you should decorrelate the expression type and the actual structure though.

On 21/07/2014 6:53 PM, Mathias Gaunard wrote:

On 15/07/14 14:45, Michael Shepanski wrote:

...

Could you clarify what you mean by "do this"? QUINCE_MAP_CLASS builds a class with exactly the data members, function members, and bases that quince requires.

Boost.Fusion defines tuple concepts, and provides utilities to adapt a class to a tuple (or define a class and adapt it at the same time).

To maximize compatibility, and to prevent users having to say that their structure is tuple-like a billion times, it's better to reuse this.

So you are suggesting that users who want to say QUINCE_MAP_CLASS be made to say BOOST_FUSION_ADAPT_STRUCT instead? I see that BOOST_FUSION_ADAPT_STRUCT asks users to repeat the members' types, in a way that QUINCE_MAP_CLASS doesn't. (Why is that, btw?) I don't (yet) see how BOOST_FUSION_ADAPT_STRUCT could be used to replace QUINCE_MAP_CLASS_WITH_BASES. Also I don't see how any of this is going to produce the mapper classes with all the specific features that the rest of quince needs. I think that would be an extra step.

Any type that is Fusion-compatible should be made compatible with your library too.

Quince already allows std::tuples to be mapped types. So I guess quince could, similarly, allow fusion sequences to be mapped types. Then a user who has a class could indeed choose to get it mapped by the circuitous route of adapting it to a Fusion sequence and then mapping that. (It wouldn't be my choice, but to each his own.)

Also, building as you do seems quite intrusive. You probably want to support a non-intrusive approach anyway.

What do you mean? --- Michael

On 07/21/2014 02:34 PM, Agustín K-ballo Bergé wrote:

On 21/07/2014 08:01 a.m., Michael Shepanski wrote:

...

On 21/07/2014 6:53 PM, Mathias Gaunard wrote:

...

On 15/07/14 14:45, Michael Shepanski wrote:

...

Could you clarify what you mean by "do this"? QUINCE_MAP_CLASS builds a class with exactly the data members, function members, and bases that quince requires.

Boost.Fusion defines tuple concepts, and provides utilities to adapt a class to a tuple (or define a class and adapt it at the same time).

To maximize compatibility, and to prevent users having to say that their structure is tuple-like a billion times, it's better to reuse this.

I see that BOOST_FUSION_ADAPT_STRUCT asks users to repeat the members' types, in a way that QUINCE_MAP_CLASS doesn't. (Why is that, btw?)

Because Fusion supports C++98. That said, there's work being done on a branch to add member type deduction via `decltype` to all ADAPT macros.

Just to note that the new macros doesn't disable the C++98 compatibility as it uses Boost.TypeOf, the emulation just work fine with C++98 compilers, and it's still possible to provide the types, as you suggested. off-topic: I'll bring the missing macros soon, I'm late on schedule. :/

Regards,

On 22/07/2014 1:57 AM, Mathias Gaunard wrote:

...

I don't (yet) see how BOOST_FUSION_ADAPT_STRUCT could be used to replace QUINCE_MAP_CLASS_WITH_BASES.

Also I don't see how any of this is going to produce the mapper classes with all the specific features that the rest of quince needs. I think that would be an extra step.

From what you described, you needed a way to list the members of a structure, like what can be done with tuples. This is what Fusion does.

Yes, I need a way to list the members of a structure. And you assumed that is all I needed -- am I right?

...

...

Any type that is Fusion-compatible should be made compatible with your library too.

Quince already allows std::tuples to be mapped types. So I guess quince could, similarly, allow fusion sequences to be mapped types. Then a user who has a class could indeed choose to get it mapped by the circuitous route of adapting it to a Fusion sequence and then mapping that. (It wouldn't be my choice, but to each his own.)

There are plenty of classes all over Boost and elsewhere that are already Fusion compatible. Why not just use this concept instead of reinventing your own?

Obviously I have given numerous answers already. I take this question as a request for a summary. In order to provide the syntax "points->x", I need a macro that generates a class with member names that match the user's member names. None of the admonitions about reinventing wheels etc. have shown how to do that without the macro. The objects I am dealing with are class-like rather than sequence-like, in this sense: they have inheritance (including multiple inheritance). I have asked how Boost.Fusion can do the work of QUINCE_MAP_CLASS_WITH_BASES and there has been no answer. I currently believe that it cannot. Most of the code in my macros is due to the foregoing two points, which are *not* reinventions of anything in Boost.Tuple. So I really think it is time we put the "reinvention" argument to bed. It's a distraction from any other arguments that are still live ... Based on my reading of http://www.boost.org/development/reuse.html , I should only introduce a dependence on another significant Boost library if there are benefits. Now, as just explained, I don't think there are benefits in the form of code savings inside quince, so the quince maintainer is not a beneficiary. Then who is? ... The more recent discussion has mentioned the user who already has classes adapted to Fusion tuples, and who would presumably like to avoid the burden of *also* typing QUINCE_MAP_CLASS or QUINCE_MAP_CLASS_WITH_BASES (aka "having to say their structure is tuple-like a billion times"). Very well, then for that person I concede that there is a benefit it in providing a new sort of quince mapper class, similar to quince::tuple_mapper, but for Fusion sequences instead of std::tuples. Let's call it quince::sequence_mapper. It would be done without macros (just as quince::tuple_mapper is done without macros), it would not support inheritance, and it would not support the "points->x" syntax (which its users probably wouldn't want anyway, since they are using Fusion syntax already). Obviously sequence_mapper would share as much as possible of its implementation with class_mapper (the thing that QUINCE_MAP_CLASS etc. define), and with tuple_mapper, just as those two already share as much as possible of their implementations with each other. As it currently stands, a quince user has the choice to represent an employee as a class or as a std::tuple. If you use a class then you get inheritance and the "points->x" syntax for members. If you use a std::tuple then you get compatibility with any other code you might have that uses std::tuples already. The proposed addition of sequence_mapper would provide compatibility with other code that uses Fusion mappers already. But the reasons for allowing *classes* are still valid imo. --- Michael

On 22/07/2014 12:02 PM, Michael Shepanski wrote:

The proposed addition of sequence_mapper would provide compatibility with other code that uses Fusion mappers already.

Dang! Of course this was meant to say "with other code that uses Fusion *sequences* already". --- Michael

On Tue, 15 Jul 2014 18:27:07 -0700, Michael Shepanski <mps@optusnet.com.au> wrote:

On 16/07/2014 10:23 AM, Mostafa wrote:

...

Can this library be used with C++03? I recall sqlpp11 being a C++11 only library with the author explicitly stating that he won't invest effort into backporting it. So this could possibly be another difference vis-a-vis sqlpp11.

Quince, and the application code that uses it, need a C++11 compiler, and a pretty new one too: see http://quince-lib.com/preparation/prerequisites.html#preparation.prerequisit... . On the other hand there is no requirement on users to learn any C++11 features. You can access all of quince's features by writing C++03 code, and feeding it to your late-model C++11 compiler.

--- Michael

Thanks. It would help if that was explicitly mentioned in the documentation. (g++ by default uses C++03.) Out of curiosity, does the library *need* to be written in C++11, and, if so, why?

On 07/16/2014 07:49 AM, Michael Shepanski wrote:

On 16/07/2014 2:14 PM, Mostafa wrote:

...

On Tue, 15 Jul 2014 18:27:07 -0700, Michael Shepanski <mps@optusnet.com.au> wrote:

...

Quince, and the application code that uses it, need a C++11 compiler, and a pretty new one too: see http://quince-lib.com/preparation/prerequisites.html#preparation.prerequisit... . On the other hand there is no requirement on users to learn any C++11 features. You can access all of quince's features by writing C++03 code, and feeding it to your late-model C++11 compiler.

Thanks. It would help if that was explicitly mentioned in the documentation.

http://quince-lib.com/queries_in_c_expressions/the_quincessential_dot_points... does say "Quince is written in C++11". :)

...

(g++ by default uses C++03.)

That's interesting. You're right, of course, and yet I didn't exert any effort to make it accept C++11. On my ubuntu machine I use Boost.Build, my user-config.jam contains this line: using gcc : 4.7 : g++-4.7 ; and somehow gcc gets invoked with the commandline option -std=c++11.

...

Out of curiosity, does the library *need* to be written in C++11, and, if so, why?

If I tried to rewrite it in C++03, its length would multiply, if only because of the need to replace variadic templates by a combinatorial explosion of overloads.

But your question was about *need*, so perhaps the uses of decltype are more relevant. I do not think I could rewrite them to avoid decltype. (Maybe somebody else could, but I found it hard enough without any such constraint.)

As I'm currently doing it for Boost.Fusion, your macro to adapt the types could take advantage of Boost.TypeOf which would be your replacement for decltype : http://www.boost.org/doc/libs/1_55_0/doc/html/typeof.html But C++11-only is fine for me. :)

Regards, --- Michael

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

On 22/07/2014 8:09 AM, Mostafa wrote:

That's what I figured, and Boost.Preprocessor could be used to take care of the variadic templates. My only concern, and maybe Michael can chime in here, would be the use of temporaries. In C++11 they could be moved, but in C++03 they may have to be copied, and I wonder if that would become costly with quince's design.

Re temporaries: no alarm bells are going off in my head. I suspect that all that's at issue are small-scale optimizations that shouldn't influence us unless measurement shows otherwise. Overall comments about replacing variadic templates with Boost.Preprocessor and decltype with Boost.TypeOf: - I experienced some difficulties with internal compiler errors, and other discrepancies between compilers. I suspect that a change to different mechanisms would produce a different set of such pragmatic problems. It may very well be that Boost.Preprocessor and Boost.TypeOf lead to *fewer* problems than variadic templates and decltype -- but the latter problems have all been worked around already. - I do like the brevity of expression in C++11. Cheers, --- Michael

[skipped a few items that seem to require no further immediate discussion] On 2014-07-16 16:11, Michael Shepanski wrote:

On 16/07/2014 10:07 PM, Roland Bock wrote: sqlpp11 exposes the concept of column aliases,

...

Yes, in sqlpp11, columns of tables have names, columns in result rows have names, parameters in prepared statements have names. They are represented as members in structs which are constructed at compile time. Thus, to access column "host" in a result row you write

const std::string host = row.host;

Personally, I like this much better than

const std::string host = row<3>;

since there might be 'protocol' and 'path' as well if the whole row represents a query (all strings) and later in the life time of the product, someone wants to add username and password. Then you have to rewrite those numbers to access members of a tuple. And the compiler cannot tell you if you mix up your numbers if the types are the same.

It is much easier to get it right using names, IMHO. I therefore believe that the name approach is friendlier for maintenance.

When you execute a query in quince, it uses the query's value mapper to convert complete rows into their C++ equivalent type -- which could be simple or complex, depending on the query's value type. So the application code that receives results looks like this (from http://quince-lib.com/queries/executing.html):

for(const point &p: points.where(points->x > 4).fetch_size(50)) std::cout << p.y << std::endl;

As you can see, the access is via meaningful names, just as with your "row.host". Sorry, I misinterpreted some piece of documentation.

What I was getting at, when I said "sqlpp11 exposes the concept of column aliases", is that --if I understand correctly-- sqlpp11 puts the user in control of when column aliases are created in the SQL. Quince takes that level of control away from the user. What happens with the names when I do a self join with quince? Personally I like to have that control, since only the developer knows

...

has all the information like which indexes can be used, number of rows, etc. I therefore like the idea, but I would want to have the complete-control version as well.

http://quince-lib.com/queries/compositionality.html has some discussion of the aspects of control that quince takes away.

If you want to be sure that the sql will only sort based on columns that you know you've indexed, or if you want to avoid inspecting every row in a table that you know will be huge, then you have nothing to fear: quince leaves you in control of those important things. Ok, cool :-)

...

sqlpp11's connectors have two options to reflect specific dialects:

a) They can provide specializations of the interpreter/serializer template for parts (or all) of the expression language. With this mechanism they can either interpret (sub-)expressions according to their dialect or they can use static asserts to reject certain constructs. b) They can extend the EDSL with features that are not in the standard. This is a rather new feature and not yet documented, but it is actually quite easy, IMHO. To get an impression, take a look at https://github.com/rbock/sqlpp11/blob/master/include/sqlpp11/select.h :

template<typename Database> using blank_select_t = statement_t<Database, select_t, no_select_flag_list_t, no_select_column_list_t, no_from_t, no_extra_tables_t, no_where_t<true>, no_group_by_t, no_having_t, no_order_by_t, no_limit_t, no_offset_t>;

With those two mechanisms, sqlpp11's connector libraries can tell the developer at runtime, if a feature is supported or not, including special features which might be available for one vendor only.

You meant to say "at compile time" just now, yes?

Ooops, yes, of course :-)

Indeed, this is something quince cannot do. In order to do it, I think I would need to make database type a static property of each query, each mapper, etc. That would be significant and sweeping complication.

I suspect that, in 99% of cases, an application only deals with databases from one vendor. A user who only wants to deal with sqlite would be better served by a version of quince that didn't provide postgresql-specific features at all, and didn't declare virtual methods to implement them. Perhaps a pragmatic solution would be to provide that sort of quince, by means of preprocessor switches. =-O

That's a solution, but it also means that you might have to change quince for each additional database dialect. sqlpp11 aims to be completely vendor neutral, offering the tools to adjust for dialects and non-standard features in the connector. I hope that in the long run this will lead to more distributed efforts :-)

...

In addition, the SQL expression can be rewritten at compile time not only to talk to string based database backends, but also to other databases. Thus sqlpp11 can be used as an SQL frontend to structs in std::vector, for instance, see https://github.com/rbock/sqlpp11-connector-stl

Due to the virtual functions in quince's backends, I assume that this would be harder to do with quince and probably not with comparable performance since the expression would be reinterpreted at runtime. But I might certainly be missing something.

This is not something I have considered. Whenever I have thought about uniform treatment of queries and containers, I have thought the opposite way: making a query act like a container (http://quince-lib.com/getting_started/executing_the_query.html)

That part is common for both approaches :-) I am not sure what will happen with the treat-container-as-databases thing, but LINQ uses that with great success and sqlpp11 offers the mechanisms to do the same.

...

As of now, sqlpp11 is not using boost::optional (there's been a long discussion about it). There are several use cases, one of which works wonderfully with boost::optional. So, I would like to add support for that, too.

[...]

Fwiw, the handling of optionals, in all situations, was one of the things I found most difficult.

Thanks for the heads up :-)

...

...

5. An application can customize the representation of data in a particular database, or a particular table. Not sure what that means. When used as an ORM?

This is explained in http://quince-lib.com/custom.html

OK. You could use your own datatypes in sqlpp11 as long as they provide the correct API (which is not documented yet, but rather lean and easy to copy/paste from existing types).

...

BTW: I am going to give a talk about sqlpp11 at CppCon in September and at MeetingC++ in December. Maybe we can have a live discussion there, too?

Best regards, Roland

That would be great, but at this stage I'd say it's unlikely I'll be at either conference. I'll let you know if anything changes. (I'm in Australia, btw.)

I am living in Germany myself, but I hope the conference will be worth the effort :-) Best, Roland

On 17/07/2014 1:29 AM, Roland Bock wrote:

...

What happens with the names when I do a self join with quince?

Ah. Then there is a problem, and there is a solution, both described at http://quince-lib.com/queries/join/self_joins.html The automatic, ever changing alias is a cool feature. Since the actual alias is unknown to the user, that requires the std::get<index> instead of using a name, but that is probably OK for joins.

...

...

Indeed, this is something quince cannot do. In order to do it, I think I would need to make database type a static property of each query, each mapper, etc. That would be significant and sweeping complication.

I suspect that, in 99% of cases, an application only deals with databases from one vendor. A user who only wants to deal with sqlite would be better served by a version of quince that didn't provide postgresql-specific features at all, and didn't declare virtual methods to implement them. Perhaps a pragmatic solution would be to provide that sort of quince, by means of preprocessor switches. =-O That's a solution, but it also means that you might have to change quince for each additional database dialect.

sqlpp11 aims to be completely vendor neutral, offering the tools to adjust for dialects and non-standard features in the connector. I hope that in the long run this will lead to more distributed efforts :-)

Yes. Iiuc this is a benefit you get by including DB vendor in the static type of your objects, i.e. a statement_t for MySQL is a different type from a statement_t for sqlite -- yes? Not exactly. In sqlpp11, the database type is only required for serialization/interpretation. If the full statement structure is known at compile time, the statement_t is independent of the database. Only if you use parts which are decided at runtime (e.g. some expensive column X is in the result row only if the user really, reallly wants it), then

On 2014-07-17 03:08, Michael Shepanski wrote: these dynamic parts require to know the database and it is convenient to do this by adding the database type to the complete statement. Thus, in at least 90% of the statements I write myself, the database template parameter of the statement is just void.

For quince I decided I preferred I wanted my queries to be simply query<point>, query<employee> etc. The price I pay is that use of vendor-inappropriate features is not caught until run-time. I concede that.

...

I am not sure what will happen with the treat-container-as-databases thing, but LINQ uses that with great success and sqlpp11 offers the mechanisms to do the same.

Would it be correct to say that you get this because sqlpp11 does not generate SQL, but rather it leaves that completely to the connector libraries, so they are free to do something completely different instead?

Yes, the connector library is completely free, but it does not have to re-invent the wheel, too. sqlpp11 has a default serializer implementation which generates standard SQL. The connector libraries only have to specialize for the things that are actually special. For instance, MySQL uses the concat function instead of the || operator. Thus it has to specialize string concatenation. This is the full serializer specialization for the MySQL connector: https://github.com/rbock/sqlpp11-connector-mysql/blob/master/include/sqlpp11... sqlpp11's EDSL is not complete yet, so there probably will be a few more entries for other special things, for instance iirc, MySQL does not support WITH, so I would disable it in that file by adding a static assert into a specialization for with_t.

I have (so far) taken the approach of centralizing as much as possible of the SQL generation in quince itself, with the backend libraries only making a contribution where there are dialectal differences.

Same in sqlpp11 as explained above. Of course, it is different if the connector does not operate on strings like the sqlpp11-connector-stl. That one has to implement a full interpreter. But of course, that is because its interpretation is entirely different from the standard string serialization, so there is no way around it.

I have not thought about this until today, but I suppose it would be possible to change quince so that it always calls out to backends to do the whole job of generating SQL and executing it; and then a special-purpose backend could decide to do something else instead.

If I took that approach, I think I would still want some common code to generate "normal" SQL, rather than repeating such code in quince_postgresql, quince_sqlite, etc.. Quince could provide such code, and it would be up to each backend library to decide whether to use it.

That's exactly what sqlpp11 is doing in a finely grained way, so that is not a yes/no decision but a 'OK, I like most of the default, but I'll change this and that detail' kind of approach for most connectors. Cheers, Roland

On 2014-07-17 08:04, Michael Shepanski wrote:

On 17/07/2014 2:48 PM, Roland Bock wrote:

...

The automatic, ever changing alias is a cool feature. Since the actual alias is unknown to the user, that requires the std::get<index> instead of using a name, but that is probably OK for joins.

What you say about std::get<index> makes me think there might be some residual misunderstanding.

It's true that joins generate tuples. E.g. (from http://quince-lib.com/queries/join.html):

const query<std::tuple<cinema, screen, movie>> combinations = join(cinemas, screens, movies);

Code that receives the results would look like this:

for (const std::tuple<cinema, screen, movie> &combo: combinations)

As you can see, the members of the received tuple are a cinema, a screen, and a movie. So the user will say std::get<>(), but it's not to access individual columns.

And in any case, if the user doesn't like tuples, he can use a collector class instead (http://quince-lib.com/queries/join/collector.html), then he doesn't need to use std::get<>() in any way at all. It is really fascinating to see that many ideas are almost identical. sqlpp11 also has a grouping some columns and give the group a name. It is called multi_column, can be accessed by name just like columns and is of course of great value in joins :-)

All of these points apply the same to normal joins or self-joins. The only difference with self-joins is the use of table_alias objects in lieu of table objects. Got it :-)

Does quince support adding columns at runtime, btw? Say, I have a table that contains some meta data and a blob. I always want to select the meta data, but whether or not to select the blob is decided at runtime.

...

Not exactly. In sqlpp11, the database type is only required for serialization/interpretation. If the full statement structure is known at compile time, the statement_t is independent of the database. Only if you use parts which are decided at runtime (e.g. some expensive column X is in the result row only if the user really, reallly wants it), then these dynamic parts require to know the database and it is convenient to do this by adding the database type to the complete statement.

Thus, in at least 90% of the statements I write myself, the database template parameter of the statement is just void.

So, if it's not too much trouble, could you walk through the steps that lead to a compile-time failure, if I try to use a feature that is not supported on the DBMS where I am trying to use it?

Since you have watched the "selected template toffees" video, you have seen all the tools already. Here's a walk through: * You have an expression tree and a context type. The latter is provided by the database * sqlpp11 provides a template function called serialize, which takes an generic expression tree and a generic context (basically any output stream). * sqlpp11 also provides "partial specializations" of these functions via a serializer template class. The partial specializations are available for all building blocks of the EDSL (e.g. tables, columns, select, where, arithmetic expressions, ...) and the generic context. You can see those at the bottom of each header file for such a building block, e.g. https://github.com/rbock/sqlpp11/blob/master/include/sqlpp11/group_by.h * If you need special behavior, you add a specialization for the respective building block and your own context type. * If your special behavior is that you do not support that feature, you just say so in a static_assert which fires when the serializer is instantiate Here's an example from https://github.com/rbock/sqlpp11-connector-sqlite3/blob/master/include/sqlpp... (sqlite does not support full outer join): template<typename Lhs, typename Rhs, typename On> struct serializer_t<sqlite3::serializer_t, join_t<outer_join_t, Lhs, Rhs, On>> { using T = join_t<outer_join_t, Lhs, Rhs, On>; static void _(const T& t, sqlite3::serializer_t& context) { static_assert(::sqlpp::wrong_t<outer_join_t, Lhs, Rhs, On>::value, "No support for outer join"); } };

...

Yes, the connector library is completely free, but it does not have to re-invent the wheel, too.

sqlpp11 has a default serializer implementation which generates standard SQL. The connector libraries only have to specialize for the things that are actually special. For instance, MySQL uses the concat function instead of the || operator. Thus it has to specialize string concatenation.

This is the full serializer specialization for the MySQL connector: https://github.com/rbock/sqlpp11-connector-mysql/blob/master/include/sqlpp11...

sqlpp11's EDSL is not complete yet, so there probably will be a few more entries for other special things, for instance iirc, MySQL does not support WITH, so I would disable it in that file by adding a static assert into a specialization for with_t.

Cool. (Btw I saw your online video about "template toffees", and it made me think I should do more with static_assert. My thinking has been "Any error I can catch at compile-time rather than run-time is awesome", but of course earlier is better, even at compile time.)

Thanks for the feedback :-) Cheers, Roland

On 2014-07-17 13:11, Michael Shepanski wrote:

On 17/07/2014 5:09 PM, Roland Bock wrote:

...

It is really fascinating to see that many ideas are almost identical. sqlpp11 also has a grouping some columns and give the group a name. It is called multi_column, can be accessed by name just like columns and is of course of great value in joins :-)

The difference, of course, is that in quince we don't speak of "columns" so much. :) :-)

Every query produces values of its "value type", which can be any type in this taxonomy: http://quince-lib.com/mapped_data_types.html#mapped_data_types.definition_ta... . It is common for a query's value type to be the same as a table's value type, but this is not the case for queries that use join() or select(). Shouldn't collections be mentioned on that page, too?

...

...

All of these points apply the same to normal joins or self-joins. The only difference with self-joins is the use of table_alias objects in lieu of table objects. Got it :-)

Does quince support adding columns at runtime, btw? Say, I have a table that contains some meta data and a blob. I always want to select the meta data, but whether or not to select the blob is decided at runtime.

Sure. Quince's select() is analogous to supplying an SQL "select list". (I say "analogous", not identical, because here again we work at the level of C++ types, without regard to whether they are single-column or multi-column.)

So it's:

struct something { metadata meta; // metadata is some mapped type defined elsewhere vector<uint8_t> blob; string other; }; QUINCE_MAP_CLASS(something, (meta)(blob)(other))

extern table<something> somethings;

// retrieve metadata only: for (const metadata &m: somethings.select(somethings->meta)) // ...

// retrieve all three parts: for (const something &s: somethings) // ...

// retrieve metadata and blob only: for (const std::tuple<meta, vector<uint8_t>> &pair: somethings.select(somethings->meta, somethings->blob)) // ...

In the last example, we could use a collector class if we don't like tuples.

Hmm. I guess my question was unclear or I misunderstood you answer. I was thinking something like auto query = somethings.select(somethings->meta) if (userWantsBlob) query.add_to_select_list(somethings-blob); for(const auto& row : query) { if (userWantsBlob) { //somehow access blob here } } We have some code in our company that has several of these dynamic fields. Multiplying the code is not really an option then.

...

Since you have watched the "selected template toffees" video, you have seen all the tools already. Here's a walk through:

* You have an expression tree and a context type. The latter is provided by the database * sqlpp11 provides a template function called serialize, which takes an generic expression tree and a generic context (basically any output stream). * sqlpp11 also provides "partial specializations" of these functions via a serializer template class. The partial specializations are available for all building blocks of the EDSL (e.g. tables, columns, select, where, arithmetic expressions, ...) and the generic context. You can see those at the bottom of each header file for such a building block, e.g.

https://github.com/rbock/sqlpp11/blob/master/include/sqlpp11/group_by.h * If you need special behavior, you add a specialization for the respective building block and your own context type. * If your special behavior is that you do not support that feature, you just say so in a static_assert which fires when the serializer is instantiate

Here's an example from https://github.com/rbock/sqlpp11-connector-sqlite3/blob/master/include/sqlpp...

(sqlite does not support full outer join):

template<typename Lhs, typename Rhs, typename On> struct serializer_t<sqlite3::serializer_t, join_t<outer_join_t, Lhs, Rhs, On>> { using T = join_t<outer_join_t, Lhs, Rhs, On>; static void _(const T& t, sqlite3::serializer_t& context) { static_assert(::sqlpp::wrong_t<outer_join_t, Lhs, Rhs, On>::value, "No support for outer join"); } };

Okay, I think I understand. When you have a complex expression tree that includes, say, a full join somewhere deep inside, then that affects the static type of the expression tree as a whole. Therefore, when you ask a connector to serialize the tree, the compiler can see the incompatibility and fail. Yes?

That's correct for queries with a static structure. For dynamic parts (like the optional blob in the example above), the compatibility would be tested when the dynamic part is added to the query.

If so, then it confirms what I suspected (albeit for different reasons): I can't make quince behave this way -- or at least the price of doing so would be very high. A complex quince::query object may have a full join buried somewhere inside, but it doesn't affect the object's static type. (It only becomes apparent when quince tries to generate the SQL, so it traverses the tree, calling virtual methods, etc.)

If you knew the database type while constructing the query, you could do those tests during construction. It probably wouldn't even be that hard with a similar mechanism as used in sqlpp11 (easy for me to say, still not knowing really much about your code). Best, Roland

On 2014-07-17 15:34, Michael Shepanski wrote:

On 17/07/2014 10:48 PM, Roland Bock wrote:

...

On 2014-07-17 13:11, Michael Shepanski wrote:

...

Every query produces values of its "value type", which can be any type in this taxonomy: http://quince-lib.com/mapped_data_types.html#mapped_data_types.definition_ta...

. It is common for a query's value type to be the same as a table's value type, but this is not the case for queries that use join() or select(). Shouldn't collections be mentioned on that page, too?

I'm not sure I understand. Do you mean STL containers? Then no, these are not mapped types (except for the special cases std::string and std::vector<uint8_t>). You can't have a table<std::list<float>>. No, sorry for the confusion, I meant the usage of collector classes.

...

...

Sure. Quince's select() is analogous to supplying an SQL "select list". (I say "analogous", not identical, because here again we work at the level of C++ types, without regard to whether they are single-column or multi-column.)

So it's:

struct something { metadata meta; // metadata is some mapped type defined elsewhere vector<uint8_t> blob; string other; }; QUINCE_MAP_CLASS(something, (meta)(blob)(other))

extern table<something> somethings;

[...] Hmm. I guess my question was unclear or I misunderstood you answer.

I was thinking something like

auto query = somethings.select(somethings->meta) if (userWantsBlob) query.add_to_select_list(somethings-blob); for(const auto& row : query) { if (userWantsBlob) { //somehow access blob here } }

Ah, I see. No, on two counts.

[...]

Iiuc you are starting with a query that gets something narrow, and then using add_to_select_list() to widen it. With quince, you would work in the opposite direction: start with a query that gets all the information anyone needs (or just start with the table itself), and use select(...) to make narrower queries as needed. Making new queries in this way is a cheap operation.

Would that cause you any problems? (I'm not sure what you mean by "Multiplying the code".)

Say I have two conditions, userWantsBlob and userWantsOther. I would want to have everything in one loop. In sqlpp11 I could say something like: auto query = dynamic_select(db, t.meta)...; if (userWantsBlob) query.add_column(t.blob); if (userWantsOther) query.add_column(t.other); for (const auto& row : db(query)) { std::cout << t.meta << '\n'; if (userWantsBlob) std::cout << t.at("blob") << '\n'; if (userWantsOther) std::cout << t.at("other") << '\n'; }; How would you do that in quince?

...

If you knew the database type while constructing the query, you could d those tests during construction. It probably wouldn't even be that hard with a similar mechanism as used in sqlpp11 (easy for me to say, still not knowing really much about your code).

As quince builds a query it always "knows" the database type in a dynamic sense, but not in a static sense.

OK Cheers, Roland

On 18/07/2014 2:26 AM, Roland Bock wrote:

...

On 2014-07-17 15:34, Michael Shepanski wrote:

...

...

Shouldn't collections be mentioned on that page, too? I'm not sure I understand. Do you mean STL containers? Then no,

On 17/07/2014 10:48 PM, Roland Bock wrote: these are not mapped types (except for the special cases std::string and std::vector<uint8_t>). You can't have a table<std::list<float>>. No, sorry for the confusion, I meant the usage of collector classes.

Oh but they are mentioned! They are in the category of "user-defined class". OK, I missed that.

Just as a std::tuple can be used to represent data in a table, and can also be put together _ad hoc_ by quince to deliver results of a join() or select(), the same can be said of user-defined classes (after they have been processed with QUINCE_MAP_CLASS). Thinking about this, I just realized why I might have such a hard time with those user-defined classes: In a earlier version (still in C++03), I used something similar to tuples to define the results of a select. That often led to inefficient code. Instead of writing a type that represented just the n columns they actually needed, people tended to use all columns all the time, just because they were too lazy to write

On 2014-07-18 03:05, Michael Shepanski wrote: this extra class. I therefore wanted the library to calculate the result types based on what is selected, which never was a big deal, but required the auto keyword to be usable. That's why in sqlpp11 you don't have to write such classes. You declare what you want to be selected inside the expression. On the other hand, of course, it is a neat feature to be able to map directly to user-defined structs. Maybe I'll find a way to do that in sqlpp11, too :-)

...

Say I have two conditions, userWantsBlob and userWantsOther. I would want to have everything in one loop.

In sqlpp11 I could say something like:

auto query = dynamic_select(db, t.meta)...; if (userWantsBlob) query.add_column(t.blob); if (userWantsOther) query.add_column(t.other);

for (const auto& row : db(query)) { std::cout << t.meta << '\n'; if (userWantsBlob) std::cout << t.at("blob") << '\n'; if (userWantsOther) std::cout << t.at("other") << '\n'; };

How would you do that in quince?

Ah, no, quince's static typing forbids anything like that. If I wanted to provide it, that would be a matter of exposing type-unsafe code that I currently keep hidden.

You lose on type-safety in sqlpp11, too, when doing something like this, but in several of our projects, such dynamic parts are a hard requirement. And it was quite painful to learn that because the type-safety was one of the main drivers for developing the library in the first place. It's been quite an interesting comparison/discussion so far. I think we shed some light on differences and similarities of quince and sqlpp11. I certainly got a some ideas for improving sqlpp11 (e.g. table definitions and mapping to user-defined structs) and stuff to think about (e.g quince's way of composing queries). Cheers, Roland

On 2014-07-18 10:02, Michael Shepanski wrote:

On 18/07/2014 5:00 PM, Roland Bock wrote:

...

Thinking about this, I just realized why I might have such a hard time with those user-defined classes: In a earlier version (still in C++03), I used something similar to tuples to define the results of a select. That often led to inefficient code. Instead of writing a type that represented just the n columns they actually needed, people tended to use all columns all the time, just because they were too lazy to write this extra class.

Sometimes the burden of selecting just the needed part of the output is not so great in quince, where you can work in multi-column types. E.g.:

for (const address &addr: employees.select(employees->home_address))

might retrieve five columns at once. Ah, ok, that certainly helps a lot, if you know the combinations typically selected. Nice!

...

I therefore wanted the library to calculate the result types based on what is selected, which never was a big deal, but required the auto keyword to be usable.

That's why in sqlpp11 you don't have to write such classes. You declare what you want to be selected inside the expression.

On the other hand, of course, it is a neat feature to be able to map directly to user-defined structs. Maybe I'll find a way to do that in sqlpp11, too :-)

The collector class aspect is a bit of sugar that I added because it turned out to be easy. On the other hand, having user-defined structs as mapped types (as in the example above) was always a core feature in my mind. I guess I see the ascent from SQL syntax to C++ syntax, and the ascent from SQL types to C++ types, as two aspects of the same ascent.

...

You lose on type-safety in sqlpp11, too, when doing something like this, but in several of our projects, such dynamic parts are a hard requirement. And it was quite painful to learn that because the type-safety was one of the main drivers for developing the library in the first place.

Funnily enough, one of my pre-quince attempts at a DAL *did* provide something similar to an untyped version of a quince query. If you executed it you would get row objects, and you'd have to figure out how to use something a bit like mappers to extract values from them. That was all because I hadn't figured out how to use tuples to provide type-safety without table-based value types. That product even had a with() method, which was like a widening select (because my thoughts about compositionality weren't so firm then).

With quince, I thought the move to full static typing was pure forward progress -- but maybe not. I thought so too, but I had to introduce ways around it for more dynamic use cases. sqlpp11 provides full static typing wherever possible and several degrees of reduced type-safety, depending on what the user needs.

...

It's been quite an interesting comparison/discussion so far. I think we shed some light on differences and similarities of quince and sqlpp11. I certainly got a some ideas for improving sqlpp11 (e.g. table definitions and mapping to user-defined structs) and stuff to think about (e.g quince's way of composing queries).

Cheers, Roland

The thing that amazes me most is that we are such a small club. When I read things online about how ORM is "the Vietnam of computer science", and the way people seemed to have coined the phrase "impedance mismatch" specifically to mean "I'm sad because I can't write queries in my programming language", I wonder why everyone isn't attempting this.

I have seen/heard about several attempts in this direction but quince and sqlpp11 are the only ones that I know that are this advanced. My guess is that the basic ideas are easy, but it quickly becomes difficult after the 27th line of code or so. Also, live has changed with C++11. Writing such a library in C++03 is horrible at best. With more people using C++11, I assume that more libraries of this kind will appear. Cheers, Roland

On 21/07/2014 12:40 PM, Gavin Lambert wrote:

Speaking for myself, here, but nowadays I punt things like database access to C# code, where just about everything is trivial (there's libraries for everything, and even if you want to hand-roll the SQL you can make a basic bare-persistence ORM in less than a page of code).

Which isn't to say that I wouldn't be thrilled if there were a good C++ database library. It's just that historically there hasn't been (to my knowledge), and I haven't minded that all that much because I usually regard C++ as useful for high-performance code and integration with third-party native libraries, and C# as useful for everything else.

Now excuse me while I don my flame-retardant coat, for daring to post such things in a C++ mailing list. ;)

No flames from me: I don't know C# so I can't participate in any C# vs. C++ thing. It sounds like you use C# for the bulk of your programming, and only break out into C++ for special purposes, so it's fair enough that you don't want to access a database from C++. Quince is designed for people who treat C++ as their home, and one of its purposes is to save them the trouble of going outdoors. I dare say that the same applies to sqlpp11. Quince also has another purpose (and this is also for people who treat C++ as their home): it allows them to rethink the boundary of what is "database access". By making it easy to use the DBMS's data-processing features, in C++ syntax and with C++ data types, it lets a C++ programmer continually renegotiate the division of labour between the DBMS and his application. It may be that a programmer already has a solution for what he calls "database access" today, but maybe the boundary of his "database access" would shift if he had a different tool. Regards, --- Michael

On 7/20/2014 10:40 PM, Gavin Lambert wrote:

On 18/07/2014 20:02, Michael Shepanski wrote:

...

The thing that amazes me most is that we are such a small club. When I read things online about how ORM is "the Vietnam of computer science", and the way people seemed to have coined the phrase "impedance mismatch" specifically to mean "I'm sad because I can't write queries in my programming language", I wonder why everyone isn't attempting this.

Speaking for myself, here, but nowadays I punt things like database access to C# code, where just about everything is trivial (there's libraries for everything, and even if you want to hand-roll the SQL you can make a basic bare-persistence ORM in less than a page of code).

Which isn't to say that I wouldn't be thrilled if there were a good C++ database library. It's just that historically there hasn't been (to my knowledge), and I haven't minded that all that much because I usually regard C++ as useful for high-performance code and integration with third-party native libraries, and C# as useful for everything else.

Now excuse me while I don my flame-retardant coat, for daring to post such things in a C++ mailing list. ;)

I have written database components in C++, C#, Python, and even Object Pascal in my career. Depending on the other database tools/components you use it can be easy or hard to create code in any of those languages and C#/.net hardly has a monopoly on database programming. I would welcome a good cross-platform C++ database library using sql, or even one based on an object-oriented database instead. I am glad to see C++ programmers working on a cross-platform C++ database library.

[Please do not mail me a copy of your followup] Michael Shepanski <mps@optusnet.com.au> spake the secret code <53C8D486.4090006@optusnet.com.au> thusly:

The thing that amazes me most is that we are such a small club. When I read things online about how ORM is "the Vietnam of computer science", and the way people seemed to have coined the phrase "impedance mismatch" specifically to mean "I'm sad because I can't write queries in my programming language", I wonder why everyone isn't attempting this.

They've been doing it in .NET for quite a while with LINQ: <http://msdn.microsoft.com/en-us/library/bb397926.aspx> ...but it required changes to the .NET runtime and the language syntax; it wasn't provided as an EDSL. -- "The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline> The Computer Graphics Museum <http://computergraphicsmuseum.org> The Terminals Wiki <http://terminals.classiccmp.org> Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>