[Spirit] Retrieve informations out of tokens
Hi, I want to create a c++ syntax analysor (for creating auto-completion tools), and to help me, I want to use the boost.spirit library. In order to learn the basics of spirit, I executed this exercise. It is now working but I want to display the tokens' characters instead of counting tokens. The problem is I don't know the properties of the argument passed to my functor (the variable "t" in the code below). What is the type of this variable and what are the public method/attributes we can use on ? This is the code of my functor : struct counter{ counter() : j(0) {j = new int();} counter(const counter &model) : j(model.j) {} ~counter() {delete int;} template <typename Token> bool operator()(Token const& t){//I want to know the type of this variable t ++i; ++(*j); std::ostringstream oss; oss << "global : " << i << " membre : " << *j;//typeid(t).name(); printf("%s\n", oss.str().c_str()); return true; } int* j;}; The variable t will be relay to my functor by the following tokenize function : counter* foncteur = new counter();lex::tokenize(first, last, word_count_functor, *foncteur); I still have other questions : I read a consistent part of the documentation but I have not been able to define the type of regex used by boost.spirit. Which type is it ? What is the more appropriate function to use to analyse c++ synthax : tokenize or tokenize _and_parse ? You seems to describe tokenize_and_parse as a more powerful tool, but is it possible to define all the syntax of c++ using a parser ? Could I find an example of c++ syntax analysor using boost.spirit ? Any help would be appreciate,Cordialy,Guillaume Bersac237bis rue de Pessac33000 Bordeaux06.65.30.89.43
On 26.06.2013, at 15:22, ... ... wrote:
Hi,
I want to create a c++ syntax analysor (for creating auto-completion tools), and to help me, I want to use the boost.spirit library.
Are you sure you want to do that? C++ syntax is horrible and extremely context-dependent. If you want to create auto-completion tools, writing a full syntax analysis is mostly wasted time. I suggest you try Clang's libraries instead. Sebastian
Hi, I know it is going to be very difficult and I probably won't go to the end, but I am doing this mostly for fun. Anyway I am interested in the lexer library. Cordialy
Hi, A good starting point for writing a lexer is the conjure tutorial: https://github.com/boostorg/spirit/tree/master/example/qi/compiler_tutorial/... and of course epoch: http://code.google.com/p/epoch-language/source/browse/#hg/EpochCompiler/Lexe... Regarding tokenize vs tokenize_and_parse: the former executes the lexer on the input, and the latter also passes the iterators received from the lexer to the qi parse expression. You could have a look at clang since it is implemented as a recursive descent parser .. just for reference. In general I think it might be quite entertaining to do the nasty and ambiguous parts of C++ with qi's dynamic parsers. regards Andreas
On Jun 26, 2013, at 9:22 AM, "... ..."
I want to create a c++ syntax analysor (for creating auto-completion tools), and to help me, I want to use the boost.spirit library. In order to learn the basics of spirit, I executed this exercise. It is now working but I want to display the tokens' characters instead of counting tokens. The problem is I don't know the properties of the argument passed to my functor (the variable "t" in the code below). What is the type of this variable and what are the public method/attributes we can use on ?
If you want Boost.Spirit support, I suggest the Spirit mailing list. ___ Rob (Sent from my portable computation engine)
Hi,
From: robertstewart@comcast.net> If you want Boost.Spirit support, I suggest the Spirit mailing list.That's a good idea, and I already did it, and no one is answering. I investigated a lot since then and maybe you could help me : I use the function tokenize with a functor I created myself. The code of the functor :struct counter{ template <typename Token> bool operator()(Token const& t){//what is the type of the variable t ??? return true; }}; And the code of my tokenize function call :counter* foncteur = new counter();char const* first = str.c_str();//char const* is the type Iteratorchar const* last = &first[str.size()];lex::tokenize(first, last, word_count_functor, *foncteur); Now I am trying to figure out how to use the variable t, the token, relay to my functor.According to the source code, the type of the token is that :boost::spirit::lex::lexertl::token< Iterator, AttributeTypes, HasState, Idtype > The token has one member function which is value(). This member function return a value of type :typedef detail::token_value_type < iterpair_type, AttributeTypes >::type token_value_type iterpair_type being an alias of :typedef iterator_range<Iterator> iterpair_type So, to sum up the informations, the type of token.value() is : detail::token_value_type < iterator_range<Iterator>, AttributeTypes >::type Here is the definition of the template parameter of the token type Iterator The type of the iterator used to access the underlying character stream. AttributeTypes A mpl sequence containing the types of all required different token values to be supported by this token type. HasState A mpl::bool_ indicating, whether this token type should support lexer states. Idtype The type to use for the token id (defaults to std::size_t). To define the type of token.value, I need to know what's the value of the template parameter : Iterator and attribute type.I already figured out that char const* is, in this case, the Iterator attribute, now I need to know what's the type of Attribute type. Could you help me understand what's the type of AttributeTypes ? Then what are the member functions and attributes of this class so that I can use it ? What does namespace detail refer to ? What is a mpl sequence ?
Any help is welcomenCordialy,Guillaume Bersac237bis rue de Pessac33000 Bordeaux06.65.30.89.43
On Jun 26, 2013, at 7:47 PM, "... ..."
From: robertstewart@comcast.net> If you want Boost.Spirit support, I suggest the Spirit mailing list.That's a good idea, and I already did it, and no one is answering. I investigated a lot since then and maybe you could help me :
I've never used Lex, but maybe you'll follow my earlier advice. I'll even point you to information on getting support: http://boost-spirit.com/home/feedback-and-support/ ___ Rob (Sent from my portable computation engine)
Have you considered using clang directly? It is designed and built for
exactly the purpose of using it with auto completion tools, etc.
http://clang.llvm.org/
Cheers!
Andrew Hundt
On Wed, Jun 26, 2013 at 9:22 AM, ... ...
Hi,
I want to create a c++ syntax analysor (for creating auto-completion tools), and to help me, I want to use the boost.spirit library. In order to learn the basics of spirit, I executed this exercise. It is now working but I want to display the tokens' characters instead of counting tokens. The problem is I don't know the properties of the argument passed to my functor (the variable "t" in the code below). What is the type of this variable and what are the public method/attributes we can use on ? This is the code of my functor : struct counter{ counter() : j(0) {j = new int();} counter(const counter &model) : j(model.j) {} ~counter() {delete int;} template <typename Token> bool operator()(Token const& t){//I want to know the type of this variable t ++i; ++(*j); std::ostringstream oss; oss << "global : " << i << " membre : " << *j;//typeid(t).name(); printf("%s\n", oss.str().c_str()); return true; } int* j;}; The variable t will be relay to my functor by the following tokenize function : counter* foncteur = new counter();lex::tokenize(first, last, word_count_functor, *foncteur); I still have other questions : I read a consistent part of the documentation but I have not been able to define the type of regex used by boost.spirit. Which type is it ? What is the more appropriate function to use to analyse c++ synthax : tokenize or tokenize _and_parse ? You seems to describe tokenize_and_parse as a more powerful tool, but is it possible to define all the syntax of c++ using a parser ? Could I find an example of c++ syntax analysor using boost.spirit ? Any help would be appreciate,Cordialy,Guillaume Bersac237bis rue de Pessac33000 Bordeaux06.65.30.89.43
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Hi,
Have you considered using clang directly? It is designed and built for exactly the purpose of using it with auto completion tools, etc. Sure if I really need to implement an auto completion tools in one of my program, I would use clang since it seems like it has been tested a lot and is a reference, but the fun is to trying (probably not succeeding, but that's not a problem) to recreate it. But I spend a lot of time trying to understand this boost.spirit, without a lot of rewards, so I am asking this question : who this framework is made for ? For high graded computer scientist of the average programmer (I consider myself as a average programmer) ?
Cordialy,Guillaume Bersac237bis rue de Pessac33000 Bordeaux06.65.30.89.43
On Jun 27, 2013, at 3:46 AM, "... ..."
Have you considered using clang directly? It is designed and built for exactly the purpose of using it with auto completion tools, etc. Sure if I really need to implement an auto completion tools in one of my program, I would use clang since it seems like it has been tested a lot and is a reference, but the fun is to trying (probably not succeeding, but that's not a problem) to recreate it. But I spend a lot of time trying to understand this boost.spirit, without a lot of rewards, so I am asking this question : who this framework is made for ? For high graded computer scientist of the average programmer (I consider myself as a average programmer) ?
Wrapping your head around Spirit can be daunting. It combines expression templates, type traits, parsing concepts, etc. Whether that's more than you can grasp only you can say. However, this list is for discussions on the development and maintenance of Boost libraries. It is not for teaching or explaining C++ or Boost libraries. ___ Rob (Sent from my portable computation engine)
Hi,
However, this list is for discussions on the development and maintenance of Boost libraries. It is not for teaching or explaining C++ or Boost libraries.
Does it mean that all my post, relating to how to use Spirit were unrelated ? If so I am sorry. But is there a Boost support forum, for end user and beginners like me ? I can't find one. Cordialy,Guillaume Bersac237bis rue de Pessac33000 Bordeaux06.65.30.89.43
On Thu, Jun 27, 2013 at 01:03:33PM +0000, ... ... wrote:
Hi,
However, this list is for discussions on the development and maintenance of Boost libraries. It is not for teaching or explaining C++ or Boost libraries.
Does it mean that all my post, relating to how to use Spirit were unrelated ? If so I am sorry. But is there a Boost support forum, for end user and beginners like me ? I can't find one.
All end-user questions are off-topic on this list. There's a generic boost-users@ list for end-user discussion, and additionally there are project-specific lists. In the case of Boost.Spirit, there's the spirit-general@ list, as well as an IRC channel on freenode. As for "forums", the closest you get is resources like StackOverflow.com, but those are not designed around back-and-forth:ing like mailing lists and IRC is. Mailing lists still exist because they still work, more or less. -- Lars Viklund | zao@acc.umu.se
participants (6)
-
... ... -
Andreas Pokorny -
Andrew Hundt -
Lars Viklund -
Rob Stewart -
Sebastian Redl