The recent discussions about a logging library have been wonderful, and they demonstrate the strengths of the boost approach. These discussions have also demonstrated that logging is a lot more complex than many of us would have probably anticipated. It seems that a) everyone wants logging and that b) no one can agree on what it is. I think that a lot of this stems from the fact that logging encompasses many competing facets (e.g. thread-safety v. performance v. macros: evil or really evil? v. the kitchen sink) and these are, naturally, difficult to completely grok and balance. So, what I'm proposing is that we step back and do some sort of requirements analysis. I don't mean anything terribly formal, but rather some place where we can try to centralize our ideas on what a logging library comprises. Trying to use the list archives to keep track of every variation of every aspect of logging is error-prone and frustrating, but something like a wiki would make it much easier to present the totality of everyone's input. As we collect and organize our thoughts, I'm hoping that a less murky picture of what we should do will emerge. What are the orthogonal aspects of logging, and which are intertwined? Are there different paradigms of logging that we should treat as distinct? What are the apparent tradeoffs of different concerns, and are there some concerns that appear in conflict but which really aren't? My sense is that, as we really clarify what all of the concerns are, we'll have a better shared sense of what everyone's looking for. We'll have a sort of lingua franca or common starting point from which to develop boost logging. Jean Daniel's recent work on organizing input from the Torjo review is a brilliant start to this kind of effort. So, these are just my thoughts on the issue. I would *love* to see a logging library in boost, and it always frustrates me (and, I imagine, others) that we can't seem to agree on often the simplest things. Maybe I'm way off target here, but I think a little high-level organization could yield big dividends in this case. -- Austin Bingham Signal & Information Sciences Laboratory Applied Research Laboratories, University of Texas at Austin 10000 Burnet Rd., Austin, TX 78758 email: abingham@arlut.utexas.edu cell: (512) 799-2444 office: (512) 835-3832
Austin Bingham wrote:
The recent discussions about a logging library have been wonderful, and they demonstrate the strengths of the boost approach. These discussions have also demonstrated that logging is a lot more complex than many of us would have probably anticipated. It seems that a) everyone wants logging and that b) no one can agree on what it is. I think that a lot of this stems from the fact that logging encompasses many competing facets (e.g. thread-safety v. performance v. macros: evil or really evil? v. the kitchen sink) and these are, naturally, difficult to completely grok and balance.
So, what I'm proposing is that we step back and do some sort of requirements analysis. I don't mean anything terribly formal, but rather some place where we can try to centralize our ideas on what a logging library comprises. Trying to use the list archives to keep track of every variation of every aspect of logging is error-prone and frustrating, but something like a wiki would make it much easier to present the totality of everyone's input.
Yes, I think it's a good idea. I don't know if a wiki is the best solution. Anyway, I have just created one: http://boost-logging.wikispaces.com/ Tomorrow, I'll add the different requirements I read from previous post. Feel free to start now!
As we collect and organize our thoughts, I'm hoping that a less murky picture of what we should do will emerge. What are the orthogonal aspects of logging, and which are intertwined? Are there different paradigms of logging that we should treat as distinct? What are the apparent tradeoffs of different concerns, and are there some concerns that appear in conflict but which really aren't?
My sense is that, as we really clarify what all of the concerns are, we'll have a better shared sense of what everyone's looking for. We'll have a sort of lingua franca or common starting point from which to develop boost logging. Jean Daniel's recent work on organizing input from the Torjo review is a brilliant start to this kind of effort.
A brilliant start would have been to come with the library written. It's a pity John did not push it to the end, but after reading Gennadiy's review, I can understand why.
So, these are just my thoughts on the issue. I would *love* to see a logging library in boost, and it always frustrates me (and, I imagine, others) that we can't seem to agree on often the simplest things. Maybe I'm way off target here, but I think a little high-level organization could yield big dividends in this case.
Yes. What do you think if some people interested in the definition of such library get together and form a group of interest? So the library do not depend from the motivation of an individual that could run out of gaz under the constant flow of Boost member's constructive but yet sometime discouraging critics. The team would write those requirements, propose them to the community and defend them in order to push this library into boost. I think this would require some boost authoritative figure to lead this effort. Anyone ? JD
JD wrote:
Yes, I think it's a good idea. I don't know if a wiki is the best solution. Anyway, I have just created one: http://boost-logging.wikispaces.com/
I might suggest we simply use the Boost Wiki instead of forking yet another site... www.boost.org/wiki Jeff
I might suggest we simply use the Boost Wiki instead of forking yet another site...
www.boost.org/wiki
Jeff
Yep, my mistake. http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?Boost.Logging Thanks Jeff for pointing this out. JD
Hello JD, Monday, April 2, 2007, 10:36:28 PM, you wrote:
Austin Bingham wrote:
The recent discussions about a logging library have been wonderful, and they demonstrate the strengths of the boost approach. These discussions have also demonstrated that logging is a lot more complex than many of us would have probably anticipated. It seems that a) everyone wants logging and that b) no one can agree on what it is. I think that a lot of this stems from the fact that logging encompasses many competing facets (e.g. thread-safety v. performance v. macros: evil or really evil? v. the kitchen sink) and these are, naturally, difficult to completely grok and balance.
So, what I'm proposing is that we step back and do some sort of requirements analysis. I don't mean anything terribly formal, but rather some place where we can try to centralize our ideas on what a logging library comprises. Trying to use the list archives to keep track of every variation of every aspect of logging is error-prone and frustrating, but something like a wiki would make it much easier to present the totality of everyone's input.
Yes, I think it's a good idea. I don't know if a wiki is the best solution. Anyway, I have just created one: http://boost-logging.wikispaces.com/
Tomorrow, I'll add the different requirements I read from previous post. Feel free to start now!
Great! I think, it's the only way to come to a viable solution. [snip]
So, these are just my thoughts on the issue. I would *love* to see a logging library in boost, and it always frustrates me (and, I imagine, others) that we can't seem to agree on often the simplest things. Maybe I'm way off target here, but I think a little high-level organization could yield big dividends in this case.
Yes. What do you think if some people interested in the definition of such library get together and form a group of interest? So the library do not depend from the motivation of an individual that could run out of gaz under the constant flow of Boost member's constructive but yet sometime discouraging critics. The team would write those requirements, propose them to the community and defend them in order to push this library into boost. I think this would require some boost authoritative figure to lead this effort. Anyone ?
Since I'm interested in having such a library in Boost, I'd be happy to participate. -- Best regards, Andrey mailto:andysem@mail.ru
Since I'm interested in having such a library in Boost, I'd be happy to participate.
It's what you did and you did well! Thank you! - Nonetheless, I see a very interesting contribution from Michael Lacher. I'm not sure the mailing list is the right place to discuss them, but I don't see any discussion page on the wiki. Concerning req id 3: Michael, I don't really get this statement. I think it's really to the library user (programmer) to decide which log shall be present or not. He's in charge of filtering the logs following the compilation configuration (debug or release). I don't think the library shall manage this by itself. Or maybe I did not get your point at all! General Thoughts, third bullet: You say: "Logging: print log messages useful for the program user. They are needed to understand unexpected conditions during usage like: "file xyz not found", "directory baz is not readable". I can't disagree more with this. IMO, logging is not at all a way to give information to the program user. In my mind, a logging library is intended only for debugging, journaling, auditing and performance measuring. Not a way to display error message or waiting message to the user. In my mind, these are totally different things. Please someone, give his POV on this, I think this is a major disagreement. - Besides, Caleb Epstein propose the definition of channel as: Multiple named channels each with configurable "threshold" (e.g. error, warning, debug, etc) Caleb, would you mind add a definition of channel on the wiki (http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?Boost.Logging) and maybe add some requirements on this? JD
On 2007-04-03, JD <jean.daniel.michaud@gmail.com> wrote:
Concerning req id 3: Michael, I don't really get this statement. I think it's really to the library user (programmer) to decide which log shall be present or not. He's in charge of filtering the logs following the compilation configuration (debug or release). I don't think the library shall manage this by itself. Or maybe I did not get your point at all!
I think the idea here is to eliminate certain kinds of information from object files/binaries. That is, if the user decides to eliminate the "top-secret debug log" from release builds, the generated binaries should give no indication that this log exists at all.
I can't disagree more with this. IMO, logging is not at all a way to give information to the program user. In my mind, a logging library is intended only for debugging, journaling, auditing and performance measuring. Not a way to display error message or waiting message to the user. In my mind, these are totally different things. Please someone, give his POV on this, I think this is a major disagreement.
You're probably going to find yourself in the minority here. In many instances, a log might be the only way communicate an error to the user; consider, say, a webserver or NFS lock daemon. More generally, I think that a generally useful logging system will be able to handle arbitrary logging and won't be concerned with domains or specific usage patterns. -- Austin Bingham Signal & Information Sciences Laboratory Applied Research Laboratories, University of Texas at Austin 10000 Burnet Rd., Austin, TX 78758 email: abingham@arlut.utexas.edu cell: (512) 799-2444 office: (512) 835-3832
Hello Austin, Tuesday, April 3, 2007, 10:55:49 PM, you wrote:
I can't disagree more with this. IMO, logging is not at all a way to give information to the program user. In my mind, a logging library is intended only for debugging, journaling, auditing and performance measuring. Not a way to display error message or waiting message to the user. In my mind, these are totally different things. Please someone, give his POV on this, I think this is a major disagreement.
You're probably going to find yourself in the minority here. In many instances, a log might be the only way communicate an error to the user; consider, say, a webserver or NFS lock daemon. More generally, I think that a generally useful logging system will be able to handle arbitrary logging and won't be concerned with domains or specific usage patterns.
I agree with Austin. I think, the "user" word is what confused you. The "user" here may be a system administrator or a technical support service specialist, for example. Such users have enough knowledge about the application that produced the log, so they are able to read it and figure out the problem. Though they are not interested in the internals of the application which only developer knows (i.e. they wouldn't understand what "i = 10, too bad :(" log record means). On the other hand, I believe such functionality may be easily achieved with proper filtering and providing separate sinks for these types of output. Michael, do you agree with me? -- Best regards, Andrey mailto:andysem@mail.ru
----- Original Message ----- From: "Andrey Semashev" <andysem@mail.ru> To: "Austin Bingham" <boost@lists.boost.org> Sent: Wednesday, April 04, 2007 7:49 AM Subject: Re: [boost] A meta-proposal for logging
Hello Austin,
Tuesday, April 3, 2007, 10:55:49 PM, you wrote:
I can't disagree more with this. IMO, logging is not at all a way to
give information to the program user. In my mind, a logging library is intended only for debugging, journaling, auditing and performance measuring. Not a way to display error message or waiting message to the user. In my mind, these are totally different things. Please someone, give his POV on this, I think this is a major disagreement.
Apologies if this is an intrusion. I have been following this thread but til now haven't had anything to offer. There seems to be some debate over what logging should properly be used for. Is it for end-users, developers or admin? My short answer to this is "one or more of the above", without forgetting that other interested parties will surely be discovered in the future (how about the legal department?). Some recent experience has directed me to the view that logging can usefully be broken into (at least) two parts. The first deals with construction, transport, storage and retrieval of logging information, the second deals with the logging requirements for the specific system being developed. The usefulness of this separation is that the former remains the same across many (a word full of hope) "specific systems". A recent implementation has resulted in a logging system that transports and stores a sequence of mixed data records (given the name a "ragged deque"). A set of record types is registered, instances are constructed, transported and stored. A minimal header with information like "received time", is prepended by the storage system. The order in which the records were received is preserved. It is difficult to say anything useful about the "specific system" requirements due to scale and the fact that they are simply not part of generic logging. Suffice to say that the new uses discovered for the log records continue to surprise me a year after the basic logging technology was ironed out, e.g. it is now delivering "alarm" type functionality. Cheers.
Andrey Semashev wrote:
Hello Austin,
Tuesday, April 3, 2007, 10:55:49 PM, you wrote:
I can't disagree more with this. IMO, logging is not at all a way to give information to the program user. In my mind, a logging library is intended only for debugging, journaling, auditing and performance measuring. Not a way to display error message or waiting message to the user. In my mind, these are totally different things. Please someone, give his POV on this, I think this is a major disagreement.
You're probably going to find yourself in the minority here. In many instances, a log might be the only way communicate an error to the user; consider, say, a webserver or NFS lock daemon. More generally, I think that a generally useful logging system will be able to handle arbitrary logging and won't be concerned with domains or specific usage patterns.
I agree with Austin. I think, the "user" word is what confused you. The "user" here may be a system administrator or a technical support service specialist, for example. Such users have enough knowledge about the application that produced the log, so they are able to read it and figure out the problem. Though they are not interested in the internals of the application which only developer knows (i.e. they wouldn't understand what "i = 10, too bad :(" log record means).
On the other hand, I believe such functionality may be easily achieved with proper filtering and providing separate sinks for these types of output. Michael, do you agree with me?
I agree with your general description, but i am not sure seperate sinks are a good idea (even if they provide a way to do it). The problem is that often temporal coherence and the sequence of log messages is important to determine the root of a problem. For example: the debug message that a socket read blocked unexpectely is much more informative if i can view it in the context of log statements telling me wich client connected/disconnected. If one lands in a debug file and the other in the access log then i will need to take special steps to view them in the correct order which is undesirable i think. The project I am currently working on has many seperate daemon programs (possibly running on different machines too), so i specifically added networked logging with a single sink, just to be able to view all the log messages in their natural order. Michael
Hello Michael, Wednesday, April 4, 2007, 5:31:29 PM, you wrote:
Andrey Semashev wrote:
I agree with Austin. I think, the "user" word is what confused you. The "user" here may be a system administrator or a technical support service specialist, for example. Such users have enough knowledge about the application that produced the log, so they are able to read it and figure out the problem. Though they are not interested in the internals of the application which only developer knows (i.e. they wouldn't understand what "i = 10, too bad :(" log record means).
On the other hand, I believe such functionality may be easily achieved with proper filtering and providing separate sinks for these types of output. Michael, do you agree with me?
I agree with your general description, but i am not sure seperate sinks are a good idea (even if they provide a way to do it). The problem is that often temporal coherence and the sequence of log messages is important to determine the root of a problem. For example: the debug message that a socket read blocked unexpectely is much more informative if i can view it in the context of log statements telling me wich client connected/disconnected. If one lands in a debug file and the other in the access log then i will need to take special steps to view them in the correct order which is undesirable i think.
The project I am currently working on has many seperate daemon programs (possibly running on different machines too), so i specifically added networked logging with a single sink, just to be able to view all the log messages in their natural order.
I see. I guess now I'm beginning to understand what channels are meant for. The channel appears to be yet another attribute which defines the "class" of the log record (i.e. an access log, debug trace, etc.). Seems to me we have to fix the set of mandatory attributes that every log record should always have and leave space for optional attributes (channels go here). I see these attributes as mandatory: - record number (increments for each record collected) - record date and time - thread name (if not set - thread id) - scope stack - severity level - message text As for optional attributes I think, it's better to leave them on user's behalf since it's too difficult to figure out the correct set of them and implement suitable interface to operate them in the unified and convenient way. But, surely, the library should leave an easy way to implement them. -- Best regards, Andrey mailto:andysem@mail.ru
I see. I guess now I'm beginning to understand what channels are meant for. The channel appears to be yet another attribute which defines the "class" of the log record (i.e. an access log, debug trace, etc.).
Actually yes, let's put it this way.
Seems to me we have to fix the set of mandatory attributes that every log record should always have and leave space for optional attributes (channels go here). I see these attributes as mandatory: - record number (increments for each record collected) - record date and time - thread name (if not set - thread id) - scope stack - severity level - message text
As for optional attributes I think, it's better to leave them on user's behalf since it's too difficult to figure out the correct set of them and implement suitable interface to operate them in the unified and convenient way. But, surely, the library should leave an easy way to implement them.
- filename - line number - process name If not mandatory, can at least be implemented by default in the library. JD
Hello JD, Thursday, April 5, 2007, 12:05:01 AM, you wrote:
I see. I guess now I'm beginning to understand what channels are meant for. The channel appears to be yet another attribute which defines the "class" of the log record (i.e. an access log, debug trace, etc.).
Actually yes, let's put it this way.
Seems to me we have to fix the set of mandatory attributes that every log record should always have and leave space for optional attributes (channels go here). I see these attributes as mandatory: - record number (increments for each record collected) - record date and time - thread name (if not set - thread id) - scope stack - severity level - message text
As for optional attributes I think, it's better to leave them on user's behalf since it's too difficult to figure out the correct set of them and implement suitable interface to operate them in the unified and convenient way. But, surely, the library should leave an easy way to implement them.
- filename - line number
These should be optional since one have to explicitly set them at each log record, which obviously may be omitted. We may support them in the library-provided macros.
- process name
This might be mandatory, although I'm not sure there's a portable way to get it without having the command line. -- Best regards, Andrey mailto:andysem@mail.ru
On 2007-04-04, Andrey Semashev <andysem@mail.ru> wrote:
I see. I guess now I'm beginning to understand what channels are meant for. The channel appears to be yet another attribute which defines the "class" of the log record (i.e. an access log, debug trace, etc.).
I think channels are often thought of as a routing mechanism for log messages. The message is passed to a channel which is connected to filters, sinks, etc. The notion is, I believe, isomorphic to some systems of tagging (in conjunction, again, with filters) that have been mentioned. So yes, it's just another attribute that used to determine how/where the message is formatted, written, and so forth.
Seems to me we have to fix the set of mandatory attributes that every log record should always have and leave space for optional attributes (channels go here). I see these attributes as mandatory: - record number (increments for each record collected) - record date and time - thread name (if not set - thread id) - scope stack - severity level - message text
I think that defining a mandatory set of attributes is going to rub a lot of users (me included) the wrong way. My sense is that we will be better served by a system that let's you dynamically (and easily!) associate attributes with a message. Moreover, it needs to be simple to do that on a global or per entity (e.g. channel, sink, message) basis. We could provide the kinds of formatters you describe with the library as they have obvious general utility. My main point, though, is that no mandated set of attributes will please everyone and may well be a huge point of contention.
As for optional attributes I think, it's better to leave them on user's behalf since it's too difficult to figure out the correct set of them and implement suitable interface to operate them in the unified and convenient way. But, surely, the library should leave an easy way to implement them.
I agree with this completely, except that (per above) it should apply to all attributes :) -- Austin Bingham Signal & Information Sciences Laboratory Applied Research Laboratories, University of Texas at Austin 10000 Burnet Rd., Austin, TX 78758 email: abingham@arlut.utexas.edu
Austin Bingham wrote: [snip]
I think that defining a mandatory set of attributes is going to rub a lot of users (me included) the wrong way. My sense is that we will be better served by a system that let's you dynamically (and easily!) associate attributes with a message. Moreover, it needs to be simple to do that on a global or per entity (e.g. channel, sink, message) basis. We could provide the kinds of formatters you describe with the library as they have obvious general utility.
My main point, though, is that no mandated set of attributes will please everyone and may well be a huge point of contention.
As for optional attributes I think, it's better to leave them on user's behalf since it's too difficult to figure out the correct set of them and implement suitable interface to operate them in the unified and convenient way. But, surely, the library should leave an easy way to implement them.
I agree with this completely, except that (per above) it should apply to all attributes :)
I totally agree to that. Except maybe that the actual log text might be mandatory or at least included by default. Michael
Hello Austin, Thursday, April 5, 2007, 12:27:39 AM, you wrote:
Seems to me we have to fix the set of mandatory attributes that every log record should always have and leave space for optional attributes (channels go here). I see these attributes as mandatory: - record number (increments for each record collected) - record date and time - thread name (if not set - thread id) - scope stack - severity level - message text
I think that defining a mandatory set of attributes is going to rub a lot of users (me included) the wrong way. My sense is that we will be better served by a system that let's you dynamically (and easily!) associate attributes with a message. Moreover, it needs to be simple to do that on a global or per entity (e.g. channel, sink, message) basis. We could provide the kinds of formatters you describe with the library as they have obvious general utility.
My main point, though, is that no mandated set of attributes will please everyone and may well be a huge point of contention.
But how would these attributes harm somebody? They are meant to be silently filled and updated by the library automatically, so they are, in fact, a guarantee to user, not the requirement. The another reason I've chosen these attributes is that they are either cumbersome to fill at user level or are constant all the time. See record number for example - a user should maintain the global counter and update it on every log record himself.
As for optional attributes I think, it's better to leave them on user's behalf since it's too difficult to figure out the correct set of them and implement suitable interface to operate them in the unified and convenient way. But, surely, the library should leave an easy way to implement them.
I agree with this completely, except that (per above) it should apply to all attributes :)
Well, there's no point to log an empty record. At least message text should be present in any record, don't you think so? Although, if there's a strong anticipation against a set of mandatory attributes, I would propose to introduce three attribute sets instead of two, as I was considering before: - Logger-associated set. Each record, collected from the logger has these. - Thread-associated set. All records made in the thread have them. - Global. All records throughout the app include attributes from this set. All three sets would be alterable in runtime. And the attribute concept should be extended too. For now I was thinking of it as of a value container. To make it suitable for things like record counter or record time it should be thought of like a value generator (which, of course, may "generate" the constant value to emulate value container case). But still, I am convinced that the message text should always be there. This fact in conjunction with its lazy construction, actually, makes it so special that it may be not an attribute at all. And as a side note, this will make it impossible to filter by the message text just because it won't be available at the time of filtering. Can we live with that? -- Best regards, Andrey mailto:andysem@mail.ru
Hi, On 4/5/07, Andrey Semashev <andysem@mail.ru> wrote:
The another reason I've chosen these attributes is that they are either cumbersome to fill at user level or are constant all the time. See record number for example - a user should maintain the global counter and update it on every log record himself.
You could say "add 'preset' counter attribute", thus avoiding that problem.
Well, there's no point to log an empty record. At least message text should be present in any record, don't you think so?
Maybe we should have a look at how Boost.Exception answered these questions, and compare it to what would be useful for a logging library.
But still, I am convinced that the message text should always be there. This fact in conjunction with its lazy construction, actually, makes it so special that it may be not an attribute at all. And as a side note, this will make it impossible to filter by the message text just because it won't be available at the time of filtering. Can we live with that?
I don't think we should live with that. Regards, Michael
If we aren't going to live with that, then I think we need two filtering points, because I, for one, don't want to pay the cost of formatting numbers etc just to throw the message away because I'm not viewing that message today. joe
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Michael Walter
I don't think we should live with that.
Regards, Michael _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 4/5/07, Greer, Joe <jgreer@doubletake.com> wrote:
If we aren't going to live with that, then I think we need two filtering points, because I, for one, don't want to pay the cost of formatting numbers etc just to throw the message away because I'm not viewing that message today.
Why can't we format on-demand? Regards, Michael
joe
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Michael Walter
I don't think we should live with that.
Regards, Michael _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Michael Walter
On 4/5/07, Greer, Joe <jgreer@doubletake.com> wrote:
If we aren't going to live with that, then I think we need two filtering points, because I, for one, don't want to pay the cost of formatting numbers etc just to throw the message away because I'm not viewing
that
message today.
Why can't we format on-demand?
[Joe] Oh, we certainly can. Unless I misunderstand you, wouldn't you then be limited to filtering out text things? For example, if I wanted to see the log lines with a "blotto 3" in them, you would have some problems, wouldn't you? Now, don't get me wrong, personally I don't see the point of filtering that kind of thing at the log level. There are plenty of tools that can scan the resultant log files and pull that info out for me. I only need a level (TRACE, LOG, WHATEVER) and a keyword of some sort. Anything more than that, I am willing to do outside the app with grep or something. joe
Hi, On 4/6/07, Greer, Joe <jgreer@doubletake.com> wrote:
From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Michael Walter
If we aren't going to live with that, then I think we need two filtering points, because I, for one, don't want to pay the cost of formatting numbers etc just to throw the message away because I'm not viewing
On 4/5/07, Greer, Joe <jgreer@doubletake.com> wrote: that
message today.
Why can't we format on-demand?
[Joe] Oh, we certainly can. Unless I misunderstand you, wouldn't you then be limited to filtering out text things? For example, if I wanted to see the log lines with a "blotto 3" in them, you would have some problems, wouldn't you?
By formatting on-demand, we would create an attribute whenever it is used. That is, if (and only if) your filter requires the text attribute, it would be generated. Or do I misunderstand you concern? Regards, Michael
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Michael Walter By formatting on-demand, we would create an attribute whenever it is used. That is, if (and only if) your filter requires the text attribute, it would be generated. Or do I misunderstand you concern?
[Joe] No, you have my concern right. I am unlikely to use such a feature and want to be sure that the filter check is as quick as possible for the kinds of things I am likely to use. In other words, my goal is as close to zero overhead as possible if a message is not desired, but if I go in and set the registry/config option to turn the log message on, then I want to see it and at that point I don't care when the message is formatted to it's final form. Are you envisioning keeping the message as a tuple of data items and manipulators and then evaluating them as late as possible? joe
Hello Michael, Friday, April 6, 2007, 1:16:58 AM, you wrote:
On 4/5/07, Greer, Joe <jgreer@doubletake.com> wrote:
If we aren't going to live with that, then I think we need two filtering points, because I, for one, don't want to pay the cost of formatting numbers etc just to throw the message away because I'm not viewing that message today.
Why can't we format on-demand?
Hmm... I don't quite understand what do you mean by "on-demand". Could you explain? I see it something like that: BOOST_LOG(logger) << "The message text"; This would essentially expand to something like: if (logger.will_write_message()) logger.strm() << "The message text"; Here will_write_message would act as follows: 1. Construct the complete attribute set from the logger's, thread's and global attribute sets. 2. Check the complete attribute set against the global filters. If it does not pass, the record is discarded not even getting to any sinks, and will_write_message returns false. No formatting takes place. 3. If the global filters pass the record, it is then tested similarly against sinks' filters. The logging lib may remember which sinks reported they will accept the record in order to avoid checking the filters second time on the actual output. If there is at least one sink that will accept the message, will_write_message returns true. Otherwise it returns false. 4. If will_write_message returned true we then format the message and put it directly to the sinks that will accept it. Is this what you mean "on-demand" or do you see it differently? -- Best regards, Andrey mailto:andysem@mail.ru
Hi, What requirements are ultimately targeted by this thread? I have had a variety of software requirements that I would broadly label as "logging" but am unclear as to whether those requirements might be met by software arising here, e.g. * Testing-only log for troublesome function, removed prior to release Free-form text, i.e. fixed content interspersed with runtime values No time stamp needed, sequence significant All output directed to a named system file Each invocation of the function overwrites the system file * Capture of high-speed events, logging unexpected sequences Fixed binary format, a list of "machine state" and "detected event" pairs Each pair representing an event with no appropriate transition Time stamped Output written to a block of shared memory * Audit trail of calls made to a set of stored procedures XML format, incl time, procedure name and parameter values Output directed to a series of files with names reflecting a connected user and session id * Audit trail for customer support and system adminstration Extended log file format from W3C (WD-logfile-960323) Output sent asynchronously to an in-process queue Forwarded from the in-process queue to a configured network service Network service directs received information to a set of files managed by an in-house library A dedicated viewing and filtering tool Export of selected time slices to customers for analysis with 3rd party tools Is Boost.Log interested in such circumstances? Thanks.
Hello Scott, Friday, April 6, 2007, 2:08:58 AM, you wrote:
Hi,
What requirements are ultimately targeted by this thread? I have had a variety of software requirements that I would broadly label as "logging" but am unclear as to whether those requirements might be met by software arising here, e.g.
* Testing-only log for troublesome function, removed prior to release Free-form text, i.e. fixed content interspersed with runtime values No time stamp needed, sequence significant All output directed to a named system file Each invocation of the function overwrites the system file * Capture of high-speed events, logging unexpected sequences Fixed binary format, a list of "machine state" and "detected event" pairs Each pair representing an event with no appropriate transition Time stamped Output written to a block of shared memory * Audit trail of calls made to a set of stored procedures XML format, incl time, procedure name and parameter values Output directed to a series of files with names reflecting a connected user and session id * Audit trail for customer support and system adminstration Extended log file format from W3C (WD-logfile-960323) Output sent asynchronously to an in-process queue Forwarded from the in-process queue to a configured network service Network service directs received information to a set of files managed by an in-house library A dedicated viewing and filtering tool Export of selected time slices to customers for analysis with 3rd party tools
Is Boost.Log interested in such circumstances?
Thanks a lot for your input in describing your use cases. Currently the discussion is about the collecting and storing part of the library, no external log viewing, filtering or extracting tools are considered for now. Although, I think, some basic tool for reading, e.g. binary logs may be provided as an example. As for the discussed part of the lib, I'd appreciate if you describe the following aspects: - The common ways of collecting data from the application. Loggers? Channels? Maybe some other sources of logging records? - The ways of storing logs (don't get this literally as "storing" somewhere - this may also include things like redirecting it to some other subsystems, on-the-fly analysis and other). What common formats of stored logs do you use? Do you use log splitting by size or time? - What common types of attributes do you use? - Do you involve log filtering? - Maybe some fresh ideas or proposals that were not highlighted in the thread? All these ideas and requirements may be noted in the Wiki: http://tinyurl.com/2fndfg -- Best regards, Andrey mailto:andysem@mail.ru
On 2007-04-05, Scott Woods <scottw@qbik.com> wrote:
Hi,
What requirements are ultimately targeted by this thread? I have had a variety of software requirements that I would broadly label as "logging" but am unclear as to whether those requirements might be met by software arising here, e.g. . . .
I started the thread, at least in part, in hopes of determining just what people were looking for :) As you point out, there seems to be a lot of ground covered by the domain broadly referred to as "logging", and it may turn out that no single implementation will ever suffice. If we can determine that, then we've really learned something important! So, to answer your question, the goal of this thread is to determine what we mean by logging. It's veered into some discussions of implementation and so forth, but I think that's important. I honestly think that this kind of open, preliminary discussion is the only way we'll ever come to grips with an adequate solution (or set thereof). So, given what I just said, your post is extremely helpful in that it lays out several disparate modes of logging. Thanks :) -- Austin Bingham Signal & Information Sciences Laboratory Applied Research Laboratories, University of Texas at Austin 10000 Burnet Rd., Austin, TX 78758 email: abingham@arlut.utexas.edu cell: (512) 799-2444 office: (512) 835-3832
Hello, I've been following this discussion for a while.
My main point, though, is that no mandated set of attributes will please everyone and may well be a huge point of contention.
But how would these attributes harm somebody? They are meant to be silently filled and updated by the library automatically, so they are, in fact, a guarantee to user, not the requirement.
If I just wanna log a timestamp and some text for the admin of my webserver to read, why would I want things like thread name and scope stack in my beeing updated and filled in all the time?
The another reason I've chosen these attributes is that they are either cumbersome to fill at user level or are constant all the time. See record number for example - a user should maintain the global counter and update it on every log record himself.
They could still be provided by the library as default values for, say, debug type logging, or easily be added by the user if desired.
Well, there's no point to log an empty record. At least message text should be present in any record, don't you think so?
I don't think so. How about I define an new attribute for my logger which is some object that I want to convert to text and format lazily? I think we should leave it as general and extensible as possible, without loosing simplicity.
Although, if there's a strong anticipation against a set of mandatory attributes, I would propose to introduce three attribute sets instead of two, as I was considering before: - Logger-associated set. Each record, collected from the logger has these. - Thread-associated set. All records made in the thread have them. - Global. All records throughout the app include attributes from this set.
All three sets would be alterable in runtime.
Agreed, but should it be to have these or should they be mandatory? Again I think it should be easy to use such sets, but there should be no mandatory list. Maybe I want to define my own set, say for a program module or something. Or are we getting to generic and complex here?
And the attribute concept should be extended too. For now I was thinking of it as of a value container. To make it suitable for things like record counter or record time it should be thought of like a value generator (which, of course, may "generate" the constant value to emulate value container case).
Sounds good.
But still, I am convinced that the message text should always be there. This fact in conjunction with its lazy construction, actually, makes it so special that it may be not an attribute at all. And as a side note, this will make it impossible to filter by the message text just because it won't be available at the time of filtering. Can we live with that?
I think we shouldn't. See above. Filtering by text should be possible to implement, but IMO not be directly provided by the library. Just some thoughts. Klaus
Hello Niko, Friday, April 6, 2007, 2:12:41 PM, you wrote:
Hello,
If I just wanna log a timestamp and some text for the admin of my webserver to read, why would I want things like thread name and scope stack in my beeing updated and filled in all the time?
Ok, I agree we should provide a global attribute set.
Well, there's no point to log an empty record. At least message text should be present in any record, don't you think so?
I don't think so. How about I define an new attribute for my logger which is some object that I want to convert to text and format lazily?
I don't see how is it different from message text. You just output your object to the logger stream and see it in the log.
I think we should leave it as general and extensible as possible, without loosing simplicity.
Agreed.
Although, if there's a strong anticipation against a set of mandatory attributes, I would propose to introduce three attribute sets instead of two, as I was considering before: - Logger-associated set. Each record, collected from the logger has these. - Thread-associated set. All records made in the thread have them. - Global. All records throughout the app include attributes from this set.
All three sets would be alterable in runtime.
Agreed, but should it be to have these or should they be mandatory?
What do you mean? The attribute set may be empty, so every attribute is optional (aside message text, which is under discussion).
Again I think it should be easy to use such sets, but there should be no mandatory list. Maybe I want to define my own set, say for a program module or something. Or are we getting to generic and complex here?
I think, there will be no restriction on constructing attributes and storing them in STL-compatible containers. Although the library itself will use only those three listed above, so user will have to add his attributes to one of these if he actually wants to log them.
But still, I am convinced that the message text should always be there. This fact in conjunction with its lazy construction, actually, makes it so special that it may be not an attribute at all. And as a side note, this will make it impossible to filter by the message text just because it won't be available at the time of filtering. Can we live with that?
I think we shouldn't. See above. Filtering by text should be possible to implement, but IMO not be directly provided by the library.
Hm. Let's put in this way. It may be possible but not though the regular filtering mechanism. A sink may actually decide wether to write the message to the output or not after the regular filtering was applied and formatting was made. This makes the difference between the message text and other attributes even greater. Otherwise I don't quite realize how to implement it in a user-friendly way. I wouldn't want to lambdify each logging record. -- Best regards, Andrey mailto:andysem@mail.ru
Hello Andrey On 06/04/2007 21:49, Andrey Semashev wrote:
Well, there's no point to log an empty record. At least message text should be present in any record, don't you think so?
I don't think so. How about I define an new attribute for my logger which is some object that I want to convert to text and format lazily?
I don't see how is it different from message text. You just output your object to the logger stream and see it in the log.
If it comes down to something like this if (logger.will_write_message()) logger.strm() << "The message text"; then forget I ever said anything. I had some other idea of how lazy logging would work in my head, but now realized it won't work.
I think, there will be no restriction on constructing attributes and storing them in STL-compatible containers. Although the library itself will use only those three listed above, so user will have to add his attributes to one of these if he actually wants to log them.
Ok.
But still, I am convinced that the message text should always be there. This fact in conjunction with its lazy construction, actually, makes it so special that it may be not an attribute at all. And as a side note, this will make it impossible to filter by the message text just because it won't be available at the time of filtering. Can we live with that?
I think we shouldn't. See above. Filtering by text should be possible to implement, but IMO not be directly provided by the library.
Hm. Let's put in this way. It may be possible but not though the regular filtering mechanism. A sink may actually decide wether to write the message to the output or not after the regular filtering was applied and formatting was made. This makes the difference between the message text and other attributes even greater.
Otherwise I don't quite realize how to implement it in a user-friendly way. I wouldn't want to lambdify each logging record.
Ok, I now realize the difference between the attributes and the message itself as you put it. Regards Niko
Hello, I have a question about the logging attributes concept. I'm thinking now about the implementation of attributes, filtering and sinks (in particular, formatting log messages in sinks) and I come to a conclusion that we have to mandate the set of types that will be used in the attributes<->logging library interface. The reason we have to do this is that the library should be able to be compiled separately from the user's code (i.e., in dll), which defines the attributes. It doesn't mean, though, that the user won't be able to define arbitrary types of attributes, but these types should be representable in one of a predefined set of types. For now I see the following types set being suitable for this purpose: - int - unsigned int - long long - unsigned long long - double - std::string or std::wstring, depending on the library configuration - boost::any. This won't be supported by the library's sinks out of box and is aimed for extensibility on the users' behalf. I know such lack of generalization is not welcome, but I can't figure out any other way to organize interaction between different dlls. Maybe there are other opinions? -- Best regards, Andrey mailto:andysem@mail.ru
Andrey Semashev wrote:
Hello,
I have a question about the logging attributes concept.
I'm thinking now about the implementation of attributes, filtering and sinks (in particular, formatting log messages in sinks) and I come to a conclusion that we have to mandate the set of types that will be used in the attributes<->logging library interface. The reason we have to do this is that the library should be able to be compiled separately from the user's code (i.e., in dll), which defines the attributes. It doesn't mean, though, that the user won't be able to define arbitrary types of attributes, but these types should be representable in one of a predefined set of types.
For now I see the following types set being suitable for this purpose: - int - unsigned int - long long - unsigned long long - double - std::string or std::wstring, depending on the library configuration - boost::any. This won't be supported by the library's sinks out of box and is aimed for extensibility on the users' behalf.
I know such lack of generalization is not welcome, but I can't figure out any other way to organize interaction between different dlls. Maybe there are other opinions?
I've been scratching my head about this. I'm currently trying a mechanism where you define a type (maskable) and all attributes need to be derived from that type. This way the filter can then apply a member from the maskable interface to determine if the message can succeed with this particular attribute.... Also there would be a member to convert to_string. So you could define a type: class line_number : public maskable { int l; ... }; then define your filter class my_filt : public filter { bool pass(line_number& line) { return line.l > my_threshold; } }; This would necessitate a constraint (every type you use for an attribute must derive from maskable) but then that's part of the framework that I cant see how to avoid.
Hello Gareth, Monday, April 9, 2007, 1:30:34 AM, you wrote:
Andrey Semashev wrote:
Hello,
I have a question about the logging attributes concept.
I'm thinking now about the implementation of attributes, filtering and sinks (in particular, formatting log messages in sinks) and I come to a conclusion that we have to mandate the set of types that will be used in the attributes<->logging library interface. The reason we have to do this is that the library should be able to be compiled separately from the user's code (i.e., in dll), which defines the attributes. It doesn't mean, though, that the user won't be able to define arbitrary types of attributes, but these types should be representable in one of a predefined set of types.
For now I see the following types set being suitable for this purpose: - int - unsigned int - long long - unsigned long long - double - std::string or std::wstring, depending on the library configuration - boost::any. This won't be supported by the library's sinks out of box and is aimed for extensibility on the users' behalf.
I know such lack of generalization is not welcome, but I can't figure out any other way to organize interaction between different dlls. Maybe there are other opinions?
I've been scratching my head about this. I'm currently trying a mechanism where you define a type (maskable) and all attributes need to be derived from that type. This way the filter can then apply a member from the maskable interface to determine if the message can succeed with this particular attribute.... Also there would be a member to convert to_string.
So you could define a type:
class line_number : public maskable { int l; ... };
then define your filter
class my_filt : public filter { bool pass(line_number& line) { return line.l > my_threshold; } };
This would necessitate a constraint (every type you use for an attribute must derive from maskable) but then that's part of the framework that I cant see how to avoid.
The necessity to derive attributes from the common base class exists either way, but it doesn't solve the problem. You see, in your example the filter would get "maskable& line", not "line_number& line" as the logging library doesn't know about "line_number" class. This means that you will have to make some type dispatch to get the real attribute type, and I'm afraid this will be a performance killer. Other two problems here are: 1. I believe the "to_string" member function you mentioned (I guess, you meant to use it to format sinks' output) should not be a part of attribute. I should be a sink's implementation detail (and in fact, it may be customizable). For example, a date that is initially represented as time_t, may be differently formatted as string by text sinks and may be dumped into file as int by binary sinks. 2. I would like filters to be as much independent from attributes as possible. IMO, in real applications the filters will be generated in very different places (and maybe modules) than the attributes and often they will not even know real attribute's type. In your example, imagine the "my_filt" doesn't know that "line_number" exists. -- Best regards, Andrey mailto:andysem@mail.ru
Hi Andrey,
You see, in your example the filter would get "maskable& line", not "line_number& line" as the logging library doesn't know about "line_number" class. This means that you will have to make some type dispatch to get the real attribute type, and I'm afraid this will be a performance killer.
It could be but I don't think it necessarily has to. My solution would be to tightly couple the filter with the attributes that are to be filtered on the basis that if the user is adding the attributes then the use knows how best to filter based on them. It seems logical to me that if some one is writing a filter based upon a specific attribute then they must know the attributes type to a certain degree.
Other two problems here are: 1. I believe the "to_string" member function you mentioned (I guess, you meant to use it to format sinks' output) should not be a part of attribute. I should be a sink's implementation detail (and in fact, it may be customizable). For example, a date that is initially represented as time_t, may be differently formatted as string by text sinks and may be dumped into file as int by binary sinks.
Yes. Your example of time is a good one. I think you are absolutely right here that it is the formatting object that should decide how to represent the attribute. This implies (like with the filter) that the user who produced the formatting object will understand enough about the attribute's type to make this feasible.
2. I would like filters to be as much independent from attributes as possible. IMO, in real applications the filters will be generated in very different places (and maybe modules) than the attributes and often they will not even know real attribute's type. In your example, imagine the "my_filt" doesn't know that "line_number" exists.
I think it is inevitable that *some* attribute type information be known by both filter producers and formatters. I just can not see a way around this whilst at the same time allowing user defined attribute types. I don't think this would necessarily be burdensome. It means that attributes will have to be documented. In your example if 'my_filt' does not know that 'line_number' exists then it will not be able to filter based upon that attribute and there is no cure for that. However anyone creating a filter that would log only a certain range of line numbers (for example) will know of and be expecting the 'line_number' attribute to be present. Of course if the expected attribute is missing the filter should not break. I tend to think that some amount of coupling is inevitable, for the most part I would imagine attributes would be well known, frequently used types like time (time_t), line_number (int) etc. and would be provided as 'standard' in the library.
Hi Gareth and others, ----- Original Message ----- From: "Gareth Buxton" <gareth.buxton@googlemail.com> To: <boost@lists.boost.org> Sent: Monday, April 09, 2007 9:13 PM Subject: Re: [boost] A meta-proposal for logging
Hi Andrey,
You see, in your example the filter would get "maskable& line", not "line_number& line" as the logging library doesn't know about "line_number" class. This means that you will have to make some type dispatch to get the real attribute type, and I'm afraid this will be a performance killer.
What constitutes death? i.e. how do you know what number of additional CPU cycles would change the mind of a potential adopter. There is no real boundary. It is way too dependent on the circumstance.
It could be but I don't think it necessarily has to. My solution would be to tightly couple the filter with the attributes that are to be filtered on the basis that if the user is adding the attributes then the use knows how best to filter based on them.
<snip>
I tend to think that some amount of coupling is inevitable, for the most part I would imagine attributes would be well known, frequently used types like time (time_t), line_number (int) etc. and would be provided as 'standard' in the library.
I would have liked to respond to the individual points raised here but there are a lot of them. If there is something useful I can add it is most likely a brief summary of a logging system recently completed. I balanced many of the issues raised here. * Logging is a stream of mixed-type records or discriminated variants. * In application code, logging looks like; log_channel << "Reached here"; log_channel << file_IO_problem; log_channel << server_snapshot(); log_channel << stack_trace(); * The stream of objects is asynchronously queued, sent over a network connection, collated, time stamped and stored using a proprietary format and file management scheme. * A dedicated viewer allows entry of arbitrary filtering expressions that typically results in a window-per record type, i.e. a fixed set of attributes. * Users can navigate inside the filtered view by entering further search criteria or by clicking on an auto-generated time breakdown of the view. * The record types are "user defined". Registration of a type includes enough information for the viewer to display full attribute information. * The logging applications (i.e. the source of log records) communicate with the server using a formalized protocol. An obvious message (or signal) within this protocol is LOG (i.e. the connecting client is transferring a log record, an instance of a discriminated variant). * Other less obvious protocol messages involve the distribution of the registered types, i.e. the logging application publishes its type information to the logging server. * The logging server accumulates and merges all the type information from all the different sources. * The logging server persists its own "super-symbol-table" to reduce traffic and also to enable the display of type information when no logging applications are connected. This might be termed a "full implementation" of logging. Some high-end requirements have certainly dragged it a long distance from where it started. Some possibly useful lessons picked up along the way; * The software machinery that provides for creation, transport, queueing, storage, retrieval and viewing of logs is far more important than a particular set of attributes. * There is no ultimate set of attributes. The set of attributes on this project changed after almost every discussion and remains in the state "to be determined". * Many of the initial requirements were eventually shown to be silly. Even now it remains very difficult to convince interested parties of this, e.g. a live window of received log records is not useful. * There are huge benefits in creating a logging system that transports and stores generic blobs (mixed-type log records). All the significant related software is "logging application" independent and inherently "forward compatible". * The cost in software complexity and dependence on formalized communcations and storage mechanisms is non-trivial. * Given another opportunity and certainly in the case of a Boost.Log implementation I would probably swing away from the "published type info" approach and go for straight text. This is mainly due to the amenable nature of text (much easier to get a google search implemented) and general software simplicity. The costs include space resources (RAM consumed in log queues and disk space) and some loss of viewing+filtering performance. Cheers.
On Fri, 6 Apr 2007 23:49:25 +0400, Andrey Semashev wrote:
Hello Niko,
Friday, April 6, 2007, 2:12:41 PM, you wrote:
Well, there's no point to log an empty record. At least message text should be present in any record, don't you think so?
I don't think so. How about I define an new attribute for my logger which is some object that I want to convert to text and format lazily?
I don't see how is it different from message text. You just output your object to the logger stream and see it in the log.
If I may jump in here, I have worked on a system where we had the specific requirement Niko has come up with. We had an exception library which logged all exceptions thrown, along with the stack trace for the exception. However, the stack trace was only output if the minimum logging level was set to DEBUG1 or lower. As we had multiple streams of output, with differing logging levels on them, we couldn't just store the message text, as we needed to determine whether or not to output the stack trace on each stream. Spencer -- <<< Eagles may soar, but weasels don't get sucked into jet engines >>> 7:27am up 41 days 14:09, 20 users, load average: 0.01, 0.02, 0.00 Registered Linux User #232457 | LFS ID 11703
Hello Spencer, Tuesday, April 10, 2007, 10:31:29 AM, you wrote:
On Fri, 6 Apr 2007 23:49:25 +0400, Andrey Semashev wrote:
Hello Niko,
Friday, April 6, 2007, 2:12:41 PM, you wrote:
Well, there's no point to log an empty record. At least message text should be present in any record, don't you think so?
I don't think so. How about I define an new attribute for my logger which is some object that I want to convert to text and format lazily?
I don't see how is it different from message text. You just output your object to the logger stream and see it in the log.
If I may jump in here, I have worked on a system where we had the specific requirement Niko has come up with. We had an exception library which logged all exceptions thrown, along with the stack trace for the exception. However, the stack trace was only output if the minimum logging level was set to DEBUG1 or lower. As we had multiple streams of output, with differing logging levels on them, we couldn't just store the message text, as we needed to determine whether or not to output the stack trace on each stream.
You may output the stack trace as an attribute (in fact, such feature is already requested). Then nothing prevents you from having two sinks: the first that outputs everything on DEBUG1 or lower severity with the trace, and the second outputs everything without it. -- Best regards, Andrey mailto:andysem@mail.ru
I can't disagree more with this. IMO, logging is not at all a way to give information to the program user. In my mind, a logging library is intended only for debugging, journaling, auditing and performance measuring. Not a way to display error message or waiting message to the user. In my mind, these are totally different things. Please someone, give his POV on this, I think this is a major disagreement.
I think this establishes two distinct uses for logging. I am thinking about logging in terms of (say) an application like Apache web server. The *user* likes to have tidy logs generated to let them know who is accessing the server, how and when. These log files are formatted and available to be fed into other applications that can produce nice statistics and graphs. To me this is what I mean by 'logging'. What you are talking about is trace debugging. For this activity I think using the logger is perfectly valid. This is an area where you might make most legitimate use of macros. But to my mind, as far as 'logging' is concerned it is only a 'niche' usage. The problem is that many developers only really use logging for this purpose and this is all they want from a logging system. However there are many medium to large systems out there that utilise logging as a core function/side-effect of their activity. Now for programmers with the task of writing these business and enterprise class applications that expect reliable and accurate logs, capable of driving statistical analysis, billing engines and the like, a standard, flexible and extensible library would be of great benefit. So it is possible we have two tasks here. 1) A basic development/debugging tool 2) A fully competent systems logging library. Now my interest lies in the second of these though I also want to be able to do the first. It would be nice to think that the one could be derived from the other for consistency. After all as a developer why use two logging systems when you can use one?
Hello JD, Tuesday, April 3, 2007, 10:30:19 PM, you wrote: [snip]
- Besides, Caleb Epstein propose the definition of channel as: Multiple named channels each with configurable "threshold" (e.g. error, warning, debug, etc)
Caleb, would you mind add a definition of channel on the wiki (http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?Boost.Logging) and maybe add some requirements on this?
I'm not sure I got the idea correctly. Do you mean that the channel outputs the record only if some condition is satisfied (e.g. log level is high enough)? How is it different from sinks and filters? JD, may be requirement #5 could be corrected: - "The library shall be able to output to different sink". Sinks? Is this what "multiplexing" above means? - "Each sink shall be link to a particular formatting". Did you mean "shall be able to have a particular formatting"? PS: Once I have some more time, I'll add more requirements there. I hope, I'll manage to look through Gareth Buxton's implementation in the vault to see new features/solutions/approaches. One thing that isn't clear for me now is the way the library should be initialized. Obviously, in most industrial-sized use cases the initialization should be made from an external source (e.g. a file with settings, system registry, etc.), but in a small-sized solutions either no or minimum initialization may be done right from code. The problem here is that I see no unified way to init the lib the first way. For some cases a simple text file would be sufficient, for others the settings should be in XML, for third ones they should be obtained not from the file at all. Therefore for now I can only think of a direct initialization from code. Any other suggestions/opinions? -- Best regards, Andrey mailto:andysem@mail.ru
On 2007-04-03, Andrey Semashev <andysem@mail.ru> wrote:
One thing that isn't clear for me now is the way the library should be initialized. Obviously, in most industrial-sized use cases the initialization should be made from an external source (e.g. a file with settings, system registry, etc.), but in a small-sized solutions either no or minimum initialization may be done right from code. The problem here is that I see no unified way to init the lib the first way. For some cases a simple text file would be sufficient, for others the settings should be in XML, for third ones they should be obtained not from the file at all. Therefore for now I can only think of a direct initialization from code. Any other suggestions/opinions?
I tend to think that the notion of an "official" configuration mechanism is of secondary importance at best, and a red-herring at worst. In the end, the logging system will be configurable via its programmatic interface, and spending too much time thinking about some configuration text file format (or whatever) will only serve to constrain our thinking on the important stuff. Given some programmatic interface (say, the ability to define channels, filter levels, sinks, etc.), I can define any number of mappings from file formats to operations on the library's interface. So, I could have code that configures my logging based on registry information, some XML file, the current headline on slashdot, or any combination thereof, but that should have no impact on how we design the interface in the first place. My point, I guess, is that we should probably not get too wrapped up in that kind of detail right now. That said, I think any final version of a logging library should include at least an example of a configuration file system, one that meets some sort of "generally useful" criterium. It's something that people seem to want. -- Austin Bingham Signal & Information Sciences Laboratory Applied Research Laboratories, University of Texas at Austin 10000 Burnet Rd., Austin, TX 78758 email: abingham@arlut.utexas.edu cell: (512) 799-2444 office: (512) 835-3832
I'm not sure I got the idea correctly. Do you mean that the channel outputs the record only if some condition is satisfied (e.g. log level is high enough)? How is it different from sinks and filters?
I think a channel is a flow of log dealing with the same matter. The warning channel ? Maybe this is over design or useless. Or maybe this is interesting, I don't know. I was looking for precision from Caleb.
JD, may be requirement #5 could be corrected: - "The library shall be able to output to different sink". Sinks? Is this what "multiplexing" above means? - "Each sink shall be link to a particular formatting". Did you mean "shall be able to have a particular formatting"?
Fixed.
One thing that isn't clear for me now is the way the library should be initialized. Obviously, in most industrial-sized use cases the initialization should be made from an external source (e.g. a file with settings, system registry, etc.), but in a small-sized solutions either no or minimum initialization may be done right from code. The
Yes I agree. Simplicity.
problem here is that I see no unified way to init the lib the first way. For some cases a simple text file would be sufficient, for others the settings should be in XML, for third ones they should be obtained not from the file at all. Therefore for now I can only think of a direct initialization from code. Any other suggestions/opinions?
Do we want to manage windows registry? Count me against that. Is there some generic way to obtain configuration from a file? _Boost.program_options_ ? Boost.Serialization? Do we really want to deal with multiple file formats? I am not convinced. I think minimal init in the code is a requirement. Logging shall be a simple task. JD
Andrey Semashev wrote:
One thing that isn't clear for me now is the way the library should be initialized. Obviously, in most industrial-sized use cases the initialization should be made from an external source (e.g. a file with settings, system registry, etc.), but in a small-sized solutions either no or minimum initialization may be done right from code. The problem here is that I see no unified way to init the lib the first way. For some cases a simple text file would be sufficient, for others the settings should be in XML, for third ones they should be obtained not from the file at all. Therefore for now I can only think of a direct initialization from code. Any other suggestions/opinions?
I would think that any program sufficiently large to warrant configuration files (or another non-hardcoded configuration machanism) probably already has one in place. And probably the worst thing you can do to the person administrating the software is adding different configuration methods. So I think the library should offer an API that allows configuration, but leave the usage of this API up to the user. One thing i would advise though is the possiblity to change these paramters on the fly (which should be easy using an API). So that log levels could be increased or decreased based on what is going on for example. Michael
Hello Michael, Wednesday, April 4, 2007, 5:40:12 PM, you wrote: [snip]
One thing i would advise though is the possiblity to change these paramters on the fly (which should be easy using an API). So that log levels could be increased or decreased based on what is going on for example.
Yes, of course. In fact, I think most of the settings should be changeable in runtime, including adding/removing sinks and filters. -- Best regards, Andrey mailto:andysem@mail.ru
JD wrote:
Since I'm interested in having such a library in Boost, I'd be happy to participate.
It's what you did and you did well! Thank you!
- Nonetheless, I see a very interesting contribution from Michael Lacher. I'm not sure the mailing list is the right place to discuss them, but I don't see any discussion page on the wiki.
Concerning req id 3: Michael, I don't really get this statement. I think it's really to the library user (programmer) to decide which log shall be present or not. He's in charge of filtering the logs following the compilation configuration (debug or release). I don't think the library shall manage this by itself. Or maybe I did not get your point at all!
I agree that it is, and it is actually a rather trivial task to write macros to remove the offending log messages. I just think that it would be nice if a logging library would already come with convenience macros that have this in mind.
General Thoughts, third bullet: You say: "Logging: print log messages useful for the program user. They are needed to understand unexpected conditions during usage like: "file xyz not found", "directory baz is not readable".
I can't disagree more with this. IMO, logging is not at all a way to give information to the program user. In my mind, a logging library is intended only for debugging, journaling, auditing and performance measuring. Not a way to display error message or waiting message to the user. In my mind, these are totally different things. Please someone, give his POV on this, I think this is a major disagreement.
Maybe my examples are badly chosen. When i say logging i probably ment what you describe as journaling (like a web server access log). To stick with the example of a webserver: while debugging i might be interested in very fine grained debug messages, without having to sift through hundreds of lines of access log telling me when i connected and which ip i used over and over again. I cannot seem the channels you mention below, but if i understand correctly this is probably exactly what i meant: * Conceptually log messages are associated to one or more channels, and by modifying the channel threshold you can specify if you are more or less interested in messages of this kind. * Each sink can specify a filter/threshold to define how suitable/desired output of the given verbosity is for this specific sink. It would even be possible to filter/classify log messages in channels by their scope, but i think this is the point where the effort off configuring it all might outweigh the actual gain.
- Besides, Caleb Epstein propose the definition of channel as: Multiple named channels each with configurable "threshold" (e.g. error, warning, debug, etc)
Caleb, would you mind add a definition of channel on the wiki (http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?Boost.Logging) and maybe add some requirements on this?
JD
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
I think the upcoming hopefully proposable Boost.Channel library can play a role in designing a flexible logging library, see: http://channel.sourceforge.net/ Also see my post here: http://article.gmane.org/gmane.comp.lib.boost.devel/156805 Thanks Shams -- "JD" <jean.daniel.michaud@gmail.com> wrote in message news:euu6g3$o4$1@sea.gmane.org...
Since I'm interested in having such a library in Boost, I'd be happy to participate.
It's what you did and you did well! Thank you!
- Nonetheless, I see a very interesting contribution from Michael Lacher. I'm not sure the mailing list is the right place to discuss them, but I don't see any discussion page on the wiki.
Concerning req id 3: Michael, I don't really get this statement. I think it's really to the library user (programmer) to decide which log shall be present or not. He's in charge of filtering the logs following the compilation configuration (debug or release). I don't think the library shall manage this by itself. Or maybe I did not get your point at all!
General Thoughts, third bullet: You say: "Logging: print log messages useful for the program user. They are needed to understand unexpected conditions during usage like: "file xyz not found", "directory baz is not readable".
I can't disagree more with this. IMO, logging is not at all a way to give information to the program user. In my mind, a logging library is intended only for debugging, journaling, auditing and performance measuring. Not a way to display error message or waiting message to the user. In my mind, these are totally different things. Please someone, give his POV on this, I think this is a major disagreement.
- Besides, Caleb Epstein propose the definition of channel as: Multiple named channels each with configurable "threshold" (e.g. error, warning, debug, etc)
Caleb, would you mind add a definition of channel on the wiki (http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?Boost.Logging) and maybe add some requirements on this?
JD
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Shams wrote:
I think the upcoming hopefully proposable Boost.Channel library can play a role in designing a flexible logging library, see: http://channel.sourceforge.net/
Also see my post here: http://article.gmane.org/gmane.comp.lib.boost.devel/156805
Thanks Shams
Shams, Yes, I read your preceding post and mail the author but I still have no answer at this time. Regards, JD
on Mon Apr 02 2007, Austin Bingham <abingham-AT-arlut.utexas.edu> wrote:
The recent discussions about a logging library have been wonderful, and they demonstrate the strengths of the boost approach. These discussions have also demonstrated that logging is a lot more complex than many of us would have probably anticipated. It seems that a) everyone wants logging and that b) no one can agree on what it is.
The author of Pantheios (http://pantheios.sourceforge.net) seems to think he really does know what it is, and has found the "sweet spot." I have no opinion about whether he's right or not, but it would be a good idea for someone who's familiar with the recent Boost discussion to review his work before proceeding too much further. -- Dave Abrahams Boost Consulting www.boost-consulting.com Don't Miss BoostCon 2007! ==> http://www.boostcon.com
The author of Pantheios (http://pantheios.sourceforge.net) seems to think he really does know what it is, and has found the "sweet spot." I have no opinion about whether he's right or not, but it would be a good idea for someone who's familiar with the recent Boost discussion to review his work before proceeding too much further.
Thanks David for your advice. So here is some stuff I read from the documentation (.chm): - "Pantheios is written in C and C++, and presents a C API along with a C++ Application Layer API." Is it a problem? - "Pantheios is released under the BSD license" Is it a problem? - "Pantheios is dependent on the STLSoft libraries, version 1.9.1 beta 20, or later. STLSoft provides the following important features: -Compiler feature discrimination, standard library feature discrimination, memory utility classes, multi-threading classes and functions, various TL extension algorithms and iterator adaptors. -The application layer uses the string access shims defined by the STLSoft main project and several sub-projects to effect the genericity, 100% type-safety and zero-overhead when not logging. -The core completely avoids memory allocation when the total extent of a logged message is less than 2048 bytes by using the highly efficient stlsoft::auto_buffer component. -Several stock back ends use STLSoft components for their implementation." Is it a problem? - Wow!! : http://minilien.com/?JXY4zQCudk I am not an expert in design and in no position in rejecting any library or whatever, and I have actually not even look into the code but IMO, just as reading few pages of the doc, I think pantheios is not what we are looking for. I would love to be wrong and that someone says, "pantheios is what we were looking for all those years, let put it into boost". I doubt it... Please, someone more experienced have a quick look and tell me I'm wrong. Thanks. JD
Hello JD, Wednesday, April 4, 2007, 11:06:06 PM, you wrote:
The author of Pantheios (http://pantheios.sourceforge.net) seems to think he really does know what it is, and has found the "sweet spot." I have no opinion about whether he's right or not, but it would be a good idea for someone who's familiar with the recent Boost discussion to review his work before proceeding too much further.
Thanks David for your advice. So here is some stuff I read from the documentation (.chm):
[snip]
I am not an expert in design and in no position in rejecting any library or whatever, and I have actually not even look into the code but IMO, just as reading few pages of the doc, I think pantheios is not what we are looking for. I would love to be wrong and that someone says, "pantheios is what we were looking for all those years, let put it into boost". I doubt it...
IMHO, we're not talking about including it into boost, but we're looking for a proper design solutions and frequently requested useful features, aren't we? -- Best regards, Andrey mailto:andysem@mail.ru
Andrey Semashev wrote:
Hello JD,
IMHO, we're not talking about including it into boost, but we're looking for a proper design solutions and frequently requested useful features, aren't we?
Well I understood David's post as : Look at pantheios before wasting your time. Anyway, did you had time to review it Andrey? Ok what I see is: Features provided by pantheios: SysLog Output supports Win32 debugger, Win32 event log, COM Error Object Property of pantheios listed on its front page: *Atomicity - all parts of a statement are emitted (or not) to the back-end transports in a single action. *Efficiency *Extensibility *Filtered *Genericity (http://www.ddj.com/dept/cpp/184401689) *100% type-safety I think for the most part it has been discussed, at least partially on the precedent post. What's new here is _SysLog_. I don't think it has been discussed, even during John's library review. And, BTW: http://www.ddj.com/dept/opensource/184403745 John, come back! JD
Hello JD, Wednesday, April 4, 2007, 11:57:29 PM, you wrote:
Andrey Semashev wrote:
Hello JD,
IMHO, we're not talking about including it into boost, but we're looking for a proper design solutions and frequently requested useful features, aren't we?
Well I understood David's post as : Look at pantheios before wasting your time.
Anyway, did you had time to review it Andrey?
Not yet. I hope I'll have a chance to get to it on the weekend. -- Best regards, Andrey mailto:andysem@mail.ru
on Wed Apr 04 2007, Andrey Semashev <andysem-AT-mail.ru> wrote:
Hello JD,
Wednesday, April 4, 2007, 11:06:06 PM, you wrote:
The author of Pantheios (http://pantheios.sourceforge.net) seems to think he really does know what it is, and has found the "sweet spot." I have no opinion about whether he's right or not, but it would be a good idea for someone who's familiar with the recent Boost discussion to review his work before proceeding too much further.
Thanks David for your advice. So here is some stuff I read from the documentation (.chm):
[snip]
I am not an expert in design and in no position in rejecting any library or whatever, and I have actually not even look into the code but IMO, just as reading few pages of the doc, I think pantheios is not what we are looking for. I would love to be wrong and that someone says, "pantheios is what we were looking for all those years, let put it into boost". I doubt it...
IMHO, we're not talking about including it into boost, but we're looking for a proper design solutions and frequently requested useful features, aren't we?
Yes, that's right. If the design is really "perfect," we can certainly do our own implementation if need be. -- Dave Abrahams Boost Consulting www.boost-consulting.com Don't Miss BoostCon 2007! ==> http://www.boostcon.com
On 2007-04-04, David Abrahams <dave@boost-consulting.com> wrote:
IMHO, we're not talking about including it into boost, but we're looking for a proper design solutions and frequently requested useful features, aren't we?
Yes, that's right. If the design is really "perfect," we can certainly do our own implementation if need be.
The pantheios system certainly has some interesting aspects as well as some shortcomings, all of which are worth considering. Let me also start by saying that the pantheios documentation is lacking and I don't have time to read the code in depth; if I'm way off target on what I say here, someone please correct me! Pantheios goes to some lengths to ensure minimal copying and memory usage, and this is certainly an area that people will be interested in for any logging library. Likewise, it has a sort of half-lazy evaluation scheme for logging arguments. That is, an object won't be converted to a string for logging unless logging is enabled. However, a function producing that object (if any) will always be called when the logging statement is made. Pantheios uses a matrix of overloaded functions (as opposed to, say, streaming operators) to implement the logging statements. This matrix is dimensioned on log severity and the number of arguments to the statement. This overloading-on-argument-count strategy is well understood and used throughout boost. The notion of hardcoding severity levels into the function name is, IMO, suboptimal and likely to generate a lot of friction. It presupposes not only the type of attributes one might want for a log message, but it also presupposes the legal values of that type. It's likely that the design described here was directed to a large degree by the goal of having a C API along with the C++. Pantheios has a system for routing log messages to different sinks. This is implemented as the interplay between what they call the front-end and back-end layers, and it appears to be an extensible system to some degree. That is, they provide the ability to send output to different destinations (debug windows, files, etc.), and other destinations can be coded to work with the API. Pantheios allows the logging of arbitrary type through the use of what they call shims. A shim appears to be nothing more than a facade pattern using overloaded function resolution to adapt types to a particular API. They seem to put a lot of stock in the technique, though I tend to think that it might be unwieldy in practice. Anyhow, this is really just the briefest of reviews of pantheios. Others should probably check it out as well, but I think that it's pretty far off the mark of what the general boost population will be looking for. We might get some good ideas from it, but it needs some serious extending. Of course, some significantly better documentation would be a good step in the right direction. -- Austin Bingham Signal & Information Sciences Laboratory Applied Research Laboratories, University of Texas at Austin 10000 Burnet Rd., Austin, TX 78758 email: abingham@arlut.utexas.edu
Hello Austin, Thursday, April 5, 2007, 5:38:14 PM, you wrote:
On 2007-04-04, David Abrahams <dave@boost-consulting.com> wrote:
IMHO, we're not talking about including it into boost, but we're looking for a proper design solutions and frequently requested useful features, aren't we?
Yes, that's right. If the design is really "perfect," we can certainly do our own implementation if need be.
The pantheios system certainly has some interesting aspects as well as some shortcomings, all of which are worth considering. Let me also start by saying that the pantheios documentation is lacking and I don't have time to read the code in depth; if I'm way off target on what I say here, someone please correct me!
[snip]
Anyhow, this is really just the briefest of reviews of pantheios. Others should probably check it out as well, but I think that it's pretty far off the mark of what the general boost population will be looking for. We might get some good ideas from it, but it needs some serious extending. Of course, some significantly better documentation would be a good step in the right direction.
After examining the documentation I agree with your review conclusions. I got the impression that many design decisions were led by the intent to be C-compatible. Some of the decisions are left unclear for me (for example, the relation between the front-end and back-ends and the way it decides which one will get the message), which, I think, is because of lack of documentation. I found no attributes (or things like that) support which seems to be fundamental for many features requested in this thread and on the Wiki page. The filtering capabilities are limited to severity level check. What I liked about the library is the set of outputs available out-of-box. The performance is better in case of disabled logging for the ability to delay formatting. Although, there is no significant difference when the logging is on. To my mind, the Boost implementation should be more extensible. It may not provide as much sink types as pantheios does, but the library should provide means to implement it. The attribute support and more elaborate filtering is a must in order to support things like channels, call stacks and log origins. -- Best regards, Andrey mailto:andysem@mail.ru
JD wrote:
The author of Pantheios (http://pantheios.sourceforge.net) seems to think he really does know what it is, and has found the "sweet spot." I have no opinion about whether he's right or not, but it would be a good idea for someone who's familiar with the recent Boost discussion to review his work before proceeding too much further.
Thanks David for your advice. So here is some stuff I read from the documentation (.chm):
<snip>
I am not an expert in design and in no position in rejecting any library or whatever, and I have actually not even look into the code but IMO, just as reading few pages of the doc, I think pantheios is not what we are looking for. I would love to be wrong and that someone says, "pantheios is what we were looking for all those years, let put it into boost". I doubt it...
Please, someone more experienced have a quick look and tell me I'm wrong.
Thanks.
JD
I wouldn't say I am more experienced but I will say that I also looked at pantheios when it was first suggested. It seems to be unnecessarily complex and introduces yet another library which I do not want or need. I believe you will have an extremely hard time making a logging library that pleases everyone. You can please some of the people some of the time but you cant please all of the people all of the time. I hope we can come up with something that will be acceptable to enough people to make it into boost soon. RVD
Richard Day wrote: [snip]
<snip>
I am not an expert in design and in no position in rejecting any library or whatever, and I have actually not even look into the code but IMO, just as reading few pages of the doc, I think pantheios is not what we are looking for. I would love to be wrong and that someone says, "pantheios is what we were looking for all those years, let put it into boost". I doubt it...
[snip]
I believe you will have an extremely hard time making a logging library that pleases everyone. You can please some of the people some of the time but you cant please all of the people all of the time. I hope we can come up with something that will be acceptable to enough people to make it into boost soon.
The way I see it there are two different things you need to consider when talking about pleasing everyone: 1) What can the logging library actually do. This includes the filtering possibilities, amount/types of sinks already included, extensibility, ... I think its pretty straightforward how to actually implement channels, sinks, ... and therefore most logging libraries supporting these features will have a very similar design. 2) How to use the logging library. This includes the setting up of the logging, and the actual logging itself. The setup is only done once, so if it requires many API calls and can get somewhat complicated (if you use any special features) it doesn't really matter. But log calls themselves will actually number in the thousands. This is what i look for first when looking at a logging library: Can I write a log message that logs what i want, when i want it and in the format i want it in a way thats easy and still looks sane ? If this criteria is not fulfilled I would not use the library under any circumstance and I guess many users will think similar. As an example that was used earlier in this list: logger << lazy(lambda::var(x) + ":" + y, _level = mpl::int_<2>()); Imho it is neither immediately apparent what this actually does (maybe to boost and C++ gurus it is) nor would i want to write something like this every time i log something. Michael Lacher
Michael Lacher wrote:
The way I see it there are two different things you need to consider when talking about pleasing everyone:
1) What can the logging library actually do. This includes the filtering possibilities, amount/types of sinks already included, extensibility, ...
I think its pretty straightforward how to actually implement channels, sinks, ... and therefore most logging libraries supporting these features will have a very similar design.
2) How to use the logging library. This includes the setting up of the logging, and the actual logging itself. The setup is only done once, so if it requires many API calls and can get somewhat complicated (if you use any special features) it doesn't really matter. But log calls themselves will actually number in the thousands. This is what i look for first when looking at a logging library:
Can I write a log message that logs what i want, when i want it and in the format i want it in a way thats easy and still looks sane ?
If this criteria is not fulfilled I would not use the library under any circumstance and I guess many users will think similar. As an example that was used earlier in this list:
logger << lazy(lambda::var(x) + ":" + y, _level = mpl::int_<2>());
Imho it is neither immediately apparent what this actually does (maybe to boost and C++ gurus it is) nor would i want to write something like this every time i log something.
Michael Lacher
I agree completely. I have no idea what that does either. I could speculate but theres no point. My needs would be both stream and macro logging. I need macro's for turning it off for release builds for example. Stream for general logging purposes. As far as configuration goes I believe some minimal support should be included such as reading in a specified text file but nothing more extravagant. If people want XML/Registry etc support it should be easy to add though. I am completely against automatic configuration. It should definitely be code initialized. Simple examples. DBG_LOG( "Null pointer we are now dying a hard death" ); vs logger << CLASS_WARNING << "File not found : " << filename << endl; With two completely different in many respects requirements satisfying everyones needs is the issue. Richard Day
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Richard Day
Simple examples.
DBG_LOG( "Null pointer we are now dying a hard death" ); vs logger << CLASS_WARNING << "File not found : " << filename << endl;
With two completely different in many respects requirements satisfying everyones needs is the issue.
[Joe] I agree. In fact, in the system I currently use, my usage pattern is a hybrid. My log messages look something like: CDBG(TRACE, "network") << "Some network related message." << endl; Where the macro invokes some tests to see if that message is desired before evaluating the rest of it. i.e. it would expand to: If (!isLoggingDesired(TRACE,"network")) {} else logger Some other ramblings I have about the system I put together for use here at work are... We write services that run 24/7 and want the log messages continually available to be written, so I put together the following system. The system is channel centric. Each channel has associated with it zero or more appenders (sinks); a formatter which formats messages for the channel; and a mechanism for determining if a message is desired. The macro (briefly described above) gathers up information associated with the log message, tests to see if the message should see the light of day, and if it should then constructs the message and passes it to the channel to be formatted and dumped out. All pretty normal stuff. The things I wished we could do with it are: We have a format that gets associated with each log. The formatter is currently constructed using a stream-like syntax which looks something like: nsi::logging::CLogger::Instance()->GetFormatter() << L"[" << nsi::logging::Channel << L"](" << nsi::logging::ThreadId << L") " << nsi::logging::Time(L"%y%m%d %H:%M:%S") << L" " << nsi::logging::FunctionName << L": " << nsi::logging::Msg << nsi::logging::fmtendl; At the time, I thought this was pretty cool. The above basically puts out the name of the channel surrounded by square brackets, followed by the thread id surrounded in parenthesis, then a timestamp in strftime format, the name of the function containing the log message (__FUNCTION__ in vc8), then the message. There are other things that I made available to go in here, but this is what we generally used. This was very convenient because we then didn't have to specify the normal stuff for each log message and it made centralized control of the information we wanted for each log messages possible. What I found though is that it is hard to change that format via a configuration file without recompiling things. I really wish I had gone ahead and made a format string for this so I could just pass in a string like "[%Chnl][%TID] %y%m%d%H%M%S %FNC: %Msg", then this could have been changed in a configuration file. I've been thinking that it would probably be possible to set up a mechanism for registering a string token with the system and a functor to provide the formatting such that this could be changed by users of the logging system. I have also wanted to be able to have different information with different sinks. For example, to the OutputDebug log, I could get rid of the timestamp, but in a file I would want almost all available information since that is more likely to be used in a postmortem situation. The other conclusion I came to was that I probably would replace the channel concept with a more generalized keyword filter. That is, in the above situation, the text in the macro selected the channel to be used. What I found in our use was that the channel wasn't really used for anything except selecting which messages were to be displayed. We never, for example, wanted "network" messages going to one file and "fileio" messages going to a different one. That meant we had an awful lot of mechanism for what basically boiled down to a filter. Today, I would probably just treat the name as a keyword and filter based on the keywords that were selected and allow more than one logger to co-exist for the purpose of having different messages going to different sinks, though we never really needed that. Anyway, that is the experiences we had with the system I wrote. The thing I want to push for as a writer of services is the ability to easily change things while the system is running. I realize this would require some hooks that would need executed periodically, to see if things changed, but don't do anything to prevent this. joe
Greer, Joe wrote:
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Richard Day
Simple examples.
DBG_LOG( "Null pointer we are now dying a hard death" ); vs logger << CLASS_WARNING << "File not found : " << filename << endl;
With two completely different in many respects requirements satisfying everyones needs is the issue.
[Joe] I agree. In fact, in the system I currently use, my usage pattern is a hybrid. My log messages look something like:
CDBG(TRACE, "network") << "Some network related message." << endl;
Maybe optional log information (channel, type, loglevel, ...) could all be passed similar to manipulators in std::stream. This would keep the basic interface clean (just << operator) and extensible. There would be no need to anticipate numbers/types of function arguments. If people prefer a functional interface, then they could go for either #define MYLOG1(LEVEL,CHANNEL,MESSAGE) logger << setLevel(LEVEL) << setChannel(CHANNEL) << MESSAGE << endl MYLOG1(WARNING,"Debug","Caught Exception: " << e.what()); or #define MYLOG2(LEVEL,CHANNEL) logger << setLevel(LEVEL) << setChannel(CHANNEL) MYLOG2(WARNING,"Debug") << "Caught Exception: " << e.what() << endl; Of course a commonly used subset of such macros could be provided. Michael
Michael Lacher wrote:
Maybe optional log information (channel, type, loglevel, ...) could all be passed similar to manipulators in std::stream. This would keep the basic interface clean (just << operator) and extensible. There would be no need to anticipate numbers/types of function arguments.
If people prefer a functional interface, then they could go for either
#define MYLOG1(LEVEL,CHANNEL,MESSAGE) logger << setLevel(LEVEL) << setChannel(CHANNEL) << MESSAGE << endl
MYLOG1(WARNING,"Debug","Caught Exception: " << e.what());
or
#define MYLOG2(LEVEL,CHANNEL) logger << setLevel(LEVEL) << setChannel(CHANNEL)
MYLOG2(WARNING,"Debug") << "Caught Exception: " << e.what() << endl;
Of course a commonly used subset of such macros could be provided.
Michael
My thinking is that macros are really just for trace during debugging for the most part. Basically so you have a way to turn them off in a release build if/when desired. Stream statements are much nicer for regular logging. No need to know arguments ahead of time and setup macros for every possibility. Level/Class/Channel etc can be set easily using manipulators. Naturally we need functions for setting things like that as well as turning sinks off and on again etc. Best to have manipulators for these as well so you can do it directly in the stream statement if/when desired as well. I would rather write a single stream statement then write logger << "Something here"; logger.turnoffwhatever() logger << "rest of the statement"; logger.turnitonagain(); logger << "Something here" << turnoffwhatever << "rest of the statement" << turnonagain; Just my 2 cents. Richard Day
Richard Day wrote:
Michael Lacher wrote:
Maybe optional log information (channel, type, loglevel, ...) could all be passed similar to manipulators in std::stream. This would keep the basic interface clean (just << operator) and extensible. There would be no need to anticipate numbers/types of function arguments.
If people prefer a functional interface, then they could go for either
#define MYLOG1(LEVEL,CHANNEL,MESSAGE) logger << setLevel(LEVEL) << setChannel(CHANNEL) << MESSAGE << endl
MYLOG1(WARNING,"Debug","Caught Exception: " << e.what());
or
#define MYLOG2(LEVEL,CHANNEL) logger << setLevel(LEVEL) << setChannel(CHANNEL)
MYLOG2(WARNING,"Debug") << "Caught Exception: " << e.what() << endl;
Of course a commonly used subset of such macros could be provided.
Michael
My thinking is that macros are really just for trace during debugging for the most part. Basically so you have a way to turn them off in a release build if/when desired. Stream statements are much nicer for regular logging. No need to know arguments ahead of time and setup macros for every possibility.
Well, my idea was basically that stream statements are the "canonical" way of writing any log message, and that convenience macros ar eprovided to provide a functional style interface for common and quick cases. Depending on the lazy evaluation and the "no global object" requirement the thing you actually stream your message into (the variable 'logger' in the example) might actually be a macro evaluating to a stream object. Michael
Richard Day wrote:
As far as configuration goes I believe some minimal support should be included such as reading in a specified text file but nothing more extravagant. If people want XML/Registry etc support it should be easy to add though.
I would add a small interface to boost::program_options but nothing more. Configuration file is not really within the scope of a logging library imho, and since boost already has a library that specifically deals with configuration not using it would be a waste of effort i think. Michael
participants (14)
-
Andrey Semashev -
Austin Bingham -
David Abrahams -
Gareth Buxton -
Greer, Joe -
JD -
Jeff Garland -
Michael Lacher -
Michael Walter -
Niko Demmel -
Richard Day -
Scott Woods -
Shams -
Spencer Collyer