"John Torjo" <john.lists@torjo.com> wrote in message news:41815A2D.3060005@torjo.com...

Johan Nilsson wrote:

...

Hi,

"John Torjo" <john.lists@torjo.com> wrote in message news:41812C6E.10903@torjo.com...

...

Dear boosters,

Finally, have had some time to update the logging lib. Get it from: http://www.torjo.com/ (of course, it comes packed with docs)

Are the docs available somewhere in single html and/or PDF format?

when you unpack the zip, they are in html format, in boost-root/libs/log/doc

I've seen that - just looking for a convenient format for printing out all of the documentation at once. // Johan

Daniel Frey <daniel.frey@aixigo.de> wrote:

...

logger("app.gui") << "something GUIsh"; Though I am not sure, if a compiler will be able to optimize checks for every operator<<() to a single check for the entire output expression.

Now replace "something GUIsh" with anExpensiveFunction(). John's code will not evaluate it when the logger is disabled. OTOH your code will always evaluate it, which is IMHO not acceptable for a logging library.

Well, that is true, though my logging needs has never required calling any expensive functions. -- Maxim Yegorushkin

Caleb Epstein <caleb.epstein@gmail.com> wrote:

All of the iostreams functions add up fast. Its a huge win to not have to evaluate them if you're not going to log the message. Its a must-have feature.

I agree. There is nothing faster than a single if statement. -- Maxim Yegorushkin

...

Would it make sense to use a shared_ptr<appender> instead of a functor for the appenders?

I think shared_ptr<appender> would be an unnecessary constraint. You always can put a shared_ptr<> inside the functor.

Yes - my thoughts exactly! Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -- v1.5 - tooltips at your fingertips (work for menus too!) + bitmap buttons (work for MessageBox too!) + tab dialogs, hyper links, lite html

At 09:50 AM 10/29/2004, Johan Nilsson wrote:

... - This opinion is probably not shared with others, but I'd prefer the library to be independent from the Boost Thread library. The dependency feels kind of the wrong way around.

Ouch! Johan's other comments are so positive, that I was starting to get excited about the library. And most of his negative comments are things I don't care a lot about or are issues that can probably be addressed later. But a dependency on any thread library is likely to throw cold water on my use. Here are the likely scenarios: * Light-duty uses in light-duty non-multithreaded programs, where introducing a threading dependency seems far too heavy-weight for the apps. You (actually, the docs) could convince me otherwise, but I'm guessing that for this kind of casual use in general you will scare off potential users with a threading dependency. * Heavy-duty industrial uses. I'm not adverse personally to a threading library dependency for these apps (which are sold as libraries); , but I'd have to get permission from managers who will be very concerned about use of a threading library clashing with customer's use of other threading libraries. Whether these concerns are valid or not is almost beside the point; it will be a hassle. These concerns can be overcome, but they would slow adoption of the logging library. Thus if you can eliminate them it would speed acceptance. Note that I'm not suggesting that all uses of the library have to eliminate the threading dependency, but that it would be nice if there was a basic subset of features that could be used without threading concerns. HTH, --Beman

On Sat, 30 Oct 2004 09:12:41 +0900, Darren Cook <darren@dcook.org> wrote:

...

...

- This opinion is probably not shared with others, but I'd prefer the library to be independent from the Boost Thread library. The dependency feels kind of the wrong way around. ... But a dependency on any thread library is likely to throw cold water on my use.

Same here.

There is no dependency on Boost.Threads unless BOOST_HAS_THREADS is defined. Is there a need for another option e.g. BOOST_LOG_HAS_THREADS that could be used to en/disable the dependency in *otherwise* multi-threaded code? -- Caleb Epstein caleb.epstein@gmail.com

Beman Dawes <bdawes <at> acm.org> writes:

But a dependency on any thread library is likely to throw cold water on my use. Here are the likely scenarios:

* Light-duty uses in light-duty non-multithreaded programs, where introducing a threading dependency seems far too heavy-weight for the apps.

There really isn't one in this case - see below.

* Heavy-duty industrial uses. I'm not adverse personally to a threading library dependency for these apps (which are sold as libraries); , but I'd have to get permission from managers who will be very concerned about use of a threading library clashing with customer's use of other threading libraries. Whether these concerns are valid or not is almost beside the point; it will be a hassle.

The library actually supports overriding the types used for logging including the threading primitives used, which (unless you use the ts_appender) is only a simple mutex and corresponding scoped lock class. So you should be able to keep those managers happy by using/wrapping other mutex/lock classes.

... it would be nice if there was a basic subset of features that could be used without threading concerns.

There is. Much of this can be addressed by better docs, but there is still some level of flux in how the internals (including the types exposed to allow alternative implementations to be assembled) fit together, so detailed docs on the architecture might take a while to arrive. In the meantime: The lib can be broken down into: 1) The logger itself. Each log has a collection of appenders and modifiers and an enabled state. 2) A manager that maintains collections of loggers, appenders, modifiers, enabled state and provides methods for changing the logging settings. 3) A collection of appender and modifier functions. Each logger has a mutex to protect its state. The log manager uses a mutex to serialise updates. It still needs to be resolved how the actual writing (calling of modifiers and appenders) should occur in a multithreaded environment. I think it should be done with no lock held by the logger (or the manager). In that case the appender and modifier functions need to be threadsafe (reentrant) so stateful functors may need to use some form of locking. Note that that isn't how the lib is at present - the log manager lock is held during writing. In all the above, in a single threaded build, a null mutex type is used. There is also the case of appenders, such as the ts_appender, which internally use boost.thread, and that can only possibly run in a multithreaded environment. FYI, the manager is a concept, which curently has 1 implementation. The concept probably needs a bit of refinement (and docs), but the key point is that alternate managers can be used if needed. Other types, including the "string" and "stream" types can also be replaced. It should be possible to build at least the following logging systems with the library: 1) The lib as presented so far. 2) A very simple logging facility, with an essentially null manager - just providing an interface to manually bolt together loggers and appenders/mutators. One form of this might be a logging facility that can only have its settings changed (eg read from a file) by an app restart, in which case even in a multithreaded app most locking can be eliminated. 3) Something quite radically different such as a log where the "stream" is something into which structured log information can be assembled, and the "string" carries that information to appenders that use that structured information (format it for textual output, write it to a database table...). 4) ... Regards Darryl Green.

Johan Nilsson <johan.nilsson <at> esrange.ssc.se> writes:

"Darryl Green" <darryl.green <at> unitab.com.au> wrote in message news:loom.20041030T055250-159 <at> post.gmane.org...

...

The library actually supports overriding the types used for logging including the threading primitives used, which (unless you use the ts_appender) is only a simple mutex and corresponding scoped lock class. So you should be able to keep those managers happy by using/wrapping other mutex/lock classes.

I don't recognize that, is it documented somewhere?

No, but it will be. The lib is a bit pre-release at present. Maybe 0.3 would be a better rev number than 1.3...

...

3) Something quite radically different such as a log where the "stream" is something into which structured log information can be assembled, and the "string" carries that information to appenders that use that structured information (format it for textual output, write it to a database table...).

Is this actually possible? How?

Have a look at log_fwd.hpp You will see that there is a default_log_manger struct which defines the char_t, string, stream types. You will also see that log_manager is a (trivial) metafunction class taking an int parameter that returns the default_log_manager type by default. In log,hpp log_manager<0>::type is used to obtain the types used throughout the logger. You can write a specialisation of log_manager<0> and include this before log.hpp to replace these types. The "stream" type needs to have a str() member function to convet it to the "string" type. In an (unreleased) version this was avoided by having a separate to_string function as part of the manager policy - it might end up that way again? Of course at most of the supplied appender and modifier functions will not work with a wildly different "string" type, although the simple write_to_file function should work with anything with a reasonable operator<< defined. HTH Darryl.

...

You can write a specialisation of log_manager<0> and include this before log.hpp to replace these types.

The "stream" type needs to have a str() member function to convet it to the "string" type. In an (unreleased) version this was avoided by having a separate to_string function as part of the manager policy - it might end up that way again?

I guess we can add that back... If it's important, certainly.

...

Of course at most of the supplied appender and modifier functions will not work with a wildly different "string" type, although the simple write_to_file function should work with anything with a reasonable operator<< defined.

Having the interface string-based makes things easy in the general case; such as when

Yup

adding modifiers (prependers? Sorry, I'm in no way friendly with the naming of the concepts yet).

sorry. I'm open to suggestions...

However it makes things extremely difficult for integration with structured log facilities. I'd really, really not like to use iostreams, sscanf or regex to extract information out of the string at the back-end, converting into correct data types etc ... ;-)

Could you be a little more specific? As Darryl said, you can change the log manager. Sorry there's no docs yet - but I should get back to work on it in about 2-3 weeks, and add docs, refine concept, etc.

I do kind of like the current implementation's interface, but I'd prefer a

structured internal format with possibilities to a stream-based interface. But again, that's just my 0.02EUR. I donot unerstand. You just said something else above...

BTW, is a single ts_appender instance meant to be used by multiple threads?

In that case the code inside ts_appender::operator() is not thread-safe ... wait a minute .. just saw that there was another lock inside ts_appender::create_thread(). Still feels like checking the same thing twice (at least). Sorry for that - I just added one more FIXME ;) Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -- v1.5 - tooltips at your fingertips (work for menus too!) + bitmap buttons (work for MessageBox too!) + tab dialogs, hyper links, lite html

On Fri, 29 Oct 2004 15:50:18 +0200, Johan Nilsson <johan.nilsson@esrange.ssc.se> wrote:

On the other side:

- I miss priority/severity levels :-(

I think the idea is that you would use separate logs instead of severity levels (e.g. app::debug, app::fatal, etc). -- Caleb Epstein caleb.epstein@gmail.com

On the other side:

- Message formatting always takes place in the caller's context.

In ~99% of the time I'd like the messages to be formatted and written to their destination in a low-priority (idle) background thread, that does not interfere with any other parts of my application. This holds true even when I don't use any threads in the application. The only time when I would not want do it this way is when the platform does not support threads. (I deliberately do not suggest any specific implementation here.) (As a side-note, I assume you've seen the ts_appender(), which does the *appending* in a different thread) That's an interesting idea. One problem that I see with the above taking place in another thread is that you don't know when it'll happen. This does not play well with: - pre-pending time - pre-pending thread ID And probably some other modifiers I'll write or other people will. As another side-note, if you really want this, you can write an appender which will modify the original string and then write it to some destination.

- I miss priority/severity levels :-(

I don't like the idea of locking you into some fixed priority/security levels. You can always create your own "app.err" (error), "app.dbg" (debug), "app.warn" logs, and log directly there (which you can then enable/disable at any time)

- I get the feeling that the logging entity always knows where the log messages are destined. I'd like (I think) a single logging "sink" where the messages can be sorted at the back-end ("appenders") to provide some looser coupling (read the comment at the bottom of this posting though).

From your point of view, you can still think of it that way. The best way to better grasp this IMO is to take a look at the examples. I would say that the coupling is as loose as possible ;) Whenever you add an appender/modifier, you don't specify which log it will be added to - you specify a logs_specification, where you can use the * wildcard (same goes for enabling/disabling). For example, to write all errors to cout, you'll say: // assuming all error log(s) contain 'err' in their string name add_appender("*err*", write_to_cout); To turn off debug logging, simply say: // assuming your debug log(s) all contain "debug" in their string name disable_logs( "*debug*");

- This opinion is probably not shared with others, but I'd prefer the library to be independent from the Boost Thread library. The dependency feels kind of the wrong way around.

This dependency is only if you use threads yourself. The library needs to be thread-safe, and it would really see awkward to reinvent the weel. Also, ts_appender() clearly needs it.

- Tests?

TODO ;)

And a question:

- Did you consider the possibility to integrate with syslog / NT event log?

Yes, in the future. Any help is welcome in this area. Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -- v1.5 - tooltips at your fingertips (work for menus too!) + bitmap buttons (work for MessageBox too!) + tab dialogs, hyper links, lite html

"John Torjo" <john.lists@torjo.com> wrote in message news:41828D7B.1080802@torjo.com...

...

On the other side:

- Message formatting always takes place in the caller's context.

In ~99% of the time I'd like the messages to be formatted and written to their destination in a low-priority (idle) background thread, that does not interfere with any other parts of my application. This holds true even when I don't use any threads in the application. The only time when I would not want do it this way is when the platform does not support threads. (I deliberately do not suggest any specific implementation here.)

(As a side-note, I assume you've seen the ts_appender(), which does the *appending* in a different thread)

I've seen it, yes.

That's an interesting idea.

One problem that I see with the above taking place in another thread is that you don't know when it'll happen. This does not play well with: - pre-pending time - pre-pending thread ID

I basically was thinking about log "messages" as data structures, consisting of least one message id (possibly consisting of facility, severity and the id itself, similar to COM hresults and VMS status codes) + insertion data. Logging a message would include creating an instance of a log item and storing it in the TSS buffer for later retrieval, formatting and writing by the worker thread. Data such as timestamps, thread id, etc is added at this point and can be selected for inclusion/exclusion in the actual writing on the back-end instead. Using message id's would also facilitate localization, which would be a good thing for end-user visible logging.

And probably some other modifiers I'll write or other people will.

As another side-note, if you really want this, you can write an appender which will modify the original string and then write it to some destination.

...

- I miss priority/severity levels :-(

I don't like the idea of locking you into some fixed priority/security levels.

Neither do I. I was more thinking in the lines of having the severity level defined as an integer, but additionally provide a set of predefined levels; e.g. DEBUG, TRACE, INFO, WARNING, ERROR, FATAL. Disabling/enabling of logs could be done by setting up either a bit mask or a trigger level.

You can always create your own "app.err" (error), "app.dbg" (debug), "app.warn" logs, and log directly there (which you can then enable/disable at any time)

...

- I get the feeling that the logging entity always knows where the log messages are destined. I'd like (I think) a single logging "sink" where the messages can be sorted at the back-end ("appenders") to provide some looser coupling (read the comment at the bottom of this posting though).

From your point of view, you can still think of it that way.

The best way to better grasp this IMO is to take a look at the examples. I would say that the coupling is as loose as possible ;)

I take your word on that, it's just that I'm trying to grip the concepts.

Whenever you add an appender/modifier, you don't specify which log it will be added to - you specify a logs_specification, where you can use the * wildcard (same goes for enabling/disabling).

Maybe it's the word "log" that confuses me. For me a log is more the final destination; e.g. the event log, a file, etc ... In this case it is more like a channel for log messages, or?

For example, to write all errors to cout, you'll say:

// assuming all error log(s) contain 'err' in their string name add_appender("*err*", write_to_cout);

To turn off debug logging, simply say:

// assuming your debug log(s) all contain "debug" in their string name disable_logs( "*debug*");

The above is exactly one of the things I like about your proposal.

...

- This opinion is probably not shared with others, but I'd prefer the library to be independent from the Boost Thread library. The dependency feels kind of the wrong way around.

This dependency is only if you use threads yourself.

Naturally, but that alone might be enough to make some people refrain from using it. I'd prefer not having to add the boost thread library to my dependencies simply because I want to log message in the background. Also, there might be special circumstances when you'd like to log messages from system invoked routines, e.g. Win32 completion routines/APCs/console control handlers, Unix signal handlers, VMS ASTs, system thread pool callbacks etc... Most of these cases are those where you are normally adviced not to do anything more than absolutely necessary, but occasionally logging from those are not out of the question.

The library needs to be thread-safe, and it would really see awkward to reinvent the weel. Also, ts_appender() clearly needs it.

Clearly needs Boost.Thread or thread support in general? I've got a gut feeling that this is one of the not-so-rare-cases when duplicating code to lessen dependencies *might* be justified. It would of course be possible to parameterize the threading implementation, but as one of your goals is to minimize compile times if might not be feasible. Do other libraries use Boost.Thread to implement thread-safety? Should they?

...

- Tests?

TODO ;)

...

And a question:

- Did you consider the possibility to integrate with syslog / NT event log?

Yes, in the future. Any help is welcome in this area.

I believe that it's not so simple to integrate with the NT event log as the current Boost.log is entirely based on pre-formatted strings. The event log is more based on message id's, insertions strings and message resource dlls, IIRC (long time since I used NT event logging). // Johan

On Mon, 01 Nov 2004 21:32:26 +0200, John Torjo <john.lists@torjo.com> wrote:

Johan wrote:

...

I basically was thinking about log "messages" as data structures, consisting of least one message id (possibly consisting of facility, severity and the id itself, similar to COM hresults and VMS status codes) + insertion data. Logging a message would include creating an instance of a log item and storing it in the TSS buffer for later retrieval, formatting and writing by the worker thread. Data such as timestamps, thread id, etc is added at this point and can be selected for inclusion/exclusion in the actual writing on the back-end instead.

So basically, after a message is added, a log_item is created, and different modifiers can fill different things of this log_item itself. Then, the appender(s) will decide which information to write to the real destination(s). This is an interesting idea.

This is how most of the logging frameworks I've used in the past have worked. There's always a free-form text field for a human-readable message, but there are numerous other fields which are filled-in automatically (perhaps even down to the level of the __FILE__ and __LINE__ where they were generated - very easy to do when using macros). Take for example the ACE_Log_Msg structure from the ACE library (just the important bits): class ACE_Log_Msg { [...] // Status of last operation (user defined) int status_; // errno when this message was created int errnum_; // Line number in file_ where the message was generated int linenum_; // Name of file where message was generated char file_[MAXPATHLEN + 1]; // The message itself char msg_[ACE_MAXLOGMSGLEN + 1]; // Pointer to thread descriptor ACE_Thread_Descriptor* thr_desc_; // Priority (severity) of this message unsigned long priority_mask_; // Name of the program static const char* program_name_; // Hostname static const char* hostname_; }; No judgements of relative goodness/badness, just for informational purposes. I would also suggest that any structured log message contain a timestamp indicating when it was generated.

Is this what you're talking about, or am I totally on the wrong track?

I think you've got the gist of Johan's point (assuming I have). -- Caleb Epstein caleb.epstein@gmail.com

On Tue, 2 Nov 2004 02:51:27 +0000 (UTC), Darryl Green <darryl.green@unitab.com.au> wrote:

Caleb Epstein <caleb.epstein <at> gmail.com> writes:

...

Take for example the ACE_Log_Msg structure from the ACE library (just the important bits):

class ACE_Log_Msg { [...]

I'm not clear on your point here - do you/have you used the ACE logging facilities? What feature do you actually want, if any, from there?

I'm not requesting any particular features, just trying to help explain what is my take on Johan Nilsson's earlier message. John apparently had a couple of questions about what Johan had written, and I thought I'd try to help clarify. Perhaps I'm muddying the waters. It *can* be useful to package up some information along with a message when logging it, and ACE_Log_Msg is just one implementation of that. I'm neither fer it nor agin it. Though I will exercise a modicum more conviction at the polls tomorrow...

I certainly don't want a logging lib that collects "interesting stuff" on the off-chance that some output formatter might use it (and there have already been complaints about coupling - something like this is going to make it worse).

Fair enough. It should be do-able to add this sort of thing in a special appender, if needed. Or see below.

If what you want is something like the ability to have a modifier add this sort of information to the message, it already exists. The only question is, in what form is the message and the data added by the modifier? There is nothing stopping you from (for example) putting XML into the message, and having the modifiers add additional elements, then having an xml processing appender pick/chose and format these elements. Now, that sounds like overkill, but in some systems that may be a great idea, because the appender is actually just chucking this data in a file, or across a network, and some log view/query/filter tool can process the XML any way it likes. A lighter-weight version might just put attrib=value entries into the message, with newlines separating them or... If that isn't enough, as I've said before, it is possible to have some completely custom "string" and "stream" if that works for you.

Very interesting. Maybe users who want a structured-type log message (which collects this extra info), could implement one as their "string" class.

Maybe what is really missing is a clear set set of requirements the libary needs to address. FWIW I think the following is a reasonable set, without ending up with an everything including the kitchen sink result:

The library must provide a logging framework with a fairly minimal, portable set of facilities with few dependencies. The logging library must be very efficient in handling disabled "logs".

The logging library must be user extensible to add new output (appender) facilities. This should be very easy and not require in-depth knowledge/study of library/its docs to do. Appenders, which are likely to have system dependencies, should not be tightly coupled with the core logging facilities.

Nicely put.

The logging library must be user extensible to add new modifier facilities. This should be very easy and not require in-depth knowledge/study of library/its docs to do. Modifiers, which are likely to have system dependencies, should not be tightly coupled with the core logging facilities.

The library must scale well. This should include the ability to (efficiently) use a very minimal form of the library in small, perhaps single-threaded only, apps as well as achieving good performance on multiprocessor platforms in multithreaded apps.

The above points cover basic, portable usage. However, there are many environments where more customisation is needed, possibly non-portably. Making some components of the library generic can address this, but beyond some (possibly useful in their own right) example/test cases, I don't think actual implementations of a wide variety of these options need be included in the library, though over time it could be extended with additional submissions.

Customisable Configuration/control of log enabled/disable, which appenders particular logs write to etc. One general purpose way of doing this at least must be provided.

Customisation of the form in which log information is provided to the log. As per earlier requirements, a stream interface at least must be provided. Another interface, perhaps one allowing some form of arbitrrary attribute/value entry might be provided at least as an example.

Customisation of the form in which log messages are transported to modifiers/appenders. Currently the lib uses std::string for this. Perhaps a more highly tuned type could be provided by default - one where modifiers directly/efficiently prepend (as well as append) through a stream interface would be useful. Appenders really only need to receive a character iterator range. However, more wildly different types should be possible and at least an example provided (eg. some form of sequence of attribute/value pairs - maybe a map<std::string,boost::any> would be a good example?).

I think this is where a structured log message-as-string object might come into play. It would just need an ostream& operator<<() and voila you have your custom log information

I actually think the current proposal goes a long way towards meeting all of these (there are some areas that need "fixing" but I think the basic design is there). Are there any other design issues that need to be resolved or is it only a case of the implementation having a few gaps/bugs?

I agree. John's submission is very nice as-is. -- Caleb Epstein caleb.epstein@gmail.com

Johan Nilsson <johan.nilsson <at> esrange.ssc.se> writes:

Could each log provide its own log item definition? The idea just popped up, so I don't know if it's good or bad.

Well, potentially I guess the "stream" type could be per log, but then I think there would still be a need to convert any/all of them into a single "string" type. To save the ievitable confusion from using string and stream to mean something completely different, perhaps relabel as follows: _stream_ becomes _message_builder_. That is, it is the object that allows data to be logged as a single message to be assembled into the message. It could be an ostringstream, data added using operator <<, the message being the underlying string. It could be an ACE_LogMsg or whatever the struct is. _string_ becomes _message_. A message is the thing you deliver to an appender (and possibly modify along the way - though I wonder if perhaps the modifier would actually be better off operating on the message_builder BEFORE it is turned into an (immutable) message. We could conceivably (but compile time expensively?) allow for a variety of message_builders. It would have to be possible to generate a single message type from all of them though. Then the appenders would need to be able to actually generate something useful from it. Stuffing a whole heap of different sorts of "messages" into a message type of boost::any isn't going to make output easy :-)

...

If that isn't enough, as I've said before, it is possible to have some completely custom "string" and "stream"

That's good, but it doesn't put off the formatting (from whatever to text) so that it can be performed in the background.

Its a question of definition of "formatting". With the non-custom version the "whatever" must be converted to a sequence of chars that some formatter can further process. With some other message type (not a string) it is clearly possible to delay (possibly to some completely offline processing if you want a binary log file) the formatting.

...

I don't think actual implementations of a wide variety of these options need be included in the library, though over time it could be extended with additional submissions.

No, but I strongly recommend that enough thought go into this process, to avoid the initial implementation preventing this without major rewrites.

Agreed. Thanks for the help in doing that ;-)

...

Appenders really only need to receive a character iterator range.

You are assuming appenders only need to work with pre-formatted text.

Well - some form of sequence of "chars", but I take your point. I really meant to suggest that for the case where the message is in this form, perhaps a type other than string could be used. I agree with you that some completely different message type should be allowed for. Everything except for the low-level mechanics of an appender (writing to a file or whatever) is coupled to the message type, and very little else in the library. So I think it is reasonable to provide/allow completely separate sets of appenders for particular message types. It might be possible to provide some simple appenders such as writing raw messages to a file or socket as simple templates. Others - like a windows event log appender - appear to be far less generic, or at least require some form of formatting/converting policy peculiar to the message type, to be provided.

...

However, more wildly different types should be possible and at least an example provided (eg. some form of sequence of attribute/value pairs - maybe a map<std::string,boost::any> would be a good example?).

That sounds interesting - even though I'd like to stay out of boost::any / boost::variant dependencies also. NIH syndrome or minimizing dependencies?

My suggestion of its use in an example is a combination of laziness (in some ways) and a clear demonstration of flexibility. If it actually performs well enough to be useful to people in its own right it qualifies as an experiment that paid off, otherwise it is a prototype for something perhaps less flexible, but more efficient? I'm assuming that an appender for this message type would have to be quite heavily templated, and pull out the fields it was interested in into their original types, by name, then format them. Whether this has any utility remains to be seen...

Also, I've got a feeling that at least boost::variant is pretty heavy on the compile times ... am I right?

I'm not an expert on those libraries. I haven't even played with variant yet. Any shouldn't be too bad at compile time - runtime performance might be an issue though!

...

I actually think the current proposal goes a long way towards meeting all

of

...

these (there are some areas that need "fixing" but I think the basic design is there). Are there any other design issues that need to be resolved or is it only a case of the implementation having a few gaps/bugs?

- The actual logging calls themselves also should be made as efficient as possible, not only the enabled/disabled checking.

Yes. I meant the requirement re scaleability to cover (some aspects) of that. The trouble with "efficiency" is that means different things to different people, especially when you start trading it off against other features. Are there any specific aspects of performance/efficiency you are concerned about?

- Localization support (perhaps not so basic).

Good point. As I thnk you said in another post, that seems to be one feature that would be directly facilitated by some form of non-string message representation. However, I can also imagine a system where the localization formatting was done by the message_builder as the log message was generated, not on output. Do you have cases where different appenders should apply different localization? If localization is too expensive to do "up front" are you sure it isn't practical to do some cheaper form of "serialisation" up front and do the localization offline? Regards Darryl.

I'd even go so far as to say that the current implementation is too general (log_manager concept etc). Why not make it even simpler and only put in some effort to make the threading support more loosely coupled (as is already in progress). I'd definitely like light-weight Win32+Posix compatible integrated threading support. Keep the naming (and concepts +

will certainly do.

implementation) simple and concrete: e.g. strings, streams, a single log manager implementation and just stick to that to make it clear what the library actually does.

I'd personally very much like some kind of structured logging functionality, and I'll probably work on that offline in the near time. I'll keep the discussions (including requirements) from here in mind and post to the list when (and if) I come up with something useful. Any interest?

Yup. Basically, I would say it would be nice to have this as a library on top of Boost.Log (maybe call it Boost.Structured Log ?) As you work out the needed bits, just let me know how I should change the lib so that it fits your scenario as well. How's that? Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -- v1.5 - tooltips at your fingertips (work for menus too!) + bitmap buttons (work for MessageBox too!) + tab dialogs, hyper links, lite html

"John Torjo" <john.lists@torjo.com> wrote in message news:41868F4A.1030406@torjo.com...

[snip side-note]

...

...

That's an interesting idea.

One problem that I see with the above taking place in another thread is that you don't know when it'll happen. This does not play well with: - pre-pending time - pre-pending thread ID

I basically was thinking about log "messages" as data structures,

consisting of least one message id (possibly consisting of facility, severity and the id itself, similar to COM hresults and VMS status codes) + insertion data. Logging a message would include creating an instance of a log item and storing it in the TSS buffer for later retrieval, formatting and writing by the worker thread. Data such as timestamps, thread id, etc is added at this point and can be selected for inclusion/exclusion in the actual writing on the back-end instead.

So basically, after a message is added, a log_item is created, and different modifiers can fill different things of this log_item itself. Then, the appender(s) will decide which information to write to the real destination(s). This is an interesting idea. You could then pass a format string to the appender - kind of like log4j. This will complicate the design - but it seems to worth it.

Is this what you're talking about, or am I totally on the wrong track?

You are definitely closing in ;-) I don't have so much spare time exactly right now, sorry. I'll try to elaborate ASAP.

...

Using message id's would also facilitate localization,

which would be a good thing for end-user visible logging.

How do you envision this?

Use message id's instead of literal strings in the interface. Postpone id -> text transformation until "append-time". Once you've got a formatted message it's much harder to translate it. Note that I'm only talking about the "human-readable" part of the log item - not including the modifier stuff (time, thread id, etc).

...

...

Whenever you add an appender/modifier, you don't specify which log it will be added to - you specify a logs_specification, where you can use the * wildcard (same goes for enabling/disabling).

Maybe it's the word "log" that confuses me. For me a log is more the

final destination; e.g.

the event log, a file, etc ... In this case it is more like a channel for log messages, or?

"log" seems to mean different things to different people ;) However, having the BOOST_LOG macro seems to be quite straightforward to people.

Yes, I've got no objections to the name of the macro. Perhaps BOOST_LOG_MESSAGE would be even better, but you'd quickly get fed up writing that.

For example, having BOOST_CHANNEL to do logging - it seems kind of misleading.

That would definitely be a bad and misleading name.

Any better name out there?

...

...

...

- This opinion is probably not shared with others, but I'd prefer the

The macro name is fine with me at least. library to be independent

...

...

from the Boost Thread library. The dependency feels kind of the wrong way around.

This dependency is only if you use threads yourself.

Naturally, but that alone might be enough to make some people refrain from using it. I'd prefer not having to add the boost thread library to my dependencies simply because I want to log message in the background. Also, there might be special circumstances when you'd like to log messages from system invoked routines, e.g. Win32 completion routines/APCs/console control handlers, Unix signal handlers, VMS ASTs, system thread pool callbacks etc...

True. Well, it's not that complex to remove the dependency on boost.thread (that is, have a certain directive, which if turned on, to ignore threads).

Ignore Boost.Thread, that is?

The problem I have now is how to update the jamfile, so that it will generate the following: - static lib + multi-thread runtime + using thread (already done) - static lib + multi-thread runtime + not using thread (problem here) - static lib + single-thread runtime + not using thread (already done) (not sure if this is just an MSVC issue)

"thread" above refers to Boost.Thread?

...

Clearly needs Boost.Thread or thread support in general?

thread support. You can override it - with your own mutex and lock

classes. And thread classes/functions (as used for e.g. the ts_appender thread)?

However, I do need to refine the log-manager concept. Also, if you provide your own mutex/lock classes, you might need to include the log source files within your project.

I'd prefer not to (of course). How about a light-weight implementation for Win32 and Posix threads?

...

...

I believe that it's not so simple to integrate with the NT event log as the current Boost.log is entirely based on pre-formatted

strings.

...

The event log is more based on message id's, insertions strings and message resource dlls, IIRC (long time since I used NT event logging).

unique IDs are not hard to generate. Anyway, I haven't used NT event log, but I don't think it should be very difficult to use.

It's actually pretty complex. From the top of my head (as I said, it's been a while since) you basically need to: - Create a message resource from a message definition file (using mc.exe) - Create a dll to hold the resource - Register an event source in the registry where you define where your message resource dll is located. - When you log a message, use the source and message id, add any insertion strings as an array of strings. Regards // Johan

I haven't browsed the new code yet but I have a couple of items in my wish list:

* Log file rotation. It's quite usual that people want a per-day (or other period) log files. * Log file size limit. In stable builds one usually wants minimum logging and don't want her log files to keep growing without limits, rather one wants a queue like log file.

Certainly, this functionality can be implemented outside of the library by a user, but I would like to have it out of the box.

Yes, it will be - rotating logs that is. I hope to have it within 2-3 weeks. Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -- v1.5 - tooltips at your fingertips (work for menus too!) + bitmap buttons (work for MessageBox too!) + tab dialogs, hyper links, lite html

Just a quick thought that came to my mind...

Did you consider using the syntax 'BOOST_LOG(dbg, "message" << foobar);' instead of 'BOOST_LOG(dbg) << "message" << foobar;'?

While I think the syntax you use looks far nicer, we ususally need the ability to be able to compile away the logging calls completely and this only can be done with the first form.

In practice, you seldom need to turn them off completely. A possibility to enable/disable them at runtime will be usually desired. This is what BOOST_LOG does (see the rest of the thread). It is equivalent to: if ( is_enabled(log)) log << whatever << you_wish; As a side-note, you can always redefine // similar to #define BOOST_LOG(x) if (false) x which will remove all log calls (probaly in release mode). Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -- v1.5 - tooltips at your fingertips (work for menus too!) + bitmap buttons (work for MessageBox too!) + tab dialogs, hyper links, lite html

...

Just a quick thought that came to my mind...

Did you consider using the syntax 'BOOST_LOG(dbg, "message" << foobar);' instead of 'BOOST_LOG(dbg) << "message" << foobar;'?

While I think the syntax you use looks far nicer, we ususally need the ability to be able to compile away the logging calls completely and this only can be done with the first form.

a point considered in detail in "Effecient C++" by Bulka & Mayhew

Unfortunately I have not read their book. Just curious, what was their conclusion? (also, note my other post to Markus) Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -- v1.5 - tooltips at your fingertips (work for menus too!) + bitmap buttons (work for MessageBox too!) + tab dialogs, hyper links, lite html

On Fri, 12 Nov 2004 17:21:10 +0000, Reece Dunn <msclrhd@hotmail.com> wrote:

Why not do: #define BOOST_LOG(id) /##/ to remove the logs at compile time?

This doesn't work, at least with gcc 3.3.2. You end up with "/ /" (e.g. slash space slash) in the proprocessed output. Additionally, the preprocessor only takes a single pass through the input, so if this *did* work as intended you'd be passing the comments through to the compiler, which would presuably emit syntax errors. -- Caleb Epstein caleb dot epstein at gmail dot com