On 2005-05-19T14:30:37-0400, Eric Goebelbecker wrote:

3) Log "rolling" is critical. In a "normal" runtime scenario very few messages are sent to the log - but when something does go wrong a history of "aw shucks" messages can be very helpful.

* Roll on date and/or size (i.e. roll every 24 hours unless the log file hits 2 GB in which case you want to roll now). * Log file name contain timestamp of when it was rolled. * User-defined action when we roll. Removal of old log files either by age, total size, and/or number of files. * Log to syslog * Run-time configuration. For instance, log normal stuff to syslog but for debugging we want to write trace data to /tmp/debug.log for some tags (suggested earlier I think). /Allan

...

* Log to syslog

I figure that's platform dependent (at least until there's a cross platform sockets library.) I was planning on writing an appender for that.

Please do! Just send it over, when it's done ;) Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -v1.6.3 (Resource Splitter) -- http://www.torjo.com/cb/ - Click, Build, Run!

* Roll on date and/or size (i.e. roll every 24 hours unless the log file hits 2 GB in which case you want to roll now).

Yup, exists now ;)

* Log file name contain timestamp of when it was rolled.

Not yet, but it should be rather easy to add.

* User-defined action when we roll. Removal of old log files either by age, total size, and/or number of files.

Not yet, but it should be rather easy to add.

* Log to syslog

To do.

* Run-time configuration. For instance, log normal stuff to syslog but for debugging we want to write trace data to /tmp/debug.log for some tags (suggested earlier I think).

Already possible. Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -v1.6.3 (Resource Splitter) -- http://www.torjo.com/cb/ - Click, Build, Run!

Cool. Will the new version build with Boost 1.32.0? On 5/22/05, John Torjo <john.lists@torjo.com> wrote:

...

3) Log "rolling" is critical. In a "normal" runtime scenario very few messages are sent to the log - but when something does go wrong a history of "aw shucks" messages can be very helpful. Since the first thing a client will do when a problems occurs is restart the app...overwriting the log will not do. Having the startup script move the log to a new name is a possibility, but less than desirable.

Note that in the latest version, there's log rolling as well, thanks to Caleb Epstein!

Best, John

-- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -v1.6.3 (Resource Splitter) -- http://www.torjo.com/cb/ - Click, Build, Run!

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

-- "Should array indices start at 0 or 1? My compromise of 0.5 was rejected without, I thought, proper consideration." - Stan Kelly-Bootle ------------------------------------ Eric Goebelbecker eric@ominor.net http://eric.ominor.net

I thought about the problem for a while and came up with my own scheme. The approach I took was to separate streaming from the log implementation, this makes the log much more flexible: LOG(1, cdbg << "Hello World!" << std::endl); Logs the message if log bit 1 is set. LOG is a macro and cdbg a stream to the Windows debug output. I use a 64 bit set of flags so that multiple log levels, 1..64 can be turned on. The log flags can be defined dynamically or statically, with the static version the log flags are a constant and the compiler optimizes out LOG statements at compile time.

-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Robert Zeh Sent: Thursday, May 19, 2005 10:46 AM To: boost@lists.boost.org Subject: [boost] Re: Logging library - your own scenario!

Our current logging is very simple. At start up, most of our applications specify a log file, which can be gotten later via a singleton. The singleton allows global access to an output stream for their logs.

There are a few things it doesn't do that we'd like in a logging framework:

1) Automatic timestamps, which are currently done manually.

2) Debug levels, so that some log messages could be ignored while others are actually written to the output stream. It would be nice if we could specify debug levels as a stream manipulator, ie:

log << debugLevel(5) << " reactor meltdown " << endl;

If the debug level is below 5, the reactor meltdown message is ignored. Sometimes we dynamically adjust the debug level, so that we can get more information from a misbehaving application.

Of course, you have to somehow specify when log level is being reset. I haven't thought that part through.

Robert Zeh Manager, Applications Development Error Free Software http:://home.earthlink.net/~rzeh

John Torjo <john.lists@torjo.com> writes:

...

Hi all,

In order to make the Logging lib as usable as it can be, I'd like to know what is *your* scenario, when using a Logging lib.

Thus, I'll do my best to make it as easy as possible to use my lib, and also tweak it to be easily usable in most common scenarios.

Even if you scenario fits an already described one, please just reply to that email with something like "mine too!", so that I'll know which scenarios are most common.

Thanks!

Best, John

-- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -v1.6.3 (Resource Splitter) -- http://www.torjo.com/cb/ - Click, Build, Run!

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Darn - forgot reply responds to boost. So, I apologize for the stupid spam. The scenario we use is something akin to the following (using macros): LOG(int level, string tag) << "whatever"; The logging (as a whole) can be enabled/disabled. When enabled, portions (according to the 'tag') as well as the level, can be selectively enabled. Levels and tags are automatically inserted into the stream (for post-filtering if desired). We generally have different logs/streams for different types of logging: errors, warnings, developer messages, determinism checks: ERROR_LOG(...) << ...; WARNING_LOG(...) << ...; etc. Which of course can be tied (at runtime) to the same stream or different streams. There's always the push for "speed", so a compile-time disabling (akin to NDEBUG for asserts) of certain streams is required. Note: we're not using boost logging, just because we've already got something that works. TJ -- Trey Jackson trey.jackson@intel.com "Never eat more than you can lift." -- Miss Piggy, a Jim Henson puppet

On 5/22/05, John Torjo <john.lists@torjo.com> wrote:

...

* Channels or topics or keywords. I think various people have asked for this by one or more of these names, and I like the idea as well. This can be implemented with separate loggers in the current implementation, but I'm not sure how I feel about that. It doesn't feel right to me.

So, a channel would be the eqivalent of the existing logger, and we'd have a singleton logger?

Pretty much, yes. I am on the fence about a singleton, but it seems to be the logical conclusion of allowing different logging topics/channels and solves the logger declare/define "problem".

...

* Simple interface using ostream::operator<<. If my classes have streaming support, they should be log-able. Something like:

BOOST_LOG (logger, topic, level) << message << goes << here

or

BOOST_LOG (logger, topic, level, message << goes << here)

How do topic and level differ?

They are orthogonal. In an application I might have topics like "network", "memory", "queues", "statistics", etc. Levels would typically be debug, info, warning, error, fatal, etc.

...

* Structured log records. In order to implement thread-safety, there must understandably be some level of buffering. I like the current ts_appender with its timed batching of writes, but you lose some information like the name of the logger to which the message was originally sent. I think it is imperative that a logging "call" is able to capture a precise timestamp, thread ID, topic, severity, and __FILE__/__LINE__ info at time the message is created.

Note: this should be doable with a custom log manager.

...

* Setup. I'm not enamored of the requirement of declaring & defining loggers (or using scoped loggers). If the channel & severity concepts become first class citizens, is there any reason not to have a singleton logger? Or perhaps a singleton logger could be made available, and users could provide their own loggers instead if they desired.

Well, I don't know. If we were to have channels, you'd still need to declare them somewhere (so I'd know which channels are there).

Assuming they could be strings, why would they need to be pre-declared?

And then, defining them was needed to be able to externally access a log (so that you can allow configuring the logs by "scripting" - by possibly reading a file, which contains how logs are configured).

I'd think they would be even simpler to configure if the interface was more dynamic (e.g. string based). The names of topics used in the configuration should be enough to configure the entire logging hierarchy.

What would the advantage of having channels be?

You can control/gate the output (e.g. adjust severity levels, perhaps even select output destinations) based on them. The more I think about it, channels really are similar or identical to your logger concept, but I'm much more comfortable/famliliar with a singleton-based logger that uses channel "names" that are strings. This removes the requirement for pre-declaring and defining loggers explicitly in code which I dislike. Adding new topics is just a matter of changing your code to log using a new topic name. -- Caleb Epstein caleb dot epstein at gmail dot com

Mark Blewett <boost <at> blewett.nildram.co.uk> writes:

Thanks for writing my request for me :) but please see additional requirements at the end.

1) Each trace (log) many have zero or more observers (appender) with independent control of what is seen.

For example our production systems run with an observer which only writes only critical/error messages to a local file. However we also provide off-site support, where we remotely connect (possibly multiple people connecting) via tcp/ip to the logging system, where each person has their own view and only sees the traces that they have enabled.

Me too (near enough).

2) Redirecting cout / cerr to a trace.

Not on my list, but nice to have.

3) Different formats (modifiiers) for different observers.

Yes. This one is annoying because it turns the modifiers/appenders list into a tree. I think this is where it becomes better to resort to a structured log message format, where modifiers are only used to add additional structured data such as timestamps to log messages, and where formatting this data is left up to the appender. So long as it is easy to compose an appender from a "dumb" writer that just receives a string (or that is just an ostream? but that gets ugly in that ostreams do formatting...) plus a formatter taking the structured log message I think this is nicer all round - users/support staff don't get: 17:56:02.123 ERROR: PID:1234 TID:4567 foobar.cpp:321 unhandled some@$%^mangled_and_even_demangled_still_incomprehensible_typename event and do get 17:56:02 ERROR: Unexpected event. Call Fred NOW. He knows where the nasty detailed log is and how to read it. (well ok, maybe not quite that descriptive, but you get the idea) Because the actual appender is constructed like this (very rough - don't take the exact method too seriously): string user_log_formatter(log_msg m); string debug_log_formatter(log_msg m); logfile_appender user_log("/var/log/myapp.log", user_log_formatter); logfile_appender debug_log("/var/log/mayapp/debug.log", debug_log_formatter);

For example for a file format we might like each line as "<date> <time> : <trace> : <level> : <message>", where as we might also like to send the same data across the network in xml / asn1 format, or even insert a row into a database table.

I see the case where the same message data is presented in different formats (eg ASN.1 vs text log entry) as being simply a marshalling format issue (assuming that the log message is structured) which is handled similarly to the above. Actually I have considered just marshalling the log message into a string as ASN.1 BER and having the formatters pull that apart if/as required. I do have some extra requirements/constraints: a) No singletons. That is, I want to be able to have logging using multiple managers with different policies in a single app. While this may sound a little weird, I need something that looks an awful lot like logging to actually log application level events, in specific formats etc independent of application debug/error logging. This is a failing of the logging lib I had been using. b) Dynamic/scoped logs are essential. I know they are in the design already, but it is one feature that doesn't seem to get many requests and, despite (e) below, one feature I can't do without. c) There are different interpretations of the named logs/keywords/tags/channels being kicked around. I'd just like to register a definite vote for the named log approach as currently implemented. It achieves everything I've ever wanted from this sort of attribute, very efficiently. If some other post filtering naming is needed (it isn't by me), it can just be another attribute in a structured log message. d) Thread safety is required (as is the ability to turn it off). Minimise lock scope and hold time. Appenders must implement their own thread safety anyway (unless a global lock is used, which would be bad) given that they can be shared by multiple logs. Except during updates to settings the logs themselves should be reenterant. e) If feature lists lead to code size/data size/performance compromises thats fine, so long as I can turn off features I don't want. Layering or policies please. Embedded systems may have a lot more memory and MIPS than they once did, but it's a long way from infinite, and an order of magnitude or more below desktops (ymmmv). I don't expect the logging lib to be tailored to this environment (which is hard to define in a general way anyhow) but configurability is key. Logging is not the deliverable - it must be efficient otherwise it will just get left out. Darryl.

Hi Mark,

1) Each trace (log) many have zero or more observers (appender) with independent control of what is seen.

For example our production systems run with an observer which only writes only critical/error messages to a local file. However we also provide off-site support, where we remotely connect (possibly multiple people connecting) via tcp/ip to the logging system, where each person has their own view and only sees the traces that they have enabled.

Yes, already there ;)

2) Redirecting cout / cerr to a trace.

For example we use some 3rd party libraries which write errors to cout / cerr etc, so we allow the option of running (via a command line option) in console mode (anything written to std output appears), or silent mode, where cout / cerr is redirected to its own trace.

True... Whew - one more thing on my todo list ;)

3) Different formats (modifiiers) for different observers.

For example for a file format we might like each line as "<date> <time> : <trace> : <level> : <message>", where as we might also like to send the same data across the network in xml / asn1 format, or even insert a row into a database table.

As to how important these would be to me: 1) & 3) would be very important, 2) would be a nicety.

As for 3), it's already there -- although the interface could be easier. As said in previous emails, I'd like to have another lib which makes configuring the logging much easier. Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -v1.6.3 (Resource Splitter) -- http://www.torjo.com/cb/ - Click, Build, Run!

He he ;) That's another lib altogether (SMART_ASSERT). I plan to enhance it and to make it use the Logging lib internally. I will certainly do so as soon as I can find some time for it (plenty of things on my plate right now) Best, John

i'd be interested in something similar to your own article here: http://www.moderncppdesign.com/publications/cuj-08-2003.html

i.e. being able to do: int a = 1; std::string b = "test"; SOME_MACRO(some_log)(a)(b);

and have: a = '1'; b = 'test';

appended to some_log

regards,

-- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -v1.6.3 (Resource Splitter) -- http://www.torjo.com/cb/ - Click, Build, Run!

We have several applications that log their activities. Usually, we have these requirements:

- specify different topics/categories or whatever you name it (like ERROR, DEBUG, ...)

To do.

- allow binding different streams to the topics

Exists.

- dynamically (during runtime) changing a severity level for each topic separately

To do.

- timestamping the log messages

Exists.

- thread safe

Already done. Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -v1.6.3 (Resource Splitter) -- http://www.torjo.com/cb/ - Click, Build, Run!

Hi Darren,

I hope I'm allowed more than one... Actually all the use cases I can

Of course you're allowed ;) The more the better!

SCENARIO A: socket server

I define ERROR, WARN and INFO and all go in the same big log file. Timestamp automatically prefixed. Three tab-separated fields (i.e. timestamp, level, message).

It is always there in production code.

How it needs to be improved: 1. Be able to hide info, or info and warn lines when viewing the log file. Alternatively write each level to its own log file, and have a viewer that is able to combine the three files (I.e. typical usage is I want to see what INFO/WARN messages led to an ERROR message). Any viewer must work on Windows and linux.

This can all be done from an external viewer. I've developed something like this -- unfortunately did not have time for improving it lately. It's Windows-only unfortunately.

2. Daily log rotation, with YYYYMMDD style datestamp going into the filename. If hourly log rotation then YYYYMMDD_HHMMSS style datestamp in the filename.

Should be possible with the latest version.

(I also want logs older than N days to be gzipped, and log older than a certain date to be deleted; but I'm happy - in fact happier - for these tasks to be done by cron job not a logging library, as it is a system administrators decision not a programmers decision)

Let's leave it to the admin for now ;)

3. Be able to start/stop a DEBUG level while the program is running.

Yup.

4. Email me when an ERROR message occurs. Different levels of error could go to different email addresses (i.e. more serious ones need to go to a mobile email address).

Doable with the current lib (if you write your appender which sends email). By the way, if anybody has written something like this and wants to share, I'd appreciate sending it to me.

SCENARIO B: debugging game tree search

The LOG macro must be able to be compiled out completely. This is code that once debugged and compiled in release mode should take a fraction of a second to run.

What I want that I don't have: Instead of writing to a log file, I want to write to a memory buffer, that only stores the last 2MB, or the last 100 messages, for instance.

Yes, Pavel Vozelinek suggested I do an appender over shmem lib. I should indeed do so.

Then when I have an assert I want to be able to output the contents of that buffer. E.g. SMART_ASSERT(node_cnt<5000 && depth<50)(depth)(node_cnt)(current_position)(history_log.output()).msg("What can it be thinking to search so deeply?");

In-memory is important as writing to a disk file is a major performance bottleneck.

Yup, to do. Best, John -- John Torjo, Contributing editor, C/C++ Users Journal -- "Win32 GUI Generics" -- generics & GUI do mix, after all -- http://www.torjo.com/win32gui/ -v1.6.3 (Resource Splitter) -- http://www.torjo.com/cb/ - Click, Build, Run!

filter to throw stuff away by content. Before accepting a log event, check if the sender has been flooding you. Aggregating of data might be helpful (last message was repeated x times, instead of logging a sequence of events combine them to one log event if they occur within some time window).

I forgot about that: I'd class it as a "nice but not essential" feature. syslog does it and it makes skimming logs easier. Darren