On Mon, 5 Apr 2010 00:27:05 -0700 (PDT), Artyom wrote:

...

I'm encountering a crash with Boost.Locale as the process terminates

I initialise the global Boost.Locale in the contructor of a class with a single static instance (the ATL module to be exact) as I don't have access to the code of my main method (it comes from the Boost.Test runner).   A static object seems to be the only logical place to put this code:

<snip>

I've tried to reproduce the bug but I couldn't: at least with MSVC9 with Release and Debug builds and under Linux. (I do not have MSVC8 you are using).

Can you provide a simple sample of the code that reproduces the issue so I can debug it?

I'll try and whip something up tonight.

Also what setup are you using:

- Debug, Release or RelWithDebInfo builds

Debug

- How do you use Boost.Locale - static version or dynamic (dll)? if you use static version are you sure you link with correct ICU library -- (debug versions of ICU library end with "d" suffix)

I'm statically linking with my compiled Boost.Locale but I'm using the ICU DLLs from the ICU site - I'm not compiling my own. These don't have the d suffix as debug builds aren't redistributable. I believe they are also compiled with MSVC9 where I'm using MSVC8. Would this cause a problem. I expected that, if it did, it would fail completely not partially work like this.

- What version of Boost.Locale you use? Taken from SVN, boost_locale.zip or boost_locale-svn_r1112.zip?

svn_r1112.zip

- What is your system locale?

Testing with en_GB and nl.

- Does program crashes when dictionaries are not loaded? (i.e. comment out: gen.add_messages_domain("swish"); ) - What ICU version do you have?

4.2.1

Can you try please following: add global destructor that would reset the global locale to standard one:

~CModule() { std::locale::global(std::locale::classic()); }

This didn't make a difference. Thanks. Alex Lamaison

...

Can you provide a simple sample of the code that reproduces the issue so I can debug it?

I'll try and whip something up tonight.

Any updates? Thanks, Artyom

I've had real trouble isolating a minimal set of code to reproduce this. It appears to be related to the static local instance:

__PURE_APPDOMAIN_GLOBAL static std::locale::_Locimp *global_locale = 0; // pointer to current locale

being deleted once for *each loaded DLL*. When I Release() a COM component in a separate DLL, the DLL is unloaded and calls tidy_global() deleting global_locale. Later the EXE terminates which again tries to delete global_locale causing an access violation.

Unfortunately, I'm nowhere near a small set of code for repro yet :(

Ok, I see, this enteres the area called COM+DLLs... where I feel quite... helpless. I really have no idea why this does not work and why something destroyed twice, because it shouldn't. But it looks like rather issue reated to runtime environment rather then to specific localization library. You probably may try to test something like that (create a facet of your own); class my_facet: std::locale::facet { public: my_facet() : std::locale::facet(0) { } std::locale::id id; }; std::locale::id my_facet::id; struct init { init() { std::locale tmp(std::locale::classic(),new my_facet); std::locale::global(tmp); }} instance; Maybe do some allocations, deallocations in it and see if it causes crash (without any usage of boost::locale), if it causes crash then it has nothing to do to library but rather to environment. -------------------------------- For unit test you may try to do something like that: void set_locale() { if(std::has_facet<boost::locale::info>(std::locale()) return; boost::locale::generator gen(); ... /// set path you need ... std::locale::global(gen("en_US.UTF-8")); } And before each test just call set_locale. It would check that `boost::locale` based locale was loaded (every such locale has boost::locale::info facet) and not reload dictionaries each time.

P.S. As you advised, I compiled my own version of ICU. Because they only ship an MSVC9 project with version 4.2 (why do projects to that?!) I've downgraded to 4.0.

Strange, because according to their documentation MSVC 2005 is supported. Artyom

On Thu, 8 Apr 2010 13:25:05 -0700 (PDT), Artyom wrote:

...

__PURE_APPDOMAIN_GLOBAL static std::locale::_Locimp *global_locale = 0; // pointer to current locale

Ok, this is interesting point... Make sure that all DLLs you are using are compiled correctly with Same MSVC runtime **DLL** (with correct import flags) and not mixed DLL/static library/

You were right about that cause of that crash: i had mixed Release and Debug DLLs (that is a problem in its own right as typically you aren't in control of other DLLs)

Ok, I see, this enteres the area called COM+DLLs... where I feel quite... helpless. I really have no idea why this does not work and why something destroyed twice, because it shouldn't.

So reproducing it wasn't too hard after all (once i realised the DLL thing). This code should crash quite reliably: http://www.doc.ic.ac.uk/~awl03/locale_test.zip Turns out its not a COM problem but a general DLL problem

But it looks like rather issue reated to runtime environment rather then to specific localization library.

Basically what happens is that the DLL sets the global locale when it is loaded. Great. We work away doing stuff and everything is fine ... Then at some point the DLL is unloaded. But the global locale is still pointing at an object whose definition was provided by the non-unloaded DLL. Disaster! The next time anything tries to use a locale (like a simple stream op) we crash with an access violation. The problem here is setting a locale 'owned' by the DLL. Is there a way round this? If anyone uses Boost.Locale with COM (or any other situation where DLLs are unloaded) they will run into this problem. Many thanks. Alex

On Sat, 10 Apr 2010 13:07:47 -0700 (PDT), Artyom wrote:

...

You were right about that cause of that crash: i had mixed Release and Debug DLLs (that is a problem in its own right as typically you aren't in control of other DLLs)

So I do not understand fixing correct runtime solved your problem or not?

No, it just stopped the double-delete

...

Basically what happens is that the DLL sets the global locale when it is loaded. Great. We work away doing stuff and everything is fine ...

Then at some point the DLL is unloaded. But the global locale is still pointing at an object whose definition was provided by the non-unloaded DLL. Disaster! The next time anything tries to use a locale (like a simple stream op) we crash with an access violation.

The problem here is setting a locale 'owned' by the DLL. Is there a way round this?

This code reproduces the issue of crash when you set global locale and then you do not unload it, at least I see that you do not call std::locale::global(std::locale::classic()) . Am I right?

Yes, I wasn't doing that.

If so.. this is your problem and not Boost.Locale one.

It can be recreated even on Linux with shared objects, if you create global locale in dynamic library and unload the library you will crash when you would try to create an locale instance - clearly.

Can you add this to the documentation becuase its not at all obvious - there is no mention of cleanup.

Does it crashes when you call on library unload:

std::locale::global(std::locale::classic());

Yes, this fixes the crash. In fact, what I'm doing now is saving the old locale and restoring that when unloading so that if tow modules use boost.locale and one unloads, its doesn't destroy boost.locale for the other.

In any case:

- It seems to be rather standard library issue then Boost.Locale one - Any DLL that would create its own facets and set global locale would have this issue.

You're probably right but it needs to be documented explicitly. I've never come across a situation before where you have to do cleanup before unloading a DLL. Now that *that* part (not crashing) is fixed, I'm running into another problem. I can see the .mo file being loaded; excellent! But when I try to translate a string, the translation mechanism fails to find it! This only happens when i set the locale statically. If I set it just before using translate() it works fine. Here is the repro code: http://www.doc.ic.ac.uk/~awl03/locale_test2.zip To see the different situations, change which commentted line declaring LocaleSetup. The problem seems to be that std::has_facet looks for a Facet with an id of 46 (0x2e). But when it gets to this line: const facet *_Facptr = _Id < _Ptr->_Facetcount ? _Ptr->_Facetvec[_Id] : 0; // null if id off end _Facetcount is 46 so it decides it's off the end. I notice that in the working version _Facet::id is 44 (0x2c). Any idea what's happening here? Many thanks. Alex

On Sun, 11 Apr 2010 07:23:10 -0700 (PDT), Artyom wrote:

...

The problem seems to be that std::has_facet looks for a Facet with an id of 46 (0x2e).  But when it gets to this line:

    const facet *_Facptr = _Id < _Ptr->_Facetcount                  ? _Ptr->_Facetvec[_Id] : 0; // null if id off end

_Facetcount is 46 so it decides it's off the end.  I notice that in the working version _Facet::id is 44 (0x2c).

Till I get to VS at evening so I'll be able to answer. Few questions:

1. Are you using statically or dynamically liked boost.locale?

Static

2. Are you using statically or dynamically liked boost.test?

Static

3. If you use dynamically liked versions do you specify correct import defines?

N/A

4. The last what parameters of CMake did you used when you build boost.locale library.

I'm afraid I don't understand the question

5. I really hope that after build you did not switch Debug/Release from the MSVC project but rather prepared two solutions with passing correct options to CMake?

I generated the single solution using CMake which puts a debug and release configuration in the same file. This is how it's supposed to work, right? I didn't run CMake twice to create two seperate solutions.

...

Any idea what's happening here?

Yes... This is what called why I love dlls very much ;-) ...snip... So it is very easy to screw everything up when using dlls and static class members...

So what happens something somehow didn't used everything correctly. So you should be extremely careful when creating or using DLLs.

I'm not using DLLs here! My repro code does it with a simple EXE. All it takes is a single static locale instance. This may even repro on Linux.

Each facet type should have specific unique id which is static member of facet class (std::locale::id).

If this id is not unique - two different ids would be created and thus there would be mismatch of facets and you would not be able to use them.

Where are these ids generated? I tried to find it but it seems that they appear by magic :P

P.P.S.: I'll take a look on the code you gave today or tomorrow.

Thanks! Alex

On Sun, 11 Apr 2010 08:05:40 -0700 (PDT), Artyom wrote:

...

...

4. The last what parameters of CMake did you used when you build     boost.locale library.

I'm afraid I don't understand the question

I mean had you passed any special parameters to CMake when you configured the library?

No, other than having to manually specify the individual ICU DLL paths.

...

I generated the single solution using CMake which puts a debug and release configuration in the same file.  This is how it's supposed to work, right? I didn't run CMake twice to create two seperate solutions.

Not really... CMake creates a solution for configured build. If you need two configurations rerun CMake and create different solution.

See solution is not more then "makefile" or a way to build something. You do not change it, you rerun CMake.

I'd suggest build libraries in clean way and add binaries to project rather then integrating project to solution.

That's not how CMake works with Visual Studio projects. There is no option to generate a specific Debug or Release version (the value of CMAKE_BUILD_TYPE) is ignored.

...

I'm not using DLLs here!  My repro code does it with a simple EXE.  All it takes is a single static locale instance. 

You are using ICU dlls. And you are using MSVCRT.DLL -- these are DLLs.

Sorry, I thought you were referring to my earlier DLL-unloading problem. Alex

On Sun, 11 Apr 2010 07:29:49 -0700 (PDT), Artyom wrote:

I've noticed something

...

Here is the repro code: http://www.doc.ic.ac.uk/~awl03/locale_test2.zip

According to project files both Debug and Release versions of Boost.Locale use release versions of ICU.

You're right. I've fixed this by removing the dependencies completely (they should only be in the EXE project) but I still have the same failure in both Debug and Release builds. Alex

On Sun, 11 Apr 2010 12:25:53 -0700 (PDT), Artyom wrote:

...

Now that *that* part (not crashing) is fixed, I'm running into another problem. I can see the .mo file being loaded; excellent! But when I try to translate a string, the translation mechanism fails to find it! This only happens when i set the locale statically. If I set it just before using translate() it works fine.

Here is the repro code: http://www.doc.ic.ac.uk/~awl03/locale_test2.zip

To see the different situations, change which commentted line declaring LocaleSetup.

Ok, you are right, I managed to reproduce it. Looks like something overerites the created locale with other one.

Maybe standard library defines global locale after it is defined in the constructor and this is just a question of an order of operation. I don't know I'll try to figure out.

I'll happily test any fix you come up with.

Note: I've tested on Linux and Windows with GCC with both static and dynamic libraries, this works. Looks like it happens only with MSVC.

In any case... I don't think that defining the global locale in static constructor is generally good idea.

Possibly not but commonly done. My workaround is to declare a static *local variable* instance of LocalSetup inside DllMain. This is created at runtime, unlike true static variables which are created at loadtime.

For unit test you may use the trick I described with testing if the current locale has boost::locale::info facet.

Thanks for the tips :) Alex