More than strange performance test results of Boost.MultiIndex
I was rewriting our production code since there were performance concerns and switched from our monstrous std::map to boost multi_index_containers which looked exactly what we need. Then when running performance tests I've noticed that once I use composite_key which includes string the performance drops in orders of magnitude, to be precise, 5 orders. Which, I would say, not reasonable at all. So I created a minimal repro case to post here and ran it on Coliru. Surprise! This behavior does not reproduce on GCC, moreover, the composite key with string for some reason was searched twice faster than int only composite. To me it sounds like a bug or in Boost.MultiIndex MS specific implementation (if such exists) or in MSVC. Tech details: VS2012, Win7x64. The code was compiled and linked targeting x64 architecture. Test output on my local machine: Populating data. Index size: 999999 Running FlyweightMICTest::deepTest Checking getTestData perf. 999999 sum of all IDs. 1M calls took 0 seconds or 125ns per call. ------------------------------------------------------------- Populating data. Index size: 999999 Running FlyweightMICTestWithString::deepTest Checking getTestData perf. 99 sum of all IDs. 99 calls took 1 seconds or 10132326ns per call. =============================================================== Coliru output (live example can be found here http://coliru.stacked-crooked.com/a/ef76d297279f15a8): Populating data. Index size: 999999 Running deepTest Checking getTestData perf. 999999 sum of all IDs. 1M calls took 0 seconds or 395ns per call. ------------------------------------------------------------- Populating data. Index size: 999999 Running deepTest Checking getTestData perf. 999999 sum of all IDs. 999999 calls took 0 seconds or 181ns per call. =============================================================== [Read More]<http://feeds.feedburner.com/~r/sizmek-blog/~6/1> [http://www.sizmek.com/Sizmek.png]<http://www.sizmek.com/>
On Tue, Nov 18, 2014 at 07:31:47AM +0000, Ernest Zaslavsky wrote:
I was rewriting our production code since there were performance concerns and switched from our monstrous std::map to boost multi_index_containers which looked exactly what we need. Then when running performance tests I've noticed that once I use composite_key which includes string the performance drops in orders of magnitude, to be precise, 5 orders. Which, I would say, not reasonable at all. So I created a minimal repro case to post here and ran it on Coliru. Surprise! This behavior does not reproduce on GCC, moreover, the composite key with string for some reason was searched twice faster than int only composite. To me it sounds like a bug or in Boost.MultiIndex MS specific implementation (if such exists) or in MSVC.
I wonder if GCC's stdlib still insists on using their abominable Copy-on-Write string implementation. In any way, ensure that your tests on Windows are against a release runtime and without the horrible heap you get when launching from a debugger (_NO_DEBUG_HEAP=1 environment variable to mitigate that).
Tech details: VS2012, Win7x64. The code was compiled and linked targeting x64 architecture. Test output on my local machine: Populating data. Index size: 999999 Running FlyweightMICTest::deepTest
Checking getTestData perf. 999999 sum of all IDs. 1M calls took 0 seconds or 125ns per call.
------------------------------------------------------------- Populating data. Index size: 999999 Running FlyweightMICTestWithString::deepTest
Checking getTestData perf. 99 sum of all IDs. 99 calls took 1 seconds or 10132326ns per call.
===============================================================
Coliru output (live example can be found here http://coliru.stacked-crooked.com/a/ef76d297279f15a8): Populating data. Index size: 999999 Running deepTest
Checking getTestData perf. 999999 sum of all IDs. 1M calls took 0 seconds or 395ns per call.
------------------------------------------------------------- Populating data. Index size: 999999 Running deepTest
Checking getTestData perf. 999999 sum of all IDs. 999999 calls took 0 seconds or 181ns per call.
-- Lars Viklund | zao@acc.umu.se
On Tuesday, November 18, 2014 10:19 PM, Lars Viklund wrote:
I wonder if GCC's stdlib still insists on using their abominable Copy-on-Write string implementation.
Looks like the SSO string is now in (as of 14-Nov), so I suspect that means it'll appear in gcc 5.0. https://gcc.gnu.org/ml/gcc-patches/2014-11/msg01785.html Ben
Do you dare to suspect me to be running against debug build!? :) just kidding. And surprise, surprise. I've found the problem. One thing I've forgot to mention and no one asked for it, It was boost version I'm using - 1.52. Don't know if it is a known bug in this version, but once I've switched to 1.57 everything started to work as expected. So, I rest my case. -----Original Message----- From: Boost-users [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Lars Viklund Sent: Tuesday, November 18, 2014 4:19 PM To: boost-users@lists.boost.org Subject: Re: [Boost-users] More than strange performance test results of Boost.MultiIndex On Tue, Nov 18, 2014 at 07:31:47AM +0000, Ernest Zaslavsky wrote:
I was rewriting our production code since there were performance concerns and switched from our monstrous std::map to boost multi_index_containers which looked exactly what we need. Then when running performance tests I've noticed that once I use composite_key which includes string the performance drops in orders of magnitude, to be precise, 5 orders. Which, I would say, not reasonable at all. So I created a minimal repro case to post here and ran it on Coliru. Surprise! This behavior does not reproduce on GCC, moreover, the composite key with string for some reason was searched twice faster than int only composite. To me it sounds like a bug or in Boost.MultiIndex MS specific implementation (if such exists) or in MSVC.
I wonder if GCC's stdlib still insists on using their abominable Copy-on-Write string implementation. In any way, ensure that your tests on Windows are against a release runtime and without the horrible heap you get when launching from a debugger (_NO_DEBUG_HEAP=1 environment variable to mitigate that).
Tech details: VS2012, Win7x64. The code was compiled and linked targeting x64 architecture. Test output on my local machine: Populating data. Index size: 999999 Running FlyweightMICTest::deepTest
Checking getTestData perf. 999999 sum of all IDs. 1M calls took 0 seconds or 125ns per call.
------------------------------------------------------------- Populating data. Index size: 999999 Running FlyweightMICTestWithString::deepTest
Checking getTestData perf. 99 sum of all IDs. 99 calls took 1 seconds or 10132326ns per call.
===============================================================
Coliru output (live example can be found here http://coliru.stacked-crooked.com/a/ef76d297279f15a8): Populating data. Index size: 999999 Running deepTest
Checking getTestData perf. 999999 sum of all IDs. 1M calls took 0 seconds or 395ns per call.
------------------------------------------------------------- Populating data. Index size: 999999 Running deepTest
Checking getTestData perf. 999999 sum of all IDs. 999999 calls took 0 seconds or 181ns per call.
-- Lars Viklund | zao@acc.umu.se _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On Tue, Nov 18, 2014 at 8:31 AM, Ernest Zaslavsky < ernest.zaslavsky@sizmek.com> wrote:
[...]. Then when running performance tests I've noticed that once I use composite_key which includes string the performance drops in orders of magnitude, to be precise, 5 orders.
Do you have a requirement to return the wstring by value? Why not by const&? Goes back to Lars' COW string remark. Also, your "random" string is not random at all, but constant in fact. Isn't that a pathological case? Unless your indexed members change often, you could also cache the hash. Especially if your string can be long. --DD
Nah, I know ☺ It was taking ages to populate test data when I was generating it really random. Actually It doesn’t really matter since it may happen in production – you will have 5-15 strings across 1M of objects. From: Boost-users [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Dominique Devienne Sent: Tuesday, November 18, 2014 4:52 PM To: boost-users Subject: Re: [Boost-users] More than strange performance test results of Boost.MultiIndex On Tue, Nov 18, 2014 at 8:31 AM, Ernest Zaslavsky <ernest.zaslavsky@sizmek.com<mailto:ernest.zaslavsky@sizmek.com>> wrote: [...]. Then when running performance tests I've noticed that once I use composite_key which includes string the performance drops in orders of magnitude, to be precise, 5 orders. Do you have a requirement to return the wstring by value? Why not by const&? Goes back to Lars' COW string remark. Also, your "random" string is not random at all, but constant in fact. Isn't that a pathological case? Unless your indexed members change often, you could also cache the hash. Especially if your string can be long. --DD
On Wed, Nov 19, 2014 at 6:00 AM, Ernest Zaslavsky < ernest.zaslavsky@sizmek.com> wrote:
[...] it may happen in production – you will have 5-15 strings across 1M of objects.
Just be aware that for low cardinality of distinct values in many rows, and a non-unique index, you can have much better performance for some access patterns with ordered indices compared to hashed ones. We had such an issue, easily fixed by going to an ordered index. FWIW. --DD
OMG, thanks, it wasn’t my intention to use hashed index there, of course I would use odered_non_unique From: Boost-users [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Dominique Devienne Sent: Wednesday, November 19, 2014 10:12 AM To: boost-users Subject: Re: [Boost-users] More than strange performance test results of Boost.MultiIndex On Wed, Nov 19, 2014 at 6:00 AM, Ernest Zaslavsky <ernest.zaslavsky@sizmek.com<mailto:ernest.zaslavsky@sizmek.com>> wrote: [...] it may happen in production – you will have 5-15 strings across 1M of objects. Just be aware that for low cardinality of distinct values in many rows, and a non-unique index, you can have much better performance for some access patterns with ordered indices compared to hashed ones. We had such an issue, easily fixed by going to an ordered index. FWIW. --DD
participants (4)
-
Ben Pope -
Dominique Devienne -
Ernest Zaslavsky -
Lars Viklund