resize() doesn't zero-initialise the new storage. In this particular case, a == result because the expression being evaluated is something like result = result + other. So resizing result is also resizing a. Hence the compare_unsigned looks at the limbs of a that have just been added by the resize, which are undefined.
It appears that this could be fixed by zeroing the new limbs in resize(), but that is unnecessary in other cases. Is it even correct that a == result is possible in this code?
Yes, inplace subtraction is OK. Here's the fix I'm testing: Index: cpp_int.hpp =================================================================== --- cpp_int.hpp (revision 78943) +++ cpp_int.hpp (working copy) @@ -964,12 +964,16 @@ result.sign(s); return; } + // This isn't used till later, but comparison has to occur before we resize the result, + // as that may also resize a or b if this is an inplace operation: + int c = a.compare_unsigned(b); + // Set up the result vector: result.resize(x); + // Now that a, b, and result are stable, get pointers to their limbs: typename cpp_int_backend<MinBits, Signed, Allocator>::const_limb_pointer pa = a.limbs(); typename cpp_int_backend<MinBits, Signed, Allocator>::const_limb_pointer pb = b.limbs(); typename cpp_int_backend<MinBits, Signed, Allocator>::limb_pointer pr = result.limbs(); bool swapped = false; - int c = a.compare_unsigned(b); if(c < 0) { swap(pa, pb); John.