compact_optional -- prompting interest

newer
[build][regression] Problems when...

Andrzej Krzemienski

25 Sep 2015 25 Sep '15

2:35 p.m.

Hi Everyone, I would like to inquire if there would be an interest in Boost in another library for storing optional objects, but working under different design goals. Compact optional T has (or can have) the same sizeof as T. It uses one indicated value of T to represent the "empty" (or "singular") value. You can declare it like this: compact_optional> oi; This reads: we have an optional int, with type int inside, where -1 represents the empty value. It can never have a genuine (non-empty value -1). This can be used, for instance, to wrap the std::string::npos into: compact_optional> With the same memory layout as std::string::size_type, but with the special syntax for managing the singular value. It is not meant to be an alternative to Boost.Optional: it targets a different application space. The (single-header) implementation is at: https://github.com/akrzemi1/compact_optional The documentation is at: https://github.com/akrzemi1/compact_optional/blob/master/documentation.md For the motivation and design rationale see this article: https://akrzemi1.wordpress.com/2015/07/15/efficient-optional-values/ Any feedback is welcome. Regards, &rzej

Show replies by date

Marcel Raad

25 Sep 25 Sep

3:07 p.m.

Andrzej Krzemienski wrote

...

I would like to inquire if there would be an interest in Boost in another library for storing optional objects, but working under different design goals.

[...]

For the motivation and design rationale see this article: https://akrzemi1.wordpress.com/2015/07/15/efficient-optional-values/

Any feedback is welcome.

Such a type would be extremely useful for me. This is exactly what I want most of my optional<bool>'s and optional<int>'s to be. -- View this message in context: http://boost.2283326.n4.nabble.com/compact-optional-prompting-interest-tp468... Sent from the Boost - Dev mailing list archive at Nabble.com.

Andrey Semashev

3:37 p.m.

On 25.09.2015 17:35, Andrzej Krzemienski wrote:

...

Hi Everyone, I would like to inquire if there would be an interest in Boost in another library for storing optional objects, but working under different design goals.

Compact optional T has (or can have) the same sizeof as T. It uses one indicated value of T to represent the "empty" (or "singular") value. You can declare it like this:

compact_optional> oi;

This reads: we have an optional int, with type int inside, where -1 represents the empty value. It can never have a genuine (non-empty value -1). This can be used, for instance, to wrap the std::string::npos into:

compact_optional>

With the same memory layout as std::string::size_type, but with the special syntax for managing the singular value.

It is not meant to be an alternative to Boost.Optional: it targets a different application space.

The idea looks interesting - I sometimes have to deal with magic values. Currently I prefer to wrap the object into optional<> or at least make the special value not so magic (i.e. so that the code always does the right thing without checking for the magic value). However, I'm not sure I agree with your rationale on the reduced interface and possibly the compact_optional naming. 1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them. 2. compact_optional does not provide direct assignment of the values of the stored type, requiring to manually construct a compact_optional-wrapped value. To me, this is too cumbersome to use while I don't see any wins from this restriction. Besides more typing, this essentially requires to use a typedef to declare and use the compact_optional variable. 3. Nitpick: the typical name for the getter operation is get(), not value(). I would also have used empty() to test for the magic value but maybe that makes you feel it like a container. 4. Regarding compact_optional naming. While the class can be used for similar purpose as optional, its interface and behavior are significantly different. Perhaps a different name would be better to avoid confusion (e.g. nullable<>). 5. A suggestion: add evp_zero and evp_empty policies. The first uses literal zero as the special value and can be used with numeric (integer and fp) and pointer types. The second uses a default constructed value as the magic value and a member empty() function to test for magic value. This could be useful with containers, strings and ranges.

Andrey Semashev

3:43 p.m.

On 25.09.2015 18:37, Andrey Semashev wrote:

...

On 25.09.2015 17:35, Andrzej Krzemienski wrote:

...
Hi Everyone, I would like to inquire if there would be an interest in Boost in another library for storing optional objects, but working under different design goals.

Compact optional T has (or can have) the same sizeof as T. It uses one indicated value of T to represent the "empty" (or "singular") value. You can declare it like this:

compact_optional> oi;

This reads: we have an optional int, with type int inside, where -1 represents the empty value. It can never have a genuine (non-empty value -1). This can be used, for instance, to wrap the std::string::npos into:

compact_optional>

With the same memory layout as std::string::size_type, but with the special syntax for managing the singular value.

It is not meant to be an alternative to Boost.Optional: it targets a different application space.

The idea looks interesting - I sometimes have to deal with magic values. Currently I prefer to wrap the object into optional<> or at least make the special value not so magic (i.e. so that the code always does the right thing without checking for the magic value).

However, I'm not sure I agree with your rationale on the reduced interface and possibly the compact_optional naming.

1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

2. compact_optional does not provide direct assignment of the values of the stored type, requiring to manually construct a compact_optional-wrapped value. To me, this is too cumbersome to use while I don't see any wins from this restriction. Besides more typing, this essentially requires to use a typedef to declare and use the compact_optional variable.

3. Nitpick: the typical name for the getter operation is get(), not value(). I would also have used empty() to test for the magic value but maybe that makes you feel it like a container.

4. Regarding compact_optional naming. While the class can be used for similar purpose as optional, its interface and behavior are significantly different. Perhaps a different name would be better to avoid confusion (e.g. nullable<>).

5. A suggestion: add evp_zero and evp_empty policies. The first uses literal zero as the special value and can be used with numeric (integer and fp) and pointer types. The second uses a default constructed value as the magic value and a member empty() function to test for magic value. This could be useful with containers, strings and ranges.

6. Also, I think unsafe_raw_value() is a too scary name. There's nothing unsafe about obtaining the special value from the wrapper. I would name it get_raw() (which is in addition to get() which is now value()).

Marcel Raad

3:55 p.m.

Andrey Semashev-2 wrote

...

1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

This is exactly the problem I sometimes have with Boost.Optional: in some use cases, I want the compiler to remind me that my special value is special and always has to be checked explicitly instead of assuming it's always the lowest possible value. That doesn't apply to operator== and operator!=, of course. Andrey Semashev-2 wrote

...

5. A suggestion: add evp_zero and evp_empty policies. The first uses literal zero as the special value and can be used with numeric (integer and fp) and pointer types. The second uses a default constructed value as the magic value and a member empty() function to test for magic value. This could be useful with containers, strings and ranges.

+1, evp_empy would also be useful for a lot of other custom classes I've seen. -- View this message in context: http://boost.2283326.n4.nabble.com/compact-optional-prompting-interest-tp468... Sent from the Boost - Dev mailing list archive at Nabble.com.

Andrey Semashev

4:17 p.m.

On 25.09.2015 18:55, Marcel Raad wrote:

...

Andrey Semashev-2 wrote

...
1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

This is exactly the problem I sometimes have with Boost.Optional: in some use cases, I want the compiler to remind me that my special value is special and always has to be checked explicitly instead of assuming it's always the lowest possible value. That doesn't apply to operator== and operator!=, of course.

In my practice most of the time you just want some kind of ordering (e.g. for binary lookup). For these cases it's useful for ordering to just work out of the box. When the actual order matters you can always provide your own operator overloads - something you will have to do either way.

Gavin Lambert

27 Sep 27 Sep

10:07 p.m.

On 26/09/2015 04:17, Andrey Semashev wrote:

...

On 25.09.2015 18:55, Marcel Raad wrote:

...
Andrey Semashev-2 wrote

...
1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

This is exactly the problem I sometimes have with Boost.Optional: in some use cases, I want the compiler to remind me that my special value is special and always has to be checked explicitly instead of assuming it's always the lowest possible value. That doesn't apply to operator== and operator!=, of course.

In my practice most of the time you just want some kind of ordering (e.g. for binary lookup). For these cases it's useful for ordering to just work out of the box. When the actual order matters you can always provide your own operator overloads - something you will have to do either way.

Isn't the solution there to specialize std::less and/or std::hash instead of providing operators? (I vaguely recall there being a nicer way to do this proposed and/or implemented in one of the newer standards, but I don't recall the details.)

Andrey Semashev

10:15 p.m.

On 28.09.2015 01:07, Gavin Lambert wrote:

...

On 26/09/2015 04:17, Andrey Semashev wrote:

...
On 25.09.2015 18:55, Marcel Raad wrote:

...
Andrey Semashev-2 wrote

...
1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

This is exactly the problem I sometimes have with Boost.Optional: in some use cases, I want the compiler to remind me that my special value is special and always has to be checked explicitly instead of assuming it's always the lowest possible value. That doesn't apply to operator== and operator!=, of course.

In my practice most of the time you just want some kind of ordering (e.g. for binary lookup). For these cases it's useful for ordering to just work out of the box. When the actual order matters you can always provide your own operator overloads - something you will have to do either way.

Isn't the solution there to specialize std::less and/or std::hash instead of providing operators? (I vaguely recall there being a nicer way to do this proposed and/or implemented in one of the newer standards, but I don't recall the details.)

That's one alternative, surely. My point is that this should "just work" by default and allow customization when something special is required.

Rob Stewart

26 Sep 26 Sep

2:10 p.m.

On September 25, 2015 11:37:10 AM EDT, Andrey Semashev wrote:

...

On 25.09.2015 17:35, Andrzej Krzemienski wrote:

...
compact_optional> oi;

This reads: we have an optional int, with type int inside, where -1 represents the empty value. It can never have a genuine (non-empty

value

...
-1). This can be used, for instance, to wrap the std::string::npos into:

compact_optional>

With the same memory layout as std::string::size_type, but with the special syntax for managing the singular value.

There are many such use cases in which a value is reserved. Codifying that in a type is helpful. Applying a typedef means one can provide a context-specific name. All of those things are good. Where this falls down is that distinct uses are the same type, so what should be distinct types are the same type and can interoperate. This is no different than when the underlying type is used, and the magic value is implied by the use case, but the point is that creating distinct types would be preferable. Therefore, tagging specializations would be appropriate.

...

1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

A policy class can also control that decision.

...

3. Nitpick: the typical name for the getter operation is get(), not value(). I would also have used empty() to test for the magic value but maybe that makes you feel it like a container.

I'd go for "is_null". The object is never empty.

...

4. Regarding compact_optional naming. While the class can be used for similar purpose as optional, its interface and behavior are significantly different. Perhaps a different name would be better to avoid confusion (e.g. nullable<>).

That fits my "is_null" suggestion, above, nicely. It fits the null pointer pattern, but doesn't fit the database notion of null, unfortunately.

...

5. A suggestion: add evp_zero and evp_empty policies. The first uses literal zero as the special value and can be used with numeric (integer and fp) and pointer types. The second uses a default constructed value

"Empty" conflicts with "null", above, of course. I dislike "empty" to mean "default constructed". ___ Rob (Sent from my portable computation engine)

Andrzej Krzemienski

4:52 p.m.

2015-09-25 17:37 GMT+02:00 Andrey Semashev :

...

On 25.09.2015 17:35, Andrzej Krzemienski wrote:

...
Hi Everyone, I would like to inquire if there would be an interest in Boost in another library for storing optional objects, but working under different design goals.

Compact optional T has (or can have) the same sizeof as T. It uses one indicated value of T to represent the "empty" (or "singular") value. You can declare it like this:

compact_optional> oi;

This reads: we have an optional int, with type int inside, where -1 represents the empty value. It can never have a genuine (non-empty value -1). This can be used, for instance, to wrap the std::string::npos into:

compact_optional>

With the same memory layout as std::string::size_type, but with the special syntax for managing the singular value.

It is not meant to be an alternative to Boost.Optional: it targets a different application space.

The idea looks interesting - I sometimes have to deal with magic values. Currently I prefer to wrap the object into optional<> or at least make the special value not so magic (i.e. so that the code always does the right thing without checking for the magic value).

Hi Andrey, thank you for your reply.

...

However, I'm not sure I agree with your rationale on the reduced interface and possibly the compact_optional naming.

1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

So this is a trade-off between convenience/expressiveness and the potential to detect unintended semantics at compile-time. If there is sufficient argumentation in favor of adding them, I can always do it. As Rob says, this can also be encoded in the policy.

...

2. compact_optional does not provide direct assignment of the values of the stored type, requiring to manually construct a compact_optional-wrapped value. To me, this is too cumbersome to use while I don't see any wins from this restriction. Besides more typing, this essentially requires to use a typedef to declare and use the compact_optional variable.

Are you proposing a member function like opt.store_raw_value(v); ?

...

3. Nitpick: the typical name for the getter operation is get(), not value(). I would also have used empty() to test for the magic value but maybe that makes you feel it like a container.

4. Regarding compact_optional naming. While the class can be used for similar purpose as optional, its interface and behavior are significantly different. Perhaps a different name would be better to avoid confusion (e.g. nullable<>).

I am not particularly tied to name compact_optional. I can be persuaded to rename it. On the other hand, I am not in favor of any names containing "null". I am not an English speaker, but no me "null" sounds like "numeric value zero", which makes sense for the pointer, but not for something that just is not. Maybe "singular" or "special".

...

5. A suggestion: add evp_zero and evp_empty policies. The first uses literal zero as the special value and can be used with numeric (integer and fp) and pointer types. The second uses a default constructed value as the magic value and a member empty() function to test for magic value. This could be useful with containers, strings and ranges.

Agreed on evp_empty, but I fail to see the advantage of evp_zero over the already existing evp_value_init. The later uses the value initialized T, which already is zero for ints, floats and pointers. Or do you expect the comparison to literal 0 to be faster? Regards, &rzej

Seth

7:04 p.m.

On 09/26/2015 06:52 PM, Andrzej Krzemienski wrote:

...

...
1. You chose not to provide relational operators for compact_optional

...
because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

So this is a trade-off between convenience/expressiveness and the potential to detect unintended semantics at compile-time. If there is sufficient argumentation in favor of adding them, I can always do it. As Rob says, this can also be encoded in the policy.

I'm in favour of the checking and explicit conversions. After all, otherwise one could literally use the underlying type with more convenience and the same level of safety. On 09/26/2015 06:52 PM, Andrzej Krzemienski wrote:

...

4. Regarding compact_optional naming. While the class can be used for

...
similar purpose as optional, its interface and behavior are significantly different. Perhaps a different name would be better to avoid confusion (e.g. nullable<>).

I agree with Andrzej; nullable<> is taken and means different things. How about 'signaling_value<>' . Yes `singular` is a good word but I don't immediately see how to synthesize a clear name for the type with it.

(Maybe discontinuous<>? Well. That is much broader again)

Nevin Liber

8:12 p.m.

On 26 September 2015 at 11:52, Andrzej Krzemienski wrote:

...

I am not particularly tied to name compact_optional.

I'm strongly against the word "optional" appearing the name, for the following reasons: - optional<int> allows me to use every single value that can be stored in an int. This doesn't. - optional<string> allows one to shorten the lifetime of the string it holds. This doesn't. - optional<T> has a nothrow default constructor. This doesn't. At best, it resembles optional only superficially. Please give it a different name. -- Nevin ":-)" Liber mailto:nevin@eviloverlord.com (847) 691-1404

work

10:46 p.m.

...

Nevin Liber wrote:

...
Andrzej Krzemienski wrote: I am not particularly tied to name compact_optional.

I'm strongly against the word "optional" appearing the name, ... At best, it resembles optional only superficially...

So very true. Carving magic values out of the actual type has always (to me) been a hack. That's what IMO boost::optional addresses perfectly. Now I am somewhat uncomfortable that Andrzej is taking that route of providing and legitimizing the hack which instead needs to be replaced with boost::optional. I might have not read the whole thread to the last letter but benefits of that pseudo-optional are still questionable to me. The only one that stood out for me was performance but I am far from convinced that boost::optional introduces any noticeable run-time penalty. I suspect in an application where boost::optional does introduce performance issues there'll be many more serious bottlenecks and restrictions to consider.

Matt Calabrese

27 Sep 27 Sep

11 a.m.

On Sat, Sep 26, 2015 at 1:12 PM, Nevin Liber wrote:

...

I'm strongly against the word "optional" appearing the name, for the following reasons:

- optional<int> allows me to use every single value that can be stored in an int. This doesn't. - optional<string> allows one to shorten the lifetime of the string it holds. This doesn't. - optional<T> has a nothrow default constructor. This doesn't.

At best, it resembles optional only superficially. Please give it a different name.

This. However, I do think that there is a place for a separate intrusive_optional as well, though the empty value should not be a valid value of the type, but rather, just be able to take advantage of its storage, and the type should not be constructed while the in the empty state. I.E. an optional reference only takes up the size of a pointer because it uses the "0" value for empty. Similarly, any type that contains a reference (or a pointer that cannot be null) also can take advantage of this by way of an intrusive optional. The user would have to provide a predicate that operates on the raw storage of the value, and access it in a way that is well-formed whether the object is constructed there or not, so it is a little bit trickier to specify properly. -- -Matt Calabrese

Matt Calabrese

11:02 a.m.

On Sun, Sep 27, 2015 at 4:00 AM, Matt Calabrese wrote:

...

Similarly, any type that contains a reference (or a pointer that cannot be null) also can take advantage of this by way of an intrusive optional.

*logically* contains a reference, I mean. I don't think you can really implement it directly via a reference.

...

-- -Matt Calabrese

Andrey Semashev

11:23 a.m.

On 27.09.2015 14:00, Matt Calabrese wrote:

...

On Sat, Sep 26, 2015 at 1:12 PM, Nevin Liber wrote:

...
I'm strongly against the word "optional" appearing the name, for the following reasons:

- optional<int> allows me to use every single value that can be stored in an int. This doesn't. - optional<string> allows one to shorten the lifetime of the string it holds. This doesn't. - optional<T> has a nothrow default constructor. This doesn't.

At best, it resembles optional only superficially. Please give it a different name.

This. However, I do think that there is a place for a separate intrusive_optional as well, though the empty value should not be a valid value of the type, but rather, just be able to take advantage of its storage, and the type should not be constructed while the in the empty state. I.E. an optional reference only takes up the size of a pointer because it uses the "0" value for empty. Similarly, any type that contains a reference (or a pointer that cannot be null) also can take advantage of this by way of an intrusive optional. The user would have to provide a predicate that operates on the raw storage of the value, and access it in a way that is well-formed whether the object is constructed there or not, so it is a little bit trickier to specify properly.

If the empty state of intrusive_optional means the governed object is not constructed then the usability of this tool will be significantly reduced. Defining the policy of empty state detection would require intricate knowledge of behavior and binary layout of the adapted type, including ABI details such as padding and vtable/virtual inheritance table pointers. For instance, you wouldn't be able to use intrusive_optional with std::string.

Matt Calabrese

8:45 p.m.

On Sun, Sep 27, 2015 at 4:23 AM, Andrey Semashev wrote:

...

If the empty state of intrusive_optional means the governed object is not constructed then the usability of this tool will be significantly reduced. Defining the policy of empty state detection would require intricate knowledge of behavior and binary layout of the adapted type, including ABI details such as padding and vtable/virtual inheritance table pointers. For instance, you wouldn't be able to use intrusive_optional with std::string.

Right, which is why you wouldn't use such an an "intrusive_optional" template in that specific case (if you made a string type you certainly could use it, though). I'm not saying intrusive_optional would replace your template, I'm just agreeing with others that if you do use one of the existing, valid values for the type, it's not really the same kind of abstraction as optional, even though it is useful, so it should probably just have a different name. Places where something like intrusive_optional (which is similar to your type, but more akin to optional) are useful are with types that you have control over. For instance, I have a "card" type that I use for representing one of 52 possible playing cards. It takes up a byte, even though a byte can hold more that 52 different values. If I put a card in an optional as-is, then it's occupying extra space for the discriminator. I'd like to use optional to represent "not-a-card" because had I baked a "not-a-card" value into the original type itself, then any function that takes an actual card would need an additional precondition and I'd likely want to assert to make sure that the card is valid. With an intrusive_optional abstraction, it makes it easy to just use the spare storage in the byte in order to represent not-a-card only when it is needed, and it takes up no extra storage than what the base abstraction does. This is a common desire when making types -- you often want a never empty guarantee so you do not wish to add an empty state to your underlying type. Ideally you could take advantage of optional in a way that would let you get an optional instance of your type without taking up extra space. Specializing optional is one way to do this, but is complicated, and really you only should need to do it via a much simpler customization point. Anyway, this is on a tangent. I am in favor of your type, I just don't think "optional" should be in the name, since your underlying object is: A) Still constructed when in the "none" state B) Uses a normally valid value to represent "none" It's pretty weird that: ////////// compact_optional foo = A(); A& underlying = *foo; underlying.bar(); // foo could technically be "none" now // even though an "A" is constructed and // has a valid value. This is a different // kind of abstraction than an optional. ////////// -- -Matt Calabrese

Andrey Semashev

9:12 p.m.

On 27.09.2015 23:45, Matt Calabrese wrote:

...

On Sun, Sep 27, 2015 at 4:23 AM, Andrey Semashev wrote:

...
If the empty state of intrusive_optional means the governed object is not constructed then the usability of this tool will be significantly reduced. Defining the policy of empty state detection would require intricate knowledge of behavior and binary layout of the adapted type, including ABI details such as padding and vtable/virtual inheritance table pointers. For instance, you wouldn't be able to use intrusive_optional with std::string.

Right, which is why you wouldn't use such an an "intrusive_optional" template in that specific case (if you made a string type you certainly could use it, though). I'm not saying intrusive_optional would replace your template,

That's not my template, it's Andrzej's. :)

...

I'm just agreeing with others that if you do use one of the existing, valid values for the type, it's not really the same kind of abstraction as optional, even though it is useful, so it should probably just have a different name. Places where something like intrusive_optional (which is similar to your type, but more akin to optional) are useful are with types that you have control over.

I suppose you could draw analogy between intrusive_optional & optional and intrusive_ptr & shared_ptr. Fair enough, intrusive_optional requires some amount of control over the adopted type, similar to intrusive_ptr. What's different is the amount of hackishness that is required to support intrusive_optional. intrusive_ptr does not require you to work with the raw storage or know the binary layout of the object while intrusive_optional does. As much as I like the idea of reusing the storage for discriminator, this just feels too fragile to me. Maybe if there was a safer interface for supporting intrusive_optional in user defined types it wouldn't feel that way.

Agustín K-ballo Bergé

9:30 p.m.

On 9/27/2015 6:12 PM, Andrey Semashev wrote:

...

On 27.09.2015 23:45, Matt Calabrese wrote: As much as I like the idea of reusing the storage for discriminator, this just feels too fragile to me. Maybe if there was a safer interface for supporting intrusive_optional in user defined types it wouldn't feel that way.

I toyed with an `intrusive_optional<T>` several years ago. It required `T` be constructible from and equality-comparable to `intrusive_optional_tag`. This tag could only be constructed by `intrusive_optional`. Regards, -- Agustín K-ballo Bergé.- http://talesofcpp.fusionfenix.com

Nevin Liber

10:08 p.m.

On 27 September 2015 at 15:45, Matt Calabrese wrote:

...

For instance, I have a "card" type that I use for representing one of 52 possible playing cards. It takes up a byte, even though a byte can hold more that 52 different values. If I put a card in an optional as-is, then it's occupying extra space for the discriminator. I'd like to use optional to represent "not-a-card" because had I baked a "not-a-card" value into the original type itself, then any function that takes an actual card would need an additional precondition and I'd likely want to assert to make sure that the card is valid. With an intrusive_optional abstraction, it makes it easy to just use the spare storage in the byte in order to represent not-a-card only when it is needed, and it takes up no extra storage than what the base abstraction does.

So, your card type already has a disengaged state it knows how to deal with. Why then do you need to wrap your card type inside another template? And if it doesn't understand the disengaged state, then you have to break a class invariant (the "byte" being in the range 1..52) to do it, which just doesn't work. -- Nevin ":-)" Liber mailto:nevin@eviloverlord.com (847) 691-1404

Matt Calabrese

10:20 p.m.

On Sun, Sep 27, 2015 at 3:08 PM, Nevin Liber wrote:

...

On 27 September 2015 at 15:45, Matt Calabrese wrote:

...
For instance, I have a "card" type that I use for representing one of 52 possible playing cards. It takes up a byte, even though a byte can hold more that 52 different values. If I put a card in an optional as-is, then it's occupying extra space for the discriminator. I'd like to use optional to represent "not-a-card" because had I baked a "not-a-card" value into the original type itself, then any function that takes an actual card would need an additional precondition and I'd likely want to assert to make sure that the card is valid. With an intrusive_optional abstraction, it makes it easy to just use the spare storage in the byte in order to represent not-a-card only when it is needed, and it takes up no extra storage than what the base abstraction does.

So, your card type already has a disengaged state it knows how to deal with. Why then do you need to wrap your card type inside another template?

No, in other words you cannot construct a card with a not-a-card state. Internally there is obviously room in storage to represent it, since we are only using 52 values on the possible set of values that can exist for a byte, but with just this abstraction there is no valid way to get there. On Sun, Sep 27, 2015 at 3:08 PM, Nevin Liber wrote:

...

And if it doesn't understand the disengaged state, then you have to break a class invariant (the "byte" being in the range 1..52) to do it, which just doesn't work.

It doesn't. The idea is that the times that you can make such a hook for intrusive_optional are where it is not UB to access that storage regardless of whether or not the object is constructed. For instance, I use an unsigned char to store the value 0 through 51 and I know that this is at the start of the storage location for the card. As the creator of the card type I can guarantee this (this is why it is intrusive). With that knowledge, I can provide some small, default set of functions that operate on raw storage (not an instance of the card type) but that can do so safely whether the object is constructed or not since I can also via hooks guarantee that there will be an unsigned char in that location regardless. -- -Matt Calabrese

Matt Calabrese

10:35 p.m.

On Sun, Sep 27, 2015 at 3:20 PM, Matt Calabrese wrote:

...

It doesn't. The idea is that the times that you can make such a hook for intrusive_optional are where it is not UB to access that storage regardless of whether or not the object is constructed. For instance, I use an unsigned char to store the value 0 through 51 and I know that this is at the start of the storage location for the card. As the creator of the card type I can guarantee this (this is why it is intrusive). With that knowledge, I can provide some small, default set of functions that operate on raw storage (not an instance of the card type) but that can do so safely whether the object is constructed or not since I can also via hooks guarantee that there will be an unsigned char in that location regardless.

Expanding on this, while it may sound hairy, in practice it should be able to be not too difficult to compose. For instance, if I make a type that contains a card, yet I did not create that card type itself, assuming that I know "card" supports intrusive optional, I would be able to easily point my own intrusive_optional hooks to the location of the "card" that I contain, and forward the functionality along. This leaves you with a higher-level type that again takes up no extra storage for the optional version, and the original type doesn't need a notion of emptiness. -- -Matt Calabrese

Nevin Liber

10:48 p.m.

On 27 September 2015 at 17:20, Matt Calabrese wrote:

...

On Sun, Sep 27, 2015 at 3:08No, in other words you cannot construct a card with a not-a-card state.

Except that every function in its implementation may assume the card is *never* in the not-a-card state. For instance, one could add ASSERT(1 <= cardIndex && cardIndex <= 52) all over the place, including in the beginning of the body of the destructor, and not expect anything to break.

...

Internally there is obviously room in storage to represent it, since we are only using 52 values on the possible set of values that can exist for a byte, but with just this abstraction there is no valid way to get there.

It isn't just about room. The class has to know about it so that it can, at a minimum, weaken its invariants to support it. And if you have to do that, I don't see any need for some generic class to wrap it. -- Nevin ":-)" Liber mailto:nevin@eviloverlord.com (847) 691-1404

Matt Calabrese

11:27 p.m.

On Sun, Sep 27, 2015 at 3:48 PM, Nevin Liber wrote:

...

On 27 September 2015 at 17:20, Matt Calabrese wrote:

...
On Sun, Sep 27, 2015 at 3:08No, in other words you cannot construct a card with a not-a-card state.

Except that every function in its implementation may assume the card is *never* in the not-a-card state.

For instance, one could add ASSERT(1 <= cardIndex && cardIndex <= 52) all over the place, including in the beginning of the body of the destructor, and not expect anything to break.

That's still true. None of these asserts would fire. The value would only ever be "0" (I'm assuming you are implying 0 is the empty state here) when the card is not i na constructed state. In other words, if someone does: "foo = nullopt", it internally checks via raw storage hooks if the object is already logically nullopt, and if not, it destroys the card, and then sets the value via the hook, which operates on raw storage. On Sun, Sep 27, 2015 at 3:48 PM, Nevin Liber wrote:

...

It isn't just about room. The class has to know about it so that it can, at a minimum, weaken its invariants to support it. And if you have to do that, I don't see any need for some generic class to wrap it.

No, it doesn't. That state will never be set while the object is in a constructed state. That is what differs between this and compact_optional (I'm off on a tangent in case that's not clear, and we could probably start a different thread). The difference is in the nature of the hooks. These hooks would deal with raw storage, and do not merely make the overall optional "nullopt" by setting a value in the underlying object. Rather, when setting to nullopt, the object itself is destroyed and we just re-use the storage in a valid and non-conflicting way to identify that we are not in a constructed state. We can check which state the overall object is in (constructed or not constructed) regardless because the hooks operate on raw storage, and with the knowledge that regardless of the underlying object being constructed or not, there is (in this specific case) an unsigned char in a very particular location. -- -Matt Calabrese

David Stone

28 Sep 28 Sep

2:52 a.m.

Matt, I think the use case you describe is more suited to the easily-specializable optional I described in the other thread (although I do prefer the interface described by Agustín K-ballo Bergé, I will have to look into this some more). You have a type that can always have a special compressed optional representation. I believe that compact_optional's use case is that in this particular case, the value of "17" is not valid and is being used as a sentinel, but it's nothing intrinsic to the type, it has to do with the usage of type.

Andrzej Krzemienski

30 Sep 30 Sep

12:14 p.m.

2015-09-27 13:00 GMT+02:00 Matt Calabrese :

...

On Sat, Sep 26, 2015 at 1:12 PM, Nevin Liber wrote:

...
I'm strongly against the word "optional" appearing the name, for the following reasons:

- optional<int> allows me to use every single value that can be stored in an int. This doesn't. - optional<string> allows one to shorten the lifetime of the string it holds. This doesn't. - optional<T> has a nothrow default constructor. This doesn't.

At best, it resembles optional only superficially. Please give it a different name.

This. However, I do think that there is a place for a separate intrusive_optional as well, though the empty value should not be a valid value of the type, but rather, just be able to take advantage of its storage, and the type should not be constructed while the in the empty state. I.E. an optional reference only takes up the size of a pointer because it uses the "0" value for empty. Similarly, any type that contains a reference (or a pointer that cannot be null) also can take advantage of this by way of an intrusive optional. The user would have to provide a predicate that operates on the raw storage of the value, and access it in a way that is well-formed whether the object is constructed there or not, so it is a little bit trickier to specify properly.

I have now implemented this capability in compact_optional. See the related docs section: https://github.com/akrzemi1/compact_optional/blob/master/documentation.md#us... Regards, &rzej

Andrzej Krzemienski

28 Sep 28 Sep

8:23 a.m.

2015-09-26 22:12 GMT+02:00 Nevin Liber :

...

On 26 September 2015 at 11:52, Andrzej Krzemienski wrote:

...
I am not particularly tied to name compact_optional.

I'm strongly against the word "optional" appearing the name, for the following reasons:

- optional<int> allows me to use every single value that can be stored in an int. This doesn't. - optional<string> allows one to shorten the lifetime of the string it holds. This doesn't. - optional<T> has a nothrow default constructor. This doesn't.

At best, it resembles optional only superficially. Please give it a different name.

Acknowledged. Regards, &rzej

Gonzalo BG

10:45 a.m.

...

5. A suggestion: add evp_zero and evp_empty policies. The first uses

Hi Andrzej, I've been using compact optional extensively for almost 2 months already and found it very useful so I'd like to see an improved version of it in Boost. My use case was an application in which the main data structor is a vector (with ~O(10^10) elements), and the application expends most of the time processing it. Compared to compact_optional, Boost.Optional requires 2x more memory, and reduces the memory bandwidth by 2x. Using a vector<int> + boost::dynamic_bitset is infinitely better than using Boost.Optional, but complicates the code significantly since there are some values that the ints in that vector cannot never take. Arguably compact_optional is a perfect fit for my use case. Things that I liked using the library so far: 1. being able to customize the sentinel type via policies: sometimes I have a signed integer that can only be positive and want to use -1 as a sentinel, and some times I want to use std::numeric_limits<...>::max(), or some other value. Andrey Semashev wrote: literal zero as the special value and can be used with numeric (integer and fp) and pointer types. The second uses a default constructed value as the magic value and a member empty() function to test for magic value. This could be useful with containers, strings and ranges. I have mixed feelings about these since someone using compact optional should explicitly know what sentinel value they are using (e.g. for things like hashing, serialization, ...). So I would rather like it to strive for making this as explicit and less magical as possible, in particular for integers and fp types. The empty policy seems very convenient though so maybe the library should include some convenient policies somewhere (this would also make for good examples of how to write policies). 2. Tags: this allowed me to strongly type my interfaces very easily, and caught a lot of errors. 3. unsafe_raw_value together with a way to query the sentinel value, which I needed to implement e.g. `std::hash` for compact_optional. The library is WIP, but I also had some points of friction using it: 4. Operators: I want my compact_optional<int> to work "as an int" but fail fast (assert, throw, terminate) on any operation involving the empty state, so I had to implement my own operators for compact optional. Still, other applications have different needs, and one might want the operators to propagate the empty value, to throw an exception instead of calling assert, or to no exist at all and result in a compiler error. Since some operators must be implemented as member functions, there should be a way to customize this using a policy. Other points: 5. Being part of Boost.Optional: I think compact_optional must be part of Boost.Optional. Stateful policies would allow Boost.Optional to be implemented on top of compact_optional, which would allow everybody to experiment with better storage for their own types. For example sizeof(optional) can be made to be sizeof(T*), and the same could be done for e.g. optional>. Right now experimenting with this is hard because Boost.Optional interface doesn't allow to customize any of this. But basing its implementation on top of compact_optional could change that. 6. Name I really don't mind about compact_optional since 99% of the time one is going to use it through an alias anyways (specifying the type, policy, and tag all the time is just too painful). If Boost.Optional is reimplemented on tope of it, one could honor std::string by calling this basic_optional instead. Vladimir Batov wrote:

...

To me having allocated -1 or std::max<int> (or whatever) to represent no-value, i.e. "no-int" feels hackish. If, as a developer, I advertise "int" to the user, IMO the user is entitled to expect the full "int" range available. If, as a developer, I do not provide the full range, the proper way (IMO) would be to introduce a new explicit type to represent the supported subset (via enum or proper sensibly-named type -- time, speed, temperature, etc.) In other words, if I cannot provide "int", I should not be saying that I do.

Using the library the best thing that I can say is that I'm providing a "compact_optional ". Vladimir Batov wrote:

...

I do have the need to be real frugal when I store data to the disk. But for that purpose I have to be far more economical and size-specific than compact_optional.

I use compact_optional to store data on disk and since sizeof(compact_optional<T>) == sizeof(T) I haven't ran into issues yet. How can one be more economical than that?

Vladimir Batov

29 Sep 29 Sep

2:04 a.m.

On 09/28/2015 08:45 PM, Gonzalo BG wrote:

...

Vladimir Batov wrote:

...
... if I cannot provide "int", I should not be saying that I do.

Using the library the best thing that I can say is that I'm providing a "compact_optional ".

Indeed. Now I realize that "compact_optional<>" introduces a new distinct type. So, my initial concern was unfounded. More so, your example of vector (with ~O(10^10) elements) seems to be a convincing use-case for such a type.

...

Vladimir Batov wrote:

...
I do have the need to be real frugal when I store data to the disk. But for that purpose I have to be far more economical and size-specific than compact_optional.

I use compact_optional to store data on disk and since sizeof(compact_optional<T>) == sizeof(T) I haven't ran into issues yet. How can one be more economical than that?

I had different economy in mind as I am forced to squeeze something represented, say, as "int" in memory into int16_t or int8_t and the likes to be stored on disk.

Gonzalo BG

7:11 a.m.

Gavin Lambert wrote:

...

On 28/09/2015 21:05, Andrzej Krzemienski wrote:

...
No, I was suggesting to allow assignment of values of type T (i.e. "opt = v;"). Having a named function like store_raw_value reduces the need in the typedef but the syntax should still be simpler. [...] In a way, this would mean and implicit conversion from T to compact_optional<T>. I do not want any kind of implicit conversion between the two (because the two are something different). I could offer:

opt.raw_value() = some_T;

This is still clunky; either an implicit conversion or simply an additional assignment operator to permit the other syntax is better.

I am against all three options, since right now my code looks like this: opt_t opt_val; opt_val = opt_t{some_T}; which is not that bad. In particular: - I don't use raw_value for assignment, I only use it for when I need unwrapped access to the storage_type independently of the optional being empty or not, e.g., to implement a hash function or for serialization. - additional assignment operator/implicit conversion: this would make the tags less useful, so I am against it since using tags extensively has made my code better. @Andrzej: Right now your implementation allows constructing a compact_optional from the sentinel value, which results in an empty compact_optional. What is the motivation for this? In my fork I've disabled it for the following reasons: - I'd rather use compact_optional's default constructor to explicitly state that i'm constructing an empty optional, this made it easier to reason about my code, - the other constructors from value_type can then hint the compiler that the compact_optional is not empty (but I did not profile so I don't know if this has any impact at all).

Andrzej Krzemienski

8:16 a.m.

2015-09-29 9:11 GMT+02:00 Gonzalo BG :

...

@Andrzej: Right now your implementation allows constructing a compact_optional from the sentinel value, which results in an empty compact_optional. What is the motivation for this?

In my fork I've disabled it for the following reasons: - I'd rather use compact_optional's default constructor to explicitly state that i'm constructing an empty optional, this made it easier to reason about my code, - the other constructors from value_type can then hint the compiler that the compact_optional is not empty (but I did not profile so I don't know if this has any impact at all).

The motivation for this was to enable an integration of compact_optional with older/other parts of the program that still use magical values: ``` using Index = compact_optional> string s = /*...*/ Index i {s.find("substr")}; ``` In the above example, I will not be changing std, but want to change to compact_optional as soon as possible. But in fact, I could provide a dedicated conversion function for this use case. This aspect is still under consideration, so I may go with your suggestion. Regards, &rzej

David Stone

30 Sep 30 Sep

12:03 a.m.

On Tue, Sep 29, 2015 at 2:16 AM, Andrzej Krzemienski wrote:

...

2015-09-29 9:11 GMT+02:00 Gonzalo BG :

...
@Andrzej: Right now your implementation allows constructing a compact_optional from the sentinel value, which results in an empty compact_optional. What is the motivation for this?

In my fork I've disabled it for the following reasons: - I'd rather use compact_optional's default constructor to explicitly

state

...
that i'm constructing an empty optional, this made it easier to reason about my code, - the other constructors from value_type can then hint the compiler that the compact_optional is not empty (but I did not profile so I don't know if this has any impact at all).

The motivation for this was to enable an integration of compact_optional with older/other parts of the program that still use magical values:

``` using Index = compact_optional> string s = /*...*/ Index i {s.find("substr")}; ```

In the above example, I will not be changing std, but want to change to compact_optional as soon as possible.

But in fact, I could provide a dedicated conversion function for this use case. This aspect is still under consideration, so I may go with your suggestion.

Regards, &rzej

In fact, I think constructing from the magic value is the primary use case for such a class. This makes it easy to integrate with, for instance, a networking protocol that has a magic value meaning "no value here". Now that I've thought about it more, compact_optional is a way of saying that you care exactly how that "not present" state is constructed.

Gonzalo BG

1:10 a.m.

I still would prefer to have a static function to explicitly construct a compact_optional from a value that _may_ be a sentinel value to indicate that the resulting optional (constructed from a value) may indeed be empty. It just seemed more in line with optional to do it this way than the other way around, even though both are equivalent in functionality. Having said this, I have only tried one of the both possible approaches here in practice, and my use case is a small portion of the use cases that compact_optional can cover. For instance string::npos is a nice use case of compact optional and I would like it to work as seamlessly as possible.

Andrey Semashev

26 Sep 26 Sep

9:34 p.m.

On Sat, Sep 26, 2015 at 7:52 PM, Andrzej Krzemienski wrote:

...

2015-09-25 17:37 GMT+02:00 Andrey Semashev :

...
However, I'm not sure I agree with your rationale on the reduced interface and possibly the compact_optional naming.

1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

So this is a trade-off between convenience/expressiveness and the potential to detect unintended semantics at compile-time. If there is sufficient argumentation in favor of adding them, I can always do it. As Rob says, this can also be encoded in the policy.

It can, but I really don't see the reason for it. Can you give an example where operators defined for type T are not sensible to compact_optional?

...

...
2. compact_optional does not provide direct assignment of the values of the stored type, requiring to manually construct a compact_optional-wrapped value. To me, this is too cumbersome to use while I don't see any wins from this restriction. Besides more typing, this essentially requires to use a typedef to declare and use the compact_optional variable.

Are you proposing a member function like opt.store_raw_value(v); ?

No, I was suggesting to allow assignment of values of type T (i.e. "opt = v;"). Having a named function like store_raw_value reduces the need in the typedef but the syntax should still be simpler. You may object that this allows the assignment to make the opt object empty (or null, or singular - not sure what terminology you prefer). This may seem counterintuitive at first glance, but only as long as you treat the type as another flavor of optional<>, which is incorrect and is one reason why the name should be changed. What compact_optional really is is just a wrapper that allows to easily distinguish a special value of the governed object from all other values, and in that light there's nothing wrong if assigning this special value to the wrapper makes it singular. Analogously, there's nothing wrong with assigning a nullptr to a pointer.

...

I am not particularly tied to name compact_optional. I can be persuaded to rename it. On the other hand, I am not in favor of any names containing "null". I am not an English speaker, but no me "null" sounds like "numeric value zero", which makes sense for the pointer, but not for something that just is not. Maybe "singular" or "special".

Well, I'm not a native speaker but AFAIK null does literally mean zero in English. However, in the programming domain I don't see null as something that is necessarily equivalent to a zero value. It's more like a 'special value' to me. E.g. in databases null is a special value that has no connection to zero at all. Even in C++ null pointers are not required to have zero value as the underlying implementation. But I'm not insisting on nullable<>. There's also a close alternative of nilable<>. Singular is the characteristic of a particular value, not the type or range of values, so it's difficult to compose a name from it (I believe, 'singularable' is not a word). Special is too generic, IMHO. We can go a different way: nav_adapter<> (where nav stands for not-a-value) or singular_adapter<>. Although I like it less than nullable.

...

...
5. A suggestion: add evp_zero and evp_empty policies. The first uses literal zero as the special value and can be used with numeric (integer and fp) and pointer types. The second uses a default constructed value as the magic value and a member empty() function to test for magic value. This could be useful with containers, strings and ranges.

Agreed on evp_empty, but I fail to see the advantage of evp_zero over the already existing evp_value_init. The later uses the value initialized T, which already is zero for ints, floats and pointers. Or do you expect the comparison to literal 0 to be faster?

Oh, it didn't occur to me until your reply that evp_value_init already covers the evp_zero use cases. Ok, good, no need for evp_zero.

Matt Hurd

11:41 p.m.

$0.02 == sentinel<> compact_optional is a good idea for 2 reasons: 1) existing code 2) performance / cache friendliness 3) portability, e.g. specific arch dependent illegal IEEE754 Naturally, one is never enough and soon more magic values may be desired... --Matt. On 27 September 2015 at 07:34, Andrey Semashev wrote:

...

...
2015-09-25 17:37 GMT+02:00 Andrey Semashev :

...
However, I'm not sure I agree with your rationale on the reduced

interface

...
and possibly the compact_optional naming.

1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

So this is a trade-off between convenience/expressiveness and the

On Sat, Sep 26, 2015 at 7:52 PM, Andrzej Krzemienski wrote: potential

...
to detect unintended semantics at compile-time. If there is sufficient argumentation in favor of adding them, I can always do it. As Rob says, this can also be encoded in the policy.

It can, but I really don't see the reason for it. Can you give an example where operators defined for type T are not sensible to compact_optional?

...
...
2. compact_optional does not provide direct assignment of the values of the stored type, requiring to manually construct a compact_optional-wrapped value. To me, this is too cumbersome to use while I don't see any wins from this restriction. Besides more typing, this essentially requires to use a typedef to declare and use the compact_optional variable.

Are you proposing a member function like opt.store_raw_value(v); ?

No, I was suggesting to allow assignment of values of type T (i.e. "opt = v;"). Having a named function like store_raw_value reduces the need in the typedef but the syntax should still be simpler.

You may object that this allows the assignment to make the opt object empty (or null, or singular - not sure what terminology you prefer). This may seem counterintuitive at first glance, but only as long as you treat the type as another flavor of optional<>, which is incorrect and is one reason why the name should be changed. What compact_optional really is is just a wrapper that allows to easily distinguish a special value of the governed object from all other values, and in that light there's nothing wrong if assigning this special value to the wrapper makes it singular. Analogously, there's nothing wrong with assigning a nullptr to a pointer.

...
I am not particularly tied to name compact_optional. I can be persuaded to rename it. On the other hand, I am not in favor of any names containing "null". I am not an English speaker, but no me "null" sounds like "numeric value zero", which makes sense for the pointer, but not for something that just is not. Maybe "singular" or "special".

Well, I'm not a native speaker but AFAIK null does literally mean zero in English. However, in the programming domain I don't see null as something that is necessarily equivalent to a zero value. It's more like a 'special value' to me. E.g. in databases null is a special value that has no connection to zero at all. Even in C++ null pointers are not required to have zero value as the underlying implementation.

But I'm not insisting on nullable<>. There's also a close alternative of nilable<>. Singular is the characteristic of a particular value, not the type or range of values, so it's difficult to compose a name from it (I believe, 'singularable' is not a word). Special is too generic, IMHO. We can go a different way: nav_adapter<> (where nav stands for not-a-value) or singular_adapter<>. Although I like it less than nullable.

...
...
5. A suggestion: add evp_zero and evp_empty policies. The first uses literal zero as the special value and can be used with numeric (integer and fp) and pointer types. The second uses a default constructed value as the magic value and a member empty() function to test for magic value. This could be useful with containers, strings and ranges.

Agreed on evp_empty, but I fail to see the advantage of evp_zero over the already existing evp_value_init. The later uses the value initialized T, which already is zero for ints, floats and pointers. Or do you expect the comparison to literal 0 to be faster?

Oh, it didn't occur to me until your reply that evp_value_init already covers the evp_zero use cases. Ok, good, no need for evp_zero.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Rob Stewart

27 Sep 27 Sep

10:35 a.m.

On September 26, 2015 5:34:59 PM EDT, Andrey Semashev wrote:

...

On Sat, Sep 26, 2015 at 7:52 PM, Andrzej Krzemienski wrote:

...
I am not particularly tied to name compact_optional. I can be persuaded to rename it. On the other hand, I am not in favor of any names containing "null". I am not an English speaker, but no me "null" sounds like "numeric value zero", which makes sense for the pointer, but not for something that just is not. Maybe "singular" or "special".

Well, I'm not a native speaker but AFAIK null does literally mean zero in English. However, in the programming domain I don't see null as something that is necessarily equivalent to a zero value. It's more like a 'special value' to me. E.g. in databases null is a special value that has no connection to zero at all. Even in C++ null pointers are not required to have zero value as the underlying implementation.

FWIW, in my experience as a native English speaker, null is rarely synonymous with zero. It connotes having no value or (legal) power.

...

But I'm not insisting on nullable<>. There's also a close alternative of nilable<>. Singular is the characteristic of a particular value, not the type or range of values, so it's difficult to compose a name from it (I believe, 'singularable' is not a word). Special is too generic, IMHO. We can go a different way: nav_adapter<> (where nav stands for not-a-value) or singular_adapter<>. Although I like it less than nullable.

"nav" is a common abbreviation for "navigation" and "navigator", so that would easily cause confusion.

...

...
...
5. A suggestion: add evp_zero and evp_empty policies. The first uses literal zero as the special value and can be used with numeric (integer and fp) and pointer types. The second uses a default constructed value as the magic value and a member empty() function to test for magic value. This could be useful with containers, strings and ranges.

Agreed on evp_empty, but I fail to see the advantage of evp_zero over the already existing evp_value_init. The later uses the value initialized T, which already is zero for ints, floats and pointers. Or do you expect the comparison to literal 0 to be faster?

Oh, it didn't occur to me until your reply that evp_value_init already covers the evp_zero use cases. Ok, good, no need for evp_zero.

An evp_zero_init (or evp_zero_initialization) would cover zeroes for built-in types and default construction of UDTs. ___ Rob (Sent from my portable computation engine)

Andrzej Krzemienski

28 Sep 28 Sep

8:05 a.m.

2015-09-26 23:34 GMT+02:00 Andrey Semashev :

...

...
2015-09-25 17:37 GMT+02:00 Andrey Semashev :

...
However, I'm not sure I agree with your rationale on the reduced

interface

...
and possibly the compact_optional naming.

1. You chose not to provide relational operators for compact_optional because you don't know how to order 'empty' values. I think you don't have to make that decision and simply forward the call to the underlying type. I mean, you always have the stored object constructed in some state and as long as it implements operators you can always use them.

So this is a trade-off between convenience/expressiveness and the

On Sat, Sep 26, 2015 at 7:52 PM, Andrzej Krzemienski wrote: potential

...
to detect unintended semantics at compile-time. If there is sufficient argumentation in favor of adding them, I can always do it. As Rob says, this can also be encoded in the policy.

It can, but I really don't see the reason for it. Can you give an example where operators defined for type T are not sensible to compact_optional?

I am trying to avoid the problem Boost.Optional (and std::experimental::optional) has, where it offers a mixed comparison between T and optional<T>. It is described in this post: https://akrzemi1.wordpress.com/2014/12/02/a-gotcha-with-optional/ Of course, because I provide no converting constructor, the mixed comparisons are removed... However, n this type I want be users to be very explicit about what they want, and it is not that obvious that they want to order the special value the same way as they would if they were using the raw type directly.

...

...
...
2. compact_optional does not provide direct assignment of the values of the stored type, requiring to manually construct a compact_optional-wrapped value. To me, this is too cumbersome to use while I don't see any wins from this restriction. Besides more typing, this essentially requires to use a typedef to declare and use the compact_optional variable.

Are you proposing a member function like opt.store_raw_value(v); ?

No, I was suggesting to allow assignment of values of type T (i.e. "opt = v;"). Having a named function like store_raw_value reduces the need in the typedef but the syntax should still be simpler.

You may object that this allows the assignment to make the opt object empty (or null, or singular - not sure what terminology you prefer). This may seem counterintuitive at first glance, but only as long as you treat the type as another flavor of optional<>, which is incorrect and is one reason why the name should be changed. What compact_optional really is is just a wrapper that allows to easily distinguish a special value of the governed object from all other values, and in that light there's nothing wrong if assigning this special value to the wrapper makes it singular. Analogously, there's nothing wrong with assigning a nullptr to a pointer.

In a way, this would mean and implicit conversion from T to compact_optional<T>. I do not want any kind of implicit conversion between the two (because the two are something different). I could offer: opt.raw_value() = some_T; Regards, &rzej

Gavin Lambert

10:40 p.m.

On 28/09/2015 21:05, Andrzej Krzemienski wrote:

...

...
No, I was suggesting to allow assignment of values of type T (i.e. "opt = v;"). Having a named function like store_raw_value reduces the need in the typedef but the syntax should still be simpler. [...] In a way, this would mean and implicit conversion from T to compact_optional<T>. I do not want any kind of implicit conversion between the two (because the two are something different). I could offer:

opt.raw_value() = some_T;

This is still clunky; either an implicit conversion or simply an additional assignment operator to permit the other syntax is better. It's also logical -- an optional type is essentially a superset of the original type (in the case of boost::optional it's a precise superset, while in this case it's technically an equivalent set with one redefined value, but since that value is supposedly invalid for the original type it's still logically a superset). The only real danger that I can see of supporting the direct assignment syntax is if someone explicitly assigns the sentinel value directly. But while that's bad style it shouldn't be undefined behaviour.

Marc Glisse

25 Sep 25 Sep

4:06 p.m.

On Fri, 25 Sep 2015, Andrzej Krzemienski wrote:

...

Hi Everyone, I would like to inquire if there would be an interest in Boost in another library for storing optional objects, but working under different design goals.

Compact optional T has (or can have) the same sizeof as T. It uses one indicated value of T to represent the "empty" (or "singular") value. You can declare it like this:

compact_optional> oi;

This reads: we have an optional int, with type int inside, where -1 represents the empty value. It can never have a genuine (non-empty value -1). This can be used, for instance, to wrap the std::string::npos into:

compact_optional>

With the same memory layout as std::string::size_type, but with the special syntax for managing the singular value.

It is not meant to be an alternative to Boost.Optional: it targets a different application space.

In my opinion, this is not an alternative to Boost.Optional because it should be part of Boost.Optional. For the interface, you could either add a second template parameter to boost::optional specifying the policy, or specialize boost::optional.

...

The (single-header) implementation is at: https://github.com/akrzemi1/compact_optional

The documentation is at: https://github.com/akrzemi1/compact_optional/blob/master/documentation.md

For the motivation and design rationale see this article: https://akrzemi1.wordpress.com/2015/07/15/efficient-optional-values/

Any feedback is welcome.

-- Marc Glisse

Andrey Semashev

4:19 p.m.

On 25.09.2015 19:06, Marc Glisse wrote:

...

On Fri, 25 Sep 2015, Andrzej Krzemienski wrote:

...
Hi Everyone, I would like to inquire if there would be an interest in Boost in another library for storing optional objects, but working under different design goals.

Compact optional T has (or can have) the same sizeof as T. It uses one indicated value of T to represent the "empty" (or "singular") value. You can declare it like this:

compact_optional> oi;

This reads: we have an optional int, with type int inside, where -1 represents the empty value. It can never have a genuine (non-empty value -1). This can be used, for instance, to wrap the std::string::npos into:

compact_optional>

With the same memory layout as std::string::size_type, but with the special syntax for managing the singular value.

It is not meant to be an alternative to Boost.Optional: it targets a different application space.

In my opinion, this is not an alternative to Boost.Optional because it should be part of Boost.Optional. For the interface, you could either add a second template parameter to boost::optional specifying the policy, or specialize boost::optional.

I'm not opposed to making it part of Boost.Optional library, but not as a specialization of optional<>. The proposed class is very different from optional<>.

work

10:45 p.m.

Andrzej, On 2015-09-26 00:35, Andrzej Krzemienski wrote:

...

I would like to inquire if there would be an interest in Boost in another library for storing optional objects, but working under different design goals.

... It uses one indicated value of T to represent the "empty" (or "singular") value.

In other words, your proposed design is intrusive as it carves something from T for its own purposes. 1) IMO non-intrusive design is by far preferred even if it comes with extra-cost (shared_ptr might serve as an example). It's not to say intrusive design has no place. It does... but seems niche as opposed to a generic solution... which Boost seems to be positioning itself (IMO). 2) Obviously that technique has been used for a long time and it's not immediately obvious to me if the extra interface adds much real value or most importantly clarity/safety. To me having allocated -1 or std::max<int> (or whatever) to represent no-value, i.e. "no-int" feels hackish. If, as a developer, I advertise "int" to the user, IMO the user is entitled to expect the full "int" range available. If, as a developer, I do not provide the full range, the proper way (IMO) would be to introduce a new explicit type to represent the supported subset (via enum or proper sensibly-named type -- time, speed, temperature, etc.) In other words, if I cannot provide "int", I should not be saying that I do. 3) I am trying to see if compact_optional might fit in my projects and fail. When in memory, I by far prefer boost::optional for its orthogonality -- it adds functionality without taking anything from T. I do have the need to be real frugal when I store data to the disk. But for that purpose I have to be far more economical and size-specific than compact_optional.

...

... It is not meant to be an alternative to Boost.Optional: it targets a different application space.

I am not sure I immediately agree with you positioning your proposed "compact_optional" as no competitor to the established "optional". After all, your proposed "compact_optional" tries to address the same problem of representing no-value. You even call it similarly. Surely, it for a reason. Having said that I can easily be wrong and I am prepared to be convinced otherwise. V.

Andrzej Krzemienski

26 Sep 26 Sep

4:31 p.m.

2015-09-26 0:45 GMT+02:00 work :

...

Andrzej,

On 2015-09-26 00:35, Andrzej Krzemienski wrote:

...
I would like to inquire if there would be an interest in Boost in another library for storing optional objects, but working under different design goals.

... It uses one indicated value of T to represent the "empty" (or "singular") value.

In other words, your proposed design is intrusive as it carves something from T for its own purposes.

Indeed, I have also considered name "intrusive_optional".

...

1) IMO non-intrusive design is by far preferred even if it comes with extra-cost (shared_ptr might serve as an example). It's not to say intrusive design has no place. It does... but seems niche as opposed to a generic solution... which Boost seems to be positioning itself (IMO).

Indeed; also compact_optional library comes with the advice that you should try boost::optional first, and only if you have performance problems with it, should you try to use compact_optional instead.

...

2) Obviously that technique has been used for a long time and it's not immediately obvious to me if the extra interface adds much real value or most importantly clarity/safety. To me having allocated -1 or std::max<int> (or whatever) to represent no-value, i.e. "no-int" feels hackish. If, as a developer, I advertise "int" to the user, IMO the user is entitled to expect the full "int" range available. If, as a developer, I do not provide the full range, the proper way (IMO) would be to introduce a new explicit type to represent the supported subset (via enum or proper sensibly-named type -- time, speed, temperature, etc.) In other words, if I cannot provide "int", I should not be saying that I do.

Ok, I think I see where you are coming from. This is an interesting topic in itself; but for now I only stick to advertising compact_optional. Using the words from your description above, when you have decided to use the hack with "magic value", you can use compact_optional to make this explicit and visible to the type-system: now "int" and "int with magic value" are never confused.

...

3) I am trying to see if compact_optional might fit in my projects and fail. When in memory, I by far prefer boost::optional for its orthogonality -- it adds functionality without taking anything from T.

And more: boost::optional does not force you to take the time and consider and declare how you want to store the no-value.

...

I do have the need to be real frugal when I store data to the disk. But for that purpose I have to be far more economical and size-specific than compact_optional.

Indeed. We could say that compact_optional gives you a "generic optimization": it is a fairly generic and mechanical optimization over boost::optional, but to maximally optimize your particular case, you need to do it manually.

...

...

...
It is not meant to be an alternative to Boost.Optional: it targets a different application space.

I am not sure I immediately agree with you positioning your proposed "compact_optional" as no competitor to the established "optional". After all, your proposed "compact_optional" tries to address the same problem of representing no-value. You even call it similarly. Surely, it for a reason.

If not anything else, this often came out as an extension request to boost::optional and std::experimental::optional. My idea of the two libraries is this: When you design a new data structure, you just use boost::optional to represent optional values. Whet it never causes performance issues, you leave it at that. Another situation is where you already encode a magic value in your type int, you consider changing it to boost::optional, but are concerned about adding potential run-time overhead. Your concerns may be misplaced, but only having a concern (valid or not) is often enough to reject a safer library solution (boost::optional). compact_optional does not give you this concerns: it gives you a different trade-off between type-safety and "performance concerns".

...

Having said that I can easily be wrong and I am prepared to be convinced otherwise.

V.

Reards, &rzej

Domagoj Saric

1 Oct 1 Oct

10:13 a.m.

On 25.9.2015. 16:35, Andrzej Krzemienski wrote:

...

Hi Everyone, I would like to inquire if there would be an interest in Boost in another library for storing optional objects, but working under different design goals.

Compact optional T has (or can have) the same sizeof as T. It uses one indicated value of T to represent the "empty" (or "singular") value. You can declare it like this:

compact_optional> oi; <...> It is not meant to be an alternative to Boost.Optional: it targets a different application space.

Hi Andrzej, +1 @ the improvement (as just one of the many improvements and 'polishes' long overdue for boost::optional) -1 @ making it a new/different class template If the improvement/change does not prevent providing the same interface and semantics as optional while offering same-or-better efficiency - then it's just simply better. IOW, I'm all for adding a new 'policies' template parameter (properly defaulted for backwards compatibility) to optional (or perhaps trait(s) classes instead of a new template parameter - w can debated that ;). The one-size-fits-all shared_ptr and make-every-one-pay-for-everything-no-matter-how-fat-it-is iostreams, filesystem, Qt,... logic is IMNHO so C#/Java like and so 'unworthy' of C++...so can we please remove that mentality from optional before it gets into the standard? As the discussions and (already implemented) proposals from a couple of years back show, eg.: http://lists.boost.org/Archives/boost/2012/01/189857.php http://lists.boost.org/Archives/boost/2009/10/156750.php there are many other possible and needed improvements for optional and it is not nearly all about memory efficiency. There are many codegen issues, for example if I work with floats that must never be NaN, I'd want to wrap them in optional _without_ the compiler suddenly adding braindead branching in my code and passing the never-nan-floats through the stack (instead of in registers like 'raw' floats). The approach with different class templates will actually cause uglier user code in the long run (as opposed to a relatively minor _one_time_ hurdle in only some projects that would be broken by a new defaulted parameter) - we'll end up with projects that use optional, pink_optional, compact_optional, smelly_optional and hal9000_optional... with someone having to track and document where and why each of those is used. In addition, the different class templates and/or explicit specializations approaches either require either: - duplication (of the shared optional interface and functionality that optional-like classes must have) - a private base/impl class w/ policies which the public 'different class templates' and/or 'explicit optional specializations' can reuse. If duplication is 'almost always evil', 'all I'm saying' is make the 'private base/impl class w/ policies' the public boost/std::optional proper... C++ >In The Kernel< Now! -- Domagoj Saric Software Architect www.LittleEndian.com

Jeremy Maitin-Shepard

2 Oct 2 Oct

5:04 a.m.

On Thu, Oct 1, 2015 at 3:13 AM, Domagoj Saric < domagoj.saric@littleendian.com> wrote:

...

The approach with different class templates will actually cause uglier user code in the long run (as opposed to a relatively minor _one_time_ hurdle in only some projects that would be broken by a new defaulted parameter) - we'll end up with projects that use optional, pink_optional, compact_optional, smelly_optional and hal9000_optional... with someone having to track and document where and why each of those is used.

It seems you are saying you want to be able to define some optional-specific policy specification called Policy, e.g. NeverNan, and then be able to say optional. This type would behavior similarly, but not identically, to optional<float>. It seems the alternative would be to just define OptionalNeverNan<T> as an alias, using whatever mechanism exists to define it. Users don't have to care if it is aliasing optional or compact_optional or whatever. Calling it optional instead of something_else doesn't really help generic code, since generic code is going to use optional<T>; it will have no way of knowing what Policy to use. If you have to tell it the optional policy to use, you could just as well tell it the full optional type to use. That is why the real win comes from specializing optional<T> for certain user-defined types T. You would call your type NeverNan<float> and then (somehow) specialize optional.

3327

Age (days ago)

3334

Last active (days ago)

List overview

Download

43 comments

17 participants

participants (17)

Agustín K-ballo Bergé
Andrey Semashev
Andrzej Krzemienski
David Stone
Domagoj Saric
Gavin Lambert
Gonzalo BG
Jeremy Maitin-Shepard
Marc Glisse
Marcel Raad
Matt Calabrese
Matt Hurd
Nevin Liber
Rob Stewart
Seth
Vladimir Batov
work