Cryptography for Boost. Boost.Crypto3.
Hello! Some time ago I promised to show something about cryptography library architecture and implementation for Boost (https://lists.boost.org/Archives/boost//2020/02/248205.php <https://lists.boost.org/Archives/boost//2020/02/248205.php>), so here I am. I’m actually glad my two innocent letters triggered such a conversation. That is what I wanted to see! Please, read this letter very carefully, because some explaining needs to be done before you go to the code. By the way. The code: https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>. Warning. This letter contains mentioning of my non-commercial organizations I develop Boost.Crypto3 with. Don’t consider that as advertising. It is not supposed to be an advertisement. So, what cryptography for Boost (and probably STL some time after) should look like? 1. It should definitely be simple to use and implemented with some generic programming spirit. That is exactly what was done. Remember std::transform? Typical usage technique for all of the modules of a suite looks as follows. Classic iterator pair with output iterator: using namespace boost::crypto3; std::string input = “Weird German 2 byte thing: Ã.“, out; encode<codec::base64>(input.begin(), input.end(), std::inserter(out, out.end())); Classic iterator pair with output value return: using namespace boost::crypto3; std::string input = “00112233445566778899aabbccddeeff”; std::string key = “000102030405060708090a0b0c0d0e0f”; std::string out = encrypt<block::rijndael<128, 128>>(input.begin(), input.end(), key.begin(), key.end()); C++20 Ranges (or Boost.Range) usage: using namespace boost::crypto3; std::vector<std::uint8_t> input = {0x27, 0x0f, 0xb1, 0x89, 0x82, 0x80, 0x0d, 0xa6, 0x40}; std::string out = encode<codec::base32>(input); // or encode<codec::base32>(input, out); What about generic encoding/encryption/signing/hashing/%name_your_action% interface for classes? You can definitely do that in case you define implicit conversion of your class to integral type. I’m thinking of introducing Inspectable concept which will require for the developer to have a function in his class which would return every class member which developer thinks needs to be encrypted/hashes/encoded/etc, converted to the particular algorithm’s format. (e.g. satisfying Integral concept for ciphers/codes/hashes). 2. It should contain only time-proven schemes implemented with well-known and proven techniques. Yes. Boost.Crypto3 (https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>) is derived from =nil; Crypto3 C++ Cryptography Suite (https://github.com/NilFoundation/crypto3 <https://github.com/NilFoundation/crypto3>), which my non-commercial foundation designed and developed for it’s projects. A little bit of information on how exactly it was derived from the original suite you can get from the readme.md in both or repositories. As far as you can see in the original repository, the suite contains some very novel schemes (e.g. Verifiable Delay Functions: https://github.com/nilfoundaton/crypto3-vdf.git <https://github.com/nilfoundaton/crypto3-vdf.git>) and it is going to contain more (threshold cryptography, zero-knowlege cryptography, cryptographic accumulators etc.). But Boost-ified version has to be a solid one. No novel schemes. Only proven ones with proven implementation techniques. Which ones? This is what Boost community is up to decide. So, the question for the community. What schemes should be included? For now we only present small set of block ciphers, hashes and codecs listed in here https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html>, https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html>, https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html <https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html>. (Or you can find the list in the include directories of particular repositories: https://github.com/nilfoundation/crypto3-block.git <https://github.com/nilfoundation/crypto3-block.git>, https://github.com/nilfoundation/crypto3-codec <https://github.com/nilfoundation/crypto3-codec.git>.git <https://github.com/nilfoundation/crypto3-codec.git>, https://github.com/nilfoundation/crypto3-hash <https://github.com/nilfoundation/crypto3-hash.git>.git <https://github.com/nilfoundation/crypto3-hash.git>). Peter (Dimov), I’m aware of your abandoned hash library implementation (https://github.com/pdimov/hash2 <https://github.com/pdimov/hash2>) based on Vinnie’s paper. I would really like to see it’s contents and concepts (probably) adapted to hash module of Boost.Crypto3. Would be a great synergy. 3. It should have a concept-based architecture. Concept-based architecture supposes every reused type trait to be formalized, and standardized. This is what exactly has been done. In particular, concepts for contents of Boost.Crypto3 are listed in sufficient documentation chapters: https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concept... <https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concepts.html>, https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html>, https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html>. Long story short. We standardize stateless policies containing all the params as constexpr ones used for functions (interface-functions like encrypt/decrypt) and actual algorithms. We also figured out that most of cryptographic algorithms are about accumulation and only then processing. So we used Accumulator concept from Boost.Accumulator to accumulate (surprise surprise) data in a format, which is required by particular algorithm to be passed into. You can get the idea of how I’ve merged the Accumulator concept in from architecture and implementation notes (for Block Ciphers) in here: https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.ht... <https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.html>. Pretty drafty as well, but better than nothing. Yes, Hash concept should be made consent with currently existing Hash concept. This is in progress. Vinnie (Falco), I remember, you’ve mentioned some more generic concepts like OneWayFunction (in here: https://lists.boost.org/Archives/boost//2020/02/248228.php <https://lists.boost.org/Archives/boost//2020/02/248228.php>). I agree, we should do that, but we simply had no time to put this fine idea to our architecture. We would really appreciate any help with that. Cristopher (Kormanyos), I’m aware of your CRC catalog library (https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog>) I like it because of how in particular it is implemented. I would really like to merge it into our CRC implementation in Crypto3.Hash. By the way. I wonder if we could sometime remove separate Boost.CRC library and use CRC from more generic Boost.Crypto3? Hypothetically. Just asking. Yeah, we have checksums being implemented in Crypto3.Hash (and in Boost.Crypto3 as well), but they were simply not verified yet, so they are hanging in a PR (https://github.com/NilFoundation/crypto3-hash/pull/68 <https://github.com/NilFoundation/crypto3-hash/pull/68>). 4. Other modules? Roadmap? Yes, there will be other modules being proposed in some time. Public Key cryptography in Boost? Yes. Based on enhanced for cryptographic purposes Boost.Multiprecision (https://github.com/NilFoundation/crypto3-multiprecision <https://github.com/NilFoundation/crypto3-multiprecision>). SSL based on Boost.Crypto3? Yes. Why not. We are already in progress. Major updates are going to be performed with merging in the Boost-ified version of bunch of cross-dependent modules of the original suite. For example, public key cryptography would require enhancing Boost.Multiprecision (which is already in progress) and introducing finite fields library (https://github.com/NilFoundation/crypto3-algebra <https://github.com/NilFoundation/crypto3-algebra>). By the way. The original =nil; Crypto3 suite is highly modular. Every cryptography field is emplaced in separate submodule. There are more than 30 of them now in total (counting closed ones)(yeah, it is huge). Would you like to keep this modularity (like introducing Boost.BlockCiphers, Boost.Hashes and Boost.Codec instead of Boost.Crypto3)? Or do we need to keep everything in header-only Boost.Crypto3? I would vote for the second option. John (Maddock), do you understand now why I was attempting to get accepted those strange PRs in Boost.Multiprecision for the last year? Degski (Sorry, I’m not aware of your name), I remember you’ve mentioned some kind of compression library of yours. I would like to see the compression in the suite. Do you think I could fork it and fit into our architecture? 5. Why do we think we can implement properly design and cryptography? Ask me :). 6. Conclusion. In some time, this would definitely result in C++ Standardization Committee proposal. It is a long way, but I hope to bring it there. What we would like to hear from Boost community is some guidance about what in particular from this field would be useful for Boost, proposals and suggestions about architecture and concepts. Let us standardize some cryptography together (starting with this). Sincerely yours, Mikhail Komarov nemo@nil.foundation
Mikhail Komarov wrote:
Peter (Dimov), I’m aware of your abandoned hash library implementation (https://github.com/pdimov/hash2 <https://github.com/pdimov/hash2>) based on Vinnie’s paper.
It's not abandoned. So far I haven't had time to finish it, but I still intend to.
By the way.> The code: https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>. I really like this design. Having done somethingvery similar (probably not-to-be published),I appreciate very much the true resource-clarityin its design. In some brief glances at the code,I see it is favoring compile-time sizes overdynamic allocation via use of <array>,<type_traits>, etc. Fast and clean! There is sensible use of constexpr too.Honeslty, this looks like an attempt at cryptothat potentially lends itself well to PC as wellas the resource-constrained world of bare-metalalike. Nice. Questions: Am I correct in noting thathashing and block ciphers in this designare seen as generalized stream transformations?If so, do you have thoughts about how (or if)this design extends to digital signaturessuch as ECC having both signature andsignature verification?Are random generators such as Fortuna,Salsa or ChaCha also stream transformationsin this paradigm?
Cristopher (Kormanyos), I’m aware of your> CRC catalog library (https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog>)> I like it because of how in particular it is implemented.> I would really like to merge it into our CRC> implementation in Crypto3.Hash. Thank you. In its present state, however,there's only bitwise calculations.The catalog, as implemented, does utilizea clear, generic, stream-oriented approachwhich would fit with your overall work.But if you need table-driven methodsor table generation, we would need todiscuss these.
Thank you for sharing this work. Kind regards, Chris On Wednesday, September 2, 2020, 2:52:42 AM GMT+2, Mikhail Komarov via Boost <boost@lists.boost.org> wrote: Hello! Some time ago I promised to show something about cryptography library architecture and implementation for Boost (https://lists.boost.org/Archives/boost//2020/02/248205.php <https://lists.boost.org/Archives/boost//2020/02/248205.php>), so here I am. I’m actually glad my two innocent letters triggered such a conversation. That is what I wanted to see! Please, read this letter very carefully, because some explaining needs to be done before you go to the code. By the way. The code: https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>. Warning. This letter contains mentioning of my non-commercial organizations I develop Boost.Crypto3 with. Don’t consider that as advertising. It is not supposed to be an advertisement. So, what cryptography for Boost (and probably STL some time after) should look like? 1. It should definitely be simple to use and implemented with some generic programming spirit. That is exactly what was done. Remember std::transform? Typical usage technique for all of the modules of a suite looks as follows. Classic iterator pair with output iterator: using namespace boost::crypto3; std::string input = “Weird German 2 byte thing: Ã.“, out; encode<codec::base64>(input.begin(), input.end(), std::inserter(out, out.end())); Classic iterator pair with output value return: using namespace boost::crypto3; std::string input = “00112233445566778899aabbccddeeff”; std::string key = “000102030405060708090a0b0c0d0e0f”; std::string out = encrypt<block::rijndael<128, 128>>(input.begin(), input.end(), key.begin(), key.end()); C++20 Ranges (or Boost.Range) usage: using namespace boost::crypto3; std::vector<std::uint8_t> input = {0x27, 0x0f, 0xb1, 0x89, 0x82, 0x80, 0x0d, 0xa6, 0x40}; std::string out = encode<codec::base32>(input); // or encode<codec::base32>(input, out); What about generic encoding/encryption/signing/hashing/%name_your_action% interface for classes? You can definitely do that in case you define implicit conversion of your class to integral type. I’m thinking of introducing Inspectable concept which will require for the developer to have a function in his class which would return every class member which developer thinks needs to be encrypted/hashes/encoded/etc, converted to the particular algorithm’s format. (e.g. satisfying Integral concept for ciphers/codes/hashes). 2. It should contain only time-proven schemes implemented with well-known and proven techniques. Yes. Boost.Crypto3 (https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>) is derived from =nil; Crypto3 C++ Cryptography Suite (https://github.com/NilFoundation/crypto3 <https://github.com/NilFoundation/crypto3>), which my non-commercial foundation designed and developed for it’s projects. A little bit of information on how exactly it was derived from the original suite you can get from the readme.md in both or repositories. As far as you can see in the original repository, the suite contains some very novel schemes (e.g. Verifiable Delay Functions: https://github.com/nilfoundaton/crypto3-vdf.git <https://github.com/nilfoundaton/crypto3-vdf.git>) and it is going to contain more (threshold cryptography, zero-knowlege cryptography, cryptographic accumulators etc.). But Boost-ified version has to be a solid one. No novel schemes. Only proven ones with proven implementation techniques. Which ones? This is what Boost community is up to decide. So, the question for the community. What schemes should be included? For now we only present small set of block ciphers, hashes and codecs listed in here https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html>, https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html>, https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html <https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html>. (Or you can find the list in the include directories of particular repositories: https://github.com/nilfoundation/crypto3-block.git <https://github.com/nilfoundation/crypto3-block.git>, https://github.com/nilfoundation/crypto3-codec <https://github.com/nilfoundation/crypto3-codec.git>.git <https://github.com/nilfoundation/crypto3-codec.git>, https://github.com/nilfoundation/crypto3-hash <https://github.com/nilfoundation/crypto3-hash.git>.git <https://github.com/nilfoundation/crypto3-hash.git>). Peter (Dimov), I’m aware of your abandoned hash library implementation (https://github.com/pdimov/hash2 <https://github.com/pdimov/hash2>) based on Vinnie’s paper. I would really like to see it’s contents and concepts (probably) adapted to hash module of Boost.Crypto3. Would be a great synergy. 3. It should have a concept-based architecture. Concept-based architecture supposes every reused type trait to be formalized, and standardized. This is what exactly has been done. In particular, concepts for contents of Boost.Crypto3 are listed in sufficient documentation chapters: https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concept... <https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concepts.html>, https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html>, https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html>. Long story short. We standardize stateless policies containing all the params as constexpr ones used for functions (interface-functions like encrypt/decrypt) and actual algorithms. We also figured out that most of cryptographic algorithms are about accumulation and only then processing. So we used Accumulator concept from Boost.Accumulator to accumulate (surprise surprise) data in a format, which is required by particular algorithm to be passed into. You can get the idea of how I’ve merged the Accumulator concept in from architecture and implementation notes (for Block Ciphers) in here: https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.ht... <https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.html>. Pretty drafty as well, but better than nothing. Yes, Hash concept should be made consent with currently existing Hash concept. This is in progress. Vinnie (Falco), I remember, you’ve mentioned some more generic concepts like OneWayFunction (in here: https://lists.boost.org/Archives/boost//2020/02/248228.php <https://lists.boost.org/Archives/boost//2020/02/248228.php>). I agree, we should do that, but we simply had no time to put this fine idea to our architecture. We would really appreciate any help with that. Cristopher (Kormanyos), I’m aware of your CRC catalog library (https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog>) I like it because of how in particular it is implemented. I would really like to merge it into our CRC implementation in Crypto3.Hash. By the way. I wonder if we could sometime remove separate Boost.CRC library and use CRC from more generic Boost.Crypto3? Hypothetically. Just asking. Yeah, we have checksums being implemented in Crypto3.Hash (and in Boost.Crypto3 as well), but they were simply not verified yet, so they are hanging in a PR (https://github.com/NilFoundation/crypto3-hash/pull/68 <https://github.com/NilFoundation/crypto3-hash/pull/68>). 4. Other modules? Roadmap? Yes, there will be other modules being proposed in some time. Public Key cryptography in Boost? Yes. Based on enhanced for cryptographic purposes Boost.Multiprecision (https://github.com/NilFoundation/crypto3-multiprecision <https://github.com/NilFoundation/crypto3-multiprecision>). SSL based on Boost.Crypto3? Yes. Why not. We are already in progress. Major updates are going to be performed with merging in the Boost-ified version of bunch of cross-dependent modules of the original suite. For example, public key cryptography would require enhancing Boost.Multiprecision (which is already in progress) and introducing finite fields library (https://github.com/NilFoundation/crypto3-algebra <https://github.com/NilFoundation/crypto3-algebra>). By the way. The original =nil; Crypto3 suite is highly modular. Every cryptography field is emplaced in separate submodule. There are more than 30 of them now in total (counting closed ones)(yeah, it is huge). Would you like to keep this modularity (like introducing Boost.BlockCiphers, Boost.Hashes and Boost.Codec instead of Boost.Crypto3)? Or do we need to keep everything in header-only Boost.Crypto3? I would vote for the second option. John (Maddock), do you understand now why I was attempting to get accepted those strange PRs in Boost.Multiprecision for the last year? Degski (Sorry, I’m not aware of your name), I remember you’ve mentioned some kind of compression library of yours. I would like to see the compression in the suite. Do you think I could fork it and fit into our architecture? 5. Why do we think we can implement properly design and cryptography? Ask me :). 6. Conclusion. In some time, this would definitely result in C++ Standardization Committee proposal. It is a long way, but I hope to bring it there. What we would like to hear from Boost community is some guidance about what in particular from this field would be useful for Boost, proposals and suggestions about architecture and concepts. Let us standardize some cryptography together (starting with this). Sincerely yours, Mikhail Komarov nemo@nil.foundation _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
It is great to see some positive feedback! Thank you! Coming to questions: 1. Yes. You are correct. Processing over the stream of various classes convertible to any Integral type (for now, we will generalize that further in some time). Internal architecture about how these streams are being processed is a little bit more complicated (includes Boost.Accumulator usage, caches data with Merkle-Damgard or Sponge constructions etc), but still being concept-based. Coming to the internal architecture. According to what we’ve been recommended for us to be able to pave the road for this set of concepts to the standard (https://www.reddit.com/r/cpp/comments/ikxgs5/nil_crypto3_conceptbased_pure_c_cryptography/g3ro8dg?utm_source=share&utm_medium=web2x&context=3 <https://www.reddit.com/r/cpp/comments/ikxgs5/nil_crypto3_conceptbased_pure_c_cryptography/g3ro8dg?utm_source=share&utm_medium=web2x&context=3>), the set of concepts used internally will be extended and modified a little bit. 2. Yes. There is even some drafty implementations of such an approach for various cryptography fields in the main =nil; Crypto3 suite (https://github.com/NilFoundation/crypto3 <https://github.com/NilFoundation/crypto3>). They were simply not yet ported to Boost-ified version (https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>). 2.1 For example, here is a drafty public key cryptography implementation (including elliptic curves operations, along with signing and verification): https://github.com/NilFoundation/crypto3-pubkey <https://github.com/NilFoundation/crypto3-pubkey>. (work in progress) 2.2 Here is a drafty cryptography-enhanced fork of Boost.Random (https://github.com/NilFoundation/crypto3-random <https://github.com/NilFoundation/crypto3-random>) which uses drafty stream ciphers (https://github.com/NilFoundation/crypto3-stream <https://github.com/NilFoundation/crypto3-stream>) being implemented with the same approach. I would really like to push stream ciphers into Boost.Crypto3 sometime, so after that I could push Boost.Random, enhanced with Salsa/ChaCha RNGs. 2.3 Same architecture is being also applied to drafty message authentication codes in here: https://github.com/NilFoundation/crypto3-mac <https://github.com/NilFoundation/crypto3-mac>. (drafty) 2.4 Password-based key derivation functions: https://github.com/NilFoundation/crypto3-pbkdf <https://github.com/NilFoundation/crypto3-pbkdf> (also drafty) 2.5 Password hashing: https://github.com/NilFoundation/crypto3-passhash <https://github.com/NilFoundation/crypto3-passhash> (also drafty) 2.6 Usual key derivation functions: https://github.com/NilFoundation/crypto3-kdf <https://github.com/NilFoundation/crypto3-kdf> (drafty) 2.7 Cipher modes: https://github.com/NilFoundation/crypto3-modes <https://github.com/NilFoundation/crypto3-modes> (less drafty) 2.8 Codecs: https://github.com/NilFoundation/crypto3-codec <https://github.com/NilFoundation/crypto3-codec> (not drafty) 2.9 And much more things using similar approach mostly listed in here: https://github.com/NilFoundation/crypto3/tree/master/libs <https://github.com/NilFoundation/crypto3/tree/master/libs>. I would really like to get a clue of what particular fields could be considered useful for Boost, so I could merge them into Boost.Crypto3 and propose for a review afterwards. Coming to CRC. We have our own implementation of CRC being fit into the general architecture (https://github.com/NilFoundation/crypto3-hash/pull/68 <https://github.com/NilFoundation/crypto3-hash/pull/68>), but the architecture is a little bit too heavy for such small-digested hashes (literally checksums) as CRC or Adler or xxHash or MurmurHash. So that is why I pretty much like the way you got it implemented. Sincerely yours, Mikhail Komarov nemo@nil.foundation
On 3 Sep 2020, at 20:55, Christopher Kormanyos <e_float@yahoo.com> wrote:
By the way. The code: https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3><https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>>.
I really like this design. Having done something very similar (probably not-to-be published), I appreciate very much the true resource-clarity in its design. In some brief glances at the code, I see it is favoring compile-time sizes over dynamic allocation via use of <array>, <type_traits>, etc. Fast and clean! There is sensible use of constexpr too. Honeslty, this looks like an attempt at crypto that potentially lends itself well to PC as well as the resource-constrained world of bare-metal alike. Nice.
Questions: Am I correct in noting that hashing and block ciphers in this design are seen as generalized stream transformations? If so, do you have thoughts about how (or if) this design extends to digital signatures such as ECC having both signature and signature verification? Are random generators such as Fortuna, Salsa or ChaCha also stream transformations in this paradigm?
Cristopher (Kormanyos), I’m aware of your CRC catalog library (https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog><https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog>>) I like it because of how in particular it is implemented. I would really like to merge it into our CRC implementation in Crypto3.Hash.
Thank you. In its present state, however, there's only bitwise calculations. The catalog, as implemented, does utilize a clear, generic, stream-oriented approach which would fit with your overall work. But if you need table-driven methods or table generation, we would need to discuss these.
Thank you for sharing this work.
Kind regards, Chris On Wednesday, September 2, 2020, 2:52:42 AM GMT+2, Mikhail Komarov via Boost <boost@lists.boost.org <mailto:boost@lists.boost.org>> wrote:
Hello!
Some time ago I promised to show something about cryptography library architecture and implementation for Boost (https://lists.boost.org/Archives/boost//2020/02/248205.php <https://lists.boost.org/Archives/boost//2020/02/248205.php><https://lists.boost.org/Archives/boost//2020/02/248205.php <https://lists.boost.org/Archives/boost//2020/02/248205.php>>), so here I am. I’m actually glad my two innocent letters triggered such a conversation. That is what I wanted to see!
Please, read this letter very carefully, because some explaining needs to be done before you go to the code.
By the way. The code: https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3><https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>>.
Warning. This letter contains mentioning of my non-commercial organizations I develop Boost.Crypto3 with. Don’t consider that as advertising. It is not supposed to be an advertisement.
So, what cryptography for Boost (and probably STL some time after) should look like?
1. It should definitely be simple to use and implemented with some generic programming spirit.
That is exactly what was done. Remember std::transform? Typical usage technique for all of the modules of a suite looks as follows.
Classic iterator pair with output iterator:
using namespace boost::crypto3; std::string input = “Weird German 2 byte thing: Ã.“, out; encode<codec::base64>(input.begin(), input.end(), std::inserter(out, out.end()));
Classic iterator pair with output value return:
using namespace boost::crypto3; std::string input = “00112233445566778899aabbccddeeff”; std::string key = “000102030405060708090a0b0c0d0e0f”; std::string out = encrypt<block::rijndael<128, 128>>(input.begin(), input.end(), key.begin(), key.end());
C++20 Ranges (or Boost.Range) usage:
using namespace boost::crypto3; std::vector<std::uint8_t> input = {0x27, 0x0f, 0xb1, 0x89, 0x82, 0x80, 0x0d, 0xa6, 0x40}; std::string out = encode<codec::base32>(input); // or encode<codec::base32>(input, out);
What about generic encoding/encryption/signing/hashing/%name_your_action% interface for classes?
You can definitely do that in case you define implicit conversion of your class to integral type. I’m thinking of introducing Inspectable concept which will require for the developer to have a function in his class which would return every class member which developer thinks needs to be encrypted/hashes/encoded/etc, converted to the particular algorithm’s format. (e.g. satisfying Integral concept for ciphers/codes/hashes).
2. It should contain only time-proven schemes implemented with well-known and proven techniques.
Yes. Boost.Crypto3 (https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3><https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>>) is derived from =nil; Crypto3 C++ Cryptography Suite (https://github.com/NilFoundation/crypto3 <https://github.com/NilFoundation/crypto3><https://github.com/NilFoundation/crypto3 <https://github.com/NilFoundation/crypto3>>), which my non-commercial foundation designed and developed for it’s projects.
A little bit of information on how exactly it was derived from the original suite you can get from the readme.md in both or repositories.
As far as you can see in the original repository, the suite contains some very novel schemes (e.g. Verifiable Delay Functions: https://github.com/nilfoundaton/crypto3-vdf.git <https://github.com/nilfoundaton/crypto3-vdf.git><https://github.com/nilfoundaton/crypto3-vdf.git <https://github.com/nilfoundaton/crypto3-vdf.git>>) and it is going to contain more (threshold cryptography, zero-knowlege cryptography, cryptographic accumulators etc.). But Boost-ified version has to be a solid one. No novel schemes. Only proven ones with proven implementation techniques. Which ones? This is what Boost community is up to decide.
So, the question for the community. What schemes should be included?
For now we only present small set of block ciphers, hashes and codecs listed in here https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html><https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html>>, https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html><https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html>>, https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html <https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html><https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html <https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html>>. (Or you can find the list in the include directories of particular repositories: https://github.com/nilfoundation/crypto3-block.git <https://github.com/nilfoundation/crypto3-block.git><https://github.com/nilfoundation/crypto3-block.git <https://github.com/nilfoundation/crypto3-block.git>>, https://github.com/nilfoundation/crypto3-codec <https://github.com/nilfoundation/crypto3-codec><https://github.com/nilfoundation/crypto3-codec.git <https://github.com/nilfoundation/crypto3-codec.git>>.git <https://github.com/nilfoundation/crypto3-codec.git <https://github.com/nilfoundation/crypto3-codec.git>>, https://github.com/nilfoundation/crypto3-hash <https://github.com/nilfoundation/crypto3-hash><https://github.com/nilfoundation/crypto3-hash.git <https://github.com/nilfoundation/crypto3-hash.git>>.git <https://github.com/nilfoundation/crypto3-hash.git <https://github.com/nilfoundation/crypto3-hash.git>>).
Peter (Dimov), I’m aware of your abandoned hash library implementation (https://github.com/pdimov/hash2 <https://github.com/pdimov/hash2><https://github.com/pdimov/hash2 <https://github.com/pdimov/hash2>>) based on Vinnie’s paper. I would really like to see it’s contents and concepts (probably) adapted to hash module of Boost.Crypto3. Would be a great synergy.
3. It should have a concept-based architecture.
Concept-based architecture supposes every reused type trait to be formalized, and standardized. This is what exactly has been done. In particular, concepts for contents of Boost.Crypto3 are listed in sufficient documentation chapters: https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concept... <https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concepts.html><https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concepts.html <https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concepts.html>>, https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html><https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html>>, https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html><https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html>>.
Long story short. We standardize stateless policies containing all the params as constexpr ones used for functions (interface-functions like encrypt/decrypt) and actual algorithms.
We also figured out that most of cryptographic algorithms are about accumulation and only then processing. So we used Accumulator concept from Boost.Accumulator to accumulate (surprise surprise) data in a format, which is required by particular algorithm to be passed into.
You can get the idea of how I’ve merged the Accumulator concept in from architecture and implementation notes (for Block Ciphers) in here: https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.ht... <https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.html><https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.html <https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.html>>. Pretty drafty as well, but better than nothing.
Yes, Hash concept should be made consent with currently existing Hash concept. This is in progress.
Vinnie (Falco), I remember, you’ve mentioned some more generic concepts like OneWayFunction (in here: https://lists.boost.org/Archives/boost//2020/02/248228.php <https://lists.boost.org/Archives/boost//2020/02/248228.php><https://lists.boost.org/Archives/boost//2020/02/248228.php <https://lists.boost.org/Archives/boost//2020/02/248228.php>>). I agree, we should do that, but we simply had no time to put this fine idea to our architecture. We would really appreciate any help with that.
Cristopher (Kormanyos), I’m aware of your CRC catalog library (https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog><https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog>>) I like it because of how in particular it is implemented. I would really like to merge it into our CRC implementation in Crypto3.Hash.
By the way. I wonder if we could sometime remove separate Boost.CRC library and use CRC from more generic Boost.Crypto3? Hypothetically. Just asking.
Yeah, we have checksums being implemented in Crypto3.Hash (and in Boost.Crypto3 as well), but they were simply not verified yet, so they are hanging in a PR (https://github.com/NilFoundation/crypto3-hash/pull/68 <https://github.com/NilFoundation/crypto3-hash/pull/68><https://github.com/NilFoundation/crypto3-hash/pull/68 <https://github.com/NilFoundation/crypto3-hash/pull/68>>).
4. Other modules? Roadmap?
Yes, there will be other modules being proposed in some time. Public Key cryptography in Boost? Yes. Based on enhanced for cryptographic purposes Boost.Multiprecision (https://github.com/NilFoundation/crypto3-multiprecision <https://github.com/NilFoundation/crypto3-multiprecision><https://github.com/NilFoundation/crypto3-multiprecision <https://github.com/NilFoundation/crypto3-multiprecision>>). SSL based on Boost.Crypto3? Yes. Why not. We are already in progress.
Major updates are going to be performed with merging in the Boost-ified version of bunch of cross-dependent modules of the original suite. For example, public key cryptography would require enhancing Boost.Multiprecision (which is already in progress) and introducing finite fields library (https://github.com/NilFoundation/crypto3-algebra <https://github.com/NilFoundation/crypto3-algebra><https://github.com/NilFoundation/crypto3-algebra <https://github.com/NilFoundation/crypto3-algebra>>).
By the way. The original =nil; Crypto3 suite is highly modular. Every cryptography field is emplaced in separate submodule. There are more than 30 of them now in total (counting closed ones)(yeah, it is huge). Would you like to keep this modularity (like introducing Boost.BlockCiphers, Boost.Hashes and Boost.Codec instead of Boost.Crypto3)? Or do we need to keep everything in header-only Boost.Crypto3? I would vote for the second option.
John (Maddock), do you understand now why I was attempting to get accepted those strange PRs in Boost.Multiprecision for the last year?
Degski (Sorry, I’m not aware of your name), I remember you’ve mentioned some kind of compression library of yours. I would like to see the compression in the suite. Do you think I could fork it and fit into our architecture?
5. Why do we think we can implement properly design and cryptography? Ask me :).
6. Conclusion.
In some time, this would definitely result in C++ Standardization Committee proposal. It is a long way, but I hope to bring it there.
What we would like to hear from Boost community is some guidance about what in particular from this field would be useful for Boost, proposals and suggestions about architecture and concepts.
Let us standardize some cryptography together (starting with this).
Sincerely yours,
Mikhail Komarov nemo@nil.foundation <mailto:nemo@nil.foundation>
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost>
Mikhail Komarov wrote:<nemo@nil.foundation>
Some time ago I promised to show something about cryptography library architecture and implementation for Boost
Here's a "meta question" about the idea of having cryptography in Boost: do we think that the "Boost process" (i.e. reviews etc.) is suitable for cryptography, where the issues are somewhat different than other domains? If I were looking for a cryptography library, I don't think that Boost's emphasis on modern C++ best-practice and the "stamp of approval" from our review process would be my top priorities. Rather, I would be looking for a track record of securely-implemented cryptography coming from acknowledged and trusted domain experts. So if I were comparing this with other libraries, my first question would be "Who is Mikhail Komarov?", followed by "what is the Nil Foundation, and why is it registered in the Cayman Islands?". Regards, Phil.
On Sep 4, 2020, at 17:06, Phil Endecott via Boost <boost@lists.boost.org> wrote:
So if I were comparing this with other libraries, my first question would be "Who is Mikhail Komarov?", followed by "what is the Nil Foundation, and why is it registered in the Cayman Islands?".
I would add to that list the question Pink Panther would have asked Q: "What is a Kochivon?" A: it is something spelled backwards
I was waiting for it. And here it is. Cheers! I actually laughed at it. But let us not get down to politics in this mailing list. Sincerely yours, Mikhail Komarov nemo@nil.foundation
On 4 Sep 2020, at 22:59, Kostas Savvidis via Boost <boost@lists.boost.org> wrote:
On Sep 4, 2020, at 17:06, Phil Endecott via Boost <boost@lists.boost.org> wrote:
So if I were comparing this with other libraries, my first question would be "Who is Mikhail Komarov?", followed by "what is the Nil Foundation, and why is it registered in the Cayman Islands?".
I would add to that list the question Pink Panther would have asked Q: "What is a Kochivon?" A: it is something spelled backwards
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Well, coming to that, that is why I propose to put into Boost.Crypto3 only proven schemes with only proven implementation techniques. Most of them, files related to them e.g. in CryptoPP, were not being changed for years. Moreover cryptography in Boost is not only about the implementation. It is also about the architecture, the set of concepts. Coming to “If I were looking for a cryptography…”. You are right, the "stamp of approval” matter nothing in case you are looking for some complex protocol/scheme, but this is not what you would use a generic-purpose cryptography library for. This particular case makes you look for the implementation which you would be able to audit by yourself. But in other cases, bringing CryptoPP/OpenSSL or simply some random library from the Internet to the project in case of need for SHA-family hash or some block cipher is not that handy, as simply use boost::crypto3::sign<ecdsa>() or boost::crypto3::hash<sha2<256>>(), which has well known, well established implementation techniques and no complex protocols at all. I would say in case you look for something complicated in cryptography, you don’t go for general-purpose libraries. And non-general purpose libraries are often being implemented by folks with acknowledgement in very particular fields. So, questions like “Who is Mikhail Komarov?” are not valid because (yeah, the most stock argument is coming) of we would have never knew the Earth is round (for example) if we followed that principle. Like, “Who the heck is Galileo Galilei to tell us that?”. And yet it moves :) (Here it is. The most stock argument in the world. Thank you, thank you. My pleasure.) But, you are right, this particular jurisdiction mentioning does bring too much questions. You are not the first one. I’m already thinking to mention some other one of ours. Sincerely yours, Mikhail Komarov nemo@nil.foundation
On 4 Sep 2020, at 17:06, Phil Endecott via Boost <boost@lists.boost.org> wrote:
Mikhail Komarov wrote:<nemo@nil.foundation>
Some time ago I promised to show something about cryptography library architecture and implementation for Boost
Here's a "meta question" about the idea of having cryptography in Boost: do we think that the "Boost process" (i.e. reviews etc.) is suitable for cryptography, where the issues are somewhat different than other domains?
If I were looking for a cryptography library, I don't think that Boost's emphasis on modern C++ best-practice and the "stamp of approval" from our review process would be my top priorities. Rather, I would be looking for a track record of securely-implemented cryptography coming from acknowledged and trusted domain experts. So if I were comparing this with other libraries, my first question would be "Who is Mikhail Komarov?", followed by "what is the Nil Foundation, and why is it registered in the Cayman Islands?".
Regards, Phil.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 9/4/20 7:06 AM, Phil Endecott via Boost wrote:
Mikhail Komarov wrote:<nemo@nil.foundation>
Some time ago I promised to show something about cryptography library architecture and implementation for Boost
Here's a "meta question" about the idea of having cryptography in Boost: do we think that the "Boost process" (i.e. reviews etc.) is suitable for cryptography, where the issues are somewhat different than other domains?
Lot's of interesting stuff in this little post.
If I were looking for a cryptography library, I don't think that Boost's emphasis on modern C++ best-practice and the "stamp of approval" from our review process would be my top priorities.
I think any C++ library should have that emphasis.
Rather, I would be looking for a track record of securely-implemented cryptography coming from acknowledged and trusted domain experts.
Hmmm - Perhaps some domains are so arcane, mathematical, that we don't have all the resources to properly evaluate them. Should this be case, I'm thinking we might want to recruit some of the missing resources. That make sure the reviewers include some specialized individuals. Not that particularly trust the individuals more than others, but I think with thinks like this not at a bad idea that all the different facets of such an evaluation be covered. Since it's a super important application area - the standards for acceptance would likely be higher than normal - e.g. no more acceptance with 2 reviews.
So if I were comparing this with other libraries, my first question would be "Who is Mikhail Komarov?",
We're evaluating the submission - not the submitter.
followed by "what is the Nil Foundation, and why is it registered in the Cayman Islands?".
LOL - very interesting. Shades of a James Bond villain. I would be curious to know about this though.
Regards, Phil.
Robert Ramey
participants (6)
-
Christopher Kormanyos
-
Kostas Savvidis
-
Mikhail Komarov
-
Peter Dimov
-
Phil Endecott
-
Robert Ramey