Enhancement to add Format Preserving Encryption (FF3-1) support

[holt-and-catch-fire]

Proposed Enhancement

Enhancement to PyCryptodome to include support for Format Preserving Encryption. Specifically, this pull request adds support for the FF3-1 algorithm as defined in the NIST Special Publication 800-38G Revision 1 (https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-38Gr1-draft.pdf).

This is one of the items listed under "Future Plans" and this update aims to implement FPE.

Implementation

The implementation is written in pure Python and follows the algorithm as outlined in NIST 800-38G (https://csrc.nist.gov/publications/detail/sp/800-38g/rev-1/draft) . NIST revised the original FF3 standard to modify the tweak (based on attacks by Durak and Vaudenay: https://eprint.iacr.org/2017/521.pdf) as well as the domain size.

Crypto/Cipher/ff3.py: Implementation of FF3-1 algorithm in PyCryptodome. It supports only the AES block cipher (in ECB mode) with keys sizes of 128, 192, and 256 bits. The constructor takes as input the alphabet, radix, and key size. The encrypt and decrypt methods take in a tweak and the plaintext or ciphertext (respectively) and returns the encrypted or decrypted result.

The implementation includes several utility functions which are used by the encrypt and decrypt algorithms:

• NUM-RADIX(X): Takes a numeral string, and returns a number, x.
• NUM(X): Takes a byte string X and returns an integer, x.
• STR-M-RADIX(x): Takes an integer and converts to a numeral string
• REV(X): Takes a numeral string X, and returns a numeral string Y which is reversed
• REVB(X): Takes a byte string, X, represented in bits, and returns a byte string Y which is reversed

The algorithm for FF3-1 calls for an approved block cipher in CBC-MAC mode. Since only a single block is ever passed to the algorithm, in practice this is equivalent to ECB mode. Only AES is presently approved by NIST for the block cipher operation. Additional ciphers could be supported, but would not be validated by NIST.

It should be noted that the NIST ACVP testing vectors (see below) limits the algorithm to an alphabet of 64 characters. The FF3-1 implementation is limited to sample cases which meet the NIST ACVP definitions.

Testing

PyCryptodome primarily utilizes the NIST CAVP test vectors. NIST is no longer publishing new CAVP test vectors, and is transitioning to the Automated Cryptographic Validation Protocol (https://pages.nist.gov/ACVP/). NIST ACVP vectors for FF3-1 were taken from (https://github.com/usnistgov/ACVP-Server/tree/master/gen-val/json-file) and all 450 test vectors are passing.

As this test support is not currently integrated within PyCryptodome, the test vectors have been added as an additional directory under pycryptodome_test_vectors/Cipher/ACVP/FF3

Documentation

Doc/src/cipher/ff3.rst: Provides documentation on FPE, alphabets, and example use cases for conducting encrypt and decrypt operations.

Licensing

Care was taken to avoid other implementations of FF3 (and FF3-1) during the development process. Only academic papers and NIST 800-38G was used in developing the implementation. It should qualify for release under the BSD-2 clause license.

Note: NIST 800-38G also recommends the FF1 algorithm which is based on FFX. Voltage Systems holds a patent claim against FF1, and thus it is not eligible to be released under the PyCryptodome BSD-2 clause license.

Comments

This is my first pull request to PyCryptodome. I would appreciate comments particularly around the test cases and documentation which I was not able to generate locally.

I appreciate your consideration.

[holt-and-catch-fire]

Converting to draft. Need to add support for pycryptodome_test_vectors and ensure ACVP tests are not ran if the module is not present.

Python 2.7 support is also failing - as this enhancement was not written to support Python 2.7 which was deprecated Jan 1, 2020.

[holt-and-catch-fire]

Marking this as ready for review and integration. The test cases now appropriately check as to whether pycryptodome_test_vectors is available, and skips the NIST Automated Cryptographic Validation Protocol (ACVP) test vectors if not available. The implementation conforms with the NIST definitions of alphabet, radix, tweakLen, and capabilities and implements the tweak changes as outlined in the latest version of the NIST 800-38G draft.

This pull request does not support Python 2.7. Python 2.7 was declared end of life on January 1, 2020, and the final release (2.7.18) was on April 20, 2020. This integration relies heavily on conversion of strings to integers and working with bytes and bytearrays, and while it is possible to write this code in a backward-compatible fashion I'd argue that it is cleaner using Python 3 native functionality.

I'll make a post on the PyCryptodome Google Groups to further discuss the enhancement.

Thanks,

Joshua Holt