Remove use of Sage, reimplement field and polynomial operations

[divergentdave]

This is a first attempt at #431. It works, and it's slower, which is to be expected, but currently it's a lot slower. Unit tests take 22 seconds instead of 2 seconds. I think I may take a second pass at polynomial interpolation, using the method of divided differences, and see if that improves the speed somewhat.

[cjpatton]

This is a first attempt at #431. It works, and it's slower, which is to be expected, but currently it's a lot slower. Unit tests take 22 seconds instead of 2 seconds. I think I may take a second pass at polynomial interpolation, using the method of divided differences, and see if that improves the speed somewhat.

22 is definitely too slow :) Another option is to just implement NTT? We only use it in FLP anyway.

[divergentdave]

Divided differences brings the time down to 10s. These are the remaining ideas I have for algorithmic speedups:

• Conditionally use NTT if the interpolation points match
• Use the extended Euclidean algorithm for multiplicative inverses, instead of Euler's theorem. Sage currently uses the extended Euclidean algorithm via GMP. (addition-chain exponentiation could speed up using Euler's theorem, but we don't care about constant timeness, and extended GCD is likely faster and simpler)
• I recall discovering that Python's modular exponentiation algorithm for 3-argument pow() is slower than what Sage has, but that might have been only relevant for much larger inputs.

I'll do the extended Euclidean algorithm next, as that is easy to do, and we'll see how far that gets us.

By the time you're done, you will have written an ad hoc, informally-specified, bug-ridden, slow implementation of one percent of SageMath.

-- https://cryptopals.com/sets/8

[divergentdave]

Test suite runtime is now 6.7s in CI and 4.1s locally for me. I think this is about as good as we can get it without a lot more complexity.

[cjpatton]

Test suite runtime is now 6.7s in CI and 4.1s locally for me. I think this is about as good as we can get it without a lot more complexity.

4 seconds is doable. Have you tried generating the test vectors? Hopefully that's quick, too.

My main concern about landing this is the runtime of PINE's (https://github.com/junyechen1996/draft-chen-cfrg-vdaf-pine/) unit tests, which at the moment take about 10 seconds. Let's prototype this in that repo before land this.

[divergentdave]

Test vector generation just takes 0.6s for me.

FWIW, here are the slowest tests. We could probably also speed things up by tweaking the dimensions they use, but I think that would be best handled separately.

Slowest test durations
----------------------------------------------------------------------
2.053s     test (tests.test_vdaf_prio3.TestPrio3SumVecWithMultiproof.test)
0.905s     test (tests.test_idpf_bbcggi21.TestIdpfBBCGGI21.test)
0.478s     test (tests.test_vdaf_prio3.TestPrio3SumVec.test)
0.219s     test_poplar1 (tests.test_vdaf_poplar1.TestPoplar1.test_poplar1)

[cjpatton]

My main concern about landing this is the runtime of PINE's (https://github.com/junyechen1996/draft-chen-cfrg-vdaf-pine/) unit tests, which at the moment take about 10 seconds. Let's prototype this in that repo before land this.

I hacked this together here: junyechen1996/draft-chen-cfrg-vdaf-pine@bfe25cd

As expected, the unit tests take longer, about twice as long (now 12 seconds). Test vector generation also takes about twice as long, now 120 seconds. Note that we generate test vectors for higher dimensional data for PINE than we do for Prio3.

I still think we should take this change, given how much more portable it makes the reference code, but let's check with @junyechen1996 to make sure this isn't a deal breaker.

[divergentdave]

For PINE's sake, there's one other algorithmic improvement we may want to consider other than those listed above: multiplying polynomials using NTT instead of the straightforward quadratic algorithm. This only matters when multiplying large polynomials.

[cjpatton]

For PINE's sake, there's one other algorithmic improvement we may want to consider other than those listed above: multiplying polynomials using NTT instead of the straightforward quadratic algorithm. This only matters when multiplying large polynomials.

Right, as we do in libprio already. How much would we have to change the spec? I wouldn't want to have to change too much to accommodate this. In particular, I wouldn't want to make the spec less clear.

[cjpatton]

@junyechen1996 is cool with this change. I'll review tomorrow.

[divergentdave]

Right, as we do in libprio already. How much would we have to change the spec? I wouldn't want to have to change too much to accommodate this. In particular, I wouldn't want to make the spec less clear.

Neither poly_mul() nor poly_interp() are excerpted in the spec, so this is all implementation details of the proof of concept.

[cjpatton]

Right, as we do in libprio already. How much would we have to change the spec? I wouldn't want to have to change too much to accommodate this. In particular, I wouldn't want to make the spec less clear.

Neither poly_mul() nor poly_interp() are excerpted in the spec, so this is all implementation details of the proof of concept.

Got it. Let's keep it that way: if anyone tells us we need to list this, then we can do so.

As far as optimizing poly_mul(), I'm fine with holding off for now.

[cjpatton]

Man, remember when we we're just had a bunch of .sage files? That was truly terrible. Thank you so much for your hard work on the software engineering for this spec.