batch verification: add ed25519 batch verification implementation #3031

id-ms · 2021-10-10T06:36:36Z

Summary

In this change, we plan to integrate the ed25519 batch verification algorithm. This will speed up the process of verifying the digital signature of several transactions by roughly 2.4X.

This PR implements the algorithm according to the following paper:
https://eprint.iacr.org/2020/1244.pdf
This PR fixes a bug in the paper (Listing 1.2) that didn't reject the points (-0,-1) and (-0,1)

Another blog article with similar background: https://hdevalence.ca/blog/2020-10-04-its-25519am

Changes Overview
Changes made on the single ed255519 verification :

Signatures that contain small order group R value will now be accepted.
use the cofactor version of the equation to validate the signature. (multiple the result by the cofactor and compare to natural element)
add ge25519_is_canonical_vartime function (check if a given y point has a canonical or non-canonical encoding ):
- optimize this function using the the quick check.
- reject two non-canonical points by comparison (points (-0,-1) and (-0,1), this resolves a bug in the paper).

Add the batch verification implementation from https://github.com/floodyberry/ed25519-donna to our fork of libsodium.
Changes made on the batch verification (to match the single verification):

reject small order A (PK)
reject non-canonical A (PK)
reject non-canonical R
reject non-canonical S
use the cofactor equation

Test Plan

The following tests were added to the libsodium's test suite.

Test against the White Paper's test vector
Test against all non-canonical options for R
Test against all non-canonical options for S

codecov-commenter · 2021-10-10T07:01:51Z

Codecov Report

Merging #3031 (c273bee) into master (93a372d) will increase coverage by 0.02%.
The diff coverage is 80.55%.

@@            Coverage Diff             @@
##           master    #3031      +/-   ##
==========================================
+ Coverage   47.57%   47.60%   +0.02%     
==========================================
  Files         370      370              
  Lines       60060    60109      +49     
==========================================
+ Hits        28572    28612      +40     
- Misses      28178    28186       +8     
- Partials     3310     3311       +1

Impacted Files	Coverage Δ
crypto/compactcert/builder.go	`63.63% <0.00%> (ø)`
crypto/compactcert/structs.go	`100.00% <ø> (ø)`
crypto/compactcert/verifier.go	`66.66% <0.00%> (ø)`
node/netprio.go	`0.00% <0.00%> (ø)`
data/transactions/verify/txn.go	`44.44% <50.00%> (+0.14%)`	⬆️
crypto/curve25519.go	`61.01% <71.42%> (-0.81%)`	⬇️
crypto/batchverifier.go	`90.62% <90.69%> (-9.38%)`	⬇️
agreement/vote.go	`81.81% <100.00%> (ø)`
config/consensus.go	`84.69% <100.00%> (+0.05%)`	⬆️
crypto/multisig.go	`45.34% <100.00%> (ø)`
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 93a372d...c273bee. Read the comment docs.

crypto/libsodium-fork/src/libsodium/crypto_core/ed25519/ref10/ed25519_ref10.c

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/batch.c

crypto/libsodium-fork/src/libsodium/crypto_core/ed25519/ref10/ed25519_ref10.c

cce · 2021-10-21T13:56:37Z

I tried running make check in libsodium-fork and it failed on the "sign" test because the expected output file (sign.exp) no longer matches the output of the sign test program (with the additional messages it prints out), even though the tests themselves pass.

id-ms · 2021-11-03T10:03:36Z

it looks like the batch verification interface uses unsigned long long *.
@cce @tsachiherman, do you think we should change it to size_t?

crypto/libsodium-fork/test/default/sign.c

cce

generally LGTM, with some questions & suggestions for changes

crypto/libsodium-fork/test/default/sign.c

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/batch.c

crypto/libsodium-fork/src/libsodium/crypto_core/ed25519/ref10/ed25519_ref10_sc.c

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/open.c

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/batch.c

cce

it looks like the batch verification interface uses unsigned long long *. @cce @tsachiherman, do you think we should change it to size_t?

It looks like they use unsigned long long for message lengths throughout most of the libsodium API (and size_t for some memory limits and some hashing APIs) so LGTM to keep mlen as unsigned long long to match the rest of the crypto_sign API.

Also, I noticed it should be const unsigned long long * to match the strict const usage in the exported API functions for input args, submitted suggested changes

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/sign_ed25519_ref10.h

crypto/libsodium-fork/src/libsodium/include/sodium/crypto_sign.h

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/sign_ed25519_ref10.h

…nt.iacr.org/2020/1244.pdf. we now reject non canonical R and accept signatures with a mixed group order (in order to impl a batch verification)

… and use it in the batch verification

… result to one of the small order points.

tsachiherman

looks good to me.

## Summary Create an update path that would enable the following features: 1. Batch Verification (#3031) 2. State Proof Keys (#2990) 3. TEAL 6 (#3397) 4. Expired Participation Keys (#2924) 5. Fix rewards calculation bug (#3403) ## Test Plan Unit tests added.

Summary The ed25519 batch verification implementation in #3031 provides a performance improvement for validating multiple signatures (such as multiple transaction signatures). Since each OneTimeSignature used by agreement votes is actually 3 ed25519 signatures, this hooks up the verifier to the batch verification implementation, yielding a ~12% performance improvement in the included benchmark on my computer. Test Plan Added benchmark, existing tests should pass.

…rand#3759) Summary The ed25519 batch verification implementation in algorand#3031 provides a performance improvement for validating multiple signatures (such as multiple transaction signatures). Since each OneTimeSignature used by agreement votes is actually 3 ed25519 signatures, this hooks up the verifier to the batch verification implementation, yielding a ~12% performance improvement in the included benchmark on my computer. Test Plan Added benchmark, existing tests should pass.

) ## Summary In this change, we plan to integrate the ed25519 batch verification algorithm. This will speed up the process of verifying the digital signature of several transactions by roughly 2.4X. This PR implements the algorithm according to the following paper: https://eprint.iacr.org/2020/1244.pdf This PR fixes a **bug** in the paper (Listing 1.2) that didn't reject the points (-0,-1) and (-0,1) Another blog article with similar background: https://hdevalence.ca/blog/2020-10-04-its-25519am **Changes Overview** Changes made on the single ed255519 verification : 1. Signatures that contain small order group R value will now be accepted. 2. use the cofactor version of the equation to validate the signature. (multiple the result by the cofactor and compare to natural element) 3. add ge25519_is_canonical_vartime function (check if a given y point has a canonical or non-canonical encoding ): - optimize this function using the the quick check. - reject two non-canonical points by comparison (points (-0,-1) and (-0,1), this resolves a bug in the paper). Add the batch verification implementation from https://github.com/floodyberry/ed25519-donna to our fork of libsodium. Changes made on the batch verification (to match the single verification): 1. reject small order A (PK) 2. reject non-canonical A (PK) 3. reject non-canonical R 4. reject non-canonical S 5. use the cofactor equation ## Test Plan The following tests were added to the libsodium's test suite. 1. Test against the White Paper's test vector 2. Test against all non-canonical options for R 3. Test against all non-canonical options for S

…rand#3759) Summary The ed25519 batch verification implementation in algorand#3031 provides a performance improvement for validating multiple signatures (such as multiple transaction signatures). Since each OneTimeSignature used by agreement votes is actually 3 ed25519 signatures, this hooks up the verifier to the batch verification implementation, yielding a ~12% performance improvement in the included benchmark on my computer. Test Plan Added benchmark, existing tests should pass.

This adds a pure-Go ed25519 BatchVerifier implementation based on the ed25519consensus library, with additional checks to preserve our ed25519 validation criteria, last updated in #3031. Like our libsodium batch verification implementation, the IsCanonicalY check here is also based on the "Taming the Many EdDSAs" paper in https://eprint.iacr.org/2020/1244 New tests added to compare ed25519 criteria results match our existing libsodium- and ed25519-donna-based batch verification implementation (from #3031 and defined in algorandfoundation/specs#60). New test helpers run test vectors with different batch sizes, taken from - 12 edge cases from "Taming the many EdDSAs" Appendix C, Table 6c, also used in our libsodium fork's tests in batch verification: add ed25519 batch verification implementation #3031 - 1025 successful cases from our libsodium fork's tests, also added in batch verification: add ed25519 batch verification implementation #3031. - 196 edge cases used to draw the 14x14 grid visualizations from the blog post "It's 255:19AM. Do you know what your validation criteria are?" and used in ed25519consensus - 768 edge cases from the Go crypto package's crypto/ed25519/ed25519vectors_test.go

This adds a pure-Go ed25519 BatchVerifier implementation based on the ed25519consensus library, with additional checks to preserve our ed25519 validation criteria, last updated in algorand#3031. Like our libsodium batch verification implementation, the IsCanonicalY check here is also based on the "Taming the Many EdDSAs" paper in https://eprint.iacr.org/2020/1244 New tests added to compare ed25519 criteria results match our existing libsodium- and ed25519-donna-based batch verification implementation (from algorand#3031 and defined in algorandfoundation/specs#60). New test helpers run test vectors with different batch sizes, taken from - 12 edge cases from "Taming the many EdDSAs" Appendix C, Table 6c, also used in our libsodium fork's tests in batch verification: add ed25519 batch verification implementation algorand#3031 - 1025 successful cases from our libsodium fork's tests, also added in batch verification: add ed25519 batch verification implementation algorand#3031. - 196 edge cases used to draw the 14x14 grid visualizations from the blog post "It's 255:19AM. Do you know what your validation criteria are?" and used in ed25519consensus - 768 edge cases from the Go crypto package's crypto/ed25519/ed25519vectors_test.go

tsachiherman reviewed Oct 11, 2021

View reviewed changes

crypto/libsodium-fork/src/libsodium/crypto_core/ed25519/ref10/ed25519_ref10.c Outdated Show resolved Hide resolved

tsachiherman reviewed Oct 11, 2021

View reviewed changes

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/batch.c Outdated Show resolved Hide resolved

tsachiherman reviewed Oct 11, 2021

View reviewed changes

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/batch.c Outdated Show resolved Hide resolved

id-ms requested a review from cce October 13, 2021 12:19

cce reviewed Oct 18, 2021

View reviewed changes

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/batch.c Outdated Show resolved Hide resolved

cce reviewed Oct 18, 2021

View reviewed changes

crypto/libsodium-fork/src/libsodium/crypto_core/ed25519/ref10/ed25519_ref10.c Outdated Show resolved Hide resolved

cce reviewed Oct 18, 2021

View reviewed changes

crypto/libsodium-fork/src/libsodium/crypto_core/ed25519/ref10/ed25519_ref10.c Outdated Show resolved Hide resolved

cce reviewed Oct 18, 2021

View reviewed changes

crypto/libsodium-fork/src/libsodium/crypto_core/ed25519/ref10/ed25519_ref10.c Outdated Show resolved Hide resolved

id-ms force-pushed the add-batch-verification-algorithm branch from 0d714f2 to 8a7a4e3 Compare November 3, 2021 08:11

cce reviewed Nov 10, 2021

View reviewed changes

crypto/libsodium-fork/test/default/sign.c Outdated Show resolved Hide resolved

cce reviewed Nov 10, 2021

View reviewed changes

cce reviewed Nov 11, 2021

View reviewed changes

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/batch.c Outdated Show resolved Hide resolved

id-ms force-pushed the add-batch-verification-algorithm branch from b4c8f30 to c1c0b76 Compare November 12, 2021 04:15

cce reviewed Nov 12, 2021

View reviewed changes

crypto/libsodium-fork/src/libsodium/crypto_sign/ed25519/ref10/batch.c Outdated Show resolved Hide resolved

cce reviewed Nov 12, 2021

View reviewed changes

id-ms force-pushed the add-batch-verification-algorithm branch from c1c0b76 to a70c996 Compare November 12, 2021 18:34

algoidan added 11 commits November 28, 2021 16:46

adding edge cases vector to tests.

309de51

changing the signature verification method. according to https://epri…

93b3b7c

…nt.iacr.org/2020/1244.pdf. we now reject non canonical R and accept signatures with a mixed group order (in order to impl a batch verification)

optimizing the points and scalars checking.

bfbf8c6

removing unnecessary point doubling

e72a6e2

add batch verification base code from libdonna.

43bb08c

batch verification now falls to single verification

92207e5

calling a random bytes functions.

3d1230e

exporting the signature and the public key validation into a function…

4cf6faa

… and use it in the batch verification

removing wrong point doubling

a0d7ab6

after calculating the batch verification equation, we now compare the…

17c899a

… result to one of the small order points.

adding more batch verification tests.

df18b15

id-ms force-pushed the add-batch-verification-algorithm branch from 96e5029 to a449a4b Compare January 6, 2022 16:21

integrate a 32bit implementation

1ba1843

id-ms force-pushed the add-batch-verification-algorithm branch from a449a4b to 1ba1843 Compare January 6, 2022 17:10

optimize the expand256_modm on 32bit

560a29a

tsachiherman previously approved these changes Jan 6, 2022

View reviewed changes

Merge branch 'master' into add-batch-verification-algorithm

66e64e8

id-ms dismissed tsachiherman’s stale review via 66e64e8 January 23, 2022 10:01

Merge branch 'master' into add-batch-verification-algorithm

c273bee

tsachiherman approved these changes Jan 31, 2022

View reviewed changes

tsachiherman changed the title ~~Add batch verification algorithm~~ batch verification: add batch verification implementation Jan 31, 2022

tsachiherman assigned id-ms Jan 31, 2022

tsachiherman added the Team Sphinx label Jan 31, 2022

tsachiherman merged commit 1dd40a8 into algorand:master Jan 31, 2022

tsachiherman mentioned this pull request Feb 10, 2022

V31 consensus upgrade #3553

Merged

onetechnical mentioned this pull request Feb 11, 2022

go-algorand 3.4.0-beta #3613

Merged

egieseke mentioned this pull request Feb 16, 2022

Script to generate go-algorand release notes #3520

Closed

onetechnical mentioned this pull request Mar 1, 2022

go-algorand 3.4.2-stable #3700

Merged

cce mentioned this pull request Mar 11, 2022

crypto: call batchVerificationImpl from OneTimeSignatureVerifier.Verify #3759

Merged

cce mentioned this pull request Mar 16, 2022

crypto: add a couple of tests for ed25519 signatures before/after verification #3783

Closed

algonautshant mentioned this pull request Sep 15, 2022

perf: add batch verification into txhandler #4549

Closed

cce mentioned this pull request Feb 22, 2023

crypto: Variable-time implementation of VRF verify #3925

Open

cce changed the title ~~batch verification: add batch verification implementation~~ batch verification: add ed25519 batch verification implementation Feb 29, 2024

cce mentioned this pull request Sep 12, 2025

crypto: add go-based BatchVerifier implementation #6440

Merged

batch verification: add ed25519 batch verification implementation #3031

batch verification: add ed25519 batch verification implementation #3031

Conversation

id-ms commented Oct 10, 2021 • edited by tsachiherman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

codecov-commenter commented Oct 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cce commented Oct 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

id-ms commented Nov 3, 2021

Uh oh!

Uh oh!

cce left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cce left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tsachiherman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

id-ms commented Oct 10, 2021 •

edited by tsachiherman

Loading

codecov-commenter commented Oct 10, 2021 •

edited

Loading

cce commented Oct 21, 2021 •

edited

Loading

cce left a comment •

edited

Loading

cce left a comment •

edited

Loading