Commitment scheme abstraction with all previous improvements

[han0110]

This PR aims to pick all previous improvements base on abstraction, to make further integration easier. Each improvement is squashed into a single commit for easier rebasing in the future, and some are slightly modified to make as less changes as possible, they includes (in order):

• Commitment scheme abstraction #77
• Remove bench regression workflow #60
• [MockProver] Use a hash map to track cell assignment #71
• Small changes for EVM circuit updates #54
• Parallelized eval_polynomial/evaluate_vanishing_polynomial #47
• test: add verify_at_rows in MockProver for improve test performance #38
• dev: add name for lookup #29
• lookup_any #8
• Poly optimizations #44, Reduce memory use in h evaluation #78
• Reduce memory use #89

With 4 more changes I made/picked:

• Use same nightly rust version as privacy-scaling-explorations/zkevm-circuits
• In MockProver::verify_*, cache the table if its identifier is same to previous one (which in most case several lookup to same table will be allocated adjacently), and check the cell indexes first in permutation check.
• MockProver::verify_at_rows_par (modified) by @spherel
• Expose necessary immutable reference fields for aggregation circuit usage.

[therealyingtong]

LGTM.

I only reviewed the final three commits; I did not review the Abstraction PR, or the already-merged PRs.

[therealyingtong]

Directly editing iter() -> par_iter() in verify_at_rows would make for a smaller diff. But I suppose that might cause conflicts in future rebases.

[han0110]

Yeah, was trying to just add things and avoid editing things for easier rebasing.

[pinkiebell]

Can we maybe include: #86 ?
Some not yet merged features in the prover crate depend on that and that's still a blocker for merging more recent changes.

[davidnevadoc]

Great work!
I went through all the changes paying special attention to the commits that are not part of a reviewed and approved PR. Overall everything LGTM :)
The only comment I'd make is that we maybe should parallelize compute_inner_product and include that change as part of this PR.

[han0110]

Can we maybe include: #86 ?
Some not yet merged features in the prover crate depend on that and that's still a blocker for merging more recent changes.

@pinkiebell In trait Param it provides read access to k and n, do you think it's sufficient? I also had an impression that we might need to truncate SRS for smaller circuit, how does that handled currently?

EDIT: I just add another method Param::downsize for the above use case.

[han0110]

The only comment I'd make is that we maybe should parallelize compute_inner_product and include that change as part of this PR.

@davidnevadoc I just realized that compute_inner_product is only used for IPA but currently we haven't needed it yet, is that possible that we spend time implementing this when we are going to use it? Since I'm not sure how to pick the threshold to parallelize and might need to investigate some time to see.

[ashWhiteHat]

I just realized that compute_inner_product is only used for IPA but currently we haven't needed it yet, is that possible that we spend time implementing this when we are going to use it? Since I'm not sure how to pick the threshold to parallelize and might need to investigate some time to see.

I think it's unnecessary to introduce parallelize to compute_inner_product.
I introduced the parallelize before but it didn't affect so much and also other solution which affects more than parallelize.
zkcrypto/bls12_381#84

This PR is enough large so we should merge this and after that, improve more.

[pinkiebell]

Can we maybe include: #86 ?
Some not yet merged features in the prover crate depend on that and that's still a blocker for merging more recent changes.

@pinkiebell In trait Param it provides read access to k and n, do you think it's sufficient?

Yes! That's all I needed :)

I also had an impression that we might need to truncate SRS for smaller circuit, how does that handled currently?

At the moment, I generated different keys for each K. I also tried a similar downsizing method by truncating, but keys generated with unsafe_setup didn't work after truncating - It basically broke commit_lagrange. Does it correctly work with the abstraction-squashed branch?

[han0110]

also tried a similar downsizing method by truncating, but keys generated with unsafe_setup didn't work after truncating - It basically broke commit_lagrange.

Did you truncate the g_lagrange by g_lagrange.truncate(smaller_n) directly?

Does it correctly work with the abstraction-squashed branch?

Yes I think so, I have tried Params::downsize in tests/plonk_api and it works.

[pinkiebell]

also tried a similar downsizing method by truncating, but keys generated with unsafe_setup didn't work after truncating - It basically broke commit_lagrange.

Did you truncate the g_lagrange by g_lagrange.truncate(smaller_n) directly?

Yes. And after I took a look at unsafe_setup it became evident why that didn't work :). I defaulted to just work with different keys instead of changing halo2 too much, but good to know that the upcoming halo2 update ships that functionality.

[CPerezz]

Awesome job mate! Thanks for working on this!