core/vm: use go-bigmodexpfix for modexp

[fjl]

This changes the modexp precompile to use a fork of math/big with some patches by @GottfriedHerold to improve worst-case performance.

[fjl]

Results are interesting:

                                                            │ modexp.old.txt │             modexp.new.txt             │
                                                            │     mgas/s     │    mgas/s     vs base                  │
PrecompiledModExp/nagydani-5-pow0x10001-Gas=285900-8            1109.0 ± ∞ ¹    619.9 ± ∞ ¹   -44.10% (p=0.008 n=5)
PrecompiledModExpEip2565/nagydani-5-pow0x10001-Gas=87381-8       338.9 ± ∞ ¹    189.5 ± ∞ ¹   -44.08% (p=0.008 n=5)
PrecompiledModExp/nagydani-4-pow0x10001-Gas=89292-8             1105.0 ± ∞ ¹    764.4 ± ∞ ¹   -30.82% (p=0.008 n=5)
PrecompiledModExpEip2565/nagydani-4-pow0x10001-Gas=21845-8       269.8 ± ∞ ¹    186.9 ± ∞ ¹   -30.73% (p=0.008 n=5)
PrecompiledModExpEip2565/nagydani-3-pow0x10001-Gas=5461-8        209.8 ± ∞ ¹    178.6 ± ∞ ¹   -14.87% (p=0.008 n=5)
PrecompiledModExp/nagydani-3-pow0x10001-Gas=30310-8             1166.0 ± ∞ ¹    993.9 ± ∞ ¹   -14.76% (p=0.008 n=5)
PrecompiledModExpEip2565/guido-2-even-Gas=2300-8                 34.84 ± ∞ ¹    35.57 ± ∞ ¹    +2.10% (p=0.008 n=5)
PrecompiledModExpEip2565/marcin-2-exp-heavy-Gas=852-8            34.25 ± ∞ ¹    35.07 ± ∞ ¹    +2.39% (p=0.008 n=5)
PrecompiledModExpEip2565/guido-1-even-Gas=2298-8                 60.06 ± ∞ ¹    62.50 ± ∞ ¹    +4.06% (p=0.008 n=5)
PrecompiledModExpEip2565/marius-1-even-Gas=2057-8                34.06 ± ∞ ¹    35.53 ± ∞ ¹    +4.32% (p=0.008 n=5)
PrecompiledModExpEip2565/marcin-3-base-heavy-Gas=677-8           45.73 ± ∞ ¹    47.86 ± ∞ ¹    +4.66% (p=0.008 n=5)
PrecompiledModExpEip2565/marcin-3-exp-heavy-Gas=765-8            43.78 ± ∞ ¹    46.23 ± ∞ ¹    +5.60% (p=0.008 n=5)
PrecompiledModExpEip2565/mod-8-exp-896-Gas=298-8                 15.24 ± ∞ ¹    16.25 ± ∞ ¹    +6.63% (p=0.008 n=5)
PrecompiledModExpEip2565/pawel-1-exp-heavy-Gas=298-8             15.09 ± ∞ ¹    16.19 ± ∞ ¹    +7.29% (p=0.008 n=5)
PrecompiledModExpEip2565/mod-32-exp-128-Gas=677-8                45.62 ± ∞ ¹    49.63 ± ∞ ¹    +8.79% (p=0.008 n=5)
PrecompiledModExp/nagydani-2-pow0x10001-Gas=10649-8              931.5 ± ∞ ¹   1031.0 ± ∞ ¹   +10.68% (p=0.008 n=5)
PrecompiledModExpEip2565/pawel-2-exp-heavy-Gas=425-8             30.07 ± ∞ ¹    33.39 ± ∞ ¹   +11.04% (p=0.008 n=5)
PrecompiledModExpEip2565/marcin-1-exp-heavy-Gas=215-8            14.56 ± ∞ ¹    16.17 ± ∞ ¹   +11.06% (p=0.008 n=5)
PrecompiledModExpEip2565/nagydani-2-pow0x10001-Gas=1365-8        119.6 ± ∞ ¹    133.8 ± ∞ ¹   +11.87% (p=0.008 n=5)
PrecompiledModExpEip2565/mod-8-exp-648-Gas=215-8                 13.59 ± ∞ ¹    15.98 ± ∞ ¹   +17.59% (p=0.008 n=5)
PrecompiledModExpEip2565/nagydani-1-pow0x10001-Gas=341-8         66.55 ± ∞ ¹    78.84 ± ∞ ¹   +18.47% (p=0.008 n=5)
PrecompiledModExp/nagydani-1-pow0x10001-Gas=3276-8               640.4 ± ∞ ¹    762.0 ± ∞ ¹   +18.99% (p=0.008 n=5)
PrecompiledModExpEip2565/pawel-3-exp-heavy-Gas=501-8             34.04 ± ∞ ¹    43.87 ± ∞ ¹   +28.88% (p=0.008 n=5)
PrecompiledModExpEip2565/pawel-4-exp-heavy-Gas=506-8             33.59 ± ∞ ¹    48.20 ± ∞ ¹   +43.50% (p=0.008 n=5)
PrecompiledModExpEip2565/mod-32-exp-32-Gas=200-8                 27.15 ± ∞ ¹    45.39 ± ∞ ¹   +67.18% (p=0.008 n=5)
PrecompiledModExpEip2565/mod-32-exp-36-Gas=200-8                 24.31 ± ∞ ¹    41.19 ± ∞ ¹   +69.44% (p=0.008 n=5)
PrecompiledModExpEip2565/mod-32-exp-40-Gas=208-8                 22.68 ± ∞ ¹    39.20 ± ∞ ¹   +72.84% (p=0.008 n=5)
PrecompiledModExpEip2565/mod-32-exp-64-Gas=336-8                 23.00 ± ∞ ¹    42.05 ± ∞ ¹   +82.83% (p=0.008 n=5)
PrecompiledModExpEip2565/mod-32-exp-65-Gas=341-8                 22.99 ± ∞ ¹    42.96 ± ∞ ¹   +86.86% (p=0.008 n=5)
PrecompiledModExpEip2565/marcin-1-balanced-Gas=200-8             22.11 ± ∞ ¹    41.81 ± ∞ ¹   +89.10% (p=0.008 n=5)
PrecompiledModExpEip2565/marcin-2-balanced-Gas=996-8             29.59 ± ∞ ¹    59.32 ± ∞ ¹  +100.47% (p=0.008 n=5)
PrecompiledModExpEip2565/mod_vul_pawel_3_exp_8-Gas=200-8         12.59 ± ∞ ¹    38.01 ± ∞ ¹  +201.91% (p=0.008 n=5)

[MariusVanDerWijden]

I have pushed it to the performance branch so we can see the difference in the performance grafana. I also built a fuzzer that has been fuzzing for a while to find both differences and big performance regressions. So far everything looks good

[chfast]

I can test this with my fuzzer too.

[MariusVanDerWijden]

LGTM