Parallel ParticleSwarm

[sgaure]

#1124

This PR enhances ParticleSwarm with a new keyword parallel::Bool = false. If set to true, the objective function will be called with a vector to fill in, and a matrix of column vectors as inputs. Thus, the user may choose to evaluate the swarm in parallel with whatever means.

[sgaure]

Btw, I forgot to mention. The commits also contain a change which allows some upper bounds to be equal to the lower bounds. This can be used as a simple way to keep some variables constant.

[pkofod]

I see, yes I also implemented something similar in NLSolvers.jl. Let me review it. Thanks

[sgaure]

The test failures seem to be unrelated. I came to think of it, should the new parallel keyword be called batched instead? It's not implementing anything parallel.

[github-actions]

Benchmark Results

	master	662a5da...	master / 662a5da...
multivariate/solvers/first_order/AdaMax	0.654 ± 0.0078 ms	0.647 ± 0.0078 ms	1.01 ± 0.017
multivariate/solvers/first_order/Adam	0.655 ± 0.0077 ms	0.645 ± 0.0079 ms	1.01 ± 0.017
multivariate/solvers/first_order/BFGS	0.224 ± 0.0049 ms	0.224 ± 0.0036 ms	1 ± 0.027
multivariate/solvers/first_order/ConjugateGradient	0.0483 ± 0.00055 ms	0.0484 ± 0.00057 ms	0.998 ± 0.016
multivariate/solvers/first_order/GradientDescent	1.71 ± 0.0092 ms	1.71 ± 0.013 ms	1 ± 0.0093
multivariate/solvers/first_order/LBFGS	0.22 ± 0.0036 ms	0.22 ± 0.0036 ms	1 ± 0.023
multivariate/solvers/first_order/MomentumGradientDescent	2.51 ± 0.013 ms	2.5 ± 0.017 ms	1 ± 0.0084
multivariate/solvers/first_order/NGMRES	0.556 ± 0.011 ms	0.554 ± 0.01 ms	1 ± 0.027
time_to_load	0.515 ± 0.0019 s	0.519 ± 0.0038 s	0.993 ± 0.0083

Benchmark Plots

A plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR.
Go to "Actions"->"Benchmark a pull request"->[the most recent run]->"Artifacts" (at the bottom).

[pkofod]

I'll look at this later today, it had slipped out of my attention.