Introduce a rule for too high NBANDS

[Andrew-S-Rosen]

When modeling very small systems (e.g. a single CO molecule in a box) with VASP, if NBANDS is too large, the electronic structure (e.g. the DOS) can be completely unphysical. I found this out the hard way when, by default, running a single node job on Cori caused NBANDS = 64 yet I was modeling a CO molecule where NBANDS = 6 might be more appropriate. In the former case, countless fictitious states were introduced into the DOS.

We should introduce some sort of rule that makes sure NBANDS isn't too high for small systems.

Brainstorming with @mkhorton yields the following possible rule:
if NBANDS > 2*default NBANDS where default NBANDS is max(NELECT/2+NIONS/2, 0.6*NELECT), then kill job and restart with the default unless LOPTICS = .TRUE., in which case use a factor of 10 as the cutoff.

[Andrew-S-Rosen]

@utf, alternatively this could be something we incorporate into atomate2 directly. That could be easier, but I could see arguments for both approaches.

[shyuep]

Thanks. As I have told Anubhav's before, everything to do with the correctness of a calculation should be in custodian. Everything that deals with automation for high throughput should be in atomate. Not all people who use custodian wants to do HT.

[utf]

I agree with @shyuep that this should be in custodian.

I would add a caveat to the logic that if NBANDS is already in the INCAR then it shouldn't be overwritten. If you are setting NBANDS for a reason you probably don't want it changed.

[yang-ruoxi]

Note that VASP will also automatically changes NBANDS depending on the nodes/cores you are running with, so it may not necessarily be a problem with atomate or custodian : link
" When executed on multiple CPUs, VASP automatically increases the number of bands, so that NBANDS is divisible by the number of CPU cores. If NCORE > 1, NBANDS is increased until it is divisible by the number of cores in a group (NCORE). If KPAR > 1, NBANDS is increased until it is divisible by the number of cores in a group."

[Andrew-S-Rosen]

@yang-ruoxi --- actually that's the problem I ran into. The issue is if you take a small molecule like CO and run on a full KNL node, VASP will automatically change NBANDS to be too large and when you have a large excess number of bands, the results can be nonphysical as in my prior example.

Edit: We clarified over Slack. Ruoxi was suggesting that even if we set NBANDS manually, VASP will still override it.

[Andrew-S-Rosen]

@yang-ruoxi: I have confirmed that even if the user changes NBANDS in the INCAR, VASP will still override it (see attached).

OUTCAR.txt

Unfortunately, I was not able to come up with a clean way of either raising a robust warning or an error. I'll keep this open for someone else to think about.

[Andrew-S-Rosen]

Just noting here that this will be an even bigger problem for CPU nodes on Perlmutter, which have 128 cores.