Implementing tiny versions of current state-of-art llms and comparing them on simple task

With train.py you can train different models or different hyperparameter settings

You can change the hyperparamers for different models from models/model_name.py

python3 train.py --model_params=[n_params]

python3 train.py [model_name] --model_params=[n_params]

python3 train.py [model_name] --tune --model_params=[n_params]

use --save if you want to save the model and losses

at the moment only 50M, 75M and 100M param models are supported

See the training graph of training run

python3 compare.py

python3 compare.py [run_name]