Saving checkpoints to Google Drive

[woctezuma]

I have trouble saving checkpoints to Google Drive, despite using the correct argument (#17).

The checkpoint is supposed to be 1.4 GB large, but it ends up being 1.2 MB, when it is effectively saved. In other cases, a seemingly empty folder is created in my checkpoint/ folder in my Google Drive.

gpt2.mount_gdrive()

checkpoint_folder = 'checkpoint/' + run_name
gpt2.copy_checkpoint_to_gdrive(checkpoint_folder=checkpoint_folder)

Is there a way to reliably download the checkpoint?

For instance, wouldn't it be better to archive it to .zip and then send it to Google Drive?

[woctezuma]

Definitely both faster and more reliable if I execute the following instead:

gpt2.mount_gdrive()

!tar -cvf MY_ARCHIVE.tar checkpoint/MY_RUN_NAME/

!scp MY_ARCHIVE.tar '/content/drive/My Drive/'

[minimaxir]

It's possible (but unexpected/undocumented) the 345M model may be too big for the Google Drive connector to handle.

Adding compression to the folder isn't a bad idea. I think you can do that within Python.

[woctezuma]

I experienced this issue with the 117M model as well when I was trying to cp checkpoint folders between Google Drive and the Colab machine. As you mentioned, with the 345M model, it is even more visible because of the large folder sizes.

The best solution is to tar the folder before copying it rather than calling shutil.copytree(). It is very fast, as you do not need to compress the folder at all, it is really just making it one big .tar archive file. It is game-changing, really. The copying is lighting fast and suddenly becomes very reliable.

As for scp vs. cp, I don't know, but scp feels safer, because the copying does not happen locally.

I do not know how to do that in Python. I could look into it.

[minimaxir]

Implementation and design in #37 looks good so closing. Thanks!