Add rebuild to CI (#415)

## Summary
<!--- This is a required section; please describe the main purpose of
this proposed code change. --->

<!---
## Details
This is an optional section; is there anything specific that reviewers
should be aware of?
--->

The current modal image always uses cache, so it does not run with the
latest dependencies.

1. We `force_build` every night, and it pulls the latest dependencies
2. For CI and main commit, it uses the cache so it is faster

## Testing Done
<!--- This is a required section; please describe how this change was
tested. --->

<!-- 
Replace BLANK with your device type. For example, A100-80G-PCIe

Complete the following tasks before sending your PR, and replace `[ ]`
with
`[x]` to indicate you have done them. 
-->

- Hardware Type: <BLANK>
- [x] run `make test` to ensure correctness
- [x] run `make checkstyle` to ensure code style
- [x] run `make test-convergence` to ensure convergence

Author: ByronHsu

.github/workflows/ci.yml

@@ -13,9 +13,12 @@ on:
     paths:
       - "src/**"
       - "test/**"
+  schedule:
+    # Runs at 00:00 UTC daily
+    - cron: '0 0 * * *'
+  workflow_dispatch:  # Enables manual trigger
 
 concurrency:
-  # This causes it to cancel previous in-progress actions on the same PR / branch,
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
   cancel-in-progress: true
 
@@ -46,6 +49,7 @@ jobs:
     env:
       MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
       MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
+      REBUILD_IMAGE: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}
 
     steps:
     - name: Checkout code
@@ -71,6 +75,7 @@ jobs:
     env:
       MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
       MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
+      REBUILD_IMAGE: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}
 
     steps:
     - name: Checkout code

dev/modal/tests.py

@@ -4,8 +4,13 @@
 
 ROOT_PATH = Path(__file__).parent.parent.parent
 
+# REBUILD_IMAGE is an environment variable that is set to "true" in the nightly build
+REBUILD_IMAGE = modal.env("REBUILD_IMAGE", default=False)
+
 image = modal.Image.debian_slim().pip_install_from_pyproject(
-    ROOT_PATH / "pyproject.toml", optional_dependencies=["dev"]
+    ROOT_PATH / "pyproject.toml",
+    optional_dependencies=["dev"],
+    force_build=REBUILD_IMAGE,
 )
 
 app = modal.App("liger_tests", image=image)

dev/modal/tests_bwd.py

@@ -4,13 +4,18 @@
 
 ROOT_PATH = Path(__file__).parent.parent.parent
 
+# REBUILD_IMAGE is an environment variable that is set to "true" in the nightly build
+REBUILD_IMAGE = modal.env("REBUILD_IMAGE", default=False)
+
 # tests_bwd is to ensure the backward compatibility of liger with older transformers
 image = (
     modal.Image.debian_slim()
     .pip_install_from_pyproject(
-        ROOT_PATH / "pyproject.toml", optional_dependencies=["dev"]
+        ROOT_PATH / "pyproject.toml",
+        optional_dependencies=["dev"],
+        force_build=REBUILD_IMAGE,
     )
-    .pip_install("transformers==4.44.2")
+    .pip_install("transformers==4.44.2", force_build=REBUILD_IMAGE)
 )
 
 app = modal.App("liger_tests", image=image)