Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 44 additions & 3 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
app.kubernetes.io/created-by: "dragonfly-operator"
name: "dragonfly-sample"
spec:
image: "ghcr.io/dragonflydb/dragonfly-weekly:latest"
image: "ghcr.io/dragonflydb/dragonfly:latest"
args: ["--cache_mode"]
replicas: 2
resources:
Expand Down Expand Up @@ -89,7 +89,27 @@ jobs:
kubectl wait -n ${{ steps.setup.outputs.namespace }} pods --selector app=dragonfly-sample --for condition=Ready --timeout=120s
kubectl describe -n ${{ steps.setup.outputs.namespace }} pod dragonfly-sample-0

- name: Run Benchmark
- name: Run Memtier Benchmark
shell: bash
run: |
kubectl apply -n ${{ steps.setup.outputs.namespace }} -f tools/benchmark/k8s-benchmark-job.yaml

- name: Version upgrade
shell: bash
run: |
# benchmark is running, wait for 30 seconds before version upgrade
sleep 30
kubectl patch dragonfly dragonfly-sample -n ${{ steps.setup.outputs.namespace }} --type merge -p '{"spec":{"image":"ghcr.io/dragonflydb/dragonfly-weekly:latest"}}'

- name: Wait for Memtier Benchmark fail
shell: bash
run: |
# Memtier benchmark run will fail at some point because old master shutdown on version upgrade
kubectl wait --for=condition=failed --timeout=120s -n ${{ steps.setup.outputs.namespace }} jobs/memtier-benchmark 2>/dev/null
kubectl logs -n ${{ steps.setup.outputs.namespace }} -f jobs/memtier-benchmark
kubectl delete -n ${{ steps.setup.outputs.namespace }} jobs/memtier-benchmark

- name: Run Memtier Benchmark again
shell: bash
run: |
kubectl apply -n ${{ steps.setup.outputs.namespace }} -f tools/benchmark/k8s-benchmark-job.yaml
Expand All @@ -116,7 +136,6 @@ jobs:
- name: Server checks
run: |
nohup kubectl port-forward -n ${{ steps.setup.outputs.namespace }} service/dragonfly-sample 6379:6379 &

pip install -r tools/requirements.txt
python3 tools/benchmark/post_run_checks.py

Expand All @@ -138,6 +157,15 @@ jobs:
command: |
kubectl logs -n ${{ steps.setup.outputs.namespace }} dragonfly-sample-1

- name: Describe dragonflydb object
uses: nick-fields/retry@v3
if: always()
with:
timeout_minutes: 1
max_attempts: 3
command: |
kubectl describe dragonflies.dragonflydb.io -n ${{ steps.setup.outputs.namespace }} dragonfly-sample

- name: Scale down to zero
if: always()
run: |
Expand All @@ -152,3 +180,16 @@ jobs:
set -x
kubectl delete namespace ${{ steps.setup.outputs.namespace }}
kubectl delete namespace dragonfly-operator-system

- name: Send notification on failure
if: failure() && github.ref == 'refs/heads/main'
shell: bash
run: |
job_link="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
message="Benchmark tests failed.\\n Job Link: ${job_link}\\n"

curl -s \
-X POST \
-H 'Content-Type: application/json' \
'${{ secrets.GSPACES_BOT_DF_BUILD }}' \
-d '{"text": "'"${message}"'"}'
2 changes: 1 addition & 1 deletion tools/benchmark/k8s-benchmark-job.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ spec:
- name: memtier
image: redislabs/memtier_benchmark:latest
args:
- memtier_benchmark --pipeline=30 --key-maximum=10000 -c 10 -t 2 --requests=500000 --expiry-range=10-100 --reconnect-interval=10000 --distinct-client-seed --hide-histogram -s dragonfly-sample
- memtier_benchmark --pipeline=30 --key-maximum=100000 -c 10 -t 2 --test-time=600 --reconnect-interval=10000 --distinct-client-seed --hide-histogram -s dragonfly-sample
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the expiry-range flag as it would create failure in checking the lag between master and replica as entries will continue expire after benchmark finished running

command:
- sh # This is important! without it memtier cannot DIG the dragonfly SVC domain
- -c
Expand Down
10 changes: 9 additions & 1 deletion tools/benchmark/post_run_checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ def main():
max_unaccounted = 200 * 1024 * 1024 # 200mb

client = redis.Redis(decode_responses=True)
info = client.info("server")
# Check version upgrade finsihed from last released version to last weekly docker build
assert info["dragonfly_version"] == "df-HEAD-HASH-NOTFOUND"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, post_run_checks.py only runs after upgrade? or just runs at the end and we reach end only after an upgrade. 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It runs at the end when memtier finish running and also the upgrade was done


info = client.info("memory")
print(f'Used memory {info["used_memory"]}, rss {info["used_memory_rss"]}')
assert info["used_memory_rss"] - info["used_memory"] < max_unaccounted
Expand All @@ -26,7 +30,11 @@ def is_zero_lag(replication_state):
time.sleep(1)
replication_state = client.info("replication")["slave0"]

assert replication_state["lag"] == 0, f"Lag is bad, expected 0, got {replication_state['lag']}"
if replication_state["lag"] != 0:
print(f"Lag is bad, expected 0, got {replication_state['lag']}")
info = client.info("all")
print(f"Info all output: {info}")
assert False


if __name__ == "__main__":
Expand Down