Get real project to run through Prefect

[milanmlft]

• Test run hic_cc project on GAE with Prefect
• Use separate dev and prod deployments (closes DEP-46, Use a single deployment for omop_es runs #55)
• Drive-by:
  • some Makefile and Dockerfile tweaks
  • better arg types and defaults for Prefect flow and tasks
  • better logging: handle [WARN] and [CRITICAL]

Closes DEP-42, #52

[linear]

DEP-46 Use a single deployment for `omop_es` runs

DEP-42 Ensure batch processing works in prefect on real project

[Copilot]

Pull Request Overview

This PR integrates a real project (hic_cc) to run through Prefect with separate dev/prod deployments. The changes enhance the Prefect workflow system with better logging, improved argument handling, and production deployment support.

• Adds separate dev and prod deployment configurations for Prefect flows
• Improves logging to handle [WARN] and [CRITICAL] log levels
• Enhances Docker build process with better branch handling and caching

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
template.env	Adds OMOP_ES_BRANCH configuration variable
prefect/test_subprocess_utilities.py	Updates tests for enhanced log level handling
prefect/test_run_omop_es.py	Adds test for prod deployment flag and fixes parameter name
prefect/run_subprocess.py	Enhances log level parsing to support WARN and CRITICAL levels
prefect/run_omop_es.py	Refactors flow with better parameter types and prod deployment support
prefect/prefect.yaml	Configures separate dev and prod deployments
prefect/Makefile	Updates make targets for improved workflow management
docker/Dockerfile	Improves Docker build process with better dependency handling
docker-compose.yml	Adds cache busting for Docker builds

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[milanmlft]

Barring some omop_es errors (potentially due to outdated omop_es version on GAE), running the hic_cc project in the prod deployment was successful: http://uclvlddpragae10:8082/runs/flow-run/3ddb44a7-33a9-4303-8f1f-c01b998921e3 (UCLH network only)

[BaptisteBR]

Barring some omop_es errors (potentially due to outdated omop_es version on GAE), running the hic_cc project in the prod deployment was successful: http://uclvlddpragae10:8082/runs/flow-run/3ddb44a7-33a9-4303-8f1f-c01b998921e3 (UCLH network only)

I've created a new PR to update the OMOP vocabs version in OMOP_ES

[stefpiatek]

🎉

[milanmlft]

Reasonably confident it's doing the correct thing now: http://uclvlddpragae10:8082/runs/flow-run/37b16ef1-24ae-4303-8183-4e53acc9fc23
So going to merge this