Skip to content

Commit 59f70bf

Browse files
samblesSaseem75awsbuild
authored
Release 1.28.13 (#1191)
* Stable/1.28.x - Allianz - Celery DB authentication with Azure Service Principal (#1190) * Stable/1.28.x - Allianz - Celery DB authentication with Azure Service Principal (#1186) * Update postgres.docker-compose.yml spa env vars updated for celery and worker * Update requirements-worker.in azure identity packaged added to worker requirement * Update requirements-worker.txt * Update celeryconf.py Service Principal Logic added * Create celery_db_backend.py custom backend for celery * Create test.md * Add files via upload workflow file uploaded * Update test.md * Create celery_readme.md * Update celery_readme.md * Update celery_readme.md * Update celeryconf.py removed password/token masking * Update celeryconf.py * Update celeryconf.py * Update celery_readme.md * Update celery_db_backend.py * Update celery_readme.md * Update celery_db_backend.py removed unwanted - methods get_connection + get_db_engine * Update celery_readme.md * Update and rename celery_db_backend.py to utils.py utils.py introduced instead of celery_db_backend.py * Update and rename celery_readme.md to readme.md * Update readme.md readme.md updated * Adjust PR * update --------- Co-authored-by: Saseem75 <[email protected]> * Set version 1.28.13 Set version 1.28.13 * Update changelog --------- Co-authored-by: Saseem75 <[email protected]> Co-authored-by: awsbuild <[email protected]>
1 parent 3abbe26 commit 59f70bf

File tree

9 files changed

+147
-13
lines changed

9 files changed

+147
-13
lines changed

‎CHANGELOG.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
OasisPlatform Changelog
22
=======================
33

4+
* [#1190](https://github.com/OasisLMF/OasisPlatform/pull/1190) - Stable/1.28.x - Allianz - Celery DB authentication with Azure Service Principal
5+
.. _`1.28.13`: https://github.com/OasisLMF/OasisPlatform/compare/1.28.12...1.28.13
6+
47
.. _`1.28.12`: https://github.com/OasisLMF/OasisPlatform/compare/1.28.11...1.28.12
58

69
`1.28.11`_

‎VERSION‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
1.28.12
1+
1.28.13

‎compose/postgres.docker-compose.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,10 @@ services:
6565
- OASIS_CELERY_DB_USER=celery
6666
- OASIS_CELERY_DB_NAME=celery
6767
- OASIS_CELERY_DB_PORT=5432
68+
- OASIS_AZURE_CLIENT_ID=spa-client
69+
- OASIS_AZURE_TENANT_ID=spa-tenant
70+
- OASIS_AZURE_CLIENT_SECRET=spa-client-secret
71+
- OASIS_AZURE_SERVICE_PRINCIPAL_USER=spa-name
6872
volumes:
6973
- ${OASIS_MEDIA_ROOT:-./docker-shared-fs}:/shared-fs:rw
7074
worker:
@@ -91,6 +95,10 @@ services:
9195
- OASIS_CELERY_DB_NAME=celery
9296
- OASIS_CELERY_DB_PORT=5432
9397
- OASIS_MODEL_DATA_DIRECTORY=/home/worker/model
98+
- OASIS_AZURE_CLIENT_ID=spa-client
99+
- OASIS_AZURE_TENANT_ID=spa-tenant
100+
- OASIS_AZURE_CLIENT_SECRET=spa-client-secret
101+
- OASIS_AZURE_SERVICE_PRINCIPAL_USER=spa-name
94102
volumes:
95103
- ${OASIS_MODEL_DATA_DIR:-./data/static}:/home/worker/model:rw
96104
- ${OASIS_MEDIA_ROOT:-./docker-shared-fs}:/shared-fs:rw

‎requirements-worker.in‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,4 @@ psycopg2-binary
1111
sqlalchemy
1212
pytest
1313
numba
14+
azure-identity

‎requirements-worker.txt‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,10 @@ attrs==22.2.0
1717
azure-core==1.26.3
1818
# via
1919
# -r requirements-worker.in
20+
# azure-identity
2021
# azure-storage-blob
22+
azure-identity==1.17.1
23+
# via -r requirements-worker.in
2124
azure-storage-blob==12.15.0
2225
# via -r requirements-worker.in
2326
billiard==3.6.4.0
@@ -62,7 +65,11 @@ configparser==5.3.0
6265
cramjam==2.6.2
6366
# via fastparquet
6467
cryptography==42.0.4
65-
# via azure-storage-blob
68+
# via
69+
# azure-identity
70+
# azure-storage-blob
71+
# msal
72+
# pyjwt
6673
exceptiongroup==1.1.1
6774
# via pytest
6875
fasteners==0.18
@@ -97,6 +104,12 @@ kombu==5.2.4
97104
# via celery
98105
llvmlite==0.39.1
99106
# via numba
107+
msal==1.32.0
108+
# via
109+
# azure-identity
110+
# msal-extensions
111+
msal-extensions==1.3.1
112+
# via azure-identity
100113
msgpack==1.0.5
101114
# via oasislmf
102115
numba==0.56.4
@@ -119,7 +132,7 @@ numpy==1.22.4
119132
# scikit-learn
120133
# scipy
121134
# shapely
122-
oasislmf[extra]==1.28.12
135+
oasislmf[extra]==1.28.13
123136
# via -r requirements-worker.in
124137
ods-tools==3.1.5
125138
# via oasislmf
@@ -148,6 +161,10 @@ pyarrow==14.0.1
148161
# via oasislmf
149162
pycparser==2.21
150163
# via cffi
164+
pyjwt[crypto]==2.10.1
165+
# via
166+
# msal
167+
# pyjwt
151168
pymysql==1.1.1
152169
# via -r requirements-worker.in
153170
pyogrio==0.10.0
@@ -171,6 +188,7 @@ requests==2.31.0
171188
# via
172189
# azure-core
173190
# forex-python
191+
# msal
174192
# oasislmf
175193
# requests-toolbelt
176194
requests-toolbelt==0.10.1
@@ -216,6 +234,7 @@ tqdm==4.65.0
216234
typing-extensions==4.5.0
217235
# via
218236
# azure-core
237+
# azure-identity
219238
# azure-storage-blob
220239
# sqlalchemy
221240
urllib3==1.26.18

‎requirements.txt‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,9 @@ azure-core==1.26.3
3333
# azure-identity
3434
# azure-storage-blob
3535
azure-identity==1.17.1
36-
# via -r /home/runner/work/OasisPlatform/OasisPlatform/requirements-server.in
36+
# via
37+
# -r /home/runner/work/OasisPlatform/OasisPlatform/requirements-server.in
38+
# -r /home/runner/work/OasisPlatform/OasisPlatform/requirements-worker.in
3739
azure-storage-blob==12.15.0
3840
# via
3941
# -r /home/runner/work/OasisPlatform/OasisPlatform/requirements-worker.in
@@ -279,7 +281,7 @@ numpy==1.22.4
279281
# scikit-learn
280282
# scipy
281283
# shapely
282-
oasislmf[extra]==1.28.12
284+
oasislmf[extra]==1.28.13
283285
# via -r /home/runner/work/OasisPlatform/OasisPlatform/requirements-worker.in
284286
ods-tools==3.1.5
285287
# via

‎src/conf/azure_auth_SP_readme.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Celery Configuration with Azure Service Principal for PostgreSQL Backend
2+
3+
## Overview
4+
This setup enables Celery to store task results in Azure PostgreSQL while authenticating securely using Azure Active Directory Service Principal (SP). This eliminates the need for database passwords, enhances security, and ensures seamless authentication with Azure PostgreSQL using Service Principal tokens
5+
6+
- Instead of a static username/password, it retrieves an Azure AD access token dynamically.
7+
8+
### Configuration Breakdown
9+
10+
11+
```/var/www/oasis/
12+
│── src/
13+
│ ├── conf/
14+
│ │ ├── celeryconf.py
15+
│ │ ├── utils.py # Configures celery backend logic for Service Principal authentication
16+
│ ├── server
17+
│ │ ├── oasisapi
18+
│ │ │ ├── settings.py # where we Configures server database backend
19+
```
20+
21+
22+
##### utils.py - Handling Azure AD Token Authentication
23+
This module is responsible for fetching Azure Active Directory access tokens and managing the PostgreSQL connection for Celery.
24+
25+
<details>
26+
<summary>Environment Variables Required (Click to expand)</summary>
27+
28+
| Variable Name | Description |
29+
|--------------------------------------|------------------------------------------------|
30+
| `AZURE_TENANT_ID` | Azure AD Tenant ID |
31+
| `AZURE_CLIENT_ID` | Azure Service Principal Client ID |
32+
| `AZURE_CLIENT_SECRET` | Azure Service Principal Client Secret |
33+
| `AZURE_SERVICE_PRINCIPAL_USER` | Username for PostgreSQL (e.g., `sp_user@db`) |
34+
| `CELERY_RESULTS_DB_BACKEND` | Database backend (e.g., `db+postgresql`) |
35+
| `DB_HOST` | PostgreSQL database host |
36+
| `DB_PORT` | PostgreSQL database port |
37+
| `DB_NAME` | Celery result backend database name |
38+
39+
</details>
40+
41+
42+
#### Troubleshooting
43+
- Error: Azure AD credentials are missing for Celery DB
44+
- Ensure AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET are set correctly.
45+
- Celery fails to connect to PostgreSQL
46+
- Verify that the Service Principal is assigned the correct roles (db_owner or db_reader).
47+
- Expired Token Error
48+
- Restart the worker to refresh the token.

‎src/conf/celeryconf.py‎

Lines changed: 28 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,14 +20,34 @@
2020
DB_NAME=settings.get('celery', 'db_name', fallback='celery.db.sqlite'),
2121
)
2222
else:
23-
CELERY_RESULT_BACKEND = '{DB_ENGINE}://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}'.format(
24-
DB_ENGINE=settings.get('celery', 'db_engine'),
25-
DB_USER=urllib.parse.quote(settings.get('celery', 'db_user')),
26-
DB_PASS=urllib.parse.quote(settings.get('celery', 'db_pass')),
27-
DB_HOST=settings.get('celery', 'db_host'),
28-
DB_PORT=settings.get('celery', 'db_port'),
29-
DB_NAME=settings.get('celery', 'db_name', fallback='celery'),
30-
)
23+
#: Attempt to retrieve Service Principal credentials
24+
service_principal_user = settings.get("celery", "AZURE_SERVICE_PRINCIPAL_USER", fallback=None)
25+
if service_principal_user:
26+
#: Initialize Celery Database Backend
27+
from src.conf.utils import CeleryAzureServicePrincipal
28+
azure_token_client = CeleryAzureServicePrincipal(settings)
29+
30+
#: Use Service Principal Authentication
31+
azure_token = azure_token_client.get_access_token()
32+
CELERY_RESULT_BACKEND = "{DB_ENGINE}://{SP_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}".format(
33+
DB_ENGINE=settings.get('celery', 'db_engine'),
34+
SP_USER=urllib.parse.quote(service_principal_user),
35+
DB_PASS=urllib.parse.quote(azure_token),
36+
DB_HOST=settings.get('celery', 'db_host'),
37+
DB_PORT=settings.get("celery", "db_port", fallback="5432"),
38+
DB_NAME=settings.get("celery", "db_name", fallback="celery"),
39+
)
40+
else:
41+
#: Fallback to Username/Password Authentication
42+
CELERY_RESULT_BACKEND = "{DB_ENGINE}://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}".format(
43+
DB_ENGINE=settings.get('celery', 'db_engine'),
44+
DB_USER=urllib.parse.quote(settings.get('celery', 'db_user')),
45+
DB_PASS=urllib.parse.quote(settings.get('celery', 'db_pass')),
46+
DB_HOST=settings.get('celery', 'db_host'),
47+
DB_PORT=settings.get('celery', 'db_port'),
48+
DB_NAME=settings.get('celery', 'db_name', fallback='celery'),
49+
)
50+
3151

3252
#: Celery config - AMQP task result expiration time
3353
CELERY_AMQP_TASK_RESULT_EXPIRES = 1000

‎src/conf/utils.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
from azure.identity import ClientSecretCredential
2+
import time # To track token expiration
3+
4+
5+
class CeleryAzureServicePrincipal:
6+
def __init__(self, settings):
7+
self.settings = settings
8+
self.credential = None
9+
self.token = None
10+
self.token_expiry = 0 # Stores expiry timestamp
11+
12+
def get_access_token(self):
13+
"""
14+
Fetches and caches a fresh access token using Azure Service Principal credentials.
15+
"""
16+
tenant_id = self.settings.get('celery', 'AZURE_TENANT_ID')
17+
client_id = self.settings.get('celery', 'AZURE_CLIENT_ID')
18+
client_secret = self.settings.get('celery', 'AZURE_CLIENT_SECRET')
19+
20+
if not all([tenant_id, client_id, client_secret]):
21+
raise ValueError("Azure AD credentials are missing for Celery DB.")
22+
23+
if self.credential is None:
24+
self.credential = ClientSecretCredential(tenant_id, client_id, client_secret)
25+
26+
# Refresh token if expired or close to expiration
27+
current_time = time.time()
28+
if self.token is None or current_time >= self.token_expiry - 300: # Refresh 5 min before expiry
29+
token_response = self.credential.get_token("https://ossrdbms-aad.database.windows.net/.default")
30+
self.token = token_response.token
31+
self.token_expiry = current_time + 3600 # Azure tokens last ~60 min
32+
33+
return self.token

0 commit comments

Comments
 (0)