awslogs driver for ECS - the newest log is not sent to CW if it is in EMF format

[tomaszdudek7]

I'm seeing a weird behaviour of awslogs driver in ECS when it comes to sending logs to CloudWatch.

My web Python app (Docker, FastAPI) writes logs to the stdout using aws-embedded-metrics-python. This works just fine when running locally in Python or in Docker.

However, in AWS ECS, if the following two logs are sent:

1. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahblah", "taskArn": "6900448c0b [...] blah
2. {"Environment": "tdudek", "_aws": {"Timestamp": 1693484861577, "CloudWatchMetrics": [{"Dimensions": [["Environment"]], "Metric [...] # correct EMF format

only 1. is seen in CW.

If I swap the order to the following:

1. {"Environment": "tdudek", "_aws": {"Timestamp": 1693484861577, "CloudWatchMetrics": [{"Dimensions": [["Environment"]], "Metric [...] # correct EMF format}
2. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahblah", "taskArn": "6900448c0b [...]}

then both 1. and 2. are seen in CW.

If I print something like this:

1. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahblah", "taskArn": "6900448c0b [...]}
2. {"Environment": "tdudek", "_aws": {"Timestamp": 1693484861577, "CloudWatchMetrics": [{"Dimensions": [["Environment"]], "Metric [...] # correct EMF format}
3. {"datetime": "2023-08-31T12:27:41.576840Z", "level": "WARNING", "message": "Blahbalah", "taskArn": "6900448c0b [...]}

Then 1., 2. and 3. are sent to CW.

It is as if awslogs ignored/buffered the last log if it happens to be in EMF format. Is this a documented / known behaviour or a bug?

[gordonpn]

Hi @tomaszdudek7, awslogs log driver is owned by the Moby project https://github.com/moby/moby/tree/master/daemon/logger/awslogs

Could you try reaching out to that project?