AWS S3 notification receiver (SQS) improvement #36315
Description
Component(s)
receiver/awss3
Is your feature request related to a problem? Please describe.
No, it is extending the functionality of the component.
Describe the solution you'd like
Current functionality for the aws s3 receiver is reading a set of files from S3 based on the configuration. This allows processing of files archived in S3. The proposal (already mentioned in #30750) is to allow the receiver to read a SQS queue for S3 notifications. That way, when a new file is created in the bucket, a notification is sent to SQS. The receiver would be reading that queue waiting for new messages and when a notification is sent, read the message, retrieve the object key from it, and get the events from the file.
The heavy lifting of the component is already present as it reads files from S3. Component extension would be to add the possibility to configure the queue to listen to, read those messages for s3 notifications and get the file to read instead of just listing the objects in the bucket.
This would allow sending files to S3 for processing, not just once but for any file created in the bucket.
Describe alternatives you've considered
No response
Additional context
To give some background, this is for example already possible for logs in vector with this source https://vector.dev/docs/reference/configuration/sources/aws_s3