Support AWS secrets in configuring Data Prepper pipelines

[chenqi0805]

Is your feature request related to a problem? Please describe.
In the current state, user credential variables such as username and password are hardcoded in pipelines.yaml. For better security, we could support AWS secret as an extension plugin, i.e.

data-prepper-config.yaml

aws:
  credentials:
    default:
       sts_role_arn: arn:aws:iam::99123456789:role/OsiDataPlaneRole
       region: us-east-1
     my-custom-role:
       sts_role_arn:
       region:
  secret: # secret extension
    my_es_secret_configuration:
      name: ...
      authentication: my-custom-role

Then our pipeline configuration can reference the key-value pairs stored in an AWS secret.
pipelines.yaml

...
sink:
  - opensearch:
      ...
      username: ${{ aws_secret:my_es_secret_configuration:USERNAME }}
      password: ${{ aws_secret:my_es_secret_configuration:PASSWORD }}

Describe the solution you'd like
We can reuse aws credential extensions when configuring the credentials for AWS secrets.

Describe alternatives you've considered (Optional)
Alternatively, we can make AWS secret extension independent of credentials plugin at the price of duplication in configuration:

aws:
  credentials:
    default:
      sts_role_arn: arn:aws:iam::99123456789:role/OsiDataPlaneRole
      region: us-east-1
    my-custom-role:
      sts_role_arn:
      region:

aws_secrets:
  secrets:
    my-secret1:
      sts_role_arn: arn:aws:iam::99123456789:role/OsiDataPlaneRole
      region: us-east-1

Additional context
Add any other context or screenshots about the feature request here.

[dlvenable]

@chenqi0805 , This proposal looks good and I like it quite a bit.

One thing we may want to consider is the way we escape the secrets. We are looking to use ${} as mechanism for getting values from events. So it may be better to not use it here. See #2719 for the proposal on this topic.

Perhaps username: {{ aws_secret:my_es_secret_configuration:USERNAME }}? Maybe there is another approach?

Also, do you have any thoughts on how to reconcile this with #947? Right now, I'm thinking that this concept is driven by a properties loader feature. So we could have either env or aws_secret as available properties loaders. Then you can have either ${{ env:MY_ENVIRONMENT }} or ${{ aws_secret:my_configuration:USERNAME }}. But, I'm interested in your thoughts.

[zied-chekir]

I tried using ${{ env:MY_ENVIRONMENT }} in the pipeline.yaml for the sink password but it does not work. I was wondering if it is implemented yet?

[dlvenable]

@zied-chekir , Support for environment variables is not yet available. That is proposed in #947. The work done for AWS Secrets does lay down a common framework that we could use for environment variables. But, some additional work is needed.

Would you be interested in contributing toward this? I'd be happy to help give some guidance.