OpenSearch source plugin

[dlvenable]

Is your feature request related to a problem? Please describe.

Some users have looked for a way to migrate data from one OpenSearch cluster to another. One of the missing components here would be the ability to retrieve data from OpenSearch. Other users have expressed a need to transform data from OpenSearch clusters.

Describe the solution you'd like

Include an OpenSearch source plugin for Data Prepper. It will need some of the following configurations.

• Connection configurations similar to the opensearch sink.
• Source index or indices. This should also support dynamic indices based on date.
• Possibly query options to filter down the data. If not specified, all data in the index would be used.
• Schedule configurations for reading data.

Additional context

This should be similar to the logstash-input-opensearch-plugin provided in the OpenSearch project.

https://opensearch.org/blog/community/2022/05/introducing-logstash-input-opensearch-plugin-for-opensearch/
https://github.com/opensearch-project/logstash-input-opensearch

[ashoktelukuntla]

Create a source plugin which would enable users to bulk read, bulk write on a scheduled manner to a given OpenSearch cluster. This plugin should be extendable to take user defined additional sources. Users should be able to create/schedule pipeline for migration of data by

Auto discovery i.e. Listing all the indexes or Take given Index
Iterate over a index , read/fetch complete data
Enrich/transform Data (Optional)
Sink (can be any say) OpenSearch using Data Prepper
Reconcile/report comparing source and sink data
Cron can be used to schedule the migration of data. Example: schedule: "* * * * *" ' will load data every minute

Additional context

Plugin should be able to take configurations data related to cluster including hostname:port, user credentials, optional - index and query (e.g. match_all) .

I would envision the following sequence of steps

cat indices - https://opensearch.org/docs/1.2/opensearch/rest-api/cat/cat-indices/
Iterate over an index
Query index i.e. match_all or scroll query for a large indices
Enrich/transform Data (Optional)
Data Prepper pipeline to ingest data in to opensearch
Report on data from sink and source
**References: **

This should be similar to the logstash-input-opensearch-plugin provided in the OpenSearch project.

https://opensearch.org/blog/community/2022/05/introducing-logstash-input-opensearch-plugin-for-opensearch/
https://github.com/opensearch-project/logstash-input-opensearch
https://opensearch.org/docs/2.4/opensearch/point-in-time/
https://opensearch.org/docs/2.4/api-reference/scroll/

Sample YML should be :

source:
      opensearch:
      version: x
      query: "..."
      indexes:
        - logs-*
      schedule: PT20S