Feature/info request datastream migration

[ryemelia]

Hi,
is there any way how to migrate datastreams via RfsMigrateDocuments?

as i see now

1. i can not target datastream name(logs-datasetv1-events) via --index-allowlist as backing index will be skipped and nothing happen

.ds-logs-datasetv1-events-000001 will be reindexed, it was not included in the allowlist: [logs-datasetv1-events]

2. i can not use backing index restore as opensearch doesn't have "add index as backing to DS" API so i will not be able to attach to to datastream

from .ds-logs-datasetv1-events-000001  to .ds-logs-datasetv1-events-000001 

3. i can not restore to existing index

2025-10-29 15:26:04,024 WARN o.o.m.b.c.OpenSearchClient [reactor-http-epoll-3] After bulk request attempt 1 on index '.ds-logs-datasetv1-events-000001', 0 more documents have succeeded, 9609 remain. The error response message was: {"took":24,"errors":true,"items":[{"index":{"_index":".ds-logs-datasetv1-events-000001","_id":"5b9356ef-6873-4ed8-8e1a-e3cbaf6bd127","status":400,"error":{"type":"illegal_argument_exception","reason":"index request with op_type=index and no if_primary_term and if_seq_no set targeting backing indices is disallowed, target corresponding data stream [logs-datasetv1-events] instead"}}},{"index":{"_index":".ds-logs-datasetv1-events-000001","_id":"24ccfe8e-4542-4a5c-8eec-8ecf2d438805","status":400,"error":{"type":"illegal_argument_exception","reason":"index request with op_type=index and no if_primary_term and if_seq_no set targeting backing indices is disallowed, target corresponding data stream [logs-datasetv1-events] instea... [truncated] ...1-events] instead"}}},{"index":{"_index":".ds-logs-datasetv1-events-000001","_id":"d1c444f1-2abe-4f6d-9e19-ae26da769ff1","status":400,"error":{"type":"illegal_argument_exception","reason":"index request with op_type=index and no if_primary_term and if_seq_no set targeting backing indices is disallowed, target corresponding data stream [logs-datasetv1-events] instead"}}},{"index":{"_index":".ds-logs-datasetv1-events-000001","_id":"dd245c91-d225-4c8a-a87a-74a42706750c","status":400,"error":{"type":"illegal_argument_exception","reason":"index request with op_type=index and no if_primary_term and if_seq_no set targeting backing indices is disallowed, target corresponding data stream [logs-datasetv1-events] instead"}}}]}

4. i can not use different output for this command to migrate from .ds-logs-datasetv1-events-000001 index to DS name(logs-datasetv1-events) instead

migrating from elasticsearch 7.17.28 to opensearch 2.19

Could someone help here please

[peternied]

Thanks for calling this out - data streams are not supported today, we have a tracking issue for work that has of yet to be picked up or worked on [MIGRATIONS-2639 | Support for Data streams].

@ryemelia what do you think about making a contribution for this feature? I have a branch [1] with a failing test case that is a good starting point.

• [1] main...peternied:opensearch-migrations:metadata-datastream-test

[AndreKurait]

Experimental DataStream Support was released in MA 2.5.12.

This is an RFS only support which means the OpenSearch DataStream will need to be created out of band.

Once a DataStream is created with the same name as on the source, docs can be extracted and migrated with the following:

1. In cdk.context.json
   Add or update the key reindexFromSnapshotExtraArgs to include --doc-transformer-config-file /shared-logs-output/transformation.json
2. In cdk.context.json
   Add or update the key reindexFromSnapshotExtraArgs to include --index-allowlist 'regex:.ds.*'
3. in cdk.context.json
   Add or update the key reindexFromSnapshotExtraArgs to include --allowed-doc-exception-types version_conflict_engine_exception
4. Trigger a deployment cdk deploy...
5. From the Migration Console, create file at shared-logs-output/transformation.json with content:

[
  {
    "JsonJSTransformerProvider": {
      "initializationResourcePath": "js/dataStreamBackingIndexTransform.js",
      "bindingsObject": "{}"
    }
  }
]

6. Trigger console backfill stop to make sure it is stopped and working state is reset
7. Trigger console backfill start

Note: This will migrate all backing indices into the data stream without regards to ordering. Validation should be made to verify performance. If ordering is required, at this point users will need to perform different migrations for different backing indices. e.g.

--index-allowlist '.ds-my-data-stream-000001'

Followed by a rollover

POST my-data-stream/_rollover

Then a subsequent migration

--index-allowlist .ds-my-data-stream-000002

and

POST my-data-stream/_rollover

etc..