[Feature Request] Allow indexes to be added to data streams (not re-indexed, migrated, etc.) for Cold Storage support or data streams are useless on Amazon Managed OpenSearch

[arianitu]

Is your feature request related to a problem? Please describe

When using Amazon UltraWarm and OpenSearch there is a feature where you can move indices to cold storage via an ISM policy. The issue is, when that index is moved to cold storage, it's removed from the data stream and there is no way to add it back.

This means data stream are essentially useless if you're using Managed Amazon OpenSearch since you cannot move a data stream to cold storage and back and keep all your dashboards and index patterns referencing the data stream working.

I thought I could use a hack, where I create an alias like so:

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": ".ds-data-stream-name-*",
        "alias": "data-stream-all"
      }
    }
  ]
}

but this fails because you cannot have an alias that includes a data stream backing indexes.

So what are you left with... you need to refer to the data stream name data-stream-name and then also the backing indexes for everything .ds-data-stream-name-* for any dashboards, etc.

This is extremely frustrating because this means we basically cannot use data streams.

Describe the solution you'd like

Allow an index to be added to an existing data stream or allow the movement of them which I assume Amazon OpenSearch could utilize.

Related component

Indexing

Describe alternatives you've considered

Alternatives:

1. When an index is moved from cold to warm, we could re-index it back into the data stream but then we lose the index dates and couldn't move it back to cold storage since it's mixed with fresh data
2. Alias on data stream and backing index so the alias could match the indexes moved back but alias do not work either.
3. Do not use data streams at all, instead use regular indexes (with aliases) and apply all ISM policies to this instead, seems super weird since data-streams solve this exact problem but we can't use it

Additional context

No response

[arianitu]

#8271