Failure stores cannot be replicated in CCR

[jbaiera]

We currently filter failure store indices out from being considered for auto follow logic when processing data streams (see #126355). This is because a write index must always be present on a data stream. If a failure index is the only index from a cluster that is replicated (i.e. no other backing indices are replicated), the resulting follower data stream is not guaranteed to satisfy this invariant. Even if all indices are replicated, the construction of the data stream to hold them is done as part of the follow operation. If follow operations are done in arbitrary order and are subject to independent failure, we can't guarantee that a data stream will always have a write index in its backing index set.

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[jbaiera]

Some suggestions on path forward:

When replicating a failure store index, if it is the first follow operation to create the data stream, then it could create an empty write index to satisfy the data stream invariant. This has the downside of potentially producing a follower data stream that is not structurally identical to the leader data stream. The upside is that this would be the simplest solution.

Alternatively, we could refactor data streams to ease this invariant of always requiring a write index. Data streams now have a feature that allows them to be lazily rolled over when they receive their first document write operation. We use this with failure stores to ensure that they are always ready to accept a write, while not actually allocating any indices up front. If we added this ability to the regular backing indices on a data stream (that a data stream could have no indices as long as it is marked for lazy rollover) then we can work on relaxing the invariant that a data stream always has a write index. Once eased, we no longer need to worry about replication order outside of the eventual ordering of the backing indices.