Updating doc mapping does not update index V2 validation

[rdettai]

Describe the bug
We can update a doc mapping with PUT api/v1/indexes/<index id>, indexing pipelines are automatically restarted, but when we hit the ingest V2 endpoint after the update, documents obeying the new doc mapping are likely be considered as invalid by existing shards that still use the previous doc mapping.

Steps to reproduce (if applicable)
Test was added in https://github.com/quickwit-oss/quickwit/pull/5739/files

Expected behavior
The best solution is likely to force the recreation of all shards for the existing index

[trinity-1686a]

do you happen to know when does the issue resolve by itself? Is that on node restart, or possibly when ingesting plan or something like that evolves?

[rdettai]

If my understanding is correct, it resolves fully when all shards existing before the doc mapping update are closed.

• On a graceful shutdown, I expect a node to close all its shards. I don't think a shard can be re-opened.
• If the graceful shutdown fail, I don't know what happens. I don't know if existing shards can be resumed.

The in-between state where some shards are accepting some docs and others not is tricky...