ES|QL Rerank Command

[afoucret]

Tracked in #123043

The RERANK command will be used to reorder the the result from a first pass query using a semantic reranker.

Usage example:

FROM movies METADATA _score
  | WHERE QSTR("star wars")
  | RERANK "star wars" ON title, overview=SUBSTRING(overview, 0, 100), actors WITH rerankerInferenceId

Development

Done:

• Initial PR - Introducing the rerank command
• Add implicit SORT / LIMIT after the RERANK command
• Licensing: no need since the licensing is enforced by the model upload
• CI fixes ([CI] EsqlSpecIT test {rerank.Reranker using multiple fields SYNC} failing #126320, [CI] EsqlSpecIT test {rerank.Reranker before a limit SYNC} failing #126322)

WIP:

Follow-ups tasks

• [ES|QL] RERANK command - Prevent unnamed fields #126603

• Scoring behavior

  • Fulltext functions after a rerank
  • Should we add the score automatically (ES|QL Reranker command #123074 (comment))?

• Add implicit LIMIT before the RERANK command

• Check if we need to move inference id resolution elsewhere (premaper) convo

• Rerank fields should be optional (use all fields from previous step if not provided

Tech debt

• Use the ModelRegistry service for inference resolution
• Can we parallelized inference and index resolution (see comment ES|QL Reranker command #123074 (review))
• Can we avoid copying the fuill page to build the inference request (ES|QL Reranker command #123074 (comment))?
• Check RerankOperatorTests.testSimpleCircuitBreaking (muted tests because flaky)
• Call the createRerankInferenceEndpoint as part of CSV tests dataset main. (ES|QL Reranker command #123074 (comment))

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Pinging @elastic/kibana-esql (ES|QL-ui)