Bump format version from 0.5 to 0.6 (#3338)

Changes the serialization version from 0.5 to 0.6 and cleans up
duplicate backward compatibility test files for older versions.

Author: imotov

.github/workflows/ui-ci.yml

@@ -26,7 +26,8 @@ jobs:
               CI=false yarn --cwd quickwit-ui build
               cargo build --features=postgres
               mkdir qwdata
-              cargo run --features=postgres -- run --service searcher --service metastore --config ../config/quickwit.yaml &
+              # Workaround for possible rust regression - https://github.com/quickwit-oss/quickwit/issues/3346
+              target/debug/quickwit run --service searcher --service metastore --config ../config/quickwit.yaml &
               yarn --cwd quickwit-ui cypress run
           - name: Lint
             command: yarn --cwd quickwit-ui lint

config/quickwit.yaml

@@ -9,7 +9,7 @@
 #
 # Config file format version.
 #
-version: 0.5
+version: 0.6
 #
 # Node ID. Must be unique within a cluster. If not set, a random node ID is generated on each startup.
 #

config/tutorials/gh-archive/index-config-for-clickhouse.yaml

@@ -1,7 +1,7 @@
 #
 # Index config file for gh-archive dataset.
 #
-version: 0.5
+version: 0.6
 
 index_id: gh-archive
 

config/tutorials/gh-archive/index-config.yaml

@@ -1,7 +1,7 @@
 #
 # Index config file for gh-archive dataset.
 #
-version: 0.5
+version: 0.6
 
 index_id: gh-archive
 

config/tutorials/gh-archive/kafka-source.yaml

@@ -1,4 +1,4 @@
-version: 0.5
+version: 0.6
 source_id: kafka-source
 source_type: kafka
 params:

config/tutorials/gh-archive/kinesis-source.yaml

@@ -1,4 +1,4 @@
-version: 0.5
+version: 0.6
 source_id: kinesis-source
 source_type: kinesis
 params:

config/tutorials/hdfs-logs/index-config-partitioned.yaml

@@ -2,7 +2,7 @@
 # Index config file for hdfs-logs dataset with partitioning configured.
 #
 
-version: 0.5
+version: 0.6
 
 index_id: hdfs-logs-partitioned
 

config/tutorials/hdfs-logs/index-config-retention-policy.yaml

@@ -2,7 +2,7 @@
 # Index config file for hdfs-logs dataset with a retention policy configured.
 #
 
-version: 0.5
+version: 0.6
 
 index_id: hdfs-logs-retention-policy
 

config/tutorials/hdfs-logs/index-config.yaml

@@ -2,7 +2,7 @@
 # Index config file for hdfs-logs dataset.
 #
 
-version: 0.5
+version: 0.6
 
 index_id: hdfs-logs
 

config/tutorials/hdfs-logs/searcher-1.yaml

@@ -1,4 +1,4 @@
-version: 0.5
+version: 0.6
 node_id: searcher-1
 listen_address: 127.0.0.1
 rest_listen_port: 7280

config/tutorials/hdfs-logs/searcher-2.yaml

@@ -1,4 +1,4 @@
-version: 0.5
+version: 0.6
 node_id: searcher-2
 listen_address: 127.0.0.1
 rest_listen_port: 7290

config/tutorials/hdfs-logs/searcher-3.yaml

@@ -1,4 +1,4 @@
-version: 0.5
+version: 0.6
 node_id: searcher-3
 listen_address: 127.0.0.1
 rest_listen_port: 7300

config/tutorials/otel-logs/index-config.yaml

@@ -3,7 +3,7 @@
 # Link: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md
 #
 
-version: 0.5
+version: 0.6
 
 index_id: otel-log-v0
 

config/tutorials/otel-trace/index-config.yaml

@@ -3,7 +3,7 @@
 # Link: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md
 #
 
-version: 0.5
+version: 0.6
 
 index_id: otel-trace-v0
 

config/tutorials/stackoverflow/index-config.yaml

@@ -1,7 +1,7 @@
 #
 # Index config file for stackoverflow dataset.
 #
-version: 0.5
+version: 0.6
 
 index_id: stackoverflow
 

config/tutorials/stackoverflow/pulsar-source.yaml

@@ -1,4 +1,4 @@
-version: 0.5
+version: 0.6
 source_id: pulsar-source
 source_type: pulsar
 params:

config/tutorials/vector-otel-logs/index-config.yaml

@@ -3,7 +3,7 @@
 # Link: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md
 #
 
-version: 0.5
+version: 0.6
 
 index_id: vector-otel-logs
 

config/tutorials/wikipedia/index-config.yaml

@@ -2,7 +2,7 @@
 # Index config file for wikipedia dataset.
 #
 
-version: 0.5
+version: 0.6
 
 index_id: wikipedia
 

docs/configuration/index-config.md

@@ -20,7 +20,7 @@ The index configuration format is YAML. When a key is absent from the configurat
 Here is a complete example suited for the HDFS logs dataset:
 
 ```yaml
-version: 0.5 # File format version.
+version: 0.6 # File format version.
 
 index_id: "hdfs"
 
@@ -387,7 +387,7 @@ The configuration of `dynamic` mode can be set via the `dynamic_mapping` paramet
 `dynamic_mapping` offers the same configuration options as when configuring a `json` field. It defaults to:
 
 ```yaml
-version: 0.5
+version: 0.6
 index_id: my-dynamic-index
 doc_mapping:
   mode: dynamic
@@ -407,7 +407,7 @@ the root of the JSON object.
 For instance, in a entirely schemaless settings, a minimal index configuration could be:
 
 ```yaml
-version: 0.5
+version: 0.6
 index_id: my-dynamic-index
 # note we did not map anything.
 doc_mapping:
@@ -492,7 +492,7 @@ Quickwit's default merge policy is the `stable_log` merge policy
 with the following parameters:
 
 ```yaml
-version: 0.5
+version: 0.6
 index_id: "hdfs"
 # ...
 indexing_settings:
@@ -520,7 +520,7 @@ of the number of merge operation a split should undergo.
 
 
 ```yaml
-version: 0.5
+version: 0.6
 index_id: "hdfs"
 # ...
 indexing_settings:
@@ -548,7 +548,7 @@ This setting is not recommended. Merges are necessary to reduce the number of sp
 :::
 
 ```yaml
-version: 0.5
+version: 0.6
 index_id: "hdfs"
 indexing_settings:
     merge_policy:
@@ -575,7 +575,7 @@ This section describes search settings for a given index.
 This section describes how Quickwit manages data retention. In Quickwit, the retention policy manager drops data on a split basis as opposed to individually dropping documents. Splits are evaluated based on their `time_range` which is derived from the index timestamp field specified in the (`indexing_settings.timestamp_field`) settings. Using this setting, the retention policy will delete a split when `now() - split.time_range.end >= retention_policy.period`
 
 ```yaml
-version: 0.5
+version: 0.6
 index_id: hdfs
 # ...
 retention:

docs/configuration/node-config.md

@@ -110,7 +110,7 @@ export QW_LISTEN_ADDRESS=0.0.0.0
 
 ```yaml
 # config.yaml
-version: 0.5
+version: 0.6
 cluster_id: quickwit-cluster
 node_id: my-unique-node-id
 listen_address: ${QW_LISTEN_ADDRESS}
@@ -120,7 +120,7 @@ rest_listen_port: ${QW_LISTEN_PORT:-1111}
 Will be interpreted by Quickwit as:
 
 ```yaml
-version: 0.5
+version: 0.6
 cluster_id: quickwit-cluster
 node_id: my-unique-node-id
 listen_address: 0.0.0.0
@@ -135,7 +135,7 @@ by default no origins are allowed.
 A wildcard, single origin or multiple origins can be specified as part of the `rest_cors_allow_origins` parameter:
 
 ```yaml
-version: 0.5
+version: 0.6
 index_id: hdfs
 
 rest_cors_allow_origins: '*'                                 # Allow all origins

docs/configuration/source-config.md

@@ -77,7 +77,7 @@ Short max poll interval durations may cause a source to crash when back pressure
 
 ```bash
 cat << EOF > source-config.yaml
-version: 0.5
+version: 0.6
 source_id: my-kafka-source
 source_type: kafka
 params:
@@ -119,7 +119,7 @@ If no region is specified, Quickwit will attempt to find one in multiple other l
 
 ```bash
 cat << EOF > source-config.yaml
-version: 0.5
+version: 0.6
 source_id: my-kinesis-source
 source_type: kinesis
 params:
@@ -148,7 +148,7 @@ The Pulsar source consumes `topics` using the client library [pulsar-rs](https:/
 
 ```bash
 cat << EOF > source-config.yaml
-version: 0.5
+version: 0.6
 source_id: my-pulsar-source
 source_type: pulsar
 params:

docs/distributed-tracing/otel-service.md

@@ -27,7 +27,7 @@ Below is the doc mapping of the `otel-trace-v0` index:
 
 ```yaml
 
-version: 0.5
+version: 0.6
 
 index_id: otel-trace-v0
 

docs/get-started/quickstart.md

@@ -87,7 +87,7 @@ And here is the complete config:
 #
 # Index config file for stackoverflow dataset.
 #
-version: 0.5
+version: 0.6
 
 index_id: stackoverflow
 

docs/get-started/tutorials/add-full-text-search-to-your-olap-db.md

@@ -51,7 +51,7 @@ We don't need to index all fields described above as `title` and `body` are the
 The `id` will be helpful for making the JOINs in ClickHouse, `created_at` and `event_type` may also be beneficial for timestamp pruning and filtering.
 
 ```yaml title="gh-archive-index-config.yaml"
-version: 0.5
+version: 0.6
 index_id: gh-archive
 # By default, the index will be stored in your data directory,
 # but you can store it on s3 or on a custom path as follows:

docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md

@@ -57,7 +57,7 @@ Now we can create a Quickwit config file.
 
 ```bash
 # Create Quickwit config file.
-echo "version: 0.5
+echo "version: 0.6
 node_id: searcher-1
 listen_address: 0.0.0.0
 metastore_uri: ${S3_PATH}
@@ -70,7 +70,7 @@ default_index_root_uri: ${S3_PATH}
 > # config.yaml
 > node_id: searcher-1
 > listen_address: 0.0.0.0
-> version: 0.5
+> version: 0.6
 > metastore_uri: ${S3_PATH}
 > default_index_root_uri: ${S3_PATH}
 >```
@@ -95,7 +95,7 @@ used by Quickwit for [splits pruning](../../overview/architecture) at query time
 Check out the [index config docs](../../configuration/index-config) for more details.
 
 ```yaml title="hdfs_logs_index_config.yaml"
-version: 0.5
+version: 0.6
 
 index_id: hdfs-logs
 
@@ -195,7 +195,7 @@ export IP_NODE_1={first-ec2-instance-public-ip}
 
 ```bash
 # configuration for our second node
-echo "version: 0.5
+echo "version: 0.6
 node_id: searcher-2
 metastore_uri: ${S3_PATH}
 default_index_root_uri: ${S3_PATH}
@@ -210,7 +210,7 @@ peer_seeds:
 
 ```bash
 # configuration for our third node
-echo "version: 0.5
+echo "version: 0.6
 node_id: searcher-3
 listen_address: 0.0.0.0
 peer_seeds:

docs/get-started/tutorials/tutorial-hdfs-logs.md

@@ -84,7 +84,7 @@ The `timestamp_field` and `tag_fields` are used by Quickwit for [splits pruning]
 Check out the [index config docs](../../configuration/index-config) for more details.
 
 ```yaml title="hdfs-logs-index.yaml"
-version: 0.5
+version: 0.6
 
 index_id: hdfs-logs
 

docs/guides/schemaless.md

@@ -25,7 +25,7 @@ As a user, you need to precisely define the list of fields to be ingested by Qui
 For instance, a reasonable mapping for an application log could be:
 
 ```yaml title=my_strict_index.yaml
-version: 0.5
+version: 0.6
 
 index_id: my_strict_index
 
@@ -70,7 +70,7 @@ By default, this catch-all configuration indexes and stores all of these fields,
 A minimalist, yet perfectly valid and useful index configuration is then:
 
 ```yaml title=my_dynamic_index.yaml
-version: 0.5
+version: 0.6
 index_id: my_dynamic_index
 doc_mapping:
   mode: dynamic
@@ -115,7 +115,7 @@ Each event type comes with its own set of attributes. Declaring our mapping as t
 Instead, we can cherry-pick the fields that are common to all of the logs, and rely on dynamic mode to handle the rest.
 
 ```yaml title=my_dynamic_index.yaml
-version: 0.5
+version: 0.6
 index_id: my_dynamic_index
 doc_mapping:
   mode: dynamic
@@ -149,7 +149,7 @@ Execute the following commands to create the index, ingest a few documents and s
 
 ```bash
 cat << EOF > my_dynamic_index.yaml
-version: 0.5
+version: 0.6
 index_id: my_dynamic_index
 doc_mapping:
   mode: dynamic
@@ -219,7 +219,7 @@ Quickwit 0.3 introduced a JSON field type to handle this use case.
 A good index configuration here could be:
 
 ```yaml title=otel_logs.yaml
-version: 0.5
+version: 0.6
 index_id: otel_logs
 doc_mapping:
   mode: dynamic

docs/ingest-data/ingest-api.md

@@ -22,7 +22,7 @@ First, let's create a schemaless index.
 ```bash
 # Create the index config file.
 cat << EOF > stackoverflow-schemaless-config.yaml
-version: 0.5
+version: 0.6
 index_id: stackoverflow-schemaless
 doc_mapping:
   mode: dynamic