etcd raftindex never consistent #19701

hongliang5316 · 2025-04-03T01:30:39Z

Bug report criteria

This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
You have read the etcd bug reporting guidelines.
Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

`server logs:`

{"level":"warn","ts":"2025-04-03T08:59:21.650442+0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"epqYPGKRnYXOGunN.50653"}
{"level":"warn","ts":"2025-04-03T08:59:28.227083+0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"JWhaPozgpsUbIhrs.50863"}
{"level":"warn","ts":"2025-04-03T08:59:48.4305+0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"LMUDarkiUhYPgYgl.51574"}
client error:
The client's watcher resp is often canceled
{"level":"warn","ts":"2025-04-03T10:24:55.036264+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00051a780/{ip}:12379","attempt":0,"error":"rpc error: code = Unauthenticated desc = etcdserver: invalid auth token"}
{"level":"warn","ts":"2025-04-03T10:25:02.132197+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00051a780/{ip}:12379","attempt":0,"error":"rpc error: code = Unauthenticated desc = etcdserver: invalid auth token"}
{"level":"warn","ts":"2025-04-03T10:25:02.217804+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00051a780/{ip}:12379","attempt":0,"error":"rpc error: code = Unauthenticated desc = etcdserver: invalid auth token"}
{"level":"warn","ts":"2025-04-03T10:25:07.030061+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00051a780/{ip}:12379","attempt":0,"error":"rpc error: code = Unauthenticated desc = etcdserver: invalid auth token"}
I am very sure that the username and password entered by the client are correct.

What did you expect to happen?

No server or client errors

How can we reproduce it (as minimally and precisely as possible)?

ETCD_NAME=etcd01
ETCD_DATA_DIR=/data/etcd
ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:12379
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
ETCD_ADVERTISE_CLIENT_URLS=http://{ip}:12379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.0.0.6:2380
ETCD_INITIAL_CLUSTER=etcd01=http://10.0.0.6:2380,etcd02=http://10.2.0.8:2380,etcd03=http://10.1.0.17:2380
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_MAX_REQUEST_BYTES=5242880
ETCD_LOG_LEVEL=info
ETCD_LOG_OUTPUTS=/data/log/etcd/etcd.log
ETCD_AUTH_TOKEN=simple
ETCD_AUTH_TOKEN_TTL=3600

ETCD_NAME=etcd02
ETCD_DATA_DIR=/data/etcd
ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:12379
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
ETCD_ADVERTISE_CLIENT_URLS=http://{ip}:12379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.2.0.8:2380
ETCD_INITIAL_CLUSTER=etcd01=http://10.0.0.6:2380,etcd02=http://10.2.0.8:2380,etcd03=http://10.1.0.17:2380
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_MAX_REQUEST_BYTES=5242880
ETCD_LOG_LEVEL=info
ETCD_LOG_OUTPUTS=/data/log/etcd/etcd.log
ETCD_AUTH_TOKEN=simple
ETCD_AUTH_TOKEN_TTL=3600

ETCD_NAME=etcd03
ETCD_DATA_DIR=/data/etcd
ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:12379
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
ETCD_ADVERTISE_CLIENT_URLS=http://{ip}:12379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.1.0.17:2380
ETCD_INITIAL_CLUSTER=etcd01=http://10.0.0.6:2380,etcd02=http://10.2.0.8:2380,etcd03=http://10.1.0.17:2380
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_MAX_REQUEST_BYTES=5242880
ETCD_LOG_LEVEL=info
ETCD_LOG_OUTPUTS=/data/log/etcd/etcd.log
ETCD_AUTH_TOKEN=simple
ETCD_AUTH_TOKEN_TTL=3600

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here
etcd Version: 3.5.12
Git SHA: e7b3bb6cc
Go Version: go1.23.7
Go OS/Arch: linux/amd64
$ etcdctl version
# paste output here

etcdctl version: 3.5.19
API version: 3.5

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

The text was updated successfully, but these errors were encountered:

joshjms · 2025-04-10T18:19:47Z

/assign

jberkus · 2025-04-10T18:21:48Z

Hey, can you retry your issue on the current patch version of 3.5? That is, 3.5.21?

hongliang5316 · 2025-04-12T03:57:13Z

@jberkus

Hello, there is still a problem, but after I disable auth, everything is ok. If I enable auth again, the same problem occurs again.

enable auth

disable auth

joshjms · 2025-04-12T05:02:32Z

I see. @hongliang5316, I'll try to help with this issue 😄 . I'll try to replicate it first, I hope you don't mind if I have any inquiries.

joshjms · 2025-04-12T11:09:51Z

Hi @hongliang5316 , can you tell me the exact command you used and what you did to enable and disable auth?

hongliang5316 · 2025-04-12T12:27:30Z

@joshjms

hello, etcdctl auth disable

joshjms · 2025-04-13T01:39:53Z

Ok, I managed to reproduce the issue. @hongliang5316 can you confirm that this is the exact issue you mentioned?

joshjms · 2025-04-13T01:40:40Z

etcd version: v3.5.21

Create a Cluster

services:
  etcd01:
    image: etcd:3.5.21-0
    container_name: etcd01
    networks:
      etcd_net:
        aliases:
          - etcd01
    ports:
      - "12379:2379"
      - "12380:2380"
    command: >
      etcd
      --name=etcd01
      --data-dir=/etcd-data
      --listen-client-urls=http://0.0.0.0:2379
      --advertise-client-urls=http://etcd01:2379
      --listen-peer-urls=http://0.0.0.0:2380
      --initial-advertise-peer-urls=http://etcd01:2380
      --initial-cluster=etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380
      --initial-cluster-state=new
      --initial-cluster-token=etcd-cluster
      --auth-token=simple
      --auth-token-ttl=3600

  etcd02:
    image: etcd:3.5.21-0
    container_name: etcd02
    networks:
      etcd_net:
        aliases:
          - etcd02
    ports:
      - "22379:2379"
      - "22380:2380"
    command: >
      etcd
      --name=etcd02
      --data-dir=/etcd-data
      --listen-client-urls=http://0.0.0.0:2379
      --advertise-client-urls=http://etcd02:2379
      --listen-peer-urls=http://0.0.0.0:2380
      --initial-advertise-peer-urls=http://etcd02:2380
      --initial-cluster=etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380
      --initial-cluster-state=new
      --initial-cluster-token=etcd-cluster
      --auth-token=simple
      --auth-token-ttl=3600

  etcd03:
    image: etcd:3.5.21-0
    container_name: etcd03
    networks:
      etcd_net:
        aliases:
          - etcd03
    ports:
      - "32379:2379"
      - "32380:2380"
    command: >
      etcd
      --name=etcd03
      --data-dir=/etcd-data
      --listen-client-urls=http://0.0.0.0:2379
      --advertise-client-urls=http://etcd03:2379
      --listen-peer-urls=http://0.0.0.0:2380
      --initial-advertise-peer-urls=http://etcd03:2380
      --initial-cluster=etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380
      --initial-cluster-state=new
      --initial-cluster-token=etcd-cluster
      --auth-token=simple
      --auth-token-ttl=3600
  client:
    image: etcd:3.5.21-0
    container_name: etcd-client
    networks:
      etcd_net:
        aliases:
          - etcd-client
    depends_on:
      - etcd01
      - etcd02
      - etcd03
    tty: true
    stdin_open: true
    entrypoint: /bin/sh

networks:
  etcd_net:
    driver: bridge

In the terminal:

docker attach etcd-client

Create User

etcdctl \
    --endpoints=http://etcd01:2379 \
    user add root:joshjmswashere
    
etcdctl \
    --endpoints=http://etcd01:2379 \
    user grant-role root root

Enable Auth

etcdctl \
    --endpoints=http://etcd01:2379 \
    auth enable

Show Status Table

etcdctl \
  --endpoints=http://etcd01:2379,http://etcd02:2379,http://etcd03:2379 \
  --user root:joshjmswashere \
  endpoint status --write-out=table

Result

+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://etcd01:2379 | b8b747c74aaea686 |  3.5.21 |   20 kB |      true |      false |         2 |         17 |                 17 |        |
| http://etcd02:2379 |  b3504381e8ba3cb |  3.5.21 |   20 kB |     false |      false |         2 |         18 |                 18 |        |
| http://etcd03:2379 | f572fdfc5cb68406 |  3.5.21 |   20 kB |     false |      false |         2 |         19 |                 19 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Disable Auth

etcdctl \
    --endpoints=http://etcd01:2379 \ 
    auth disable

Show Status Table

etcdctl \
  --endpoints=http://etcd01:2379,http://etcd02:2379,http://etcd03:2379 \
  --user root:joshjmswashere \
  endpoint status --write-out=table

Result

{"level":"warn","ts":"2025-04-13T01:31:55.499544Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000024000/etcd01:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: authentication is not enabled"}
{"level":"warn","ts":"2025-04-13T01:31:55.500523Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000024000/etcd01:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: authentication is not enabled"}
{"level":"warn","ts":"2025-04-13T01:31:55.503404Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000024000/etcd01:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: authentication is not enabled"}
{"level":"warn","ts":"2025-04-13T01:31:55.505539Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000024000/etcd01:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: authentication is not enabled"}
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://etcd01:2379 | b8b747c74aaea686 |  3.5.21 |   20 kB |      true |      false |         2 |         22 |                 22 |        |
| http://etcd02:2379 |  b3504381e8ba3cb |  3.5.21 |   20 kB |     false |      false |         2 |         22 |                 22 |        |
| http://etcd03:2379 | f572fdfc5cb68406 |  3.5.21 |   20 kB |     false |      false |         2 |         22 |                 22 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

cc @henrybear327 pls lmk if i did anything wrong thx!

hongliang5316 · 2025-04-13T01:49:04Z

@joshjms

Hello, Yes, I do the same as you. This should be abnormal.

henrybear327 · 2025-04-13T09:21:57Z

etcd version: v3.5.21

Create a Cluster

services:
  etcd01:
    image: etcd:3.5.21-0
    container_name: etcd01
    networks:
      etcd_net:
        aliases:
          - etcd01
    ports:
      - "12379:2379"
      - "12380:2380"
    command: >
      etcd
      --name=etcd01
      --data-dir=/etcd-data
      --listen-client-urls=http://0.0.0.0:2379
      --advertise-client-urls=http://etcd01:2379
      --listen-peer-urls=http://0.0.0.0:2380
      --initial-advertise-peer-urls=http://etcd01:2380
      --initial-cluster=etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380
      --initial-cluster-state=new
      --initial-cluster-token=etcd-cluster
      --auth-token=simple
      --auth-token-ttl=3600

  etcd02:
    image: etcd:3.5.21-0
    container_name: etcd02
    networks:
      etcd_net:
        aliases:
          - etcd02
    ports:
      - "22379:2379"
      - "22380:2380"
    command: >
      etcd
      --name=etcd02
      --data-dir=/etcd-data
      --listen-client-urls=http://0.0.0.0:2379
      --advertise-client-urls=http://etcd02:2379
      --listen-peer-urls=http://0.0.0.0:2380
      --initial-advertise-peer-urls=http://etcd02:2380
      --initial-cluster=etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380
      --initial-cluster-state=new
      --initial-cluster-token=etcd-cluster
      --auth-token=simple
      --auth-token-ttl=3600

  etcd03:
    image: etcd:3.5.21-0
    container_name: etcd03
    networks:
      etcd_net:
        aliases:
          - etcd03
    ports:
      - "32379:2379"
      - "32380:2380"
    command: >
      etcd
      --name=etcd03
      --data-dir=/etcd-data
      --listen-client-urls=http://0.0.0.0:2379
      --advertise-client-urls=http://etcd03:2379
      --listen-peer-urls=http://0.0.0.0:2380
      --initial-advertise-peer-urls=http://etcd03:2380
      --initial-cluster=etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380
      --initial-cluster-state=new
      --initial-cluster-token=etcd-cluster
      --auth-token=simple
      --auth-token-ttl=3600
  client:
    image: etcd:3.5.21-0
    container_name: etcd-client
    networks:
      etcd_net:
        aliases:
          - etcd-client
    depends_on:
      - etcd01
      - etcd02
      - etcd03
    tty: true
    stdin_open: true
    entrypoint: /bin/sh

networks:
  etcd_net:
    driver: bridge

In the terminal:

docker attach etcd-client

Create User

etcdctl \
    --endpoints=http://etcd01:2379 \
    user add root:joshjmswashere
    
etcdctl \
    --endpoints=http://etcd01:2379 \
    user grant-role root root

Enable Auth

etcdctl \
    --endpoints=http://etcd01:2379 \
    auth enable

Show Status Table

etcdctl \
  --endpoints=http://etcd01:2379,http://etcd02:2379,http://etcd03:2379 \
  --user root:joshjmswashere \
  endpoint status --write-out=table

Result

+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://etcd01:2379 | b8b747c74aaea686 |  3.5.21 |   20 kB |      true |      false |         2 |         17 |                 17 |        |
| http://etcd02:2379 |  b3504381e8ba3cb |  3.5.21 |   20 kB |     false |      false |         2 |         18 |                 18 |        |
| http://etcd03:2379 | f572fdfc5cb68406 |  3.5.21 |   20 kB |     false |      false |         2 |         19 |                 19 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Disable Auth

etcdctl \
    --endpoints=http://etcd01:2379 \ 
    auth disable

Show Status Table

etcdctl \
  --endpoints=http://etcd01:2379,http://etcd02:2379,http://etcd03:2379 \
  --user root:joshjmswashere \
  endpoint status --write-out=table

Result

{"level":"warn","ts":"2025-04-13T01:31:55.499544Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000024000/etcd01:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: authentication is not enabled"}
{"level":"warn","ts":"2025-04-13T01:31:55.500523Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000024000/etcd01:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: authentication is not enabled"}
{"level":"warn","ts":"2025-04-13T01:31:55.503404Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000024000/etcd01:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: authentication is not enabled"}
{"level":"warn","ts":"2025-04-13T01:31:55.505539Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000024000/etcd01:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: authentication is not enabled"}
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://etcd01:2379 | b8b747c74aaea686 |  3.5.21 |   20 kB |      true |      false |         2 |         22 |                 22 |        |
| http://etcd02:2379 |  b3504381e8ba3cb |  3.5.21 |   20 kB |     false |      false |         2 |         22 |                 22 |        |
| http://etcd03:2379 | f572fdfc5cb68406 |  3.5.21 |   20 kB |     false |      false |         2 |         22 |                 22 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

cc @henrybear327 pls lmk if i did anything wrong thx!

Thanks for the nice write-up! ;)

/cc @jberkus @ahrtr @serathius @fuweid

joshjms · 2025-04-13T11:44:02Z

The same issue is also encountered when using etcd:3.6.0-rc.2-0. I think it's important to note that there is no stale data despite the difference in Raft index (i.e. when querying from any node, the latest update is still shown even though the raft index shown in the table is different).

etcdctl \
    --endpoints=http://etcd03:2379 \
    --user root:joshjmswashere \
    put counter 1
OK
etcdctl \
    --endpoints=http://etcd02:2379 \
    --user root:joshjmswashere \
    get counter
counter
1
etcdctl \
    --endpoints=http://etcd02:2379 \
    --user root:joshjmswashere \
    put counter 2
OK
etcdctl \
    --endpoints=http://etcd03:2379 \
    --user root:joshjmswashere \
    get counter
counter
2

ahrtr · 2025-04-14T07:41:23Z

This isn't an issue. It's expected behaviour. When Auth is enabled, each time when client connects to the server, it will authenticate and get a token, so the raftIndex will advance by 1; and etcdctl endpoint stauts get all members's status one by one, so next member will always get a bigger raftIndex.

$ ./bin/etcdctl --endpoints=http://127.0.0.1:22379,http://127.0.0.1:32379,http://127.0.0.1:2379 endpoint status -w table --user root:root
+------------------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
|        ENDPOINT        |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+------------------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| http://127.0.0.1:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 |           3.6.0 |   98 kB |  98 kB |                    0% |   0 B |     false |      false |         2 |         33 |                 33 |        |                          |             false |
| http://127.0.0.1:32379 | fd422379fda50e48 | 3.6.0-alpha.0 |           3.6.0 |   98 kB |  98 kB |                    0% |   0 B |      true |      false |         2 |         34 |                 34 |        |                          |             false |
|  http://127.0.0.1:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 |           3.6.0 |   98 kB |  98 kB |                    0% |   0 B |     false |      false |         2 |         35 |                 35 |        |                          |             false |
+------------------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+


$ ./bin/etcdctl --endpoints=http://127.0.0.1:22379,http://127.0.0.1:32379,http://127.0.0.1:2379 endpoint status -w table --user root:root
+------------------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
|        ENDPOINT        |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+------------------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| http://127.0.0.1:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 |           3.6.0 |   98 kB |  98 kB |                    0% |   0 B |     false |      false |         2 |         36 |                 36 |        |                          |             false |
| http://127.0.0.1:32379 | fd422379fda50e48 | 3.6.0-alpha.0 |           3.6.0 |   98 kB |  98 kB |                    0% |   0 B |      true |      false |         2 |         37 |                 37 |        |                          |             false |
|  http://127.0.0.1:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 |           3.6.0 |   98 kB |  98 kB |                    0% |   0 B |     false |      false |         2 |         38 |                 38 |        |                          |             false |
+------------------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+


$ ./bin/etcdctl --endpoints=http://127.0.0.1:22379,http://127.0.0.1:32379,http://127.0.0.1:2379 endpoint status -w table --user root:root
+------------------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
|        ENDPOINT        |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+------------------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| http://127.0.0.1:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 |           3.6.0 |   98 kB |  98 kB |                    0% |   0 B |     false |      false |         2 |         39 |                 39 |        |                          |             false |
| http://127.0.0.1:32379 | fd422379fda50e48 | 3.6.0-alpha.0 |           3.6.0 |   98 kB |  98 kB |                    0% |   0 B |      true |      false |         2 |         40 |                 40 |        |                          |             false |
|  http://127.0.0.1:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 |           3.6.0 |   98 kB |  98 kB |                    0% |   0 B |     false |      false |         2 |         41 |                 41 |        |                          |             false |
+------------------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+

joshjms · 2025-04-14T07:58:58Z

Ah I see, thanks @ahrtr

hongliang5316 added the type/bug label Apr 3, 2025

siyuanfoundation added the stage/triaged label Apr 10, 2025

k8s-ci-robot assigned joshjms Apr 10, 2025

ahrtr closed this as completed Apr 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etcd raftindex never consistent #19701

etcd raftindex never consistent #19701

hongliang5316 commented Apr 3, 2025 •

edited

Loading

paste your configuration here

joshjms commented Apr 10, 2025

jberkus commented Apr 10, 2025 •

edited

Loading

hongliang5316 commented Apr 12, 2025 •

edited

Loading

joshjms commented Apr 12, 2025

joshjms commented Apr 12, 2025

hongliang5316 commented Apr 12, 2025 •

edited

Loading

joshjms commented Apr 13, 2025 •

edited

Loading

joshjms commented Apr 13, 2025 •

edited

Loading

hongliang5316 commented Apr 13, 2025

henrybear327 commented Apr 13, 2025 •

edited

Loading

Create a Cluster

Create User

Enable Auth

Show Status Table

Result

Disable Auth

Show Status Table

Result

joshjms commented Apr 13, 2025 •

edited

Loading

ahrtr commented Apr 14, 2025

joshjms commented Apr 14, 2025

etcd raftindex never consistent #19701

etcd raftindex never consistent #19701

Comments

hongliang5316 commented Apr 3, 2025 • edited Loading

Bug report criteria

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Etcd version (please run commands below)

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Relevant log output

joshjms commented Apr 10, 2025

jberkus commented Apr 10, 2025 • edited Loading

hongliang5316 commented Apr 12, 2025 • edited Loading

enable auth

disable auth

joshjms commented Apr 12, 2025

joshjms commented Apr 12, 2025

hongliang5316 commented Apr 12, 2025 • edited Loading

joshjms commented Apr 13, 2025 • edited Loading

joshjms commented Apr 13, 2025 • edited Loading

Create a Cluster

Create User

Enable Auth

Show Status Table

Result

Disable Auth

Show Status Table

Result

hongliang5316 commented Apr 13, 2025

henrybear327 commented Apr 13, 2025 • edited Loading

Create a Cluster

Create User

Enable Auth

Show Status Table

Result

Disable Auth

Show Status Table

Result

joshjms commented Apr 13, 2025 • edited Loading

ahrtr commented Apr 14, 2025

joshjms commented Apr 14, 2025

hongliang5316 commented Apr 3, 2025 •

edited

Loading

jberkus commented Apr 10, 2025 •

edited

Loading

hongliang5316 commented Apr 12, 2025 •

edited

Loading

hongliang5316 commented Apr 12, 2025 •

edited

Loading

joshjms commented Apr 13, 2025 •

edited

Loading

joshjms commented Apr 13, 2025 •

edited

Loading

henrybear327 commented Apr 13, 2025 •

edited

Loading

joshjms commented Apr 13, 2025 •

edited

Loading