You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
`Self-managed` is also used in the context of Cloud vs on-prem and it can
be confusing.
`KRaft` is a cute combination of `Kafka Raft` and it's pronounced like `craft`
(as in `craftsmanship`).
Reviewers: Colin P. McCabe <[email protected]>, Jose Sancio <[email protected]>, Chia-Ping Tsai <[email protected]>, Ron Dagostino <[email protected]>
It is now possible to run Apache Kafka without Apache ZooKeeper! We call this mode [self-managed mode](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum). It is currently *EARLY ACCESS AND SHOULD NOT BE USED IN PRODUCTION*, but it is available for testing in the Kafka 2.8 release.
5
+
It is now possible to run Apache Kafka without Apache ZooKeeper! We call this the [Kafka Raft metadata mode](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum), typically shortened to `KRaft mode`.
6
+
`KRaft` is intended to be pronounced like `craft` (as in `craftsmanship`). It is currently *EARLY ACCESS AND SHOULD NOT BE USED IN PRODUCTION*, but it
7
+
is available for testing in the Kafka 2.8 release.
6
8
7
-
When the Kafka cluster is in self-managed mode, it does not store its metadata in ZooKeeper. In fact, you do not have to run ZooKeeper at all, because it stores its metadata in a Raft quorum of controller nodes.
9
+
When the Kafka cluster is in KRaft mode, it does not store its metadata in ZooKeeper. In fact, you do not have to run ZooKeeper at all, because it stores its metadata in a KRaft quorum of controller nodes.
8
10
9
-
Self-managed mode has many benefits -- some obvious, and some not so obvious. Clearly, it is nice to manage and configure one service rather than two services. In addition, you can now run a single process Kafka cluster. Most important of all, self-managed mode is more scalable. We expect to be able to [support many more topics and partitions](https://www.confluent.io/kafka-summit-san-francisco-2019/kafka-needs-no-keeper/) in this mode.
11
+
KRaft mode has many benefits -- some obvious, and some not so obvious. Clearly, it is nice to manage and configure one service rather than two services. In addition, you can now run a single process Kafka cluster.
12
+
Most important of all, KRaft mode is more scalable. We expect to be able to [support many more topics and partitions](https://www.confluent.io/kafka-summit-san-francisco-2019/kafka-needs-no-keeper/) in this mode.
10
13
11
14
# Quickstart
12
15
13
16
## Warning
14
-
Self-managed mode in Kafka 2.8 is provided for testing only, *NOT* for production. We do not yet support upgrading existing ZooKeeper-based Kafka clusters into this mode. In fact, when Kafka 3.0 is released, it will not be possible to upgrade your self-managed clusters from 2.8 to 3.0. There may be bugs, including serious ones. You should *assume that your data could be lost at any time* if you try the early access release of self-managed mode.
17
+
KRaft mode in Kafka 2.8 is provided for testing only, *NOT* for production. We do not yet support upgrading existing ZooKeeper-based Kafka clusters into this mode. In fact, when Kafka 3.0 is released,
18
+
it will not be possible to upgrade your KRaft clusters from 2.8 to 3.0. There may be bugs, including serious ones. You should *assume that your data could be lost at any time* if you try the early access release of KRaft mode.
15
19
16
20
## Generate a cluster ID
17
21
The first step is to generate an ID for your new cluster, using the kafka-storage tool:
@@ -25,8 +29,8 @@ xtzWWN4bTjitpL3kfd9s5g
25
29
The next step is to format your storage directories. If you are running in single-node mode, you can do this with one command:
26
30
27
31
~~~~
28
-
$ ./bin/kafka-storage.sh format -t <uuid> -c ./config/self-managed/server.properties
29
-
Formatting /tmp/self-managed-combined-logs
32
+
$ ./bin/kafka-storage.sh format -t <uuid> -c ./config/kraft/server.properties
33
+
Formatting /tmp/kraft-combined-logs
30
34
~~~~
31
35
32
36
If you are using multiple nodes, then you should run the format command on each node. Be sure to use the same cluster ID for each one.
@@ -35,10 +39,10 @@ If you are using multiple nodes, then you should run the format command on each
35
39
Finally, you are ready to start the Kafka server on each node.
[2021-02-26 15:37:11,071] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
40
44
[2021-02-26 15:37:11,294] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
41
-
[2021-02-26 15:37:11,466] INFO [Log partition=@metadata-0, dir=/tmp/self-managed-combined-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
45
+
[2021-02-26 15:37:11,466] INFO [Log partition=@metadata-0, dir=/tmp/kraft-combined-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
42
46
[2021-02-26 15:37:11,509] INFO [raft-expiration-reaper]: Starting (kafka.raft.TimingWheelExpirationService$ExpiredOperationReaper)
43
47
[2021-02-26 15:37:11,640] INFO [RaftManager nodeId=1] Completed transition to Unattached(epoch=0, voters=[1], electionTimeoutMs=9037) (org.apache.kafka.raft.QuorumState)
44
48
...
@@ -54,8 +58,8 @@ Created topic foo.
54
58
# Deployment
55
59
56
60
## Controller Servers
57
-
In self-managed mode, only a small group of specially selected servers can act as controllers (unlike the ZooKeeper-based mode, where any server can become the
58
-
Controller). The specially selected controller servers will participate in the metadata quorum. Each self-managed controller server is either active, or a hot
61
+
In KRaft mode, only a small group of specially selected servers can act as controllers (unlike the ZooKeeper-based mode, where any server can become the
62
+
Controller). The specially selected controller servers will participate in the metadata quorum. Each controller server is either active, or a hot
59
63
standby for the current active controller server.
60
64
61
65
You will typically select 3 or 5 servers for this role, depending on factors like cost and the number of concurrent failures your system should withstand
@@ -65,16 +69,19 @@ controllers, you can tolerate 1 failure; with 5 controllers, you can tolerate 2
65
69
## Process Roles
66
70
Each Kafka server now has a new configuration key called `process.roles` which can have the following values:
67
71
68
-
* If `process.roles` is set to `broker`, the server acts as a self-managed broker.
69
-
* If `process.roles` is set to `controller`, the server acts as a self-managed controller.
70
-
* If `process.roles` is set to `broker,controller`, the server acts as both a self-managed broker and a self-managed controller.
71
-
* If `process.roles` is not set at all then we are assumed to be in ZooKeeper mode. As mentioned earlier, you can't currently transition back and forth between ZK mode and self-managed mode without reformatting.
72
+
* If `process.roles` is set to `broker`, the server acts as a broker in KRaft mode.
73
+
* If `process.roles` is set to `controller`, the server acts as a controller in KRaft mode.
74
+
* If `process.roles` is set to `broker,controller`, the server acts as both a broker and a controller in KRaft mode.
75
+
* If `process.roles` is not set at all then we are assumed to be in ZooKeeper mode. As mentioned earlier, you can't currently transition back and forth between ZooKeeper mode and KRaft mode without reformatting.
72
76
73
77
Nodes that act as both brokers and controllers are referred to as "combined" nodes. Combined nodes are simpler to operate for simple use cases and allow you to avoid
74
-
some of the fixed memory overheads associated with JVMs. The key disadvantage is that the controller will be less isolated from the rest of the system. For example, if activity on the broker causes an out of memory condition, the controller part of the server is not isolated from that OOM condition.
78
+
some fixed memory overheads associated with JVMs. The key disadvantage is that the controller will be less isolated from the rest of the system. For example, if activity on the broker causes an out of
79
+
memory condition, the controller part of the server is not isolated from that OOM condition.
75
80
76
81
## Quorum Voters
77
-
All nodes in the system must set the `controller.quorum.voters` configuration. This identifies the quorum controller servers that should be used. All the controllers must be enumerated. This is similar to how, when using ZooKeeper, the `zookeeper.connect` configuration must contain all the ZooKeeper servers. Unlike with the ZK config, however, `controller.quorum.voters` also has IDs for each node. The format is id1@host1:port1,id2@host2:port2, etc.
82
+
All nodes in the system must set the `controller.quorum.voters` configuration. This identifies the quorum controller servers that should be used. All the controllers must be enumerated.
83
+
This is similar to how, when using ZooKeeper, the `zookeeper.connect` configuration must contain all the ZooKeeper servers. Unlike with the ZooKeeper config, however, `controller.quorum.voters`
84
+
also has IDs for each node. The format is id1@host1:port1,id2@host2:port2, etc.
78
85
79
86
So if you have 10 brokers and 3 controllers named controller1, controller2, controller3, you might have the following configuration on controller1:
Each broker and each controller must set `controller.quorum.voters`. Note that the node ID supplied in the controller.quorum.voters configuration must match that supplied to the server. So on controller1, node.id must be set to 1, and so forth. Note that there is no requirement for controller IDs to start at 0 or 1. However, the easiest and least confusing way to allocate node IDs is probably just to give each server a numeric ID, starting from 0.
94
+
Each broker and each controller must set `controller.quorum.voters`. Note that the node ID supplied in the `controller.quorum.voters` configuration must match that supplied to the server.
95
+
So on controller1, node.id must be set to 1, and so forth. Note that there is no requirement for controller IDs to start at 0 or 1. However, the easiest and least confusing way to allocate
96
+
node IDs is probably just to give each server a numeric ID, starting from 0.
88
97
89
98
Note that clients never need to configure `controller.quorum.voters`; only servers do.
90
99
91
100
## Kafka Storage Tool
92
101
As described above in the QuickStart section, you must use the `kafka-storage.sh` tool to generate a cluster ID for your new cluster, and then run the format command on each node before starting the node.
93
102
94
-
This is different from how Kafka has operated in the past. Previously, Kafka would format blank storage directories automatically, and also generate a new cluster UUID automatically. One reason for the change is that auto-formatting can sometimes obscure an error condition. For example, under UNIX, if a data directory can't be mounted, it may show up as blank. In this case, auto-formatting would be the wrong thing to do.
103
+
This is different from how Kafka has operated in the past. Previously, Kafka would format blank storage directories automatically, and also generate a new cluster UUID automatically. One reason for the change
104
+
is that auto-formatting can sometimes obscure an error condition. For example, under UNIX, if a data directory can't be mounted, it may show up as blank. In this case, auto-formatting would be the wrong thing to do.
95
105
96
-
This is particularly important for the metadata log maintained by the controller servers. If two controllers out of three controllers were able to start with blank logs, a leader might be able to be elected with nothing in the log, which would cause all metadata to be lost.
106
+
This is particularly important for the metadata log maintained by the controller servers. If two controllers out of three controllers were able to start with blank logs, a leader might be able to be elected with
107
+
nothing in the log, which would cause all metadata to be lost.
97
108
98
109
# Missing Features
99
-
We do not yet support generating or loading KIP-630 metadata snapshots. This means that after a while, the time required to restart a broker will become very large. This is a known issue and we are working on completing snapshots for the next release.
110
+
We do not yet support generating or loading KIP-630 metadata snapshots. This means that after a while, the time required to restart a broker will become very large. This is a known issue and we are working on
111
+
completing snapshots for the next release.
100
112
101
-
We also don't support any kind of upgrade right now, either to or from self-managed mode. This is another important gap that we are working on.
113
+
We also don't support any kind of upgrade right now, either to or from KRaft mode. This is another important gap that we are working on.
102
114
103
115
Finally, the following Kafka features have not yet been fully implemented:
104
116
@@ -119,8 +131,8 @@ If you encounter an issue, you might want to take a look at the metadata log.
119
131
One way to view the metadata log is with kafka-dump-log.sh tool, like so:
Another tool for examining the metadata logs is the Kafka metadata shell. Just like the ZooKeeper shell, this allows you to inspect the metadata of the cluster.
0 commit comments