DEVX-1531: add client config files and get all settings from configs across clients, update cloud client readmes (#441)

* add client config files and get all settings from configs across clients
add example.config files to each client, get all settings from config files, update readmes

* change .ccloud to .confluent, update config file instructions for clients

Author: laura-brouckman

clients/cloud/c/README.md

@@ -2,17 +2,11 @@
 
 Produce messages to and consume messages from a Kafka cluster using the C client [librdkafka](https://github.com/edenhill/librdkafka).
 
-
 # Prerequisites
 
 * [librdkafka](https://github.com/edenhill/librdkafka) installed on your machine, see [installation instructions](https://github.com/edenhill/librdkafka/blob/master/README.md#instructions).
-
-To run this example, create a local file with configuration parameters to connect to your Kafka cluster, which can be on your local host, [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), or any other cluster.
-If this is a Confluent Cloud cluster, you must have:
-
-* Access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud) cluster
-* Local file with configuration parameters to connect to your Confluent Cloud instance ([how do I find those?](https://docs.confluent.io/current/cloud/using/config-client.html#librdkafka-based-c-clients?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud)).
-
+* Create a local file (e.g. at `$HOME/.confluent/librdkafka.config`) with configuration parameters to connect to your Kafka cluster, which can be on your local host, [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), or any other cluster.  Follow [these detailed instructions](https://github.com/confluentinc/configuration-templates/tree/master/README.md) to properly create this file. 
+* If you are running on Confluent Cloud, you must have access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud) cluster
 
 # Build the example applications
 
@@ -24,20 +18,6 @@ cc   consumer.c common.c json.c -o consumer -lrdkafka -lm
 cc   producer.c common.c json.c -o producer -lrdkafka -lm
 ```
 
-# Create a configuration file
-
-The configuration file must contain the bootstrap servers and
-the SASL username and password, as shown in your Confluent Cloud settings.
-
-Additional configuration properties are supported, see [CONFIGURATION.md](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) for the full list.
-
-```bash
-$ cat $HOME/.ccloud/example.config
-bootstrap.servers=<broker-1,broker-2,broker-3>
-sasl.username=<api-key-id>
-sasl.password=<secret-access-key>
-```
-
 # Example 1: Hello World!
 
 In this example, the producer writes JSON data to a topic in Confluent Cloud.
@@ -47,7 +27,7 @@ The consumer reads the same topic from Confluent Cloud and keeps a rolling sum o
 1. Run the producer, passing in arguments for (a) the topic name, and (b) the local file with configuration parameters to connect to your Confluent Cloud instance:
 
 ```bash
-$ ./producer test1 $HOME/.ccloud/example.config
+$ ./producer test1 $HOME/.confluent/librdkafka.config
 Creating topic test1
 Topic test1 successfully created
 Producing message #0 to test1: alice={ "count": 1 }
@@ -78,7 +58,7 @@ Message delivered to test1 [0] at offset 9 in 22.81ms: { "count": 10 }
 2. Run the consumer, passing in arguments for (a) the same topic name as used above, (b) the local file with configuration parameters to connect to your Confluent Cloud instance. Verify that the consumer received all the messages, then press Ctrl-C to exit.
 
 ```bash
-$ ./consumer test1 $HOME/.ccloud/example.config
+$ ./consumer test1 $HOME/.confluent/librdkafka.config
 Subscribing to test1, waiting for assignment and messages...
 Press Ctrl-C to exit.
 Received message on test1 [0] at offset 0: { "count": 1 }

clients/cloud/c/common.c

@@ -53,19 +53,6 @@ rd_kafka_conf_t *read_config (const char *config_file) {
         }
 
         conf = rd_kafka_conf_new();
-
-        /* Set up basic Confluent Cloud connectivity parameters,
-         * we expect the bootstrap.servers, sasl.username, and sasl.password
-         * to be specified in the configuration file. */
-        if (rd_kafka_conf_set(conf, "security.protocol", "SASL_SSL",
-                              errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK ||
-            rd_kafka_conf_set(conf, "sasl.mechanisms", "PLAIN",
-                              errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK) {
-                fprintf(stderr, "Configuration failed: %s\n", errstr);
-                rd_kafka_conf_destroy(conf);
-                return NULL;
-        }
-
         /* Read configuration file, line by line. */
         while (fgets(buf, sizeof(buf), fp)) {
                 char *s = buf;

clients/cloud/clojure/README.md

@@ -8,22 +8,8 @@ For more information, please see the [application development documentation](htt
 
 * Java 8 or higher (Clojure 1.10 recommends using Java 8 or Java 11)
 * The [Leiningen](https://leiningen.org/#install) tool to compile and run the demos
-
-To run this example, create a local file with configuration parameters to connect to your Kafka cluster, which can be on your local host, [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), or any other cluster.
-If this is a Confluent Cloud cluster, you must have:
-
-* Access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud) cluster
-* Local file with configuration parameters to connect to your Confluent Cloud instance ([how do I find those?](https://docs.confluent.io/current/cloud/using/config-client.html#librdkafka-based-c-clients?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud)). Format the file as follows:
-
-```bash
-$ cat ~/.ccloud/example.config
-bootstrap.servers=<broker-1,broker-2,broker-3>
-sasl.username=<api-key-id>
-sasl.password=<secret-access-key>
-ssl.endpoint.identification.algorithm=https
-security.protocol=SASL_SSL
-sasl.mechanism=PLAIN
-```
+* Create a local file (e.g. at `$HOME/.confluent/java.config`) with configuration parameters to connect to your Kafka cluster, which can be on your local host, [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), or any other cluster.  Follow [these detailed instructions](https://github.com/confluentinc/configuration-templates/tree/master/README.md) to properly create this file. 
+* If you are running on Confluent Cloud, you must have access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud) cluster
 
 # Example 1: Hello World!
 
@@ -34,7 +20,7 @@ The consumer reads the same topic from Confluent Cloud and keeps a rolling sum o
 1. Run the producer, passing in arguments for (a) the local file with configuration parameters to connect to your Confluent Cloud instance and (b) the topic name:
 
 ```shell
-$ lein producer ~/.ccloud/example.config test1
+$ lein producer $HOME/.confluent/java.config test1
 …
 Producing record: alice 	{"count":0}
 Producing record: alice 	{"count":1}
@@ -62,7 +48,7 @@ Produced record to topic test1 partiton [0] @ offest 9
 2. Run the consumer, passing in arguments for (a) the local file with configuration parameters to connect to your Confluent Cloud instance and (b) the same topic name as used above. Verify that the consumer received all the messages:
 
 ```shell
-$ lein consumer ~/.ccloud/example.config test1
+$ lein consumer $HOME/.confluent/java.config test1
 …
 Waiting for message in KafkaConsumer.poll
 Consumed record with key alice and value {"count":0}, and updated total count to 0

clients/cloud/clojure/src/io/confluent/examples/clients/clj/consumer.clj

@@ -11,7 +11,8 @@
 (defn- build-properties [config-fname]
   (with-open [config (jio/reader config-fname)]
     (doto (Properties.)
-      (.putAll {ConsumerConfig/KEY_DESERIALIZER_CLASS_CONFIG "org.apache.kafka.common.serialization.StringDeserializer"
+      (.putAll {ConsumerConfig/GROUP_ID_CONFIG, "clojure_example_group"
+                ConsumerConfig/KEY_DESERIALIZER_CLASS_CONFIG "org.apache.kafka.common.serialization.StringDeserializer"
                 ConsumerConfig/VALUE_DESERIALIZER_CLASS_CONFIG "org.apache.kafka.common.serialization.StringDeserializer"})
       (.load config))))
 

clients/cloud/confluent-cli/README.md

@@ -9,20 +9,8 @@ Produce messages to and consume messages from a Kafka cluster using [Confluent C
 
 * [Confluent Platform 5.4](https://www.confluent.io/download/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), which includes the Confluent CLI
 
-To run this example, create a local file with configuration parameters to connect to your Kafka cluster, which can be on your local host, [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), or any other cluster.
-If this is a Confluent Cloud cluster, you must have:
-
-* Access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud) cluster
-* Initialize a properties file at `$HOME/.ccloud/config` with configuration to your Confluent Cloud cluster:
-
-```shell
-$ cat $HOME/.ccloud/config
-bootstrap.servers=<BROKER ENDPOINT>
-ssl.endpoint.identification.algorithm=https
-security.protocol=SASL_SSL
-sasl.mechanism=PLAIN
-sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="<API KEY>" password\="<API SECRET>";
-```
+* Create a local file (e.g. at `$HOME/.confluent/java.config`) with configuration parameters to connect to your Kafka cluster, which can be on your local host, [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), or any other cluster.  Follow [these detailed instructions](https://github.com/confluentinc/configuration-templates/tree/master/README.md) to properly create this file. 
+* If you are running on Confluent Cloud, you must have access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud) cluster
 
 
 # Example 1: Hello World!
@@ -34,12 +22,12 @@ The consumer reads the same topic from Confluent Cloud.
 1. Create the topic in Confluent Cloud
 
 ```bash
-$ kafka-topics --bootstrap-server `grep "^\s*bootstrap.server" $HOME/.ccloud/config | tail -1` --command-config $HOME/.ccloud/config --topic test1 --create --replication-factor 3 --partitions 6
+$ kafka-topics --bootstrap-server `grep "^\s*bootstrap.server" $HOME/.confluent/java.config | tail -1` --command-config $HOME/.confluent/java.config --topic test1 --create --replication-factor 3 --partitions 6
 ```
 
 2. Run the [Confluent CLI producer](https://docs.confluent.io/current/cli/command-reference/confluent-produce.html#cli-confluent-produce?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), writing messages to topic `test1`, passing in additional arguments:
 
-* `--cloud`: write messages to the Confluent Cloud cluster specified in `$HOME/.ccloud/config`
+* `--cloud`: write messages to the Confluent Cloud cluster specified in `$HOME/.confluent/java.config`
 * `--property parse.key=true --property key.separator=,`: pass key and value, separated by a comma
 
 ```bash
@@ -58,7 +46,7 @@ When you are done, press `<ctrl>-d`.
 
 2. Run the [Confluent CLI consumer](https://docs.confluent.io/current/cli/command-reference/confluent-consume.html#cli-confluent-consume?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), reading messages from topic `test1`, passing in additional arguments:
 
-* `--cloud`: read messages from the Confluent Cloud cluster specified in `$HOME/.ccloud/config`
+* `--cloud`: read messages from the Confluent Cloud cluster specified in `$HOME/.confluent/java.config`
 * `--property print.key=true`: print key and value (by default, it only prints value)
 * `--from-beginning`: print all messages from the beginning of the topic
 
@@ -93,14 +81,14 @@ Note that your VPC must be able to connect to the Confluent Cloud Schema Registr
     # View the list of registered subjects
     $ curl -u <SR API KEY>:<SR API SECRET> https://<SR ENDPOINT>/subjects
 
-    # Same as above, as a single bash command to parse the values out of $HOME/.ccloud/config
-    $ curl -u $(grep "^schema.registry.basic.auth.user.info" $HOME/.ccloud/config | cut -d'=' -f2) $(grep "^schema.registry.url" $HOME/.ccloud/config | cut -d'=' -f2)/subjects
+    # Same as above, as a single bash command to parse the values out of $HOME/.confluent/java.config
+    $ curl -u $(grep "^schema.registry.basic.auth.user.info" $HOME/.confluent/java.config | cut -d'=' -f2) $(grep "^schema.registry.url" $HOME/.confluent/java.config | cut -d'=' -f2)/subjects
     ```
 
-3. Add the following parameters to your local Confluent Cloud configuration file (``$HOME/.ccloud/config``). In the output below, substitute values for `<SR API KEY>`, `<SR API SECRET>`, and `<SR ENDPOINT>`.
+3. Add the following parameters to your local Confluent Cloud configuration file (``$HOME/.confluent/java.config``). In the output below, substitute values for `<SR API KEY>`, `<SR API SECRET>`, and `<SR ENDPOINT>`.
 
     ```shell
-    $ cat $HOME/.ccloud/config
+    $ cat $HOME/.confluent/java.config
     ...
     basic.auth.credentials.source=USER_INFO
     schema.registry.basic.auth.user.info=<SR API KEY>:<SR API SECRET>
@@ -111,7 +99,7 @@ Note that your VPC must be able to connect to the Confluent Cloud Schema Registr
 4. Create the topic in Confluent Cloud
 
 ```bash
-$ kafka-topics --bootstrap-server `grep "^\s*bootstrap.server" $HOME/.ccloud/config | tail -1` --command-config $HOME/.ccloud/config --topic test2 --create --replication-factor 3 --partitions 6
+$ kafka-topics --bootstrap-server `grep "^\s*bootstrap.server" $HOME/.confluent/java.config | tail -1` --command-config $HOME/.confluent/java.config --topic test2 --create --replication-factor 3 --partitions 6
 ```
 
 5. Run the [Confluent CLI producer](https://docs.confluent.io/current/cli/command-reference/confluent-produce.html#cli-confluent-produce?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), writing messages to topic `test2`, passing in additional arguments. The additional Schema Registry parameters are required to be passed in as properties instead of a properties file due to https://github.com/confluentinc/schema-registry/issues/1052.

clients/cloud/confluent-cli/confluent-cli-ccsr-example.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-CONFIG_FILE=~/.ccloud/config
+CONFIG_FILE=$HOME/.confluent/java.config
 
 source ../../../utils/helper.sh
 

clients/cloud/confluent-cli/confluent-cli-example.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-CONFIG_FILE=~/.ccloud/config
+CONFIG_FILE=$HOME/.confluent/java.config
 
 source ../../../utils/helper.sh 
 

clients/cloud/csharp/Program.cs

@@ -28,18 +28,6 @@ namespace CCloud
 {
     class Program
     {
-        /// <summary>
-        ///     Extract the value associated with key <paramref name="key"/> from the
-        ///     sasl.jaas.config (<paramref name="jaasConfig"/>) configuration value.
-        /// </summary>
-        static string ExtractJaasValue(string jaasConfig, string key)
-        {
-            var beginToken = key + "\\=\"";
-            var startIdx = jaasConfig.IndexOf(beginToken);
-            var endIdx = jaasConfig.IndexOf("\"", startIdx + beginToken.Length);
-            return jaasConfig.Substring(startIdx + beginToken.Length, endIdx - startIdx - beginToken.Length);
-        }
-
         static async Task<ClientConfig> LoadConfig(string configPath, string certDir)
         {
             try
@@ -49,14 +37,15 @@ static async Task<ClientConfig> LoadConfig(string configPath, string certDir)
                     .ToDictionary(
                         line => line.Substring(0, line.IndexOf('=')),
                         line => line.Substring(line.IndexOf('=') + 1));
-
+                Enum.TryParse(cloudConfig["sasl.mechanisms"], out SaslMechanism saslMechanism);
+                Enum.TryParse(cloudConfig["security.protocol"], out SecurityProtocol securityProtocol);
                 var clientConfig = new ClientConfig
                 {
                     BootstrapServers = cloudConfig["bootstrap.servers"].Replace("\\", ""),
-                    SaslMechanism = SaslMechanism.Plain,
-                    SecurityProtocol = SecurityProtocol.SaslSsl,
-                    SaslUsername = ExtractJaasValue(cloudConfig["sasl.jaas.config"], "username"),
-                    SaslPassword = ExtractJaasValue(cloudConfig["sasl.jaas.config"], "password")
+                    SaslMechanism = saslMechanism,
+                    SecurityProtocol = securityProtocol,
+                    SaslUsername = cloudConfig["sasl.username"],
+                    SaslPassword = cloudConfig["sasl.password"],
                 };
 
                 if (certDir != null)

clients/cloud/csharp/README.md

@@ -6,21 +6,8 @@ Produce messages to and consume messages from a Kafka cluster using the .NET Pro
 # Prerequisites
 
 * [.NET Core 2.1](https://dotnet.microsoft.com/download) or higher to run the demo application
-
-To run this example, create a local file with configuration parameters to connect to your Kafka cluster, which can be on your local host, [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), or any other cluster.
-If this is a Confluent Cloud cluster, you must have:
-
-* Access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud) cluster
-* Initialize a properties file at `$HOME/.ccloud/config` with configuration to your Confluent Cloud cluster:
-
-```shell
-$ cat $HOME/.ccloud/config
-bootstrap.servers=<BROKER ENDPOINT>
-ssl.endpoint.identification.algorithm=https
-security.protocol=SASL_SSL
-sasl.mechanism=PLAIN
-sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="<API KEY>" password\="<API SECRET>";
-```
+* Create a local file (e.g. at `$HOME/.confluent/librdkafka.config`) with configuration parameters to connect to your Kafka cluster, which can be on your local host, [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud), or any other cluster.  Follow [these detailed instructions](https://github.com/confluentinc/configuration-templates/tree/master/README.md) to properly create this file. 
+* If you are running on Confluent Cloud, you must have access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clients-ccloud) cluster
 
 # Example
 
@@ -40,10 +27,10 @@ Run the example application, passing in arguments for (a) whether to produce or
 $ dotnet build
 
 # Run the producer (Windows)
-$ dotnet run produce test1 %HOMEPATH%/.ccloud/config /path/to/curl/cacert.pem
+$ dotnet run produce test1 $HOME/.confluent/librdkafka.config /path/to/curl/cacert.pem
 
 # Run the producer (other)
-$ dotnet run produce test1 $HOME/.ccloud/config
+$ dotnet run produce test1 $HOME/.confluent/librdkafka.config
 ```
 
 You should see:
@@ -78,10 +65,10 @@ Run the consumer, passing in arguments for (a) whether to produce or consume (co
 
 ```shell
 # Run the consumer (Windows)
-$ dotnet run consume test1 %HOMEPATH%/.ccloud/config /path/to/curl/cacert.pem
+$ dotnet run consume test1 $HOME/.confluent/librdkafka.config /path/to/curl/cacert.pem
 
 # Run the consumer (other)
-$ dotnet run consume test1 $HOME/.ccloud/config
+$ dotnet run consume test1 $HOME/.confluent/librdkafka.config
 ```
 
 You should see: