Updated README and cleanup

alollo-ca · alollo-ca · commit 39a79bbed00d · 2024-07-19T16:54:15.000+02:00
diff --git a/README.md b/README.md
@@ -70,25 +70,43 @@ The script `01-enable-cdc.sql` enable Change Data Capture on the aforementioned
 All data gathered by Change Data Capture will be send to Event Hubs, so create an Azure Event Hubs in your Azure Subscription. Using the [Azure Cloud Shell](https://shell.azure.com/) Bash:
 
 ```bash
+# Set vars
+RESOURCE_GROUP=debezium
+LOCATION=eastus
+EVENTHUB_NAMESPACE=debezium
+EVENTHUB_SCHEMA_HISTORY=schemahistory
+
 # create group
 az group create \
-    --name debezium \
-    --location eastus
+    --name $RESOUCE_GROUP \
+    --location $LOCATION
 
-# create eventhuvbs with kafka enabled
+# create eventhub namespace with kafka enabled
 az eventhubs namespace create \
-    --name debezium \
-    --resource-group debezium \
-    --location eastus \
+    --name $EVENTHUB_NAMESPACE \
+    --resource-group $RESOURCE_GROUP \
+    --location $LOCATION \
     --enable-kafka
+
+# create eventhub for schema history
+az eventhubs eventhub create \
+    --resource-group $RESOURCE_GROUP
+    --namespace $EVENTHUB_NAMESPACE \
+    --name $EVENTHUB_SCHEMA_HISTORY \
+    --partition-count 1 \
+    --cleanup-policy Delete \
+    --retention-time-in-hours 168
 ```
 
 Later in the configuration process you'll need the EventHubs connection string, so grab it and store it somewhere:
 
 ```bash
+RESOURCE_GROUP=debezium
+EVENTHUB_NAMESPACE=debezium
+
 az eventhubs namespace authorization-rule keys list \
-    --resouce-group debezium \
-    --namespace-name debezium \
+    --resouce-group $RESOURCE_GROUP \
+    --namespace-name $EVENTHUB_NAMESPACE \
     --name RootManageSharedAccessKey \
     --query "primaryConnectionString" \
     --output tsv
@@ -108,11 +126,11 @@ Docker Compose will use `.env` to get the environment variables values used in t
 
 ```bash
 DEBEZIUM_VERSION=2.7
-EVENTHUB_NAMESPACE=debezium
-EH_CONNECTION_STRING=
+EVENTHUB_NAMESPACE=<eventhub_namespace>
+EVENTHUB_CONNECTION_STRING=<eventhub_connection_string>
 ```
 
-Copy it and create a new `.env` file. Leave the version set to `2.7`. Change the `EVENTHUB_NAMESPACE` to the EventHubs name you created before. Also set `EH_CONNECTION_STRING` to hold the EventHubs connection string you got before. Make sure not to use any additional quotes or double quotes.
+Copy it and create a new `.env` file. Leave the version set to `2.7`. Change the `EVENTHUB_NAMESPACE` to the EventHubs name you created before. Also set `EVENTHUB_CONNECTION_STRING` to hold the EventHubs connection string you got before. Make sure not to use any additional quotes or double quotes.
 
 #### The .yaml file
 
@@ -165,7 +183,7 @@ Once the startup has finished, you'll see something like
 [Worker clientId=connect-1, groupId=1] Finished starting connectors and tasks   [org.apache.kafka.connect.runtime.distributed.DistributedHerder]
 ```
 
-you will see three topics (or EventHub to use the Azure EventHubs nomenclature):
+you will see four topics (or EventHub to use the Azure EventHubs nomenclature):
 
 ```bash
 az eventhubs eventhub list \
@@ -179,30 +197,36 @@ and the result will show:
 - debezium_configs
 - debezium_offsets
 - debezium_statuses
+- schemahistory
 
 to explore Azure Event Hubs is strongly suggest to download and use [Service Bus Explorer](https://github.com/paolosalvatori/ServiceBusExplorer)
 
 #### Register SQL Server Connector
 
 Now that Debezium is running, the SQL Server Connector (which is used both for connecting to Azure SQL or SQL Server) can be registered. Before doing that, make sure to specify the correct connection for your SQL Server instance in a file named `sqlserver-connector-config.json`. You can create one using the template file [sqlserver-connector-config.json.template](debezium/sqlserver-connector-config.json.template) file.
 
-Please note that, the script [debezium/on-prem/register-connector.ps1](debezium/on-prem/register-connector.ps1) will take care of generating the Debezium SQL Server connector configuration JSON file, and then will use it to register a new connector on Debezium.
-The generated file will be stored in the `debezium` folder.
-
-If you are using the Wide World Importers database, the only values you have to change are:
+Make sure to change the following properties to match your SQL Server configuration:
 
 ```json
 "database.hostname": "<sql_server_name>.database.windows.net",
 "database.names": "<db_name>",
+"database.user" : "<debezium_user_name>",
+"database.password" : "<debezium_user_password>",
 ```
 
-If you are following the step-by-step guide using a database of yours, make sure to also correctly set values for
+You would need to replace `<sql_server_name>`, `<db_name>`, `<debezium_user_name>`, and `<debezium_user_password>` with proper values.
+
+Also, make sure to change the following properties to match your Azure Event Hub configuration:
 
 ```json
-"database.user" : "<debezium_user_name>",
-"database.password" : "<debezium_user_password>",
+"schema.history.internal.kafka.bootstrap.servers": "<eventhub_namespace>.servicebus.windows.net:9093",
+"schema.history.internal.kafka.topic": "<eventhub_schema_history>",
+"schema.history.internal.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"<eventhub_connectionstring>\";",
+"schema.history.internal.producer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"<eventhub_connectionstring>\";",
 ```
 
+You would need to replace `<eventhub_namespace>`, `<eventhub_schema_history>`, and `<eventhub_connectionstring>` with proper values.
+
 All the other values used are explained in detail here:
 
 [SQL Server Connector Configuration Values](./documentation/SQL-Server-Connector-Configuration-Value.md)
diff --git a/debezium/azure/00-deploy-debezium.sh b/debezium/azure/00-deploy-debezium.sh
@@ -4,10 +4,10 @@
 set -euo pipefail
 
 DEBEZIUM_VERSION=2.7
-RESOURCE_GROUP="dbzrg"
-EVENTHUB_NAMESPACE="dbzeventhub"
-EVENTHUB_SCHEMA_HISTORY="dbzschemahistory"
-CONTAINER_NAME="dbzcontainer"
+RESOURCE_GROUP="debezium"
+EVENTHUB_NAMESPACE="debezium"
+EVENTHUB_SCHEMA_HISTORY="schemahistory"
+CONTAINER_NAME="debezium"
 LOCATION="WestUS2"
 
 echo "deploying resource group"
@@ -33,7 +33,7 @@ az eventhubs eventhub create \
     --output none
 
 echo "gathering eventhubs connection string"
-EH_CONNECTION_STRING=`az eventhubs namespace authorization-rule keys list --resource-group $RESOURCE_GROUP --name RootManageSharedAccessKey --namespace-name $EVENTHUB_NAMESPACE --output tsv --query 'primaryConnectionString'`
+EVENTHUB_CONNECTION_STRING=`az eventhubs namespace authorization-rule keys list --resource-group $RESOURCE_GROUP --name RootManageSharedAccessKey --namespace-name $EVENTHUB_NAMESPACE --output tsv --query 'primaryConnectionString'`
 
 echo "deploying debezium container"
 az container create \
@@ -57,13 +57,13 @@ az container create \
 		CONNECT_REQUEST_TIMEOUT_MS=60000 \
 		CONNECT_SECURITY_PROTOCOL=SASL_SSL \
 		CONNECT_SASL_MECHANISM=PLAIN \
-		CONNECT_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=\"${EH_CONNECTION_STRING}\";" \
+		CONNECT_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=\"${EVENTHUB_CONNECTION_STRING}\";" \
 		CONNECT_PRODUCER_SECURITY_PROTOCOL=SASL_SSL \
 		CONNECT_PRODUCER_SASL_MECHANISM=PLAIN \
-		CONNECT_PRODUCER_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=\"${EH_CONNECTION_STRING}\";" \
+		CONNECT_PRODUCER_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=\"${EVENTHUB_CONNECTION_STRING}\";" \
 		CONNECT_CONSUMER_SECURITY_PROTOCOL=SASL_SSL \
 		CONNECT_CONSUMER_SASL_MECHANISM=PLAIN \
-		CONNECT_CONSUMER_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=\"${EH_CONNECTION_STRING}\";"
+		CONNECT_CONSUMER_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=\"${EVENTHUB_CONNECTION_STRING}\";"
  
 echo "eventhub connection string"
-echo $EH_CONNECTION_STRING
+echo $EVENTHUB_CONNECTION_STRING
diff --git a/debezium/azure/01-register-connector.sh b/debezium/azure/01-register-connector.sh
@@ -5,42 +5,6 @@ set -euo pipefail
 
 RESOURCE_GROUP="dbzrg"
 CONTAINER_NAME="dbzcontainer"
-EVENTHUB_NAMESPACE="dbzeventhub"
-
-echo "generating Debezium SQL Server connector configuration JSON file"
-EH_CONNECTION_STRING=`az eventhubs namespace authorization-rule keys list --resource-group $RESOURCE_GROUP --name RootManageSharedAccessKey --namespace-name $EVENTHUB_NAMESPACE --output tsv --query 'primaryConnectionString'`
-cat > ../sqlserver-connector-config.json << EOF
-{
-  "name": "wwi",
-  "config": {
-    "snapshot.mode": "schema_only",
-    "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
-    "database.hostname": "dmmssqlsrv.database.windows.net",
-    "database.port": "1433",
-    "database.user": "debezium-wwi",
-    "database.password": "Abcd1234!",
-    "database.names": "WideWorldImportersStandardCDC",
-    "driver.encrypt": "false",
-    "driver.trustServerCertificate": "true",
-    "schema.history.internal.kafka.bootstrap.servers": "${EVENTHUB_NAMESPACE}.servicebus.windows.net:9093",
-    "schema.history.internal.kafka.topic": "dbzschemahistory",
-    "schema.history.internal.consumer.security.protocol": "SASL_SSL",
-    "schema.history.internal.consumer.sasl.mechanism": "PLAIN",
-    "schema.history.internal.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=\"${EH_CONNECTION_STRING}\";",
-    "schema.history.internal.producer.security.protocol": "SASL_SSL",
-    "schema.history.internal.producer.sasl.mechanism": "PLAIN",
-    "schema.history.internal.producer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=\"${EH_CONNECTION_STRING}\";",
-    "table.include.list": "Sales.Orders,Warehouse.StockItems",
-    "tombstones.on.delete": "false",
-    "topic.prefix": "SQLAzure",
-    "transforms": "Reroute",
-    "transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
-    "transforms.Reroute.topic.regex": "(.*)",
-    "transforms.Reroute.topic.replacement": "wwi"
-  }
-}
-EOF
-echo "done"
 
 echo "finding debezium ip"
 DEBEZIUM_IP=`az container show --resource-group $RESOURCE_GROUP --name $CONTAINER_NAME --output tsv --query "ipAddress.ip"`
diff --git a/debezium/on-prem/.env.sample b/debezium/on-prem/.env.sample
diff --git a/debezium/on-prem/.env.template b/debezium/on-prem/.env.template
@@ -1,3 +1,3 @@
 DEBEZIUM_VERSION=2.7
-EH_NAME=debezium
-EH_CONNECTION_STRING=Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=
+EVENTHUB_NAMESPACE=<eventhub_namespace>
+EVENTHUB_CONNECTION_STRING=<eventhub_connection_string>
diff --git a/debezium/on-prem/docker-compose.yaml b/debezium/on-prem/docker-compose.yaml
@@ -5,7 +5,7 @@ services:
     ports:
      - 8083:8083
     environment:
-     - BOOTSTRAP_SERVERS=${EH_NAME}.servicebus.windows.net:9093
+     - BOOTSTRAP_SERVERS=${EVENTHUB_NAMESPACE}.servicebus.windows.net:9093
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=debezium_configs
      - OFFSET_STORAGE_TOPIC=debezium_offsets
@@ -15,10 +15,10 @@ services:
      - CONNECT_REQUEST_TIMEOUT_MS=60000
      - CONNECT_SECURITY_PROTOCOL=SASL_SSL
      - CONNECT_SASL_MECHANISM=PLAIN
-     - CONNECT_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="$$ConnectionString" password="${EH_CONNECTION_STRING}";
+     - CONNECT_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="$$ConnectionString" password="${EVENTHUB_CONNECTION_STRING}";
      - CONNECT_PRODUCER_SECURITY_PROTOCOL=SASL_SSL
      - CONNECT_PRODUCER_SASL_MECHANISM=PLAIN
-     - CONNECT_PRODUCER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="$$ConnectionString" password="${EH_CONNECTION_STRING}";
+     - CONNECT_PRODUCER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="$$ConnectionString" password="${EVENTHUB_CONNECTION_STRING}";
      - CONNECT_CONSUMER_SECURITY_PROTOCOL=SASL_SSL
      - CONNECT_CONSUMER_SASL_MECHANISM=PLAIN
-     - CONNECT_CONSUMER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="$$ConnectionString" password="${EH_CONNECTION_STRING}";
+     - CONNECT_CONSUMER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="$$ConnectionString" password="${EVENTHUB_CONNECTION_STRING}";
diff --git a/debezium/on-prem/register-connector.ps1 b/debezium/on-prem/register-connector.ps1
@@ -1,38 +1,2 @@
-$RESOURCE_GROUP="dbzrg"
-$EVENTHUB_NAMESPACE="dbzeventhub"
-
-$EH_CONNECTION_STRING=az eventhubs namespace authorization-rule keys list --resource-group $RESOURCE_GROUP --name RootManageSharedAccessKey --namespace-name $EVENTHUB_NAMESPACE --output tsv --query 'primaryConnectionString'
-
-$SQLSERVER_CONNECTOR_CONFIG="{
-`"name`": `"wwi`",
-  `"config`": {
-    `"snapshot.mode`": `"schema_only`",
-    `"connector.class`": `"io.debezium.connector.sqlserver.SqlServerConnector`",
-    `"database.hostname`": `"dmmssqlsrv.database.windows.net`",
-    `"database.port`": `"1433`",
-    `"database.user`": `"debezium-wwi`",
-    `"database.password`": `"Abcd1234!`",
-    `"database.names`": `"WideWorldImportersStandardCDC`",
-    `"driver.encrypt`": `"false`",
-    `"driver.trustServerCertificate`": `"true`",
-    `"schema.history.internal.kafka.bootstrap.servers`": `"${EVENTHUB_NAMESPACE}.servicebus.windows.net:9093`",
-    `"schema.history.internal.kafka.topic`": `"dbzschemahistory`",
-    `"schema.history.internal.consumer.security.protocol`": `"SASL_SSL`",
-    `"schema.history.internal.consumer.sasl.mechanism`": `"PLAIN`",
-    `"schema.history.internal.consumer.sasl.jaas.config`": `"org.apache.kafka.common.security.plain.PlainLoginModule required username=\`"`$ConnectionString\`" password=\`"${EH_CONNECTION_STRING}\`";`",
-    `"schema.history.internal.producer.security.protocol`": `"SASL_SSL`",
-    `"schema.history.internal.producer.sasl.mechanism`": `"PLAIN`",
-    `"schema.history.internal.producer.sasl.jaas.config`": `"org.apache.kafka.common.security.plain.PlainLoginModule required username=\`"`$ConnectionString\`" password=\`"${EH_CONNECTION_STRING}\`";`",
-    `"table.include.list`": `"Sales.Orders,Warehouse.StockItems`",
-    `"tombstones.on.delete`": `"false`",
-    `"topic.prefix`": `"SQLAzure`",
-    `"transforms`": `"Reroute`",
-    `"transforms.Reroute.type`": `"io.debezium.transforms.ByLogicalTableRouter`",
-    `"transforms.Reroute.topic.regex`": `"(.*)`",
-    `"transforms.Reroute.topic.replacement`": `"wwi`"
-  }
-}"
-$SQLSERVER_CONNECTOR_CONFIG | Out-File '..\sqlserver-connector-config.json'
-
 $JSON = Get-Content '..\sqlserver-connector-config.json' | Out-String
 Invoke-RestMethod http://localhost:8083/connectors/ -Method POST -Body $JSON -ContentType "application/json"
diff --git a/debezium/on-prem/register-connector.sh b/debezium/on-prem/register-connector.sh
@@ -1,43 +1,5 @@
 #!/bin/sh
 
-RESOURCE_GROUP="dbzrg"
-EVENTHUB_NAMESPACE="dbzeventhub"
-
-echo "generating Debezium SQL Server connector configuration JSON file"
-EH_CONNECTION_STRING=`az eventhubs namespace authorization-rule keys list --resource-group $RESOURCE_GROUP --name RootManageSharedAccessKey --namespace-name $EVENTHUB_NAMESPACE --output tsv --query 'primaryConnectionString'`
-cat > ../sqlserver-connector-config.json << EOF
-{
-  "name": "wwi",
-  "config": {
-    "snapshot.mode": "schema_only",
-    "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
-    "database.hostname": "dmmssqlsrv.database.windows.net",
-    "database.port": "1433",
-    "database.user": "debezium-wwi",
-    "database.password": "Abcd1234!",
-    "database.names": "WideWorldImportersStandardCDC",
-    "driver.encrypt": "false",
-    "driver.trustServerCertificate": "true",
-    "schema.history.internal.kafka.bootstrap.servers": "${EVENTHUB_NAMESPACE}.servicebus.windows.net:9093",
-    "schema.history.internal.kafka.topic": "dbzschemahistory",
-    "schema.history.internal.consumer.security.protocol": "SASL_SSL",
-    "schema.history.internal.consumer.sasl.mechanism": "PLAIN",
-    "schema.history.internal.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=\"${EH_CONNECTION_STRING}\";",
-    "schema.history.internal.producer.security.protocol": "SASL_SSL",
-    "schema.history.internal.producer.sasl.mechanism": "PLAIN",
-    "schema.history.internal.producer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=\"${EH_CONNECTION_STRING}\";",
-    "table.include.list": "Sales.Orders,Warehouse.StockItems",
-    "tombstones.on.delete": false,
-    "topic.prefix": "SQLAzure",
-    "transforms": "Reroute",
-    "transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
-    "transforms.Reroute.topic.regex": "(.*)",
-    "transforms.Reroute.topic.replacement": "wwi"
-  }
-}
-EOF
-echo "done"
-
 curl -i -X POST \
     -H "Accept:application/json" \
     -H "Content-Type:application/json" \
diff --git a/debezium/sqlserver-connector-config.json.template b/debezium/sqlserver-connector-config.json.template
@@ -11,7 +11,7 @@
       "driver.encrypt": "false",
       "driver.trustServerCertificate": "true",
       "schema.history.internal.kafka.bootstrap.servers": "<eventhub_namespace>.servicebus.windows.net:9093",
-      "schema.history.internal.kafka.topic": "dbzschemahistory",
+      "schema.history.internal.kafka.topic": "<eventhub_schema_history>",
       "schema.history.internal.consumer.security.protocol": "SASL_SSL",
       "schema.history.internal.consumer.sasl.mechanism": "PLAIN",
       "schema.history.internal.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"<eventhub_connectionstring>\";",
