Skip to content

Commit d130f8a

Browse files
ayoubbenz-dbayoubbenz-db
authored
add Overwatch multi-workspace deployment on Azure (#55)
* add overwatch deployment module and example on Azure * create a container for cluster logs per workspace and use SPN in the overwatch job * add more details to the README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * refactor variables * remove vars values * refactor variables * Update README.md * add overwatch module and example to the main README * spliting the overwatch deployment into multiple modules * add dependency between terraform resources * automatically generate providers.tf and main.tf from workspaces to monitor that are given into params * add terraform required variables for Azure Overwatch example * update README * Update README.md * add variable types * update README format * add description and type of variables used in module * update outputs description and README * update outputs description and README * add descriptions to variables.tf & outputs.tf and update the README file * add description to variables and output for the module adb-overwatch-regional-config and update its README * update the adb-overwatch example and remove the unused variable * add description to variables for the module adb-overwatch-ws-to-monitor and update its README * add description to variables for the module adb-overwatch-ws-to-monitor and update its README * update adb-overwatch example * update adb-overwatch example README * update adb-overwatch example README * update adb-overwatch example README * update adb-overwatch example README * update adb-overwatch example README * remove unused variable * update vars types * fix terrafrom dependency * remove token parameter * update vars types * update README with the new overwatch modules * fix the overwatch_deployment_config.csv by adding the column auditlogprefix_source_path * fix the overwatch_deployment_config.csv by adding the column auditlogprefix_source_path * fix overwatch_deployment_template.csv generation * fix dependency with azurerm_key_vault_access_policy and fix api_url&workspace_url generation
1 parent a63c95e commit d130f8a

40 files changed

+4989
-0
lines changed

‎README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ The folder `examples` contains the following Terraform implementation examples :
4242
| Azure | [adb-teradata](examples/adb-teradata/) | ADB with single VM Teradata integration |
4343
| Azure | [adb-uc](examples/adb-uc/) | ADB Unity Catalog Process |
4444
| Azure | [adb-unity-catalog-basic-demo](examples/adb-unity-catalog-basic-demo/) | ADB Unity Catalog end to end demo including UC metastore setup, Users/groups sync from AAD to databricks account, UC Catalog, External locations, Schemas, & Access Grants |
45+
| Azure | [adb-overwatch](examples/adb-overwatch/) | Overwatch multi-workspace deployment on Azure |
4546
| AWS | [aws-workspace-basic](examples/aws-workspace-basic/) | Provisioning AWS Databricks E2 |
4647
| AWS | [aws-workspace-with-firewall](examples/aws-workspace-with-firewall/) | Provisioning AWS Databricks E2 with an AWS Firewall |
4748
| AWS | [aws-exfiltration-protection](examples/aws-exfiltration-protection/) | An implementation of [Data Exfiltration Protection on AWS](https://www.databricks.com/blog/2021/02/02/data-exfiltration-protection-with-databricks-on-aws.html) |
@@ -68,6 +69,11 @@ The folder `modules` contains the following Terraform modules :
6869
| Azure | [adb-with-private-link-standard](modules/adb-with-private-link-standard/) | Provisioning Databricks on Azure with Private Link - Standard deployment |
6970
| Azure | [adb-exfiltration-protection](modules/adb-exfiltration-protection/) | A sample implementation of [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) |
7071
| Azure | [adb-with-private-links-exfiltration-protection](modules/adb-with-private-links-exfiltration-protection/) | Provisioning Databricks on Azure with Private Link and [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) |
72+
| Azure | [adb-overwatch-regional-config](modules/adb-overwatch-regional-config/) | Overwatch regional configuration on Azure |
73+
| Azure | [adb-overwatch-mws-config](modules/adb-overwatch-mws-config/) | Overwatch multi-workspace deployment on Azure |
74+
| Azure | [adb-overwatch-main-ws](modules/adb-overwatch-main-ws/) | Main Overwatch workspace deployment |
75+
| Azure | [adb-overwatch-ws-to-monitor](modules/adb-overwatch-ws-to-monitor/) | Overwatch deployment on the Azure workspace to monitor |
76+
| Azure | [adb-overwatch-analysis](modules/adb-overwatch-analysis/) | Overwatch analysis notebooks deployment on Azure |
7177
| AWS | [aws-workspace-basic](modules/aws-workspace-basic/) | Provisioning AWS Databricks E2 |
7278
| AWS | [aws-databricks-base-infra](modules/aws-databricks-base-infra/) | Provisioning AWS Infrastructure to be used for the deployment of a Databricks E2 workspace |
7379
| AWS | [aws-databricks-unity-catalog](modules/aws-databricks-unity-catalog/) | Provisioning the AWS Infrastructure and setting up the metastore for Databricks Unity Catalog |

‎examples/adb-overwatch/README.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Deploying Overwatch on Azure Databricks
2+
3+
This example contains Terraform code used to deploy Overwatch using the following modules :
4+
- [adb-overwatch-regional-config](../../modules/adb-overwatch-regional-config)
5+
- [adb-overwatch-mws-config](../../modules/adb-overwatch-mws-config)
6+
- [adb-overwatch-main-ws](../../modules/adb-overwatch-main-ws)
7+
- [adb-overwatch-ws-to-monitor](../../modules/adb-overwatch-ws-to-monitor)
8+
- [adb-overwatch-analysis](../../modules/adb-overwatch-analysis)
9+
10+
11+
## Example content
12+
13+
This code uses the [multi-workspace deployment of Overwatch](https://databrickslabs.github.io/overwatch/deployoverwatch/cloudinfra/azure/#reference-architecturehttps://databrickslabs.github.io/overwatch/deployoverwatch/cloudinfra/azure/#reference-architecture). Overwatch runs in a dedicated, or existing, Azure Databricks workspace, and monitors the specified workspaces in the config file [overwatch_deployment_config.csv](./overwatch_deployment_config.csv). This configuration file is generated automatically by the module [adb-overwatch-ws-to-monitor](../../modules/adb-overwatch-ws-to-monitor).
14+
15+
![Overwatch_Arch_Azure](https://user-images.githubusercontent.com/103026825/230571464-5892c5c7-82c2-4808-9003-61b501b75f69.png?raw=true)
16+
17+
The deployment is structured as followed :
18+
* Use an existing **Resource group**
19+
* Deploy **Eventhubs** topic per workspace, that could be in the same **Eventhubs** namespace
20+
* Deploy **Storage Accounts**, one for the cluster logs and one for Overwatch database output
21+
* Deploy the dedicated **Azure Databricks** workspace, or use an existing one for Overwatch, with some Databricks quick-start notebooks to analyse the results
22+
* Deploy **Azure Key Vault** to store the secrets
23+
* Configure **Role Assignments** and **mounts** to attribute the necessary permissions
24+
* Configure **Diagnostic Logs** on the Databricks workspaces to monitor
25+
26+
> **Note**
27+
> As Terraform requires providers and modules to be declared statically before deploying the resources, we are using in this example a [bash script](./dynamic_providers_modules_generation.sh)
28+
> that generates the provider configurations for N workspaces along with the modules references.
29+
30+
## How to use
31+
32+
1. Configure the workspaces that will be observed by Overwatch in [workspaces_to_monitor.json](./workspaces_to_monitor.json)
33+
2. Make the script [dynamic_providers_modules_generation.sh](./dynamic_providers_modules_generation.sh) executable : `chmod +x dynamic_providers_modules_generation.sh`
34+
3. Update the `terraform.tfvars` file with your environment values
35+
4. Run the script [dynamic_providers_modules_generation.sh](./dynamic_providers_modules_generation.sh) : `./dynamic_providers_modules_generation.sh`. This will dynamically generate `providers_ws_to_monitor.tf` and `main_ws_to_monitor.tf` files with the right terraform setup for all the workspaces defined in [workspaces_to_monitor.json](./workspaces_to_monitor.json)
36+
5. Run `terraform init` to initialize terraform and get provider ready
37+
6. Run `terraform plan` to check the resources that are affected
38+
7. Run `terraform apply` to create the resources

‎examples/adb-overwatch/dynamic_providers_modules_generation.sh

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
#!/bin/bash
2+
3+
# Load JSON file into variable
4+
json=$(cat workspaces_to_monitor.json)
5+
6+
# set the name of the module's source
7+
module_source="../../modules/adb-overwatch-ws-to-monitor"
8+
modules_list=(module.adb-overwatch-mws-config)
9+
10+
# Loop through JSON objects
11+
for row in $(echo "${json}" | jq -r '.[] | @base64'); do
12+
# Decode JSON object
13+
_jq() {
14+
echo "${row}" | base64 --decode | jq -r "${1}"
15+
}
16+
17+
# Generate Terraform Databricks provider configuration
18+
workspace_name=$(_jq '.workspace_name')
19+
host=$(_jq '.host')
20+
21+
echo -e "provider \"databricks\" {
22+
alias = \"$workspace_name\"
23+
host = \"$host\"
24+
}\n" >> providers_ws_to_monitor.tf
25+
26+
# Set the name of the module dynamically
27+
module_name="adb-overwatch-monitor-$workspace_name"
28+
29+
# Set the name of the provider and the Databricks value you want to use
30+
provider_name="databricks"
31+
databricks_provider_value="databricks.$workspace_name"
32+
33+
# Generate the JSON block
34+
json_block=$(cat <<EOF
35+
module "$module_name" {
36+
source = "$module_source"
37+
38+
providers = {
39+
$provider_name = $databricks_provider_value
40+
}
41+
42+
adb_ws_name = "$workspace_name"
43+
random_string = random_string.strapp.result
44+
tenant_id = var.tenant_id
45+
rg_name = var.rg_name
46+
overwatch_spn_app_id = var.overwatch_spn_app_id
47+
ehn_name = module.adb-overwatch-regional-config.ehn_name
48+
ehn_auth_rule_name = module.adb-overwatch-regional-config.ehn_ar_name
49+
logs_sa_name = module.adb-overwatch-regional-config.logs_sa_name
50+
akv_name = module.adb-overwatch-regional-config.akv_name
51+
databricks_secret_scope_name = var.databricks_secret_scope_name
52+
etl_storage_prefix = module.adb-overwatch-mws-config.etl_storage_prefix
53+
active = var.active
54+
api_waiting_time = var.api_waiting_time
55+
automated_dbu_price = var.automated_dbu_price
56+
enable_unsafe_SSL = var.enable_unsafe_SSL
57+
error_batch_size = var.error_batch_size
58+
excluded_scopes = var.excluded_scopes
59+
interactive_dbu_price = var.interactive_dbu_price
60+
jobs_light_dbu_price = var.jobs_light_dbu_price
61+
max_days = var.max_days
62+
proxy_host = var.proxy_host
63+
proxy_password_key = var.proxy_password_key
64+
proxy_password_scope = var.proxy_password_scope
65+
proxy_port = var.proxy_port
66+
proxy_user_name = var.proxy_user_name
67+
sql_compute_dbu_price = var.sql_compute_dbu_price
68+
success_batch_size = var.success_batch_size
69+
thread_pool_size = var.thread_pool_size
70+
auditlog_prefix_source_path = var.auditlog_prefix_source_path
71+
72+
depends_on = [module.adb-overwatch-regional-config]
73+
}
74+
EOF
75+
)
76+
77+
echo -e "$json_block\n" >> main_ws_to_monitor.tf
78+
79+
element="module.adb-overwatch-monitor-$workspace_name"
80+
modules_list+=("$element")
81+
82+
done
83+
84+
echo "workspace_name,workspace_id,workspace_url,api_url,cloud,primordial_date,etl_storage_prefix,etl_database_name,consumer_database_name,secret_scope,secret_key_dbpat,auditlogprefix_source_path,eh_name,eh_scope_key,interactive_dbu_price,automated_dbu_price,sql_compute_dbu_price,jobs_light_dbu_price,max_days,excluded_scopes,active,proxy_host,proxy_port,proxy_user_name,proxy_password_scope,proxy_password_key,success_batch_size,error_batch_size,enable_unsafe_SSL,thread_pool_size,api_waiting_time" > overwatch_deployment_config.csv
85+
86+
dependencies="["$(IFS=, ; echo "${modules_list[*]}")"]"
87+
88+
echo -e "resource \"databricks_dbfs_file\" \"overwatch_deployment_config\" {
89+
provider = databricks.adb-ow-main-ws
90+
91+
source = \"\${path.module}/overwatch_deployment_config.csv\"
92+
path = \"/mnt/\${module.adb-overwatch-mws-config.databricks_mount_db_name}/config/overwatch_deployment_config.csv\"
93+
depends_on = $dependencies
94+
}" >> main_ws_to_monitor.tf

‎examples/adb-overwatch/main.tf

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
resource "random_string" "strapp" {
2+
length = 5
3+
lower = true
4+
upper = false
5+
special = false
6+
}
7+
8+
9+
module "adb-overwatch-regional-config" {
10+
source = "../../modules/adb-overwatch-regional-config"
11+
12+
random_string = random_string.strapp.result
13+
rg_name = var.rg_name
14+
overwatch_spn_app_id = var.overwatch_spn_app_id
15+
ehn_name = var.ehn_name
16+
logs_sa_name = var.logs_sa_name
17+
key_vault_prefix = var.key_vault_prefix
18+
overwatch_spn_secret = var.overwatch_spn_secret
19+
}
20+
21+
22+
module "adb-overwatch-main-ws" {
23+
source = "../../modules/adb-overwatch-main-ws"
24+
25+
subscription_id = var.subscription_id
26+
rg_name = var.rg_name
27+
use_existing_ws = var.use_existing_overwatch_ws
28+
overwatch_ws_name = var.overwatch_ws_name
29+
}
30+
31+
32+
module "adb-overwatch-mws-config" {
33+
source = "../../modules/adb-overwatch-mws-config"
34+
providers = {
35+
databricks = databricks.adb-ow-main-ws
36+
}
37+
38+
tenant_id = var.tenant_id
39+
rg_name = var.rg_name
40+
overwatch_spn_app_id = var.overwatch_spn_app_id
41+
overwatch_ws_name = var.overwatch_ws_name
42+
akv_name = module.adb-overwatch-regional-config.akv_name
43+
databricks_secret_scope_name = var.databricks_secret_scope_name
44+
latest_dbr_lts = module.adb-overwatch-main-ws.latest_lts
45+
random_string = random_string.strapp.result
46+
ow_sa_name = var.ow_sa_name
47+
48+
depends_on = [module.adb-overwatch-main-ws, module.adb-overwatch-regional-config]
49+
}
50+
51+
52+
module "adb-overwatch-analysis" {
53+
source = "../../modules/adb-overwatch-analysis"
54+
providers = {
55+
databricks = databricks.adb-ow-main-ws
56+
}
57+
58+
rg_name = var.rg_name
59+
overwatch_ws_name = var.overwatch_ws_name
60+
61+
depends_on = [module.adb-overwatch-main-ws]
62+
}

‎examples/adb-overwatch/overwatch_deployment_config.csv

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
workspace_name,workspace_id,workspace_url,api_url,cloud,primordial_date,etl_storage_prefix,etl_database_name,consumer_database_name,secret_scope,secret_key_dbpat,auditlogprefix_source_path,eh_name,eh_scope_key,interactive_dbu_price,automated_dbu_price,sql_compute_dbu_price,jobs_light_dbu_price,max_days,excluded_scopes,active,proxy_host,proxy_port,proxy_user_name,proxy_password_scope,proxy_password_key,success_batch_size,error_batch_size,enable_unsafe_SSL,thread_pool_size,api_waiting_time

‎examples/adb-overwatch/providers.tf

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
terraform {
2+
required_providers {
3+
azurerm = {
4+
source = "hashicorp/azurerm"
5+
}
6+
7+
databricks = {
8+
source = "databricks/databricks"
9+
}
10+
}
11+
}
12+
13+
provider "azurerm" {
14+
features {}
15+
subscription_id = var.subscription_id
16+
}
17+
18+
provider "databricks" {
19+
alias = "adb-ow-main-ws"
20+
host = module.adb-overwatch-main-ws.adb_ow_main_ws_url
21+
}

‎examples/adb-overwatch/terraform.tfvars

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
subscription_id = ""
2+
tenant_id = ""
3+
overwatch_spn_app_id = ""
4+
overwatch_spn_secret = ""
5+
ehn_name = ""
6+
logs_sa_name = ""
7+
ow_sa_name = ""
8+
key_vault_prefix = ""
9+
rg_name = ""
10+
overwatch_ws_name = ""
11+
use_existing_overwatch_ws = false

0 commit comments

Comments
 (0)