feat: Airflow Listener integration (#604)

* mount listener volumes for relevant roles, type depending on listener class

* add podrefs to write webserver endpoint to configmap

* moved listener-class to webserver role

* move config map creation out of role-loop

* merge conflicts

* regenerate nix

* assign listener-class to all roles and consider all ports

* fixed podref name for workers

* added integration test

* changelog

* added docs

* Update rust/operator-binary/src/crd/mod.rs

Co-authored-by: Malte Sander <[email protected]>

* Update rust/operator-binary/src/crd/mod.rs

Co-authored-by: Malte Sander <[email protected]>

* Update rust/operator-binary/src/crd/mod.rs

Co-authored-by: Malte Sander <[email protected]>

* Update rust/operator-binary/src/crd/mod.rs

Co-authored-by: Malte Sander <[email protected]>

* corrected callout comments and regenerate charts

* use ephemeral listeners classes; fix test missing dimension

* allow config map endpoints to be removed if cluster is stopped

* review feedback

* check for port lower-bound

* reworked to only offer listener for webserver role

* reworked docs

* Update docs/modules/airflow/pages/usage-guide/listenerclass.adoc

Co-authored-by: Malte Sander <[email protected]>

* Update rust/operator-binary/src/crd/mod.rs

Co-authored-by: Malte Sander <[email protected]>

* added service check to test

* correctly flatten WebserverConfig

* use pvcs for externally reachable endpoints

* use group listener for webservers

* replace enum with string; use consistent webserver address for all listener classes

* removed unused test account

* Update docs/modules/airflow/pages/usage-guide/listenerclass.adoc

Co-authored-by: Natalie Klestrup Röijezon <[email protected]>

* Update tests/templates/kuttl/opa/41_check-authorization.py

Co-authored-by: Natalie Klestrup Röijezon <[email protected]>

* Update tests/templates/kuttl/oidc/login.py

Co-authored-by: Natalie Klestrup Röijezon <[email protected]>

* review feedback: tests

* review feedback: remove metrics port and re-work conditions

* use custom listeners to stay independent of future changes to standard classes

* make custom listener classes namespace-specific

* changes as per decision 51

* fixed tests

---------

Co-authored-by: Malte Sander <[email protected]>
Co-authored-by: Natalie Klestrup Röijezon <[email protected]>

Author: 3 people

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 ### Added
 
+- Added listener support for Airflow ([#604]).
 - Adds new telemetry CLI arguments and environment variables ([#613]).
   - Use `--file-log-max-files` (or `FILE_LOG_MAX_FILES`) to limit the number of log files kept.
   - Use `--file-log-rotation-period` (or `FILE_LOG_ROTATION_PERIOD`) to configure the frequency of rotation.
@@ -28,6 +29,7 @@
 
 [#600]: https://github.com/stackabletech/airflow-operator/pull/600
 [#601]: https://github.com/stackabletech/airflow-operator/pull/601
+[#604]: https://github.com/stackabletech/airflow-operator/pull/604
 [#607]: https://github.com/stackabletech/airflow-operator/pull/607
 [#608]: https://github.com/stackabletech/airflow-operator/pull/608
 [#613]: https://github.com/stackabletech/airflow-operator/pull/613

deploy/helm/airflow-operator/crds/crds.yaml

@@ -584,23 +584,6 @@ spec:
                       default: false
                       description: for internal use only - not for production use.
                       type: boolean
-                    listenerClass:
-                      default: cluster-internal
-                      description: |-
-                        This field controls which type of Service the Operator creates for this AirflowCluster:
-
-                        * cluster-internal: Use a ClusterIP service
-
-                        * external-unstable: Use a NodePort service
-
-                        * external-stable: Use a LoadBalancer service
-
-                        This is a temporary solution with the goal to keep yaml manifests forward compatible. In the future, this setting will control which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) will be used to expose the service, and ListenerClass names will stay the same, allowing for a non-breaking change.
-                      enum:
-                        - cluster-internal
-                        - external-unstable
-                        - external-stable
-                      type: string
                     loadExamples:
                       default: false
                       description: Whether to load example DAGs or not; defaults to false. The examples are used in the [getting started guide](https://docs.stackable.tech/home/nightly/airflow/getting_started/).
@@ -1338,6 +1321,10 @@ spec:
                           description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
                           nullable: true
                           type: string
+                        listenerClass:
+                          description: This field controls which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) is used to expose the webserver.
+                          nullable: true
+                          type: string
                         logging:
                           default:
                             containers: {}
@@ -1555,6 +1542,10 @@ spec:
                                 description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
                                 nullable: true
                                 type: string
+                              listenerClass:
+                                description: This field controls which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) is used to expose the webserver.
+                                nullable: true
+                                type: string
                               logging:
                                 default:
                                   containers: {}

deploy/helm/airflow-operator/templates/roles.yaml

@@ -84,6 +84,17 @@ rules:
       - customresourcedefinitions
     verbs:
       - get
+  - apiGroups:
+      - listeners.stackable.tech
+    resources:
+      - listeners
+    verbs:
+      - get
+      - list
+      - watch
+      - patch
+      - create
+      - delete
   - apiGroups:
       - {{ include "operator.name" . }}.stackable.tech
     resources:

docs/modules/airflow/pages/usage-guide/listenerclass.adoc

@@ -1,18 +1,19 @@
 = Service exposition with ListenerClasses
 :description: Configure Airflow service exposure with ListenerClasses: cluster-internal, external-unstable, or external-stable.
 
-Airflow offers a web UI and an API, both are exposed by the webserver process under the `webserver` role.
-The Operator deploys a service called `<name>-webserver` (where `<name>` is the name of the AirflowCluster) through which Airflow can be reached.
-
-This service can have three different types: `cluster-internal`, `external-unstable` and `external-stable`.
-Read more about the types in the xref:concepts:service-exposition.adoc[service exposition] documentation at platform level.
-
-This is how the listener class is configured:
+The operator deploys a xref:listener-operator:listener.adoc[Listener] for the Webserver pod.
+The listener defaults to only being accessible from within the Kubernetes cluster, but this can be changed by setting `.spec.webservers.config.listenerClass`:
 
 [source,yaml]
 ----
 spec:
-  clusterConfig:
-    listenerClass: cluster-internal  # <1>
+  webservers:
+    config:
+      listenerClass: external-unstable  # <1>
+  schedulers:
+    ...
+  celeryExecutors:
+    ...
 ----
-<1> The default `cluster-internal` setting.
+<1> Specify a ListenerClass, such as `external-stable`, `external-unstable`, or `cluster-internal` (the default setting is `cluster-internal`).
+This can be set only for the webservers role.