Skip to content

Document automatically create network policies to allow Prometheus scraping traffic for workloads annotated with prometheus.io/scrape #292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 15, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,51 @@ would block existing traffic like **loadgenerator** -> **frontend**.



## Auto-generating network policies for Prometheus server
When an intent generates a network policy that can block Prometheus from querying metrics, the intents operator by default
will automatically generate network policies to enable Prometheus to access the metrics port on this workload.

The way Otterize identifies that a workloads (pods, services) would be blocked from Prometheus is composed of 2 conditions:
1. The workload has the `prometheus.io/scrape` annotation set to `true`
2. `prometheusServerConfigs` is set in the intents operator configuration to identify the Prometheus server, as [documented in the intents operator](/reference/intents-operator).

To disable this feature, consult the [documentation for the intents operator](/reference/intents-operator).


Let's look at an example.
In our cluster we have client, server and Prometheus server.
The server has the annotations `prometheus.io/scrape: "true"` and `prometheus.io/port: "8080"`.

By applying the following intents file:
```yaml
apiVersion: k8s.otterize.com/v2beta1
kind: ClientIntents
metadata:
name: client
namespace: prometheus-test
spec:
workload:
name: client
targets:
- service:
name: server
```

Otterize will generate a network policy allowing access from the **client** workload to the **server** service.
Once a network policy matches a pod, other traffic not allowed by existing network policies to the **server** will be blocked.
In our case, this means that the Prometheus server will not be able to communicate with the server and scrape its metrics.

To resolve this, Otterize will automatically generate a network policy allowing traffic from the **Prometheus server** to the **server**.
This is based on the presence of the `prometheus.io/scrape: "true"` annotation in the **server** and the `prometheusServerConfigs` configuration
that identifies the Prometheus server.

Important note:
If the **server** does not have the `prometheus.io/port` annotation, the network policy generated by Otterize will allow communication between the
**Prometheus server** and the **server** on all ports exposed by the server. This is not a best practice, so it's recommended to set the port explicitly
if known.



## How intents translate to network policies
Let's follow an example scenario and track how Otterize configures network policies when we apply intents.
### Deploy example
Expand Down
23 changes: 22 additions & 1 deletion docs/reference/intents-operator/README.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ to access the server. This label is used as the selector to determine which clie
pod.

Learn more: [Network policies deep dive](/features/network-mapping-network-policies/reference/Network-Policies-Deep-Dive)

### Handling Third Party or Infrastructure traffic
Otterize automates network policies that enables third party or infrastructure traffic.
Otterize can currently automatically handle [external traffic](#Handling external traffic) as well as [Prometheus monitoring traffic](#Handling Prometheus traffic).

#### Handling external traffic
The intents operator has automatic behavior for allowing external traffic for pods which have indicated that they are supposed to accept external traffic, such as by creating a `Service` of type `NodePort` or `LoadBalancer`, or an `Ingress` resource.

Expand All @@ -50,7 +55,23 @@ As the intents operator creates network policies, and the semantics of network p

This meant that if you had no network policies on a pod, and created ClientIntents for that pod, then external traffic would be blocked. To make it easy to enable pod-to-pod traffic without affecting expected external traffic, the intents operator automatically detects resources of kind `Service` of type `NodePort` or `LoadBalancer`, or an `Ingress` resource, and if it creates the first network policy to affect those pods, it also creates a network policy that allows external traffic to those pods, as specified by the external `Service`/`Ingress` - for example, it only allows traffic to the specified ports, not all traffic.

This behavior can be disabled using the Helm chart's values.
This behavior can be disabled using the Helm chart's values - `automateThirdPartyNetworkPolicies`.

#### Handling Prometheus traffic
The intents operator has automatic behavior for allowing Prometheus traffic to pods which have indicated that they are supposed to accept Prometheus traffic, by adding the `prometheus.io/scrape: "true"` annotation to the pod or service.

As the intents operator creates network policies, and the semantics of network policies dictate that:
1. if no network policies apply to a pod, then all traffic is allowed.
2. once any network policy applies to a pod, only the traffic explicitly allowed in the policy is allowed

This meant that if you had no network policies on a pod, and created ClientIntents for that pod, then Prometheus scraping would be blocked. To make it easy to enable traffic for Prometheus metrics scraping, the intents operator automatically detects resources with `prometheus.io/scrape: "true"` annotation, and if it creates the first network policy to affect those pods, it also creates a network policy that allows Prometheus scraping.

You must set `prometheusServerConfigs` in the Helm chart, so Otterize would be able to identify the Prometheus server in the cluster and restrict the network policy to only allow traffic from Prometheus.

This behavior can be disabled using the Helm chart's values - `automateThirdPartyNetworkPolicies`, or to only disable Prometheus, make sure `prometheusServerConfigs` is not set.

**Note**: If you haven't specified `prometheus.io/port` in the pod or service, Otterize will match Prometheus' behavior by allowing all ports.
To improve security, specify `prometheus.io/port` to require Prometheus to only connect to that port. Otterize will the only allow access to that port.

### AWS IAM
The intents operator enables automatic management of AWS IAM roles and policies according to the
Expand Down
Loading