Document real world examples of using relabel_config

[davidmankin]

WRT http://prometheus.io/docs/operating/configuration/#%3Crelabel_config%3E

The documentation for relabel_config is very clinical and hard to understand. (At least it is accurate, as far as I can tell.)

In both relabel_configs and metric_relabel_configs some real-world examples would go a long way. (In fact, relabel_configs doesn't even have a paragraph. metric_relabel_configs says "One use for this is to blacklist time series that are too expensive to ingest" but gives no hint at how to accomplish that.)

It would be great if the examples gave some typical uses for relabeling, and included examples of relabeling a kubernetes_sd_config or similar.

I found the relabelling much easier to understand after I found this blog post which contained a few sparse examples: https://prometheus.io/blog/2015/06/01/advanced-service-discovery/ but, being a blog post instead of docs, it's already out of date.

[fabxc]

I agree that more examples would are necessary and that blog posts come with the issue of going out of date.
I'm strongly in favor for keeping the core docs clinical and precise though.

If I want to look something up and know what I'm looking for, this shouldn't be disturbed by all the examples. One doesn't want to scan through a lot of text that may or may not contain feature specification.

A guide section would probably be the way to go for having up-to-date examples with more detailed explanation.

As so often I see the issue that this (as the current documentation) will need versioning at some point.
Any suggestions about decent software for that are welcome.

[davidmankin]

Further, it'd be great if there could be some suggestions or howto on developing, testing, and debugging relabel configs. At least point out that the applied relabels appear the /status page and the original input labels appear in a tooltip. (I have no idea how to go about testing/debugging metric relabelling. Is there a similar feature?)

[davidmankin]

The Kubernetes documentation is versioned with the release. In github they have a header that says "warning this is the HEAD documentation" and then they strip it out when snapshotting the docs for a release.

[davidmankin]

I don't mind clinical & precise reference docs as long as they include a reference to a guide section of the docs.

[marekbruchaty]

I'm finding it very difficult to understand the relabeling functionality from the official docs. Few straight forward examples would be extremely useful.

[grobie]

The lack of examples and readability of our docs is one of the most common things I hear from people at meetups.

Fortunately with interactive web pages it's not a black/white or concise=no-examples question. I don't see any reason we can't provide additional examples on the same page which only get shown upon click or something like that.

[brian-brazil]

The challenge with relabel examples is that they quickly get into configuration management philosophy, even when hidden they'd quickly become unwieldy as there's just so many possibilities.

I'm for a strong separation between reference docs and guides. I've seen where they've gotten mixed for the sake of getting people started, and it ended up making everyone unhappy.

being a blog post instead of docs, it's already out of date.

Our blog posts are meant to be up to date. Please file a bug for any inaccuracies you find.

[leonerd]

If it's any help I just stumbled upon this bug while struggling with a lack of docs. After much fiddling and flailing I've managed to make my use-case work.

My problem was I wanted to relabel bridge_JOBNAME_METRIC_NAME_HERE into bridge_METRIC_NAME_HERE{job=JOBNAME}. I've solved it by:

    metric_relabel_configs:
      # Extract the bridge name into the job label
      - source_labels: ["__name__"]
        regex: "bridge_([^_]+)_.*"
        target_label: "job"
        replacement: "$1"
      # Rewrite the label name to remove the bridge name out of it
      - source_labels: ["__name__"]
        regex: "bridge_[^_]+_(.*)"
        target_label: "__name__"
        replacement: "$1"

[brian-brazil]

You might want to throw that up on http://tips.robustperception.io/

[buchanae]

I'm still dumbfounded by using relabeling and service discovery. Normally I would just hit the +1 button, but this issue is nearly 2 years old.

[discordianfish]

Well, about time to look into this again :)
I agree with @grobie, we can both have a concise reference docs and some more practical guide/tips/examples. I remember vividly when I introduced Prometheus at Docker, the biggest concern was lack of examples/snippets to get people started. From a UX perspective, I think some sidebar you can open additionally to the reference, which shows further tips/examples.

I think it would be neat to have a separate repo for this with lower review requirements, so changes to the example don't need to be vetted by maintainers. A wiki might be even better, but something that is mapped to the docs and viable side by side (so you see right away when things don't add up and examples need updating). Something that gets loaded via js, so doesn't make the reference docs slower/bigger?

[bekir-murat]

Afaik, there is no need for fancy UI stuff, even simple links to examples in a git repo would go a long way.
We decided to use Prometheus for our monitoring infrastructure but we are questioning our choice now that we are digging deeper and using more advanced features. Our main gripes are with configuration and discovery.

[jamtur01]

I wrote the getting started as a guide - https://prometheus.io/docs/prometheus/latest/getting_started/ (#895) and added a guide section - it'd be very easy to add further guides: using exporters, relabelling, service discovery, and querying all spring to mind as ideas.

[bekir-murat]

Thank you for this, I used the getting started guide for our initial setup.
Would it still be a getting started guide though, if these subjects were added ?