Adam Whitlock

Whiticism at its finest

Prometheus Labels and Consul

Having the ability to combine Prometheus and Consul to allow discovery of monitoring endpoints is wonderful, but it can come with some pain points. Depending on how Consul is setup and the information passed from your applications, Prometheus may display endpoints that will never have any metrics emitted. This creates a lot of cruft in the targets view, making it look like there is more wrong than right. It’s easily fixed with the Prometheus relabel_config directives that are part of the Prometheus configuration. relabel_config allows you to change any labels discovered by Consul and specify target labels – like __address__ – and add correct information. Relabeling rules used to drop or keep specific metrics allow you to hide what you don’t want, keeping the interface and metrics system manageable.

The regex supplied in relabel_config uses the Go implementation of regex and can only be used for a single relabel operation at a time. You cannot chain multiple relabel rules, such as renaming one label and then modifying that result. While chaining result modification seems like it would be ideal, really it would complicate matters. If you feel that making multiple chained relabeling is the only true way, then you are in for a big surprise: create a new job_name directive in your configuration. Through creating a new job_name, you will be able to specifically target all operations that have to occur in that single job. If you need to have all detected hosts have the same port, relabel everything in that job and drop what doesn’t match. It will be much easier to do that than manage multiple chained rules in the future. I mean, this isn’t Nagios… right?

Creating Target Groups by Job Name

Depending on how many services we have detected, we might want to create a per-service target group instead of lumping ALL services into individual jobs. This rule example highlights how we can take a label name and create a service-to-job result. There is an additional definition for dropping consul from the discovered service list, which isn’t required but might be helpful.

  - job_name: microservices

    # You MUST specify the source of configurations on a per-job_name basis
    consul_sd_configs:
      - server: 'localhost:8500'

    relabel_configs:
      # Service names become the job name
      - source_labels: [__meta_consul_service]
        target_label:  job

      # Hide Consul from the targets list
      - source_labels: [__meta_consul_service]
        regex:  '(^consul)'
        target_label: job
        action: drop

Prometheus Node Exporter and Consul Port

Any of the hosts in Consul and detected by Prometheus should have node-exporter running. When detected, this job will create a new list of nodes and use the default node-exporter port of 9100.

  - job_name: node-exporter
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default, otherwise delete this directive
    static_configs:
      - targets: ['localhost:9100']

    # You MUST specify the source of configurations on a per-job_name basis
    consul_sd_configs:
      - server: 'consul.domain.com:8500'

    # All hosts detected will use port 9100 for node-exporter
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '(.*):(.*)'
        target_label:  '__address__'
        # The first regex result is used for the replacement rule, which
        # is the host address without port. 9100 is appended to the address
        replacement:   '${1}:9100'

Creating Targets with Consul Tags

This is an example of targeting a specific consul tag – apache – and associating a specific port to check for metrics. This example also highlights renaming the target to something easier to view in the targets list, which we’ll want ‘apache-httpd2’.

  - job_name: apache_service

    # You MUST specify the source of configurations on a per-job_name basis
    consul_sd_configs:
      - server: 'consul.domain.com:8500'

    relabel_configs:
      # Target 'apache' services
      - source_labels: ['__meta_consul_tags']
        regex:  '.*,apache,.*' 
        target_label: job
        # This will drop all targets that do not match the regex rule,
        # leaving only the 'apache' targets
        action: 'keep'
      # Use port 9117, used by Prometheus Apache Exporter
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '(.*):(.*)'
        target_label:  '__address__'
        replacement:   '${1}:9117'
      # Will be listed as 'apache-httpd2' in the targets view
      - source_labels: ['__meta_consul_tags']
        target_label: job
        replacement: 'apache-httpd2' 

Supporting documents: * Prometheus - Relabel Config