[ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks #10490

DrAuYueng · 2022-03-27T02:00:08Z

Fixes docker sd service misssing in shared mode(#10141).
Provide a feature called MatchFirstNetwork that allows matching the first network if the container has multiple networks defined, thus avoiding collecting duplicate targets(#8831).

Signed-off-by: DrAuYueng [email protected]

…#10490

bboreham · 2024-01-16T11:24:24Z

We discussed this in the bug-scrub meeting. It looks useful, but the intention is a little hard to understand.
@DrAuYueng could you write a paragraph from the end-user's point of view, when they would add this setting and what it will do?

@roidelapluie can you take a look?

bwplotka · 2024-01-16T11:26:51Z

docs/configuration/configuration.md

@@ -718,6  718,9 @@ tls_config:
 # The host to use if the container is in host networking mode.
 [ host_networking_host: <string> | default = "localhost" ]

+# Whether filter networks by user defined network mode if the container has multiple networks.
+[ filter_networks: <boolean> | default = true ]


(During Prometheus bug scrub meeting): This flag seems too fuzzy e.g. "filter" can both mean filter out, and filter in (choose). Also this option is doing more than that (if no user defined network then something else is performed).

@bwplotka Perhaps the configuration is defined as match_first_network more appropriate.

DrAuYueng · 2024-01-17T14:11:06Z

We discussed this in the bug-scrub meeting. It looks useful, but the intention is a little hard to understand. @DrAuYueng could you write a paragraph from the end-user's point of view, when they would add this setting and what it will do?

In shared network mode, the current container has no network definition，for example:

{
	 "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {}
        }
}

As shown above, networks are undefined and the shared network is defined in HostConfig, which corresponds to NetworkMode, for example:

{
	 "HostConfig": {
            "NetworkMode": "container:62de3e27e51e1ccebc273b33706ba1c9e8f4d68507257c83b9bc27262f0533b8"
     }
}

If the container to which networkmode points to has the specified network, NetworkMode points to a container that has a specified network, the container's network is the destination network, otherwise the loop continues.

If the container defines multiple networks, for Prometheus, only one available network is needed to obtain monitoring metrics, so a configuration is required here to allow users to match the first network to avoid duplicate collection targets.

bboreham · 2024-01-17T15:51:50Z

Thanks. This reads like a programmer-level description to me.
If I am someone running a few apps under Docker, and wanting to use Prometheus, what would be a suitable description?

The description should reference the new setting (i.e. filter_networks or match_first_network).

DrAuYueng · 2024-01-18T12:58:29Z

Match the first network if the container has multiple networks defined, thus avoiding collecting duplicate targets.

@bboreham @bwplotka Is this description clear and easy to understand?

bboreham · 2024-01-18T13:53:47Z

Much better, thanks.

johntdyer · 2024-05-20T18:42:04Z

Any idea when this might be merged?

beorn7 · 2024-05-21T15:55:46Z

@bwplotka @bboreham what's your state on this?

beorn7 · 2024-06-18T11:41:04Z

@bwplotka @bboreham ping'ing again.

bboreham

I don't know enough about Docker behaviour to say whether this valid, but the code looks plausible. I'm ok to merge and see if anyone complains.

bboreham · 2024-06-20T11:27:55Z

Looks like something else changed underneath:

cannot use prometheus.NewRegistry() (value of type *prometheus.Registry) as discovery.DiscovererMetrics value in argument to NewDockerDiscovery: *prometheus.Registry does not implement discovery.DiscovererMetrics (wrong type for method Register)

DrAuYueng · 2024-06-20T12:04:20Z

Looks like something else changed underneath:

cannot use prometheus.NewRegistry() (value of type *prometheus.Registry) as discovery.DiscovererMetrics value in argument to NewDockerDiscovery: *prometheus.Registry does not implement discovery.DiscovererMetrics (wrong type for method Register)

I'll take care of it.

DrAuYueng · 2024-06-20T13:42:16Z

/hold for local tests.

DrAuYueng · 2024-06-21T02:39:57Z

I'am confused about golangci-lint check result:

discovery/moby/docker.go:33: File is not `goimports`-ed with -local github.com/prometheus/prometheus (goimports)
	"github.com/prometheus/common/model"

I did do the goimport processing, but nothing changed

goimports -w discovery/moby/docker.go

Am I missing something?

DrAuYueng · 2024-06-21T10:46:38Z

discovery/moby/docker.go

@@ -73,7  76,8 @@ type DockerSDConfig struct {
 	Filters            []Filter `yaml:"filters"`
 	HostNetworkingHost string   `yaml:"host_networking_host"`

-	RefreshInterval model.Duration `yaml:"refresh_interval"`
+	RefreshInterval   model.Duration `yaml:"refresh_interval"`
+	MatchFirstNetwork bool           `yaml:"match_first_network "`


There's an extra space.

`yaml:"match_first_network "`

DrAuYueng · 2024-06-21T10:55:56Z

Network shared mode:

version: '3.7'
networks:
  mysql_net:
    name: mysql_net
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.20.0.0/24
services:
  mysql:
    image: mysql:5.7.29
    restart: always
    networks:
      mysql_net:
        ipv4_address: 172.20.0.10
    environment:
      - MYSQL_ROOT_PASSWORD=password
      - MYSQL_DATABASE=database
    hostname: mysql
    container_name: mysql
    volumes:
      - "./config/my.cnf:/etc/my.cnf"
      - "./config/data:/var/lib/mysql"
      - "./config/init_sql:/docker-entrypoint-initdb.d/"
  mysqlexporter:
    image: prom/mysqld-exporter
    container_name: exporter__mysql
    depends_on:
      - mysql
    network_mode: service:mysql
    command:
     - "--mysqld.username=root:password"
     - "--mysqld.address=mysql:3306"
    environment:
      - DATA_SOURCE_NAME=root:password@(mysql:3306)/database

The result looks good:

DrAuYueng · 2024-06-21T11:01:27Z

Multi networks and shared mode:

version: '3.7'
networks:
  mysql_net:
    name: mysql_net
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.20.0.0/24
  mysql_net1:
    name: mysql_net1
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.21.0.0/24
services:
  mysql:
    image: mysql:5.7.29
    restart: always
    networks:
      mysql_net:
        ipv4_address: 172.20.0.10
      mysql_net1: {}
    environment:
      - MYSQL_ROOT_PASSWORD=password
      - MYSQL_DATABASE=database
    hostname: mysql
    container_name: mysql
    volumes:
      - "./config/my.cnf:/etc/my.cnf"
      - "./config/data:/var/lib/mysql"
      - "./config/init_sql:/docker-entrypoint-initdb.d/"
  mysqlexporter:
    image: prom/mysqld-exporter
    container_name: exporter__mysql
    depends_on:
      - mysql
    network_mode: service:mysql
    command:
     - "--mysqld.username=root:password"
     - "--mysqld.address=mysql:3306"
    environment:
      - DATA_SOURCE_NAME=root:password@(mysql:3306)/database

The result looks good:

DrAuYueng · 2024-06-21T11:04:36Z

If the match_first_network is false, the result is:

Signed-off-by: [email protected] <[email protected]>

DrAuYueng · 2024-06-26T10:44:52Z

/hold cancel.
Ping @bboreham.

cs8898 · 2024-08-12T13:02:27Z

For others requiring multiple networks, and got to that PR because the default value changed, this is how i managed my scrape config originally

scrape_configs:
  - job_name: "docker-containers"
    docker_sd_configs:
      - host: unix:///var/run/docker.sock # You can also use http/https to connect to the Docker daemon.
    relabel_configs:
      # Only keep containers that have a `prometheus-job` label.
      - source_labels: [__meta_docker_container_label_prometheus_job]
        regex: . 
        action: keep
      # Only keep containers in "monitoring" network
      - source_labels: [__meta_docker_network_name]
        regex: monitoring
        action: keep
      # Extract Port or overwrite `prometheus-port` label.
      - source_labels: [__address__]
        regex: '[^\s] :(\d )'
        target_label: __tmp_port
      - source_labels: [__meta_docker_container_label_prometheus_port]
        target_label: __tmp_port
        action: keepequal
      # Use the task labels that are prefixed by `prometheus-`.
      - regex: __meta_docker_container_label_prometheus_(. )
        action: labelmap
        replacement: $1
      - target_label: host
        replacement: hostname01 # static hostname
      - source_labels: ['__meta_docker_container_label_com_docker_compose_project']
        target_label: 'project'
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'
      - source_labels: ['__meta_docker_container_label_com_docker_compose_container_number']
        target_label: 'container_number'

DrAuYueng · 2024-08-12T13:19:32Z

@cs8898 You can use multiple networks by setting the configuration match_first_network to false.
The configuration is as follows:

scrape_configs:
  - job_name: 'docker-containers'
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        match_first_network: false

cs8898 · 2024-08-12T13:24:14Z

@DrAuYueng already did that, just added my config as reference, because it basically solves the same "problem".
Was quiet interesting to see the "old" behavior breaking, thats how i endet up in this issue.

So all fine, at least the docu was updated, so i was able, to find the relevant config key quiet fast

DrAuYueng force-pushed the fix-docker-sd-service-missing branch from 55ccc78 to 9f8da93 Compare April 1, 2022 11:32

DrAuYueng changed the title ~~Fixes docker sd service misssing in shared mode~~ Fixes docker sd service misssing in shared mode and deduplicate networks Apr 1, 2022

DrAuYueng changed the title ~~Fixes docker sd service misssing in shared mode and deduplicate networks~~ Fixes docker sd service misssing in shared mode and deduplicate targets by network Apr 1, 2022

DrAuYueng force-pushed the fix-docker-sd-service-missing branch from 966be77 to 43e9527 Compare April 3, 2022 04:16

DrAuYueng force-pushed the fix-docker-sd-service-missing branch from 43e9527 to 2164cfe Compare April 16, 2022 09:04

DrAuYueng mentioned this pull request Apr 16, 2022

Docker SD missing service: and container: networking modes #10141

Closed

james9001 added a commit to james9001/loki that referenced this pull request Jan 28, 2023

Incorporate fix for Docker SD by DrAuYueng from prometheus/prometheus…

cabad1c

…#10490

roidelapluie self-assigned this Jul 18, 2023

rgroothuijsen mentioned this pull request Nov 13, 2023

Promtail - Docker service discovery is not working for containers that use a certain network_mode directive grafana/loki#10739

Open

bwplotka reviewed Jan 16, 2024

View reviewed changes

DrAuYueng force-pushed the fix-docker-sd-service-missing branch 4 times, most recently from 1fc57f3 to 5328fc8 Compare January 21, 2024 01:40

bboreham mentioned this pull request Jan 23, 2024

Deduplicate containers attached to multiple networks #9111

Closed

krajorama mentioned this pull request Jun 18, 2024

docker_sd: Deduplicate containers attached to multiple networks #8831

Open

bboreham previously approved these changes Jun 20, 2024

View reviewed changes

DrAuYueng dismissed bboreham’s stale review via 5963477 June 20, 2024 13:35

DrAuYueng force-pushed the fix-docker-sd-service-missing branch from b43f477 to 8ba405d Compare June 21, 2024 01:40

DrAuYueng force-pushed the fix-docker-sd-service-missing branch 2 times, most recently from 071ccd3 to ca1215c Compare June 21, 2024 02:11

DrAuYueng force-pushed the fix-docker-sd-service-missing branch from ca1215c to ab10917 Compare June 21, 2024 03:49

DrAuYueng commented Jun 21, 2024

View reviewed changes

rebase main and adjust the configuration

0d25931

Signed-off-by: [email protected] <[email protected]>

DrAuYueng force-pushed the fix-docker-sd-service-missing branch from ab10917 to 0d25931 Compare June 21, 2024 11:10

bboreham approved these changes Jun 21, 2024

View reviewed changes

bboreham changed the title ~~Fixes docker sd service misssing in shared mode and deduplicate targets by network~~ [ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks Jun 26, 2024

bboreham merged commit c5040c5 into prometheus:main Jun 26, 2024
25 checks passed

charleskorn mentioned this pull request Jun 27, 2024

Merge upstream Prometheus at c5040c5 grafana/mimir-prometheus#655

Merged

DrAuYueng mentioned this pull request Aug 11, 2024

[release 2.54] [BUGFIX] Docker SD: fix the issue of failing to match the first network when the container is reconnected to a new network #14654

Merged

slashpai mentioned this pull request Aug 14, 2024

Docker SD: add MatchFirstNetwork field prometheus-operator/prometheus-operator#6839

Closed

DrAuYueng deleted the fix-docker-sd-service-missing branch October 9, 2024 12:42

jiekun mentioned this pull request Oct 31, 2024

vmagent: Add match_first_network support for docker service discovery VictoriaMetrics/VictoriaMetrics#7398

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks #10490

[ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks #10490

DrAuYueng commented Mar 27, 2022 •

edited

Loading

bboreham commented Jan 16, 2024

bwplotka Jan 16, 2024

DrAuYueng Jan 17, 2024

DrAuYueng commented Jan 17, 2024

bboreham commented Jan 17, 2024

DrAuYueng commented Jan 18, 2024

bboreham commented Jan 18, 2024

johntdyer commented May 20, 2024

beorn7 commented May 21, 2024

beorn7 commented Jun 18, 2024

bboreham left a comment

bboreham commented Jun 20, 2024

DrAuYueng commented Jun 20, 2024

DrAuYueng commented Jun 20, 2024

DrAuYueng commented Jun 21, 2024

DrAuYueng Jun 21, 2024

DrAuYueng Jun 21, 2024

DrAuYueng commented Jun 21, 2024

DrAuYueng commented Jun 21, 2024

DrAuYueng commented Jun 21, 2024

DrAuYueng commented Jun 26, 2024

cs8898 commented Aug 12, 2024

DrAuYueng commented Aug 12, 2024

cs8898 commented Aug 12, 2024

[ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks #10490

[ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks #10490

Conversation

DrAuYueng commented Mar 27, 2022 • edited Loading

bboreham commented Jan 16, 2024

bwplotka Jan 16, 2024

Choose a reason for hiding this comment

DrAuYueng Jan 17, 2024

Choose a reason for hiding this comment

DrAuYueng commented Jan 17, 2024

bboreham commented Jan 17, 2024

DrAuYueng commented Jan 18, 2024

bboreham commented Jan 18, 2024

johntdyer commented May 20, 2024

beorn7 commented May 21, 2024

beorn7 commented Jun 18, 2024

bboreham left a comment

Choose a reason for hiding this comment

bboreham commented Jun 20, 2024

DrAuYueng commented Jun 20, 2024

DrAuYueng commented Jun 20, 2024

DrAuYueng commented Jun 21, 2024

DrAuYueng Jun 21, 2024

Choose a reason for hiding this comment

DrAuYueng Jun 21, 2024

Choose a reason for hiding this comment

DrAuYueng commented Jun 21, 2024

DrAuYueng commented Jun 21, 2024

DrAuYueng commented Jun 21, 2024

DrAuYueng commented Jun 26, 2024

cs8898 commented Aug 12, 2024

DrAuYueng commented Aug 12, 2024

cs8898 commented Aug 12, 2024

DrAuYueng commented Mar 27, 2022 •

edited

Loading