0

I "inherited" old code for a server running several docker containers. Grafana and Prometheus are used to monitor stats, as well as node-exporter. Unfortunately, the node-exporter container is shown as down in Prometheus. The error message is Get "http://IP:9101/metrics": context deadline exceeded

The node-exporter container is the only one with network_mode host, the rest of the containers is in a user-defined bridge network. When I try to curl the metrics endpoint from the host machine with curl localhost:9091/metrics, it works. In the prometheus.yml, the scrape_config for node is defined as follows:

- job_name: node
    scrape_interval: 30s
    static_configs:
      - targets: ['HOST_NETWORK_IP:9101']

I have to get the connection between node_exporter and prometheus running and I have the feeling that the solution is most likely something simple, but I can´t figure it out, as I don´t really have much experience with docker networking. Any help would be appreciated!

Clarifications: The HOST_NETWORK_IP is the IP of the Open Stack Instance. However, normally, queries are routed through traefik. I am not quite sure where the exporter is listening, but it should be the default. The only change to listening in the docker-compose for the exporter is -web.listen-address=:9101. I didn´t set it up to run with host networking, so I am not a 100% sure about the reasoning behind it, however from my research it seems to me that this is necessary to get metrics from the host system.

LizzAlice
  • 111
  • 2
  • 1
    You should verify where the exporter is listening using `netstat`. You should also check all firewall rules of any kind. If you can access it using `localhost`, that’s unfortunately meaningless for Prometheus. You need to use `HOST_NETWORK_IP` in your tests as a start and in the end it needs to be reachable from the Prometheus container, which is in a different network namespace. – Daniel B Aug 03 '23 at 19:03

0 Answers0