documentation is not clear on how you connect to the gateway from a push client

[JohnBasrai]

I started the container with the following command line

docker run --name prom-agg-gateway -d -p 8888:8888 -p 80:80 ghcr.io/zapier/prom-aggregation-gateway

This maps the two ports it uses by default to the host. Docker logs say:

$ docker logs prom-agg-gateway
2023/10/16 16:35:25 lifecycle server listening at :8888
2023/10/16 16:35:25 api server listening at :80

The readme says to use one of the stand standard clients. I tried the python one.

https://github.com/prometheus/client_python

This page has the following example of how to push metrics.

from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
registry = CollectorRegistry()
g = Gauge('job_last_success_unixtime', 'Last time a batch job successfully finished', registry=registry)
g.set_to_current_time()
push_to_gateway('localhost:9091', job='batchA', registry=registry)

I tired adjusting this client code to use each of the two ports that appeared in the log for prom-aggregation-gateway (80 & 8888) and both of them get the same 404 error.

urllib.error.HTTPError: HTTP Error 404: Not Found

For reference here is the help message I get from the container:

$ docker exec -it prom-agg-gateway /prom-aggregation-gateway --help
prometheus aggregation gateway

Usage:
  prom-aggregation-gateway [flags]
  prom-aggregation-gateway [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  start       starts up the server
  version     Show version information

Flags:
      --AuthUsers strings        List of allowed auth users and their passwords comma separated
                                  Example: "user1=pass1,user2=pass2"
      --apiListen string         Listen for API requests on this host/port. (default ":80")
      --cors string              The 'Access-Control-Allow-Origin' value to be returned. (default "*")
  -h, --help                     help for prom-aggregation-gateway
      --lifecycleListen string   Listen for lifecycle requests (health, metrics) on this host/port (default ":8888")

Use "prom-aggregation-gateway [command] --help" for more information about a command.

There is no mention here or in the readme on what the two ports are but if I were to take a guess I think the second one above --lifecycleListen string is the one I should be using since it mentions metrics.

Please help.

[djeebus]

The docs make a quick mention here to send metrics to /metrics/, with the implication that http://localhost/, but we could do a better job of calling out exactly where to send the metrics. You can also add labels in that url by sending metrics to /metrics/label1/value1/label2/value2/, etc.

The "lifecycle server" only exists to give kubernetes a place to send probes to, and expose internal metrics. The metrics exposed by the lifecycle server describe the health and activity of the gateway itself, as opposed to metrics that are forwarded by and aggregated for the gateway's clients.

[JohnBasrai]

Thanks for the reply,
Here is what I tried and what I see in response

$ docker run -d -p9091:9091 prom/pushgateway
$ echo '
http_requests_total{method="post",code="200"} 1027
http_errors_total{method="post",code="500"} 6
'
+ curl --data-binary @- http://localhost:9091/metrics/domain/sometest.com/instance/nginx-1
404 page not found

Thanks again.

[djeebus]

You're using the prometheus pushgateway there, not prom-aggregation-gateway. Theirs requires that your path includes a job parameter first:

[~]$ echo '  
http_requests_total{method="post",code="200"} 1027
http_errors_total{method="post",code="500"} 6     
'| curl --data-binary @- http://localhost:9091/metrics/job/some-job/domain/sometest.com/instance/nginx-1 -v
*   Trying 127.0.0.1:9091...
* Connected to localhost (127.0.0.1) port 9091 (#0)
> POST /metrics/job/some-job/domain/sometest.com/instance/nginx-1 HTTP/1.1
> Host: localhost:9091
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 99
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 200 OK
< Date: Tue, 17 Oct 2023 17:48:36 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact

prom-aggregation-gateway, on the other hand, allows you to add the /job/$job label pair anywhere in the url, or omit it altogether.

[Samreay]

Hey @JohnBasrai, did you manage to figure this out in the end? Even reading over the comments left by @djeebus I'm still unable to get a docker-compose setup working that can connect. I've tried pushing metrics to / (and have also tried pushing to /metrics) endpoint on host ports 9091, 80, and 8888, but all are giving connection refused. This is the current service configuration, and I've been changing the endpoint I hit in the python service to try and get something that works.

To try and narrow this down, I've got a docker compose:

version: '3.9'
services:

  pushgateway:
    container_name: pushgateway
    image: ghcr.io/zapier/prom-aggregation-gateway:v0.9.0
    environment:
      - PAG_APILISTEN=":9091"
    ports:
      - 9091:9091

  exportmetrics:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: exportmetrics
    environment:
      - PORT=9091
    depends_on:
      - pushgateway

A python file: dinky.py:

import os
import time

from prometheus_client import CollectorRegistry, Counter, push_to_gateway

registry = CollectorRegistry()
counter = Counter("some_counter", "A counter", registry=registry)
counter.inc()
time.sleep(1)
endpoint = f"http://pushgateway:{os.environ['PORT']}"
push_to_gateway(endpoint, job="flows", registry=registry)

And a Dockerfile:

FROM python:3.12-slim
RUN pip install prometheus-client
COPY dinky.py /dinky.py
CMD ["python", "/dinky.py"]

Some findings:

• I don't know what I'm supposed to set PAG_APILISTEN to.
  • The example says the default is ":80", but if you set PAG_APILISTEN=":80", you get error while serving api: listen tcp: address "80": missing port in address
  • If you set PAG_APILISTEN="80" you get error while serving api: listen tcp: address "80": missing port in address
  • If I add I set it to localhost:80 or 0.0.0.0:80 or similar, still failures.
  • Documentation on what this should be could be clarified.
• Option 2: Don't change the port, just remap it. If I remove the env var in the pushgateway and simply map the port via 9091:80 and then try to push to 9091 in the python file, this fails with urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
• Option 3: Let's just override the command. As per the docs, I thought it should be command: ["prom-aggregation-gateway", "--appListen=:9091"], but this gives Error: unknown command so maybe not.
• Option 4: get inside the push gateway and manually run it via docker run -it --entrypoint sh ghcr.io/zapier/prom-aggregation-gateway:v0.9.0 and then /prom-aggregation-gateway --appListen :9091. Alas this returns Error: unknown flag: --appListen

I have also tried all the above with an explicit bridge network in docker-compose.

From these steps, I feel like there is something large missing for the documentation, or there might be possible bugs in setting the listening port.