Harbor replication not working

[Jeansen]

I run helm-charts-oci-proxy standalone, e.g. with this command:

docker run --rm --name helm-charts-oci-proxy -p 9000:9000 -e USE_TLS=true -v /data/cert/proxy-ng.cert:/certs/registry.pem:ro -v /data/cert/proxy-ng.key:/certs/registry-key.pem:ro 8gears.container-registry.com/library/helm-charts-oci-proxy

I did a test pull like this:

helm pull oci://proxy-ng:9000/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator

And this works. The logs show

proxy-2024/06/10 20:34:56 listening HTTP over TLS serving on port 9000
proxy-2024/06/10 20:35:00 download index: https://open-telemetry.github.io/opentelemetry-helm-charts/index.yaml
proxy-2024/06/10 20:35:00 downloading : https://open-telemetry.github.io/opentelemetry-helm-charts/index.yaml
proxy-2024/06/10 20:35:01 searching index for opentelemetry-operator with reference
proxy-2024/06/10 20:35:01 downloading : https://github.com/open-telemetry/opentelemetry-helm-charts/releases/download/opentelemetry-operator-0.62.0/opentelemetry-operator-0.62.0.tgz
proxy-2024/06/10 20:35:01 GET - /v2/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/tags/list
proxy-2024/06/10 20:35:01 HEAD - /v2/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/manifests/0.62.0
proxy-2024/06/10 20:35:01 GET - /v2/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/manifests/sha256:6e610e6d75e2e9838de76cd3930751417b41df47d5c8ec989b3ee74165c751a8
proxy-2024/06/10 20:35:01 GET - /v2/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
proxy-2024/06/10 20:35:01 GET - /v2/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/blobs/sha256:52fcda92afdc2d221213d45e2449a7f8123df0358ed898760fa229d10f1450e1

I followed along for the replication with Harbor, created an endpoint and a replication rule. I even removed the filter so everything would be replicated.

When I run the replication, I see the following logs:

proxy-2024/06/10 20:38:13 GET - /v2/
proxy-2024/06/10 20:38:13 download index: https://open-telemetry.github.io/index.yaml
proxy-2024/06/10 20:38:13 downloading : https://open-telemetry.github.io/index.yaml
proxy-2024/06/10 20:38:13 GET - /open-telemetry.github.io/v2/_catalog?n=1000

But...nothing will be replicated. Replication is a success, but 0 items have been replicated. I also cannot find anything specific in the Harbor logs ....

I am aware of the default settings, e.g. that manifests are cleaned up after 60 seconds. So, I did a replication directly after a test pull with helm. But also raising the timeouts did not help at all.

[Vad1mo]

How does your replication look like? Can you share screenshots of replication and registry?

[Jeansen]

Full URL is https://proxy-ng:9000/open-telemetry.github.io

Source filter was set to open-telemetry.github.io/**, then * and finaly I left it empty. Nothing worked so far.

[HaveFun83]

try this:

Endpoint:

• Endpoint URL: https://proxy-ng:9000

Replication:

• Source Registry: Name of the chartproxy Endpoint

To get all charts
Source resource filter:

• Name: open-telemetry.github.io/opentelemetry-helm-charts/**
• Tag: matching **

To get specific chart
Source resource filter:

• Name: open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator
• Tag: matching **

[Jeansen]

@HaveFun83 Thanks, but did not help. When I do it your way, I also do not see any downloads in the log, only:

proxy-2024/06/24 20:05:37 GET - /v2/
proxy-2024/06/24 20:05:37 GET - /v2/_catalog?n=1000
proxy-2024/06/24 20:06:22 cleanup cycle
proxy-2024/06/24 20:06:47 GET - /v2/
proxy-2024/06/24 20:06:47 GET - /v2/_catalog?n=1000
proxy-2024/06/24 20:06:52 GET - /v2/
proxy-2024/06/24 20:06:52 GET - /v2/

With the previous settings, at least something is being download on the proxy side ( see the logs excerpt below) but also then, there's nothing in Harbor:

proxy-2024/06/24 20:09:40 GET - /open-telemetry.github.io/v2/
proxy-2024/06/24 20:09:59 GET - /v2/
proxy-2024/06/24 20:09:59 download index: https://open-telemetry.github.io/index.yaml
proxy-2024/06/24 20:09:59 downloading : https://open-telemetry.github.io/index.yaml
proxy-2024/06/24 20:09:59 GET - /open-telemetry.github.io/v2/_catalog?n=1000
proxy-2024/06/24 20:10:22 cleanup cycle
proxy-2024/06/24 20:11:22 cleanup cycle
proxy-2024/06/24 20:11:52 GET - /v2/
proxy-2024/06/24 20:11:52 GET - /open-telemetry.github.io/v2/
proxy-2024/06/24 20:12:22 cleanup cycle

[HaveFun83]

@Jeansen

can you double check your config according to the logs your replication still point to the wrong path
open-telemetry.github.io instead of open-telemetry.github.io/opentelemetry-helm-charts

Also don't put any path behind your harbor endpoint url configuration
just plain https://proxy-ng:9000

I tried the following replication

Source registry: <your chartproxy endpoint>
Source resource filter
  Name: open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator
  Tag: 0.63.0
  Resource: image
Destination
  Namespace: <your harbor project>
  Flattening: Flatten All Levels

and it works without any problems

[Jeansen]

@HaveFun83 Oh..yay........ your are right.

With open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator it now works, before it was open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/**.

[Jeansen]

So, I tried around a bit more. What works for me, is the one above, that is I have the following set:

Source resource filter
Name: open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator

Also, this works fine:

Source resource filter
Name: open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-{operator,collector}

After running the replication, I have both charts with all their versions replicated in Harbor.

But this does not work. Nothing will be replicated:

Source resource filter
Name: open-telemetry.github.io/opentelemetry-helm-charts/**

But replication is indicated as a success.

@HaveFun83 Does this work for you?

[Jeansen]

@HaveFun83 I hope you are not too busy but maybe you could have a look at my last answer? Thank you :-)

[HaveFun83]

@Jeansen sorry for the late response.

When i use

Source resource filter
Name: open-telemetry.github.io/opentelemetry-helm-charts/**

The replication will only pull the opentelemetry-operator nothing else.
Maybe you need to configure one replication per chart?