Add Proxy Settings for Crawler

[Sinterdial]

Describe the feature you'd like

In certain regions of the world, internet censorship is very strict, so proxies must be used to access some very popular websites.

However, when I tried to use the environment variables HTTP_PROXY and HTTPS_PROXY to route container traffic through a proxy, it had no effect. I hope there can be configuration options added in the .env file to enable this functionality.

Describe the benefits this would bring to existing Hoarder users

Users around the world

Can the goal of this request already be achieved via other means?

Yes, this could perhaps be achieved by setting the proxy in the Docker daemon's configuration file, but I only want to enable the proxy for specific containers

Have you searched for an existing open/closed issue?

• I have searched for existing issues and none cover my fundamental request

Additional context

No response

[MohamedBassem]

you should be able to setup a proxy server on the chrome container (where most of the fetching happens) using chrome flags. Check #420 out

[KortanZ]

Is there any way to setup a proxy server when using browserless? It will be great if i could reuse my browserless service :D.

[MohamedBassem]

@KortanZ seems like you can by adding it at the end of the browserless URL you give to hoarder (https://docs.browserless.io/recipes/proxies#specifying-the-proxy)

[primejava]

I have added the proxy-server parameter in both the command and environment sections of the Chrome container. After successfully starting the container, I verified that the proxy is working by executing the command curl -I www.youtube.com inside the container. However, when I paste a URL into Hoarder, I still encounter the error Crawling job failed: TimeoutError: Navigation timeout of 30000 ms exceeded

[Sinterdial]

I have added the proxy-server parameter in both the command and environment sections of the Chrome container. After successfully starting the container, I verified that the proxy is working by executing the command curl -I www.youtube.com inside the container. However, when I paste a URL into Hoarder, I still encounter the error Crawling job failed: TimeoutError: Navigation timeout of 30000 ms exceeded

same

[waynexia]

In the docker environment the networking is a bit different. You need a special IP 172.17.0.1 to access the host network (my handbook)

[seancheung]

I have added the proxy-server parameter in both the command and environment sections of the Chrome container. After successfully starting the container, I verified that the proxy is working by executing the command curl -I www.youtube.com inside the container. However, when I paste a URL into Hoarder, I still encounter the error Crawling job failed: TimeoutError: Navigation timeout of 30000 ms exceeded

Like @waynexia said, networks in docker containers are different. You could use extra_hosts if the proxy server is not in the same docker network:

extra_hosts:
  - 'proxy_server:192.168.0.232' # give the ip a hostname
command:
  - --proxy-server='http=proxy_server:20171;https=proxy_server:20171' # use hostname instead of ip

[maidou-00]

I have added the proxy-server parameter in both the command and environment sections of the Chrome container. After successfully starting the container, I verified that the proxy is working by executing the command curl -I www.youtube.com inside the container. However, when I paste a URL into Hoarder, I still encounter the error Crawling job failed: TimeoutError: Navigation timeout of 30000 ms exceeded

Like @waynexia said, networks in docker containers are different. You could use extra_hosts if the proxy server is not in the same docker network:

extra_hosts:

• 'proxy_server:192.168.0.232' # give the ip a hostname
  command:
• --proxy-server='http=proxy_server:20171;https=proxy_server:20171' # use hostname instead of ip

I‘ve tried this but it didn't work...not sure what's wrong:

extra_hosts:
      - "shadowsocks:172.83.0.1"
command:
      - --proxy-server="https=shadowsocks:1080" # use hostname instead of ip

[dannongruver]

any update on this? i experience the same issue. proxy settings on env vars (like any other docker container) but crawling and other errors (eg downloading adblock). proxy works b/c curl works from within the container.

[MohamedBassem]

This is coming in the next release: 360ef9d