lychee shows network error: forbidden for valid links

[Rizwan-Hasan]

Suddenly lychee shows this error for valid links. But these links are valid and also accessible from the browser.

❯ lychee --max-concurrency 1 --no-progress --verbose "work/ok.txt"
✗ [403] https://catboost.ai/ | Network error: Forbidden
✗ [403] https://catboost.ai/en/docs/concepts/python-reference_datasets_msrank | Network error: Forbidden

Issues found in 1 input. Find details below.

[work/ok.txt]:
✗ [403] https://catboost.ai/ | Network error: Forbidden
✗ [403] https://catboost.ai/en/docs/concepts/python-reference_datasets_msrank | Network error: Forbidden

🔍 2 Total ✅ 0 OK 🚫 2 Errors (HTTP:2)

Contents of work/ok.txt

https://catboost.ai/
https://catboost.ai/en/docs/concepts/python-reference_datasets_msrank

Lychee version

❯ lychee --version
lychee 0.10.1

[mre]

Hum, looks like it's an issue on your end. 🤔 At least it works over here:

❯❯❯ lychee --max-concurrency 1 --no-progress --verbose "work/ok.txt"
✔ [200] https://catboost.ai/
✔ [200] https://catboost.ai/en/docs/concepts/python-reference_datasets_msrank

🔍 2 Total ✅ 2 OK 🚫 0 Errors

Can you try from a different network? Or maybe reconnect to your wifi? Maybe it also was just a temporary hickup?

[Rizwan-Hasan]

Hi @mre ,

It originally happened on github action. Same happened after I tried it on my local pc. Ran it several time on github action too and still the same.
Here's the recent github action log https://github.com/Rizwan-Hasan/clearml-docs/runs/7825833483
And this is 3 days earlier https://github.com/Rizwan-Hasan/clearml-docs/runs/7787255012

We need it to work on GitHub Action.

[mre]

Oh I see. No clue what's going on there yet.

[Rizwan-Hasan]

Any idea how to make it work?

[Rizwan-Hasan]

I just tried it on another cloud server and same error.

[mre]

What's strange is that it works for me. So it must be something that's wrong with the client-side.
Some thoughts:

• Did you try a different user-agent?
• Has any configuration been changed for catboost.ai recently? A firewall rule or a server config for example?
• If the catboost.ai docs aren't self-hosted, maybe your hosting provider changed a setting.
• Can you check the catboost.ai logs if you have access?
• When was the last time it worked? Was it a permanent error from then on or did it occasionally work for a while?

I'm afraid there ain't much I can do on my end other than asking questions. 😅

[Rizwan-Hasan]

On your thoughts,

• Tried with a different user-agent (used the same user-agent my browser has) and the error continues.
• We don't have any access to catboost.ai as it's not ours. So, can't tell if anything has changed there.
• I've just checked one GitHub Action log of 2 months ago and the error on these two links was present there.

[Rizwan-Hasan]

I've created a new git repository to make a mock test. And the result is negative.
Repository: https://github.com/Rizwan-Hasan/test-lychee-links
Logs: https://github.com/Rizwan-Hasan/test-lychee-links/runs/7826437363

[Rizwan-Hasan]

Hi @mre ,

Any update?

[mre]

Nope, no update. I don't think I can help much here. 🙁 It works over here, so it's definitely a problem with the server preventing certain clients from getting access. It's not something we can fix in lychee.

[Rizwan-Hasan]

Understood. But it's not working on my local PC and Linode server also. I'm confused. So, three places in a row it's not working.

[mre]

Yeah that's definitely weird. If anyone can help out narrow this down, please run the following command on your machine and report back the result.

echo 'https://catboost.ai' | lychee -

[Rizwan-Hasan]

With lychee it's not working

❯ echo 'https://catboost.ai' | lychee -
Issues found in 1 input. Find details below.

[stdin]:
✗ [403] https://catboost.ai/ | Network error: Forbidden

🔍 1 Total ✅ 0 OK 🚫 1 Error (HTTP:1)

On the other hand working with curl

❯ curl -I 'https://catboost.ai'        
HTTP/1.1 200 OK
Content-Length: 79368
Content-Security-Policy: default-src 'none'; script-src 'unsafe-eval' 'unsafe-inline' 'nonce-EzZqyXy1OG/i2ADhnTKeSw==' mc.yandex.ru social.yandex.ru yastatic.net; style-src 'unsafe-inline' mc.yandex.ru yastatic.net; img-src 'self' data: avatars.yandex.net avatars.mds.yandex.net avatars.mdst.yandex.net mc.yandex.ru ext.captcha.yandex.net yastatic.net; connect-src 'self' mc.yandex.ru; frame-src www.youtube.com video.yandex.ru player.video.yandex.net; media-src ext.captcha.yandex.net; font-src yastatic.net; report-uri https://csp.yandex.net/csp?from=promo-catboost-2017&yandex_login=undefined&yandexuid=undefined;
Content-Type: text/html; charset=utf-8
Date: Tue, 16 Aug 2022 09:35:03 GMT
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block

[mre]

Does this work for you?

 echo 'https://catboost.ai' | lychee --user-agent 'curl/7.79.1' -