Dependency download fails on Linux: (403) Forbidden

[Zaratusa]

Hi, I've generated a Dockerfile to build a custom version 4.27.2 on Linux in Gitlab CI with the following settings:

# This file was generated by ue4-docker version 0.0.106 with the following options:
# 
# - combine: true
# - credential_mode: "secrets"
# - excluded_components: {"ddc": false, "debug": true, "templates": true}
# - gitdependencies_args: "--exclude=Android --exclude=Mac --exclude=Win32 --exclude=Win64"
# - source_mode: "copy"
# 
# This Dockerfile combines the steps for the following images:
#
# - ue4-build-prerequisites
# - ue4-source
# - ue4-minimal

Running the Setup.sh causes the following error, even when running it without any excludes:

Failed to download 'http://cdn.unrealengine.com/dependencies/UnrealEngine-16546836/01aeef9d6f354d0b789fe0f90786078b0395495f': The remote server returned an error: (403) Forbidden. (WebException)

I've never had any problems downloading the dependencies on Windows or MacOS. Is this a known issue on Linux?

[TBBle]

I also get an Access Denied error for that URL from my Windows-based Chome from Japan. So possibly an issue with Epic's CDN.

[Zaratusa]

Read more about the fix here: https://forums.unrealengine.com/t/upcoming-disruption-of-service-impacting-unreal-engine-users-on-github/1155880

[slonopotamus]

Niiiice. This effectively means they broke all git tags.

[slonopotamus]

We're in trouble. I did some Python (#320) so we repair existing UE tags. The problem is that it doesn't work. Because we need to authenticate when we're downloading Commit.gitdeps.xml attached to a release.

[Zaratusa]

As mentioned before, the thread describes the fix: Manually update the Engine/Build/Commit.gitdeps.xml in your custom Engine. There is currently no automatic fix for this issue.

[tongwenbo]

Hello, thanks for the information. I have replaced this file on my local windows UE repo and it worked. Do you know where should I replace this file to in the Ubuntu machine where I run 'ue4-docker build'? Should I first clone the repo of Unreal Engine mannually first and replace this file and then run 'ue4-docker build'? Thanks a lot!

[jcastillo-lazr]

Hello, thanks for the information. I have replaced this file on my local windows UE repo and it worked. Do you know where should I replace this file to in the Ubuntu machine where I run 'ue4-docker build'? Should I first clone the repo of Unreal Engine manually first and replace this file and then run 'ue4-docker build'? Thanks a lot!

I have the same question. Right now I have local copy of UE4 but I would like to do modify that file in the container. The image is not created correctly therefore I cannot get inside the container and modify that file manually. I'm not sure if I should try to hack the python package and the Dockerfile.

[jonititan]

If I understand correctly this is an issue for anyone using ue4-docker and won't be fixed. Perhaps it needs including in the documentation?
I've got this issue as well but as a beginner to this area its a little more tricky to fix since its only documented in an issue and not mentioned in the docs.

[muety]

Any chances to get this fixed?

[TBBle]

This should only be a problem for users using custom (local) UE4 forks. In that case, you can apply Epic's fix to your local source, and things will work again. Per that announcement, UE4 4.25 and later custom forks should be able to merge (or cherry-pick) the latest tag from Epic and get the fixed file, but I haven't tried that myself.

Trying to automate this in ue4-docker is a hassle because we either need access to the user's GitHub credentials to extract the relevant file (which is what blocked #320), which usually isn't available when building from local forks; or we'd need to have available every updated Commit.gitdeps.xml for each release version, and I don't think the license allows us to distribute those.

Potentially we could add a flag that lets the ue4-docker user override Commit.gitdeps.xml, but that's pretty specific and requires the user to go download the correct file, in which case they may as well just commit it to their local fork and the problem is fixed.

If anyone is trying to build an old commit ID directly, rather than using local source, and hitting this issue, then #320's approach would be feasible, as you're already providing GitHub credentials for the download stage. If people are in this situation, then an approach like #320's could work, and if it's gated on providing GitHub credentials, then it'd work for people doing local source builds too if they additionally provide GitHub credentials.

To be clear, my understanding is that this only affects people (re)building the ue4-source image (and maybe ue4-minimal?), it didn't retroactively break existing images.

[jonititan]

Yes following the instructions to build airsim from source.
https://microsoft.github.io/AirSim/docker_ubuntu/

[muety]

I also don't think I'm using a custom fork. I'm really not versed about UE4 and its build process, was just following CARLA's docs to build a Docker image. I simply did pip3 install ue4-docker and ran into the above issue. Possibly the pip package is outdated or something?