Apply fails on Gitlab 16.4.0 with "Pull request must be mergeable before running apply."

[syst0m]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Running atlantis apply on an approved Gitlab PR fails intermittently with Apply Failed: Pull request must be mergeable before running apply..

Reproduction Steps

1. Run atlantis apply on approved PR
2. Atlantis comments with error: **Apply Failed**: Pull request must be mergeable before running apply.

Logs

Logs

August 23, 2023 at 17:03 (UTC+1:00)	{"level":"error","ts":"2023-08-23T16:03:28.369Z","caller":"events/instrumented_project_command_runner.go:84","msg":"Failure running apply operation: Pull request must be mergeable before running apply.","json":{"repo":"ume-platform-engineering/tf-env-data-engineering","pull":"189"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:84\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Apply\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:46\ngithub.com/runatlantis/atlantis/server/events.runProjectCmds\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:48\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:166\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:298"}

August 23, 2023 at 17:03 (UTC+1:00)	{"level":"info","ts":"2023-08-23T16:03:29.799Z","caller":"events/automerger.go:20","msg":"not automerging because project at dir \"XXX\", workspace \"default\" has status \"apply_errored\"","json":{"repo":"XXX","pull":"189"}}

August 23, 2023 at 17:03 (UTC+1:00)	{"level":"error","ts":"2023-08-23T16:03:32.912Z","caller":"logging/simple_logger.go:163","msg":"invalid key: e4d66b69-e7c7-495a-87fd-20f677f74b13","json":{},"stacktrace":"github.com/runatlantis/atlantis/server/logging.(*StructuredLogger).Log\n\tgithub.com/runatlantis/atlantis/server/logging/simple_logger.go:163\ngithub.com/runatlantis/atlantis/server/controllers.(*JobsController).respond\n\tgithub.com/runatlantis/atlantis/server/controllers/jobs_controller.go:92\ngithub.com/runatlantis/atlantis/server/controllers.(*JobsController).getProjectJobsWS\n\tgithub.com/runatlantis/atlantis/server/controllers/jobs_controller.go:70\ngithub.com/runatlantis/atlantis/server/controllers.(*JobsController).GetProjectJobsWS\n\tgithub.com/runatlantis/atlantis/server/controllers/jobs_controller.go:83\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2122\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.0/mux.go:210\ngithub.com/urfave/negroni/v3.Wrap.func1\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:59\ngithub.com/urfave/negroni/v3.HandlerFunc.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:33\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP\n\tgithub.com/runatlantis/atlantis/server/middleware.go:70\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/urfave/negroni/v3.(*Recovery).ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/recovery.go:210\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/urfave/negroni/v3.(*Negroni).ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:111\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2936\nnet/http.(*conn).serve\n\tnet/http/server.go:1995"}

Environment details

• Atlantis version: v0.23.2 & v0.23.4
• Deployment method: ecs
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: no
• Gitlab 16.4.0

Atlantis server-side config file:

{
    "taskDefinitionArn": "arn:aws:ecs:XXX:XXX",
    "containerDefinitions": [
        {
            "name": "atlantis",
            "image": "XXX/atlantis:v0.24.3-0bcdeaa1",
            "cpu": 2048,
            "memory": 4096,
            "memoryReservation": 128,
            "portMappings": [
                {
                    "containerPort": 4141,
                    "hostPort": 4141,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "ATLANTIS_ALLOW_REPO_CONFIG",
                    "value": "false"
                },
                {
                    "name": "ATLANTIS_HIDE_PREV_PLAN_COMMENTS",
                    "value": "false"
                },
                {
                    "name": "ATLANTIS_WRITE_GIT_CREDS",
                    "value": "true"
                },
                {
                    "name": "ATLANTIS_SILENCE_NO_PROJECTS",
                    "value": "false"
                },
                {
                    "name": "ATLANTIS_GITLAB_USER",
                    "value": "XXX"
                },
                {
                    "name": "ATLANTIS_LOG_LEVEL",
                    "value": "debug"
                },
                {
                    "name": "ATLANTIS_AUTOMERGE",
                    "value": "true"
                },
                {
                    "name": "ATLANTIS_BITBUCKET_USER",
                    "value": ""
                },
                {
                    "name": "ATLANTIS_REPO_CONFIG_JSON",
                    "value": "{\"repos\":[{\"allow_custom_workflows\":true,\"allowed_overrides\":[\"workflow\"],\"apply_requirements\":[\"undiverged\",\"mergeable\",\"approved\"],\"id\":\"/.*/\",\"repo_config_file\":\"atlantis.yaml\"}],\"workflows\":{\"default\":{\"apply\":{\"steps\":[\"apply\",{\"run\":\"[ ! -z \\\"$PROJECT_NAME\\\" ] \\u0026\\u0026 export TAG=$PROJECT_NAME || export TAG=last-applied \\u0026\\u0026 git config --global user.name XXXn \\u0026\\u0026 git config --global user.email XXX \\u0026\\u0026 git fetch --tags -f \\u0026\\u0026 git fetch --all --tags \\u0026\\u0026 (git tag --delete $TAG || true) \\u0026\\u0026 git tag -a $TAG -m \\\"Tagged automatically by atlantis\\\" \\u0026\\u0026 (git push origin --delete $TAG || true) \\u0026\\u0026 git push origin $TAG\"}]},\"plan\":{\"steps\":[\"init\",\"plan\"]}}}}"
                },
                {
                    "name": "ATLANTIS_PARALLEL_POOL_SIZE",
                    "value": "50"
                },
                {
                    "name": "ATLANTIS_REPO_ALLOWLIST",
                    "value": "XXX"
                },
                {
                    "name": "ATLANTIS_GITLAB_HOSTNAME",
                    "value": "gitlab.com"
                },
                {
                    "name": "ATLANTIS_DEFAULT_TF_VERSION",
                    "value": "v0.13.4"
                },
                {
                    "name": "ATLANTIS_GH_APP_ID",
                    "value": ""
                },
                {
                    "name": "ATLANTIS_BITBUCKET_BASE_URL",
                    "value": ""
                },
                {
                    "name": "ATLANTIS_PORT",
                    "value": "4141"
                },
                {
                    "name": "ATLANTIS_GH_USER",
                    "value": ""
                },
                {
                    "name": "ATLANTIS_ATLANTIS_URL",
                    "value": "https://XXX"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "secrets": [
XXX
]

Repo atlantis.yaml file:

version: 3
projects:

  - name: platform
    dir: XXX/platform
    autoplan:
      when_modified: [ "*.tf" ]
      enabled: true
...

Additional Context

Issue started happening on Atlantis v0.23.2. Still happening after upgrading to v0.23.4.

• fix(gitlab): Add backwards compatible mergability check #3277

[jamengual]

I will try 0.25.0 and see.

…

On Thu, Aug 31, 2023 at 3:07 AM Tomislav Tomašić ***@***.***> wrote: Community Note - Please vote on this issue by adding a 👍 reaction <https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/> to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you! - Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request. - If you are interested in working on this issue or have submitted a pull request, please leave a comment. ------------------------------ Overview of the Issue Running atlantis apply on an approved Gitlab PR fails intermittently with Apply Failed: Pull request must be mergeable before running apply.. Reproduction Steps 1. Run atlantis apply on approved PR 2. Atlantis comments with error: **Apply Failed**: Pull request must be mergeable before running apply. Logs Logs August 23, 2023 at 17:03 (UTC+1:00) {"level":"error","ts":"2023-08-23T16:03:28.369Z","caller":"events/instrumented_project_command_runner.go:84","msg":"Failure running apply operation: Pull request must be mergeable before running apply.","json":{"repo":"ume-platform-engineering/tf-env-data-engineering","pull":"189"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:84\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Apply\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:46\ngithub.com/runatlantis/atlantis/server/events.runProjectCmds\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:48\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:166\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:298 <http://github.com/runatlantis/atlantis/server/events.RunAndEmitStats%5Cn%5Ctgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:84%5Cngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Apply%5Cn%5Ctgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:46%5Cngithub.com/runatlantis/atlantis/server/events.runProjectCmds%5Cn%5Ctgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:48%5Cngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run%5Cn%5Ctgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:166%5Cngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand%5Cn%5Ctgithub.com/runatlantis/atlantis/server/events/command_runner.go:298>"} August 23, 2023 at 17:03 (UTC+1:00) {"level":"info","ts":"2023-08-23T16:03:29.799Z","caller":"events/automerger.go:20","msg":"not automerging because project at dir \"XXX\", workspace \"default\" has status \"apply_errored\"","json":{"repo":"XXX","pull":"189"}} August 23, 2023 at 17:03 (UTC+1:00) {"level":"error","ts":"2023-08-23T16:03:32.912Z","caller":"logging/simple_logger.go:163","msg":"invalid key: ***@***.******@***.******@***.******@***.******@***.******@***.******@***.******@***.***/negroni.go:111\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2936\nnet/http.(*conn).serve\n\tnet/http/server.go:1995 ***@***.******@***.******@***.******@***.******@***.******@***.******@***.******@***.***/negroni.go:111%5Cnnet/http.serverHandler.ServeHTTP%5Cn%5Ctnet/http/server.go:2936%5Cnnet/http.(*conn).serve%5Cn%5Ctnet/http/server.go:1995>"} Environment details - Atlantis version: v0.23.2 & v0.23.4 - Deployment method: ecs - If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: no - Gitlab 16.4.0 Atlantis server-side config file: { "taskDefinitionArn": "arn:aws:ecs:XXX:XXX", "containerDefinitions": [ { "name": "atlantis", "image": "XXX/atlantis:v0.24.3-0bcdeaa1", "cpu": 2048, "memory": 4096, "memoryReservation": 128, "portMappings": [ { "containerPort": 4141, "hostPort": 4141, "protocol": "tcp" } ], "essential": true, "environment": [ { "name": "ATLANTIS_ALLOW_REPO_CONFIG", "value": "false" }, { "name": "ATLANTIS_HIDE_PREV_PLAN_COMMENTS", "value": "false" }, { "name": "ATLANTIS_WRITE_GIT_CREDS", "value": "true" }, { "name": "ATLANTIS_SILENCE_NO_PROJECTS", "value": "false" }, { "name": "ATLANTIS_GITLAB_USER", "value": "XXX" }, { "name": "ATLANTIS_LOG_LEVEL", "value": "debug" }, { "name": "ATLANTIS_AUTOMERGE", "value": "true" }, { "name": "ATLANTIS_BITBUCKET_USER", "value": "" }, { "name": "ATLANTIS_REPO_CONFIG_JSON", "value": "{\"repos\":[{\"allow_custom_workflows\":true,\"allowed_overrides\":[\"workflow\"],\"apply_requirements\":[\"undiverged\",\"mergeable\",\"approved\"],\"id\":\"/.*/\",\"repo_config_file\":\"atlantis.yaml\"}],\"workflows\":{\"default\":{\"apply\":{\"steps\":[\"apply\",{\"run\":\"[ ! -z \\\"$PROJECT_NAME\\\" ] \\u0026\\u0026 export TAG=$PROJECT_NAME || export TAG=last-applied \\u0026\\u0026 git config --global user.name XXXn \\u0026\\u0026 git config --global user.email XXX \\u0026\\u0026 git fetch --tags -f \\u0026\\u0026 git fetch --all --tags \\u0026\\u0026 (git tag --delete $TAG || true) \\u0026\\u0026 git tag -a $TAG -m \\\"Tagged automatically by atlantis\\\" \\u0026\\u0026 (git push origin --delete $TAG || true) \\u0026\\u0026 git push origin $TAG\"}]},\"plan\":{\"steps\":[\"init\",\"plan\"]}}}}" }, { "name": "ATLANTIS_PARALLEL_POOL_SIZE", "value": "50" }, { "name": "ATLANTIS_REPO_ALLOWLIST", "value": "XXX" }, { "name": "ATLANTIS_GITLAB_HOSTNAME", "value": "gitlab.com" }, { "name": "ATLANTIS_DEFAULT_TF_VERSION", "value": "v0.13.4" }, { "name": "ATLANTIS_GH_APP_ID", "value": "" }, { "name": "ATLANTIS_BITBUCKET_BASE_URL", "value": "" }, { "name": "ATLANTIS_PORT", "value": "4141" }, { "name": "ATLANTIS_GH_USER", "value": "" }, { "name": "ATLANTIS_ATLANTIS_URL", "value": "https://XXX" } ], "mountPoints": [], "volumesFrom": [], "secrets": [XXX] Repo atlantis.yaml file: version: 3projects: - name: platform dir: XXX/platform autoplan: when_modified: [ "*.tf" ] enabled: true ... Additional Context Issue started happening on Atlantis v0.23.2. Still happening after upgrading to v0.23.4. - #3277 <#3277> — Reply to this email directly, view it on GitHub <#3722>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3ERG5OCLOBLNCOO4SZS3XYBO5HANCNFSM6AAAAAA4F2G4FE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[saraangelmurphy]

Hi, my organization is also experiencing this issue after upgrading to Gitlab v16.4.0. It affects the latest version of atlantis (v0.25.0), as well as version v0.24.2. The only workaround is to disable required approvals altogether in repos.yaml, which of course is not ideal.

Overview of the Issue

When executing atlantis apply in a comment for a gitlab merge request, atlantis fails with the message "Apply Failed: Pull request must be mergeable before running apply"

Reproduction Steps

1. Create a gitlab v16.4.0 instance with a webhook to communicate with atlantis
2. Create a merge request to a repository, with atlantis' repos.yaml file configured to require approvals
3. Obtain a merge request approval
4. Execute atlantis apply
5. atlantis fails with the message "Apply Failed: Pull request must be mergeable before running apply"

Logs

From systemctl status atlantis:

Sep 26 16:28:26 cit-prod-atlantis atlantis[13665]: {"level":"warn","ts":"2023-09-26T16:28:26.049Z","caller":"events/apply_command_runner.go:111","msg":"unable to get pull request status: fetching mergeability status for repo: tf-infrastructure-gcp/bv-infra, and pull number: 1026: GET https://gitlab.c.bvnt.co/api/v4/projects/274: 500 {message: 500 Internal Server Error}. Continuing with mergeable and approved assumed false","json":{"repo":"tf-infrastructure-gcp/bv-infra","pull":"1026"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\t/home/runner/work/atlantis/atlantis/server/events/apply_command_runner.go:111\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\t/home/runner/work/atlantis/atlantis/server/events/command_runner.go:298"}

Environment details

• Atlantis version: v0.25.0
• Deployment method: binary on EC2 AWS isntance
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: yes
• Atlantis flags: /home/atlantis/atlantis server --config /home/atlantis/atlantis.yaml

Atlantis server-side config file:

root@cit-prod-atlantis:/home/atlantis# cat atlantis.yaml
---
# Minimum required settings
atlantis-url: https://atlantis.c.bvnt.co:4141
gitlab-hostname: gitlab.c.bvnt.co
gitlab-token: '<SNIP>'
gitlab-user: atlantisbot
gitlab-webhook-secret: '<SNIP>'
repo-whitelist: 'gitlab.c.bvnt.co/*'
# Optional Settings
data-dir: /home/atlantis/.atlantis
default-tf-version: v0.11.15
port: 4141
repo-config: /home/atlantis/repos.yaml
ssl-cert-file: /home/atlantis/ssl/wc.c.bvnt.co.crt
ssl-key-file: /home/atlantis/ssl/wc.c.bvnt.co.key
# This setting means that atlantis will merge the destination branch into the source before running plans
checkout-strategy: merge
...

Repo atlantis.yaml file:

---
version: 3
projects:
- dir: prod/gcp-hashicorp-vault
  workflow: vault
  terraform_version: v1.5.5
  autoplan:
    when_modified: ["*.tf"]
.... # Repeated 3-400 times
.... # this also occurs in repos with only one or two atlantis "projects"

repos.yaml file

root@cit-prod-atlantis:/home/atlantis# cat repos.yaml
repos:
- id: /.*/
  apply_requirements: [approved] # This is the line we removed to work around this issue
  allowed_overrides: [workflow]

# Global workflows
workflows:
  default:
    plan:
      steps:
        - run: /home/atlantis/atlantis-verify.sh plan $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - init
        - plan
        - run: /home/atlantis/bv-conftest.sh
    apply:
      steps:
        - run: /home/atlantis/atlantis-verify.sh apply $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - apply
  vault:
    plan:
      steps:
        - env:
            name: VAULT_TOKEN
            command: /home/atlantis/vault-login.sh
        - run: /home/atlantis/atlantis-verify.sh plan $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - init:
            extra_args: ["-reconfigure"]
        - plan
        - run: /home/atlantis/bv-conftest.sh
    apply:
      steps:
        - env:
            name: VAULT_TOKEN
            command: /home/atlantis/vault-login.sh
        - run: /home/atlantis/atlantis-verify.sh apply $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - apply
  tfvars:
    plan:
      steps:
        - run: /home/atlantis/atlantis-verify.sh plan $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - run: /bin/cp "$DIR/_tfvars/$WORKSPACE.tfvars" "$DIR/temp-tfvars-file-copy.auto.tfvars"
        - init
        - plan
        - run: /home/atlantis/bv-conftest.sh
    apply:
      steps:
        - run: /home/atlantis/atlantis-verify.sh apply $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - apply
  tfvars_vault:
    plan:
      steps:
        - env:
            name: VAULT_TOKEN
            command: /home/atlantis/vault-login.sh
        - run: /home/atlantis/atlantis-verify.sh plan $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - run: /bin/cp "$DIR/_tfvars/$WORKSPACE.tfvars" "$DIR/temp-tfvars-file-copy.auto.tfvars"
        - init
        - plan
        - run: /home/atlantis/bv-conftest.sh
    apply:
      steps:
        - env:
            name: VAULT_TOKEN
            command: /home/atlantis/vault-login.sh
        - run: /home/atlantis/atlantis-verify.sh apply $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - apply
  global_sgrules:
    plan:
      steps:
        - env:
            name: VAULT_TOKEN
            command: /home/atlantis/vault-login.sh
        - run: /home/atlantis/atlantis-verify.sh plan $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - init
        - plan
    apply:
      steps:
        - env:
            name: VAULT_TOKEN
            command: /home/atlantis/vault-login.sh
        - run: /home/atlantis/atlantis-verify.sh apply $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - apply:
            extra_args: ['--parallelism=1']
  vaultforbluevoyantproduction:
    plan:
      steps:
        - env:
            name: VAULT_TOKEN
            command: /home/atlantis/vault-login.sh
        - run: /home/atlantis/atlantis-verify.sh plan $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - init:
            extra_args: ["-reconfigure"]
        - plan
        - run: /home/atlantis/bv-conftest.sh
    apply:
      steps:
        - env:
            name: VAULT_TOKEN
            command: /home/atlantis/vault-login.sh
        - run: /home/atlantis/atlantis-verify.sh apply $COMMENT_ARGS $BASE_REPO_OWNER $BASE_REPO_NAME
        - apply

Additional Context

#3277 seems related, but has not fixed the problem fully.

[syst0m]

@jamengual Any updates on this? Thanks!

I will try 0.25.0 and see.

[jamengual]

I do not use Gitlab sadly, never got to try this @lukemassa do you had this issue?

[lukemassa]

I don't, but if I had to guess I'd say it's related to the fact that gitlab's determination of whether an MR is "mergeable" is, as far as I understand it, asynchronous, so there might be times where if atlantis catches it at the wrong time it's still "figuring out" whether it's mergeable. Let me see if I can reproduce this on my setup and get back to you

[lukemassa]

So I took a look and I am unable to reproduce this on either 0.24.2, 0.25.0 or main. However, my company's gitlab is on v16.2.3-ee, whereas the reports says 16.4.0. Additionally, a line in the logs pasted above jumped out at me which is:

GET https://gitlab.c.bvnt.co/api/v4/projects/274: 500 {message: 500 Internal Server Error}

This makes me think there's a gitlab bug, not an atlantis bug.

@saraangelmurphy What happens when you go to https://gitlab.c.bvnt.co/api/v4/projects/274 in your browser? In my case the analogous URL gets me something that looks like:

{"id":17829,"description":null,"name":"Lmassa Test Atlantis" ...

Also for what it's worth I think this is the line that's failing: https://github.com/runatlantis/atlantis/blob/main/server/events/vcs/gitlab_client.go#L293

Otherwise, I'm happy to dig into this bug when my company upgrades their gitlab instance to 16.4.0.

[infernaltechnology]

Given the critical vulnerability in previous versions, I hope that you guys prioritize the upgrade

https://nvd.nist.gov/vuln/detail/CVE-2023-5009

[saraangelmurphy]

@lukemassa you are entirely correct, that URL does return a 500 error in gitlab 16.4.0. We've raised a ticket with them, but it does certainly appear to be a regression on their end if the projects API works fine on 16.2.X.

[jamengual]

ohhhh Gitlab.....is still better than Bitbucket

Let's see what they say in your ticket, if you can link it here @saraangelmurphy that will be useful.

[syst0m]

So I took a look and I am unable to reproduce this on either 0.24.2, 0.25.0 or main. However, my company's gitlab is on v16.2.3-ee, whereas the reports says 16.4.0. Additionally, a line in the logs pasted above jumped out at me which is:

GET https://gitlab.c.bvnt.co/api/v4/projects/274: 500 {message: 500 Internal Server Error}

This makes me think there's a gitlab bug, not an atlantis bug.

@saraangelmurphy What happens when you go to https://gitlab.c.bvnt.co/api/v4/projects/274 in your browser? In my case the analogous URL gets me something that looks like:

{"id":17829,"description":null,"name":"Lmassa Test Atlantis" ...

Also for what it's worth I think this is the line that's failing: https://github.com/runatlantis/atlantis/blob/main/server/events/vcs/gitlab_client.go#L293

Otherwise, I'm happy to dig into this bug when my company upgrades their gitlab instance to 16.4.0.

I don't see a 500 response in the logs, like @saraangelmurphy does.

[lukemassa]

I was able to to get my hands on a server with 16.4.0 and was unfortunately unable to reproduce either the original issue or the 500 issue. Especially with the original issue being intermittent makes it tricky. Any more logs or attempts to isolate would be helpful.

[saraangelmurphy]

So this ended up being an issue with an incomplete upgrade to gitlab, that was due to SQL migration scripts failing during the upgrade when applying constraints to gitlab project regex rules. Specifically, regexes needed to be <521 characters, we had several that were over, the presence of these project push rules during the upgrade broke the projects API.

once we removed the excessively lengthy push rule regexes and reran the migrations, the projects API started responding again and atlantis was happy.

https://gitlab.com/gitlab-org/gitlab/-/issues/426066#note_1575868016

I also want to applaud @lukemassa @jamengual , and the other maintainers of atlantis! While my thanks is poor reward compared to financial support, the alacrity of responses here is better than we get from many paid projects. Atlantis is a fantastic project, and we are profoundly grateful for everyone's hard work making this a production-ready service that solves many needs.

[jamengual]

I'm so glad to hear you solved the issue and that we were able to help but I have to say that without @lukemassa we could not have done it, we need more people like that in this project and the world too.

[lukemassa]

@saraangelmurphy that's great to hear!

@syst0m any chance that saraangelmurphy's was helpful in addressing your issue? If not let us know if you have any luck isolating the issue or finding a way to reproduce it consistently.