"Can't add a job to a build that is already closed."

[thibaultcha]

Run coverallsapp/github-action@master
Using lcov file: ./lcov.info
Error: Bad response: 422 {"message":"Can't add a job to a build that is already closed. Build 5217978935 is closed. See docs.coveralls.io/parallel-builds","error":true}

What does this mean? All I did was restart a GitHub Action run because some other jobs accidentally failed. Now all jobs pass but for some obscure reason, coveralls refuses the coverage data. Why? It's never, ever been a problem before to restart jobs, coveralls just typically absorb the new data into a new job. So what triggers this?

[afinetooth]

Hi @thibaultcha, we made a change on Apr 27 that prevents adding new coverage reports to closed builds. Practically speaking, this means that, if you have a parallel build with several jobs, let's say 3, you must send all 3 coverage reports (jobs) before sending the request to the (close) parallel build webhook, telling Coveralls to close the build.

If you happen to make that request to the parallel build webhook to close the build before another job is sent, or, say, between jobs 2 and 3.

This has cropped up for some users who had their CI's configured in a way that they were (inadvertently) closing the build in each of their parallel jobs.

Can you help me understand your use case here? Any insight into why your build might have become closed before you send an (additional) job with another coverage report?

One quick fix here might be upgrading to the latest version of the Github Action. v2+ is a major update, and uses an entirely new integration under the hood, our Universal Coverage Reporter, but it should work fine with your setup as far as I can tell.

[afinetooth]

P.S. @thibaultcha, it looks like that build above was deleted, so I can't find your project from that. If you'd like to share the Coveralls URL for your project, I'll be happy to look into it more closely. If your project is private, or sensitive, feel free to email us at support@coveralls.io and just mention this issue.

[thibaultcha]

Can you help me understand your use case here? Any insight into why your build might have become closed before you send an (additional) job with another coverage report?

As far as I can remember I think I just restarted a GHA build because of some other failures (not in unit tests). Our unit tests then re-computed coverage data and sent it back to coveralls.

The project is private indeed, we might have to get in touch with support. I just experienced another issue by having:

1. 6/7 unit test jobs succeed
2. Restarted the failing unit job (which re-computed coverage), and which failed at the "finished" step:

Sending to coveralls {
  repo_token: '***',
  carryforward: 'unit-ngx_1.25.0-wasmer-ssl-debug-no_hup,unit-ngx_1.25.0-wasmer-ssl-no_debug-no_hup,unit-ngx_1.25.0-wasmtime-ssl-debug-hup,unit-ngx_1.25.0-v8-ssl-debug-no_hup,unit-openresty_1.21.4.1-wasmtime-ssl-debug-no_hup,unit-ngx_1.21.6-wasmer-ssl-debug-no_hup,unit-ngx_1.25.0-wasmer-no_ssl-no_debug-no_hup',
  repo_name: 'Kong/ngx_wasm_module',
  payload: { build_num: '5250257722', status: 'done' }
}
/home/runner/work/_actions/coverallsapp/github-action/master/dist/index.js:40831
                        throw new Error('Parallel webhook error:' + err + ', ' + JSON.stringify(data));
                        ^

Error: Parallel webhook error:Error: {"status":500,"error":"Internal Server Error"}, {"status":500,"error":"Internal Server Error"}
    at Request._callback (/home/runner/work/_actions/coverallsapp/github-action/master/dist/index.js:40831:31)
    at Request.self.callback (/home/runner/work/_actions/coverallsapp/github-action/master/dist/index.js:24734:22)
    at Request.emit (node:events:527:28)
    at Request.<anonymous> (/home/runner/work/_actions/coverallsapp/github-action/master/dist/index.js:25703:10)
    at Request.emit (node:events:527:28)
    at IncomingMessage.<anonymous> (/home/runner/work/_actions/coverallsapp/github-action/master/dist/index.js:25625:12)
    at Object.onceWrapper (node:events:641:28)
    at IncomingMessage.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

Our finished step looks like so:

  finish:
    needs: tests
    runs-on: ubuntu-latest
    steps:
      - name: Coveralls Finished
        uses: coverallsapp/github-action@master
        with:
          github-token: ${{ secrets.github_token }}
          parallel-finished: true
          carryforward: "unit-ngx_1.25.0-wasmer-ssl-debug-no_hup,unit-ngx_1.25.0-wasmer-ssl-no_debug-no_hup,unit-ngx_1.25.0-wasmtime-ssl-debug-hup,unit-ngx_1.25.0-v8-ssl-debug-no_hup,unit-openresty_1.21.4.1-wasmtime-ssl-debug-no_hup,unit-ngx_1.21.6-wasmer-ssl-debug-no_hup,unit-ngx_1.25.0-wasmer-no_ssl-no_debug-no_hup"

[afinetooth]

Ok, thanks. Looking into it here.

[afinetooth]

Hi @thibaultcha. We think the more recent issue was infrastructure-related and has been fixed as described here.

Please let me know if that's not the case for you. I checked the private accounts associated with your Coveralls user account and see builds from today for each, so I'm hopeful we've addressed the issue.

Update

Still checking into the original issue for you. Trying to determine whether this is the correct behavior for a re-run from CI. Does not seem like it, but will feedback with further insights / updates.

[afinetooth]

Hi @thibaultcha.

tl;dr:
An update and a workaround.

Details:
An update on the original issue:

Your issue has prompted an internal discussion about whether, and how, we could walk back this "closed" state of a parallel build. To clarify, the expected behavior is, indeed, currently, what you experienced. If the build was closed, you will not be able to submit additional jobs to it, or re-run it in CI and have that generate a new build.

We think the latter case (re-running the build in CI) should probably be allowed, but we are trying to determine what information would allow us to safely know we can replace the existing build (with identical sha) and whether we need to hold it until the new build completes, and roll back if it does not, etc. Concerns like that.

So, I'm afraid there will be some more time required for the internal discussion and any changes that come out of it. But I will keep you updated here on that.

In the meantime, there is a workaround here, which is simply to delete the old build, at which point you should be able to re-run the CI build without issue.

Here's where to delete a build in the UI. Example:
https://coveralls.io/builds/33054356

Let me know if you have any issues with that.

[thibaultcha]

Thanks for the update! I guess I don't really understand why this changed? In the past, we've been able to restart as many builds as we want and just get a new coverage stats from our coverage tools (whether it be coveralls or codecov). Can this not be done anymore? The reason why builds in the same PR are restarted is because once we get coverage data from our first CI run, typically our workflow involves adding new tests to increase coverage and force-push the branch to the PR to get a new coverage status, and iterate until all new lines in the PR are covered as desired. Is this workflow incompatible with Coveralls? Or do we now need to manually delete the CI build to run the PR's CI again?

[thibaultcha]

Well, I just tried it on a branch I have been working on and got the HTTP 422 again, so it seems to be the case that this changed recently because until now, we never had an issue restarting builds :/

[afinetooth]

Hi @thibaultcha. Without getting into the minutiae, this change relates to addressing a different set of issues, and is also part of a recent refactor, which now just means that we can't quickly re-wind the functionality to support CI re-runs at this moment.

That said, we are actively discussing this and I am trying to determine an ETA on that that I can share with you ASAP.

In the meantime, I have a doubt that I'd like to share and get some more clarification on your use case, because, aside from the workaround I offered, which, I admit, is inefficient, I'm thinking about a more standard use case we see where users iterate on PRs:

So the "doubt": I don't understand why iterating on your PRs would require you to re-run old CI builds. I say this because the use case I'm familiar with, for doing that, is to configure CI to build pull_request commits on your PR branches (branches that target your default branch). What happens then, during iteration, in steps, is as follows:

1. User commits some (first) changes to a (new) PR branch - If CI is configured to build push builds everywhere, then CI builds that commit, and send Coveralls a coverage report for the first commit on "New PR Branch". If CI is only configured to build push commits on the default branch, then no build and no coverage report for this commit.
2. User opens a PR on the new PR branch - If CI is configured to build pull_request commits on PR branches (recommended), then CI generates a new pull_request build, which sends a coverage report to Coveralls, and we generate one of our pull_request builds (new build).
3. User makes changes to PR - Based on feedback, including coverage reports, user makes changes to the PR in the form of one or more push commits to "New PR Branch" and, again, depending on CI config, CI will, or won't, build those commits and send coverage reports to Coveralls. (Generally speaking, for cleanliness and simplicity, we recommend not building push commits on your PR branches, just pull_request commits.)
4. New pull_request builds are generated by CI - So, if CI is configured to build pull_request commits on your PR branches, then a new pull_request build will be generated after each change (push commit) to "New PR Branch." These (new) pull_request builds send their coverage reports to Coverall, which generates new Coveralls builds for each CI build.
5. And so on...

So, following the model above, it's my feeling that you should be able to iterate on PRs by generating new pull_request builds along the way, without needing to rebuild any old ones.

Thanks in advance for helping me understand your use case.

[nebolsin]

@thibaultcha here's a workaround which should help with the issue you're experiencing. Add the following block to your Github Actions workflow (I recommend placing it at top level right above the jobs: section, so it applies to both normal coverage upload and build finalisation job):

env:
  COVERALLS_SERVICE_NUMBER: ${{ github.run_id }}-${{ github.run_attempt }}

I would also suggest upgrading Coveralls Action to v2. You're currently using master, which is v1 (stable).

This should unblock your workflow while we're analyzing the use-case and working on long-term solution.

[afinetooth]

Thanks @nebolsin. @thibaultcha, pls let us know how it goes.

[thibaultcha]

Thank you both for the help and suggestions! My whole team is out until Tuesday but I will try it out, maybe sooner if I get a chance to.