node_exporter 1.5.0 release SIGKILL immediately on macOS M1/arm64

[petemounce]

Host operating system: output of uname -a

$ uname -a
Darwin Peters-MBP-2.cust.communityfibre.co.uk 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000 arm64

node_exporter version: output of node_exporter --version

No output. SIGKILL. See below.

node_exporter command line flags

--help

node_exporter log output

No output. SIGKILL. See below.

Are you running node_exporter in Docker?

No.

What did you do that produced an error?

Ran the binary with --help

What did you expect to see?

The --help usage output.

What did you see instead?

The process was SIGKILL'd.

Full trace

pete at Peters-MBP-2 in ~/src/ef/_ (pete-node-exporter●)
$ uname -psm
Darwin arm64 arm

pete at Peters-MBP-2 in ~/src/ef/_ (pete-node-exporter●)
$ curl --location https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.darwin-arm64.tar.gz | tar xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 4411k  100 4411k    0     0  9061k      0 --:--:-- --:--:-- --:--:-- 15.0M

pete at Peters-MBP-2 in ~/src/ef/_ (pete-node-exporter●)
$ ./node_exporter-1.5.0.darwin-arm64/node_exporter --help
[1]    33812 killed     ./node_exporter-1.5.0.darwin-arm64/node_exporter --help

Other notes

I ran the same sequence on an AWS amd64 mac without error. This only happens on M1 macs (my laptop; AWS-hosted).

This happened for 1.4.0 as well but I saw the fresh release and figured I'd try it before reporting.

[SuperQ]

We haven't updated the osx build tools in a while. I guess we need to look into that.

[discordianfish]

Are you sure this isn't System Integrity Protection? I'm not a mac expert but that looks like when that hit me in the past.

[petemounce]

I'm not; how would I verify?

[rtuin]

I can confirm this issue, tried 1.5.0, 1.4.1 and 1.3.1. All the behaviour as described.
Perhaps this is specifically on macOS Ventura (released in Oct '22).

[petemounce]

(I'm on Monterey.)

[discordianfish]

Yeah likely code signing issue. I'm not a mac expert, dunno if its only on M1 or general depending on the macos version but unsigned binaries get sigkilled, e.g: nodejs/node#40827 (comment)

[pboiseau]

I'm having the same issue. Do you know a way to solve it ?

[discordianfish]

@pboiseau Did you try the link above?

[matthiasr]

Is this the same issue as #2217?

There is something node exporter specific going on here. Comparing with the statsd exporter (released both before and after node exporter 1.5.0):

❯ codesign -vvv node_exporter
node_exporter: valid on disk
node_exporter: satisfies its Designated Requirement
❯ codesign -vvv ~/Downloads/node_exporter-1.5.0.darwin-arm64/node_exporter
/Users/matthiasrampke/Downloads/node_exporter-1.5.0.darwin-arm64/node_exporter: invalid signature (code or signature have been modified)
In architecture: arm64
❯ codesign -vvv ~/Downloads/statsd_exporter-0.23.1.darwin-arm64/statsd_exporter
/Users/matthiasrampke/Downloads/statsd_exporter-0.23.1.darwin-arm64/statsd_exporter: valid on disk
/Users/matthiasrampke/Downloads/statsd_exporter-0.23.1.darwin-arm64/statsd_exporter: satisfies its Designated Requirement
❯ codesign -vvv ~/Downloads/statsd_exporter-0.22.8.darwin-arm64/statsd_exporter
/Users/matthiasrampke/Downloads/statsd_exporter-0.22.8.darwin-arm64/statsd_exporter: valid on disk
/Users/matthiasrampke/Downloads/statsd_exporter-0.22.8.darwin-arm64/statsd_exporter: satisfies its Designated Requirement

[pboiseau]

@discordianfish I've tried but I have this error

codesign -s - node_exporter
node_exporter: internal error in Code Signing subsystem

[discordianfish]

Unfortunately I have no idea how this mac code signing works. If someone has some suggestion what we can do better/different, let me know

[petemounce]

I'm only a little familiar; it's a PITA.

• https://federicoterzi.com/blog/automatic-code-signing-and-notarization-for-macos-apps-using-github-actions/ - for the Apple toolchain manner, which assumes running on Apple hardware (to get the toolchain)

• https://gregoryszorc.com/blog/2022/08/08/achieving-a-completely-open-source-implementation-of-apple-code-signing-and-notarization/ - this claims to support code-signing from non-Apple hardware, which would be compelling since Apple hardware costs more.

  https://gregoryszorc.com/docs/apple-codesign/main/ for end-user docs

In both cases, guidance is to be extremely cautious with one's Apple Developer certs. If they leak, other people can sign their own binaries with said certs leading to all sorts of mischief.

[pboiseau]

I found a solution.

I share with you an ansible script that I have done to sign the app on Apple M1 in order to prevent the process from being SIGKILL.

You need to have a Apple developer account and create a Developer ID Application certificate.

- name: Get developer ID application certificate
  ansible.builtin.shell: |
    echo {{ apple_developer_certificate }} > developerID_application.txt
    base64 -d -i developerID_application.txt -o developerID_application.cer
    rm developerID_application.txt

- name: Check if certificate already exist in keychain
  ansible.builtin.shell: |
    security find-certificate - c "Developer ID Application: <Name Of Your Certificate (XXXXXXXXX)>" -a -p /Library/Keychains/System.keychain
  failed_when: is_certificate.rc != 0 and is_certificate.rc != 44
  register: is_certificate

- name: Import developer ID application certificate
  ansible.builtin.shell: security import developerID_application.cer -k /Library/Keychains/System.keychain
  when: is_certificate.rc != 0

- name: Sign node_exporter binary
  become: true
  ansible.builtin.shell: "codesign -s - {{ node_exporter_binary_install_dir }}/node_exporter"

- name: Verify node_exporter signature
  ansible.builtin.shell: "codesign -vvvv {{ node_exporter_binary_install_dir }}/node_exporter"
  register: result
  failed_when: result.rc == 1

[discordianfish]

It would be nice if we could sign the releases as well. Dunno what that would involve, probably need to get some key signed by apple. If someone is familiar with the process, let me know! Even better if someone wants to submit a PR to add this to CI.

[flichtenheld]

I just ran into this problem as well when switching to macOS 13. One existing work-around is to use the homebrew version (https://formulae.brew.sh/formula/node_exporter#default) which is correctly signed.