Skip to content

Commit f966a8f

Browse files
ryanaslettryanaslett
authored
Orka packer image updates (#4163)
* ansible: add new uncompromised smartos hosts * orka: update base image and image name for osx13 arm test image * orka: configure image to allow sudo without password prompting * orka: set system hostname * orka: allow pubkey authentication with ssh * orka: add pipx, orka-vm-tools, and tap2junit. Ensure arm homebrew can run older scripts that expect intel homebrew paths * orka: add ccache configuration * ansible: add host entry for node-www * orka: update README.md with updated documentation * orka: add orkaconnect.sh utility script * orka: add the packer templates for release and test * orka: ignore secrets * orka: add files for packer * orka: extraneous data in old file * orka: remove old hcl templates, update github workflow * orka: add dummy secret files for the workflow
1 parent 56676c0 commit f966a8f

15 files changed

+726
-772
lines changed

‎.github/workflows/orka-templates.yml‎

Lines changed: 24 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,28 @@ jobs:
2222
- name: Set up Packer
2323
uses: hashicorp/setup-packer@1aa358be5cf73883762b302a3a03abd66e75b232 #v3.1.0
2424

25+
- name: Create dummy Packer variables file
26+
working-directory: orka/templates
27+
run: |
28+
cat > variables.auto.pkrvars.hcl <<'EOF'
29+
orka_endpoint = "https://mock-orka-endpoint"
30+
xcode_version = "mock-xcode_version"
31+
macos_version = "mock-macos_version"
32+
orka_source_image = "mock-orka_source_image"
33+
ssh_new_password = "mock-ssh_new_password"
34+
ssh_image_password = "mock-ssh_image_password"
35+
ssh_new_public_key = "mock-new_public_key"
36+
EOF
37+
- name: Mock secret files for validate
38+
working-directory: orka/templates
39+
run: |
40+
mkdir -p files/secrets
41+
: > "files/secrets/Apple Developer ID Node.js Foundation.p12"
42+
: > "files/secrets/id_rsa"
43+
2544
- name: Initialize Packer
45+
env:
46+
ORKA_AUTH_TOKEN: 'mock-orka-auth-token'
2647
run: |
2748
for file in $(find . -name '*.pkr.hcl'); do
2849
echo "Initializing $file"
@@ -32,25 +53,10 @@ jobs:
3253

3354
- name: Validate Packer templates
3455
env:
35-
ORKA_ENDPOINT: 'https://mock-orka-endpoint'
3656
ORKA_AUTH_TOKEN: 'mock-orka-auth-token'
37-
SSH_DEFAULT_USERNAME: 'mock-ssh-default-username'
38-
SSH_DEFAULT_PASSWORD: 'mock-ssh-default-password'
39-
SSH_TEST_PASSWORD: 'mock-ssh-test-password'
40-
SSH_RELEASE_PASSWORD: 'mock-ssh-release-password'
41-
SSH_TEST_PUBLIC_KEY: 'mock-ssh-test-public-key'
42-
SSH_RELEASE_PUBLIC_KEY: 'mock-ssh-release-public-key'
57+
working-directory: orka/templates
4358
run: |
4459
for file in $(find . -name '*.pkr.hcl'); do
4560
echo "Validating $file"
46-
vars="-var orka_endpoint=$ORKA_ENDPOINT -var orka_auth_token=$ORKA_AUTH_TOKEN -var ssh_default_username=$SSH_DEFAULT_USERNAME"
47-
48-
if echo "$file" | grep -q "release"; then
49-
vars="$vars -var ssh_release_password=$SSH_RELEASE_PASSWORD -var ssh_release_public_key=$SSH_RELEASE_PUBLIC_KEY"
50-
elif echo "$file" | grep -q "test"; then
51-
vars="$vars -var ssh_test_password=$SSH_TEST_PASSWORD -var ssh_test_public_key=$SSH_TEST_PUBLIC_KEY -var ssh_default_password=$SSH_DEFAULT_PASSWORD"
52-
fi
53-
54-
packer validate $vars $file || exit 1
55-
done
56-
working-directory: orka/templates
61+
packer validate -var-file=variables.auto.pkrvars.hcl "$file" || exit 1
62+
done

‎.gitignore‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,6 @@ ansible/host_vars/*
1616
Pipfile.lock
1717

1818
# Orka secrets files including naming mutations
19-
orka/*/.env*
19+
orka/*/.env*
20+
/orka/templates/files/secrets/*
21+
/orka/templates/*.pkrvars.hcl

‎ansible/inventory.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,10 @@ hosts:
200200
ip: 172.16.9.3
201201
ansible_ssh_common_args: '-o ProxyCommand="ssh -i ~/.ssh/nodejs_build_test -W %h:%p [email protected]"'
202202
ansible_user: root
203+
smartos22-x64-3:
204+
ip: 172.16.9.3
205+
ansible_ssh_common_args: '-o ProxyCommand="ssh -i ~/.ssh/nodejs_build_test -W %h:%p [email protected]"'
206+
ansible_user: root
203207
smartos23-x64-4:
204208
ip: 172.16.9.3
205209
ansible_ssh_common_args: '-o ProxyCommand="ssh -i ~/.ssh/nodejs_build_test -W %h:%p [email protected]"'
@@ -208,6 +212,10 @@ hosts:
208212
ip: 172.16.9.3
209213
ansible_ssh_common_args: '-o ProxyCommand="ssh -i ~/.ssh/nodejs_build_test -W %h:%p [email protected]"'
210214
ansible_user: root
215+
smartos23-x64-6:
216+
ip: 172.16.9.3
217+
ansible_ssh_common_args: '-o ProxyCommand="ssh -i ~/.ssh/nodejs_build_test -W %h:%p [email protected]"'
218+
ansible_user: root
211219

212220

213221
- osuosl:

‎orka/templates/README.md‎

Lines changed: 48 additions & 150 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Once installed, you can verify the installation by running the following command
1010
packer --version
1111
```
1212

13-
While writing this document, the latest version of Packer is `1.11.2`.
13+
While writing this document, the latest version of Packer is `1.14.2`.
1414

1515
## Install dependencies
1616

@@ -22,42 +22,65 @@ packer init .
2222

2323
## Access the Orka environment
2424

25-
You need to connect to the Orka VPN. You can find the instructions in the secrets repository.
25+
1. You need to connect to the Orka VPN. You can find the instructions in the secrets repository. @TODO
26+
2. Authenticate the cluster with `orka3 login` -> this will give a url to access to login to macstadium. This login lasts for 3600s.
27+
3. Once logged into macstadium, you can `orka3 user get-token` to get a user token to do other things, like build images.
2628

27-
## Load the environment variables
29+
## Authenticate to ghcr.io
2830

29-
You need to load the environment variables:
31+
Some Macstadium base images are stored at ghcr.io (github's container registry). To allow packer to seamlessly pull
32+
those images, you must provide the orka3 cli with a github personal access token (PAT). See [here]https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic
33+
```
34+
orka3 regcred add https://ghcr.io --username GITHUB_USERNAME --password PAT_TOKEN
35+
```
3036

31-
1. Get the `.env` file from the secrets repository. You will find the instructions in the repository.
32-
2. Copy the `.env` file to this directory.
33-
3. Run the following command:
34-
```shell
35-
source .env
36-
```
37-
4. Verify that the environment variables are loaded by running the following command:
38-
```shell
39-
echo $ORKA_ENDPOINT
40-
echo $ORKA_AUTH_TOKEN
41-
echo $SSH_DEFAULT_USERNAME
42-
echo $SSH_DEFAULT_PASSWORD
43-
echo $SSH_TEST_PASSWORD
44-
echo $SSH_TEST_PUBLIC_KEY
45-
```
37+
## Load the build variables
38+
39+
You need to configure the variables.auto.pkrvars.hcl file
40+
41+
1. Get the `orka.variables.auto.pkrvars.hcl` file from the secrets repository. There is one in release, and one in test
42+
2. Copy the `orka.variables.auto.pkrvars.hcl` to `variables.auto.pkrvars.hcl` file to this directory.
43+
44+
## Load the file secrets
45+
46+
We need the private key for node-www for the release images, as well as the apple developer certificate for code signing.
47+
48+
1. Copy the `secrets/build/release/staging_id_rsa_private.key` to orka/templates/files/secrets/id_rsa
49+
2. Go to the `build/release` folder in the secrets repo
50+
3. Extract from secrets/build/release and put it in this repo (adjust the orka path in this command): `dotgpg cat Apple\ Developer\ ID\ Node.js\ Foundation.p12.base64 | base64 -D > orka/templates/files/secrets/Apple\ Developer\ ID\ Node.js\ Foundation.p12`
51+
52+
## Download Xcode to the shared vm storage
53+
54+
1. Full Xcode installation
55+
56+
Xcode Command-line tools are not enough to perform a full notarization cycle, full Xcode must be fully installed on the release images.
57+
58+
* Login to https://developer.apple.com using the [email protected] account
59+
* Download Xcode: https://developer.apple.com/download/more/ - find non-beta version, open Developer Tools in browser, Networking tab, start download (then cancel), in Networking tab "Copy as cURL" (available in Chrome & FF)
60+
* On OSX 15 we currently install 16.4
61+
* Manually launch one of the existing VM's (Arm ones are faster)
62+
* `orka3 images list` to see available images
63+
* `orka3 vm deploy --image IMAGE_NAME` to deploy a new image
64+
* NOTE: don't try to connect to an existing image as jenkins may delete it while you're working on it.
65+
* Connect to the VM with ssh, and navigate to /Volumes/orka/Xcode
66+
* use orkaconnect.sh VM_NAME (ie. `orkaconnect.sh vm-h1tcv`)
67+
* Execute the curl command to download Xcode, save it to a file named Xcode_{VERSION}.xip i.e. Xcode_16.4.xip
68+
* This is where packer will look when installing xcode in the image.
4669

4770
## Validate the template
4871

49-
You can validate a specific template by running the following command:
72+
You can validate a specific template by running the following command (replace test with release if doing release images)
5073

5174
```shell
52-
packer validate -var "orka_endpoint=$ORKA_ENDPOINT" -var "orka_auth_token=$ORKA_AUTH_TOKEN" -var "ssh_default_username=$SSH_DEFAULT_USERNAME" -var "ssh_default_password=$SSH_DEFAULT_PASSWORD" -var "ssh_test_password=$SSH_TEST_PASSWORD" -var "ssh_release_password=$SSH_RELEASE_PASSWORD" -var "ssh_release_public_key=$SSH_RELEASE_PUBLIC_KEY" -var "ssh_test_public_key=$SSH_TEST_PUBLIC_KEY" <template_name>
53-
```
75+
ORKA_AUTH_TOKEN=$(orka3 user get-token) packer validate -var-file=variables.auto.pkrvars.hcl macos-test.pkr.hcl
76+
```
5477

5578
## Build the image
5679

5780
You can build a specific template by running the following command:
5881

5982
```shell
60-
packer build -var "orka_endpoint=$ORKA_ENDPOINT" -var "orka_auth_token=$ORKA_AUTH_TOKEN" -var "ssh_default_username=$SSH_DEFAULT_USERNAME" -var "ssh_default_password=$SSH_DEFAULT_PASSWORD" -var "ssh_test_password=$SSH_TEST_PASSWORD" -var "ssh_release_password=$SSH_RELEASE_PASSWORD" -var "ssh_release_public_key=$SSH_RELEASE_PUBLIC_KEY" -var "ssh_test_public_key=$SSH_TEST_PUBLIC_KEY" <template_name>
83+
ORKA_AUTH_TOKEN=$(orka3 user get-token) packer build -var-file=variables.auto.pkrvars.hcl macos-test.pkr.hcl
6184
```
6285

6386
## Continuous Integration
@@ -70,6 +93,8 @@ We don't plan to build the images in the CI pipeline. The images are built manua
7093

7194
Orka provides a base image that we need to customize to our needs.
7295

96+
Note that orka3 remote-image command is only for interacting with x64 images. arm64 images are at ghcr.io: https://github.com/macstadium/orka-images
97+
7398
1. find the image that you want to extend by running the following command:
7499
```shell
75100
orka3 remote-image list
@@ -95,130 +120,3 @@ Orka provides a base image that we need to customize to our needs.
95120
orka3 vm delete <vm_name>
96121
```
97122
Note: Don't delete the vm until you have saved the image, check by running the command `orka3 image list`
98-
99-
100-
101-
### Manual Steps for all the images
102-
103-
1. Update Sudoers file:
104-
105-
this requires `NOPASSWD` to be added to the sudoers file to enable elevation
106-
107-
`sudo visudo`
108-
and change:
109-
`%admin ALL = (ALL) ALL`
110-
to
111-
`%admin ALL = (ALL) NOPASSWD:ALL`
112-
113-
2. Allow ssh access
114-
115-
```bash
116-
sudo systemsetup -setremotelogin on
117-
```
118-
3. Install xcode
119-
120-
```bash
121-
sudo xcode-select --install
122-
```
123-
124-
Do a an update using the UI. Check the available updates and install them (click in "more info"). Note that you don't want to update the OS, just the software.
125-
126-
### Manual Steps for the release images
127-
128-
1. Full Xcode installation
129-
130-
Xcode Command-line tools are not enough to perform a full notarization cycle, full Xcode must be installed manually.
131-
132-
As root:
133-
134-
* Download Xcode: https://developer.apple.com/download/more/ - find non-beta version, open Developer Tools in browser, Networking tab, start download (then cancel), in Networking tab "Copy as cURL" (available in Chrome & FF)
135-
* On OSX 13 we currently install 14.13.1.
136-
* Go to downloads folder, decompress the xip file (double click) and delete the xip file
137-
* Move the Xcode.app to /Applications
138-
* Open xcode, accept the license, install the built-in components and close xcode
139-
* `sudo xcode-select --switch /Applications/Xcode.app`
140-
* `sudo xcodebuild -license` - accept license
141-
* `git` - check that git is working (confirming license has been accepted)
142-
* Empty the trash
143-
144-
145-
2. OSX Keychain Profile
146-
147-
Unblok the keychain:
148-
149-
```bash
150-
security unlock-keychain -u /Library/Keychains/System.keychain
151-
```
152-
153-
Create a keychain profile (`NODE_RELEASE_PROFILE`) for the release machine:
154-
155-
```bash
156-
sudo xcrun notarytool store-credentials NODE_RELEASE_PROFILE \
157-
--apple-id XXXX \
158-
--team-id XXXX \
159-
--password XXXX \
160-
--keychain /Library/Keychains/System.keychain
161-
```
162-
163-
Note: `XXXX` values are found in `secrets/build/release/apple.md`
164-
165-
The expected output is:
166-
167-
```
168-
This process stores your credentials securely in the Keychain. You reference these credentials later using a profile name.
169-
170-
Validating your credentials...
171-
Success. Credentials validated.
172-
Credentials saved to Keychain.
173-
To use them, specify `--keychain-profile "NODE_RELEASE_PROFILE" --keychain /Library/Keychains/System.keychain`
174-
```
175-
176-
3. Signing certificates
177-
178-
* Go to the `build/release` folder in the secrets repo.
179-
* Extract from secrets/build/release: `dotgpg cat Apple\ Developer\ ID\ Node.js\ Foundation.p12.base64 | base64 -D > /tmp/Apple\ Developer\ ID\ Node.js\ Foundation.p12`
180-
* Transfer to release machine (scp to /tmp)
181-
* `sudo security import /tmp/Apple\ Developer\ ID\ Node.js\ Foundation.p12 -k /Library/Keychains/System.keychain -T /usr/bin/codesign -T /usr/bin/productsign -P 'XXXX'` (where XXXX is found in secrets/build/release/apple.md) (`security unlock-keychain -u /Library/Keychains/System.keychain` _may_ be required prior to running this command).
182-
183-
4. Validating certificates are in date and valid
184-
185-
1. `security -i unlock-keychain` Enter the password for the machine located in secrets
186-
2. `security find-certificate -c "Developer ID Application" -p > /tmp/app.cert` outputs the PEM format of the cert so we can properly inspect it
187-
3. `security find-certificate -c "Developer ID Installer" -p > /tmp/installer.cert`
188-
4. `openssl x509 -inform PEM -text -in /tmp/app.cert | less`
189-
5. `openssl x509 -inform PEM -text -in /tmp/installer.cert | less`
190-
6. `security find-identity -p codesigning -v`
191-
192-
The steps 4 and 5 will show the details of the certificates allowing to see expiry dates.
193-
194-
Example:
195-
196-
```
197-
Not Before: Jan 22 03:40:05 2020 GMT
198-
Not After : Jan 22 03:40:05 2025 GMT
199-
```
200-
201-
The step 6 will show the list of certificates available on the machine.
202-
203-
Example:
204-
205-
```
206-
1) XXXXXXXXXXX "Developer ID Application: Node.js Foundation (XXXXXXX)"
207-
1 valid identities found
208-
```
209-
210-
5. Change the default password
211-
212-
Use the password found in the secrets repository to change the default password:
213-
214-
```shell
215-
passwd
216-
```
217-
218-
Also change the keychain password:
219-
220-
```shell
221-
security set-keychain-password
222-
```
223-
224-
**:warning: IMPORTANT** We do this step manually at this point and not while using Packer because we added already sensitive information to the image.

‎orka/templates/files/admin_uid_change.sh‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/bin/bash
2+
source /Users/admin/.zprofile
3+
if [ $IMAGE_ARCHITECTURE = x64 ]; then
4+
echo "Re-numerating admin account. This takes a while to chmod"
5+
dscl . -change /Users/admin UniqueID 501 107
6+
time find /Users/admin -uid 501 -exec chown -h 107 {} \;
7+
dscl . create /Groups/ci
8+
dscl . create /Groups/ci gid 107
9+
dscl . create /Groups/ci passwd '*'
10+
dscl . create /Groups/ci GroupMembership admin
11+
echo "Finally done."
12+
fi

‎orka/templates/files/com.mount9p.plist‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
3+
<plist version="1.0">
4+
<dict>
5+
<key>Label</key>
6+
<string>com.mount9p.plist</string>
7+
<key>RunAtLoad</key>
8+
<true/>
9+
<key>StandardErrorPath</key>
10+
<string>/var/log/mount_9p_error.log</string>
11+
<key>StandardOutPath</key>
12+
<string>/var/log/mount_9p.log</string>
13+
<key>ProgramArguments</key>
14+
<array>
15+
<string>/bin/bash</string>
16+
<string>-c</string>
17+
<string>mkdir -p /Volumes/orka && mount_9p orka</string>
18+
</array>
19+
</dict>
20+
</plist>

‎orka/templates/files/ssh_config.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Host node-www
2+
HostName direct.nodejs.org
3+
User staging
4+
IdentityFile ~/.ssh/id_rsa

0 commit comments

Comments
 (0)