docs: Tutorial: Go coverage profiles for e2e tests

docs-v2/content/en/docs/builders/builder-types/ko.md

@@ -471,6 +471,13 @@ please consider filing an
 [issue](https://github.com/GoogleContainerTools/skaffold/issues/new)
 that describes your use case.
 
+### Collecting coverage profiles from integration tests
+
+Go 1.20 introduced support for collecting coverage profile data from running Go
+application when running integration or end-to-end tests. To see how you can
+use the `ko` builder to configure this, see the tutorial
+[Go integration test coverage profiles]({{< relref "/docs/tutorials/go-integration-coverage" >}}).
+
 ### SBOM synthesis and upload
 
 The `ko` CLI by default generates a software bill of materials (SBOM) and

docs-v2/content/en/docs/tutorials/go-integration-coverage.md

@@ -0,0 +1,235 @@
+---
+title: "Go integration test coverage profiles"
+linkTitle: "Go integration test coverage profiles"
+weight: 100
+---
+
+This tutorial describes how to use Skaffold to collect
+[coverage profile data](https://go.dev/testing/coverage/)
+from Go applications when running
+[integration tests](https://go.dev/testing/coverage/#glos-integration-test).
+These more comprehensive tests, often called end-to-end tests, are run against
+a deployed application, typically testing multiple user journeys.
+
+## Background
+
+Go 1.20 introduced support for collecting coverage profile data from running Go
+applications. To enable coverage collection, build the binary with the `-cover`
+flag. The application records coverage profile data in a local directory set by
+the `GOCOVERDIR` environment variable.
+
+When the application runs on Kubernetes, there is an additional challenge of
+copying the coverage profile data files to permanent storage before the pod
+terminates.
+
+By default, the coverage profile data files are written on application exit.
+This tutorial shows how you can send a signal to write these files without
+exiting the application, and then copy the files out of the pods.
+
+## Steps
+
+Skaffold orchestrates the steps of:
+
+1.  Building binary and the container image, with support for collecting
+    coverage profiles.
+2.  Deploying the application to a Kubernetes cluster.
+3.  Running the integration tests.
+4.  Sending the signal to write coverage profile data files.
+5.  Collecting the counter-data files from the application pods.
+
+For steps 3-5, this tutorial uses Skaffold
+[lifecycle hooks]({{<relref "/docs/lifecycle-hooks" >}})
+to run these steps automatically.
+
+## The example application
+
+This tutorial refers to the files in the
+[`go-integration-coverage`](https://github.com/GoogleContainerTools/skaffold/tree/main/examples/go-integration-coverage)
+example.
+
+You may find it helpful to refer to these files as you go through this
+tutorial.
+
+## Sending signals for writing coverage profile data files
+
+By default, coverage profile data files are only written on application exit,
+specifically on return from `main.main()` or by calling `os.Exit()`. This is
+problematic in a Kubernetes pod, as the  application exit triggers pod
+termination.
+
+To work around this, add a signal handler to the application. This handler
+writes the coverage profile data files when it receives the configured signal,
+using the functions in the built-in
+[`coverage` package](https://pkg.go.dev/runtime/coverage).
+It also clears (resets) the counters, which can be useful if you want separate
+coverage profile reports for different sets of tests.
+
+The snippet below is a Go function that sets up a signal handler. It uses the
+[`SIGUSR1`](https://www.gnu.org/software/libc/manual/html_node/Miscellaneous-Signals.html)
+signal, but you can use another signal in your application.
+
+{{% highlight go "hl_lines=3 8 12-13" %}}
+// Note: This snippet omits error handling for brevity.
+func SetupCoverageSignalHandler() {
+	coverDir, exists := os.LookupEnv("GOCOVERDIR")
+	if !exists {
+		return
+	}
+	c := make(chan os.Signal)
+	signal.Notify(c, syscall.SIGUSR1)
+	go func() {
+		for {
+			<-c
+			coverage.WriteCountersDir(coverDir)
+			coverage.ClearCounters() // only works with -covermode=atomic
+        }
+	}()
+}
+{{% / highlight %}}
+
+You can call this function from your `main.main()` function to set up the
+signal handler early on in the application lifecycle.
+
+If the `GOCOVERDIR` environment variable is not set, the function returns
+without setting up the signal handler. This means that you can control enabling
+and disabling the signal handler by whether this environment variable is set.
+
+## Building the binary and the container image
+
+To build the container image with support for coverage profile collection,
+compile the binary with the `-cover` flag, and optionally also the `-covermode`
+flag.
+
+The image must contain the `tar` command to enable copying the counter-data
+files from the pod.
+
+The following snippet shows how to configure the image build using the Skaffold
+[ko builder]({{<relref "/docs/builders/builder-types/ko" >}}):
+
+{{% readfile file="samples/builders/ko-flags-cover.yaml" %}}
+
+Using other builders is also possible, by adding the flags to the `go build`
+command or by setting the `GOFLAGS` environment variable.
+
+## Running the integration tests
+
+The integration tests can be implemented in a number of ways, since they do not
+run in-process with the application.
+
+For instance, you can implement them using Go tests, a shell script with a
+sequence of `curl` commands against an HTTP server, or other integration and
+end-to-end test frameworks.
+
+Use Skaffold post-deploy hooks to run the tests automatically after deploying
+the application. These hooks can run either on the
+[`host`]({{<relref "/docs/lifecycle-hooks#before-deploy-and-after-deploy" >}})
+where you run Skaffold, or in the deployed
+[`container`]({{<relref "/docs/lifecycle-hooks#before-deploy-and-after-deploy-1" >}}).
+
+This tutorial uses a `host` hook that runs a shell script. The shell script
+sets up port-forwarding to the service and then runs the integration test. The
+arguments to the shell script are used to configure port forwarding.
+
+For this tutorial, the integration test is simply a `curl` command that sends a
+HTTP request to the application.
+
+{{% highlight yaml "hl_lines=4" %}}
+    hooks:
+      after:
+      - host:
+          command: ["./integration-test/run.sh", "service/go-integration-coverage", "default", "4503", "80"]
+          os: [darwin, linux]
+{{% / highlight %}}
+
+The arguments to the shell script are:
+
+1.  the Kubernetes resource to port-forward to, e.g., `service/myapp` or
+    `deployment/myapp` (required),
+2.  the namespace of the Kubernetes resource (defaults to `default`),
+3.  the local port (defaults to `4503`), and
+4.  the remote port (defaults to `8080`).
+
+After running the integration tests, a `container` hook sends `SIGUSR1` to the
+application process (PID 1) using the `kill` command:
+
+{{% highlight yaml "hl_lines=2" %}}
+      - container:
+          command: ["kill", "-USR1", "1"]
+          podName: go-integration-coverage-*
+          containerName: app
+{{% / highlight %}}
+
+The `podName` and `containerName` fields are required and must match the values
+from the Pod spec in your Kubernetes manifest.
+
+If you create multiple pods, the hook will run in all matching pods.
+
+## Copying coverage profile data files
+
+A `host` post-deploy hook runs a shell script that copies the counter-data
+files from the pods to the host where you run Skaffold:
+
+{{% highlight yaml "hl_lines=2" %}}
+      - host:
+          command: ["./integration-test/coverage.sh"]
+          os: [darwin, linux]
+{{% / highlight %}}
+
+First, the shell script below locates all pods deployed by the Skaffold run
+using a selector on the
+[`skaffold.dev/run-id` label]({{<relref "/docs/tutorials/skaffold-resource-selector" >}}).
+
+Next, the script iterates over the pods and uses `kubectl exec` to run `tar` in
+the containers to package up the counter-data files and pipe them to the host.
+On the other end of the pipe, `tar` extracts the files to a report directory on
+the host where you run Skaffold.
+
+Finally, the `go tool covdata` command reports the coverage as percentage on
+the terminal.
+
+Skaffold provides the
+[`SKAFFOLD_KUBE_CONTEXT` and `SKAFFOLD_RUN_ID` environment variables]({{<relref "/docs/lifecycle-hooks#environment-variables" >}})
+to the shell script.
+
+## Profiles
+
+The Go binary must be compiled with the `-cover` flag to collect coverage
+metrics. However, you may not want to use this flag when compiling for
+production use.
+
+Additionally, to simplify metrics reporting, you may want to only specify one
+replica in the Kubernetes Deployment resource.
+
+Skaffold [profiles](https://skaffold.dev/docs/environment/profiles/) enable
+different configurations for different contexts.
+
+The `skaffold.yaml` file for this tutorial contains a `coverage` profile that
+overrides the base configuration as follows:
+
+1.  Specify a base image that contains the `tar` command. `tar` is required to
+    copy the coverage profile data files from the pod.
+
+2.  Build the Go binary with the
+    [`-cover` and `-covermode` flags](https://go.dev/blog/cover).
+
+3.  Patch the Deployment resource to add a volume and volume mount to the pod
+    template spec for the coverage profile data files. This tutorial uses
+    [Kustomize](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/)
+    to patch the resource, but you can use another tool for this in your own
+    environment.
+
+4.  Add post-deploy hooks for running integration tests and collecting coverage
+    profile data.
+
+To activate the profile, add the flag `--profile coverage` to Skaffold
+commands.
+
+## Running the steps
+
+To run the steps, follow the instructions in the
+[README.md](https://github.com/GoogleContainerTools/skaffold/tree/main/examples/go-integration-coverage).
+
+## References
+
+- [Go: Coverage profiling support for integration tests](https://go.dev/testing/coverage/)
+- [`runtime/coverage` package in the Go standard library](https://pkg.go.dev/runtime/coverage)

docs-v2/content/en/samples/builders/ko-flags-cover.yaml

@@ -0,0 +1,6 @@
+build:
+  artifacts:
+  - image: foo
+    ko:
+      fromImage: gcr.io/distroless/base-debian11:debug
+      flags: ["-cover", "-covermode=atomic"]

integration/examples/go-integration-coverage/.gitignore

@@ -0,0 +1,16 @@
+# Copyright 2023 The Skaffold Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+.kpt-pipeline/
+reports/

integration/examples/go-integration-coverage/README.md

@@ -0,0 +1,49 @@
+# skaffold-go-integration-coverage
+
+Example showing how to use Skaffold and ko to collect coverage profiles from
+[integration tests](https://go.dev/testing/coverage/#glos-integration-test),
+often called end-to-end tests, for Kubernetes workloads written in Go.
+
+For a detailed explanation of how this example works, see the tutorial
+[Go integration test coverage profiles](https://skaffold.dev/docs/tutorials/go-integration-coverage/)
+on the Skaffold website.
+
+## Requirements
+
+- Go v1.20 or later
+- Skaffold v2 or later
+
+## Usage
+
+1.  If you are using a remote Kubernetes cluster, configure Skaffold to use
+    your image registry:
+
+    ```shell
+    export SKAFFOLD_DEFAULT_REPO=[your image registry, e.g., gcr.io/$PROJECT_ID]
+    ```
+
+    You can skip this step if you are using a local Kubernetes cluster such as
+    kind or minikube.
+
+2.  Build the container image with support for coverage profile collection,
+    deploy the Kubernetes resource, run the integration tests, and collect the
+    coverage profile data:
+
+    ```shell
+    skaffold run --profile=coverage
+    ```
+
+    The coverage profile data files will be in the directory `reports`.
+
+## Cleaning up
+
+When you are done, remove the resources from your cluster:
+
+```shell
+skaffold delete
+```
+
+## References
+
+- [Go: Coverage profiling support for integration tests](https://go.dev/testing/coverage/)
+- [`runtime/coverage` package in the Go standard library](https://pkg.go.dev/runtime/coverage)

integration/examples/go-integration-coverage/coverage.go

@@ -0,0 +1,74 @@
+// Copyright 2023 The Skaffold Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	"log"
+	"os"
+	"os/signal"
+	"runtime/coverage"
+	"syscall"
+)
+
+var onlyOneCoverageSignalHandler = make(chan struct{})
+
+// SetupCoverageSignalHandler creates a channel and relays the provided signals
+// to this channel. It also starts a goroutine that receives on that channel.
+// When the goroutine receives a signal, it writes profile data files to the
+// directory specified by the `GOCOVERDIR` environment variable. After writing
+// the data, the goroutine clears the coverage counters.
+//
+// Clearing the counters is only possible if the binary was built with
+// `-covermode=atomic`.
+//
+// If no signals are provided as arguments, the function defaults to relaying
+// `SIGUSR1`.
+//
+// If the `GOCOVERDIR` environment variable is _not_ set, this function does
+// nothing.
+//
+// References:
+// - https://go.dev/testing/coverage/
+// - https://pkg.go.dev/runtime/coverage
+func SetupCoverageSignalHandler(signals ...os.Signal) {
+	close(onlyOneCoverageSignalHandler) // panics when called twice
+
+	// Default to USR1 signal if no signals provided in the function argument.
+	if len(signals) < 1 {
+		signals = []os.Signal{syscall.SIGUSR1}
+	}
+
+	// Set up the signal handler only if GOCOVERDIR is set.
+	coverDir, exists := os.LookupEnv("GOCOVERDIR")
+	if !exists {
+		return
+	}
+
+	log.Printf("Configuring coverage profile data signal handler, listening for %v", signals)
+	c := make(chan os.Signal)
+	signal.Notify(c, signals...)
+	go func() {
+		for {
+			signal := <-c
+			log.Printf("Got %v, writing coverage profile data files to %q", signal, coverDir)
+			if err := coverage.WriteCountersDir(coverDir); err != nil {
+				log.Printf("Could not write coverage profile data files to the directory %q: %+v", coverDir, err)
+			}
+			if err := coverage.ClearCounters(); err != nil {
+				log.Printf("Could not reset coverage counter variables: %+v", err)
+			}
+		}
+	}()
+}