Skip to content

Commit c22d478

Browse files
author
Felipe Sodré
authored
feat(rosetta): Local cluster setup for Rosetta (dfinity#3485)
This PR introduces a script that manages a local kubernetes set up that runs Rosetta (icp and icrc) with prod images as well as locally built images. The cluster also provides a Grafana instance connected to the rosetta nodes that allows us to easily track system utilization (CPU, Memory, IO) as well as other custom metrics. The intent is to have a quick way to investigate issues and visualize the impact of local changes against the prod versions. See the README.md file for more details.
1 parent b2888f2 commit c22d478

File tree

13 files changed

+2060
-0
lines changed

13 files changed

+2060
-0
lines changed

‎rs/rosetta-api/icrc1/BUILD.bazel

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ load("@rules_pkg//:pkg.bzl", "pkg_tar")
66
load("@rules_pkg//pkg:mappings.bzl", "pkg_attributes", "pkg_mkdirs")
77
load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_library", "rust_test")
88
load("//bazel:defs.bzl", "rust_test_suite_with_extra_srcs")
9+
load("//rs/tests:system_tests.bzl", "oci_tar")
910

1011
package(default_visibility = ["//visibility:public"])
1112

@@ -277,6 +278,12 @@ expand_template(
277278
],
278279
)
279280

281+
oci_tar(
282+
name = "icrc_rosetta_image.tar",
283+
image = ":ic_icrc_rosetta_image",
284+
repo_tags = ["icrc-rosetta:image"],
285+
)
286+
280287
# Push image with:
281288
# $ bazel run //<path>:push_ic_icrc_rosetta_image
282289
#

‎rs/rosetta-api/local/cluster/.helmignore

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Patterns to ignore when building packages.
2+
# This supports shell glob matching, relative path matching, and
3+
# negation (prefixed with !). Only one pattern per line.
4+
.DS_Store
5+
# Common VCS dirs
6+
.git/
7+
.gitignore
8+
.bzr/
9+
.bzrignore
10+
.hg/
11+
.hgignore
12+
.svn/
13+
# Common backup files
14+
*.swp
15+
*.bak
16+
*.tmp
17+
*.orig
18+
*~
19+
# Various IDEs
20+
.project
21+
.idea/
22+
*.tmproj
23+
.vscode/

‎rs/rosetta-api/local/cluster/Chart.yaml

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
apiVersion: v2
2+
name: cluster
3+
description: A Helm chart for Kubernetes
4+
5+
# A chart can be either an 'application' or a 'library' chart.
6+
#
7+
# Application charts are a collection of templates that can be packaged into versioned archives
8+
# to be deployed.
9+
#
10+
# Library charts provide useful utilities or functions for the chart developer. They're included as
11+
# a dependency of application charts to inject those utilities and functions into the rendering
12+
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
13+
type: application
14+
15+
# This is the chart version. This version number should be incremented each time you make changes
16+
# to the chart and its templates, including the app version.
17+
# Versions are expected to follow Semantic Versioning (https://semver.org/)
18+
version: 0.1.0
19+
20+
# This is the version number of the application being deployed. This version number should be
21+
# incremented each time you make changes to the application. Versions are not expected to
22+
# follow Semantic Versioning. They should reflect the version the application is using.
23+
# It is recommended to use it with quotes.
24+
appVersion: "1.31.0"

‎rs/rosetta-api/local/cluster/README.md

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# Local Rosetta API Cluster
2+
3+
This directory contains scripts and configurations for setting up a local Kubernetes cluster using Minikube to deploy the Rosetta API services. The cluster includes monitoring tools such as Prometheus, cAdvisor, and Grafana.
4+
5+
## Purpose
6+
7+
The purpose of this cluster is to provide a local development and testing environment for the Rosetta API services. It allows developers to deploy and test their services in a controlled environment with monitoring capabilities.
8+
9+
## What It Does
10+
11+
- Sets up a Minikube cluster with a specified profile.
12+
- Installs Prometheus, cAdvisor, and Grafana for monitoring.
13+
- Deploys the Rosetta API services using Helm charts.
14+
- Optionally loads local Docker images for the services.
15+
- Forwards ports for Prometheus, Grafana, and the Rosetta API services.
16+
17+
## Tools
18+
19+
### Minikube
20+
21+
Minikube is a tool that runs a single-node Kubernetes cluster on your local machine. It is useful for development and testing purposes.
22+
23+
### Prometheus
24+
25+
Prometheus is an open-source monitoring and alerting toolkit. It is used to collect and store metrics from various sources, including applications and infrastructure.
26+
27+
### cAdvisor
28+
29+
cAdvisor (Container Advisor) is an open-source container resource usage and performance analysis agent. It provides insights into the resource usage and performance characteristics of running containers.
30+
31+
### Grafana
32+
33+
Grafana is an open-source platform for monitoring and observability. It provides a web-based interface for visualizing and analyzing metrics collected by Prometheus and other data sources.
34+
35+
## Usage
36+
37+
### Prerequisites
38+
39+
The `deploy.sh` script eventually uses Docker, Minikube, Kubectl and Helm, but fear not, it will assist you in installing those if it detects they are absent.
40+
It was only tested on Ubuntu servers.
41+
42+
WARNING: The script *doesn't* work when run from this repository's dev container (at `./ci/container/`).
43+
44+
## Deploying Prod Images
45+
46+
To create and set-up the local cluster and install the production containers for ICP-Rosetta and ICRC1-Rosetta pointing at DFINITY's test ledgers, simply do:
47+
```bash
48+
./deploy.sh [options]
49+
```
50+
51+
You can make the rosetta nodes point to other ledgers by using these flags:
52+
- `--icp-ledger <ledger_id>`: Set the ICP Ledger ID (default: `xafvr-biaaa-aaaai-aql5q-cai`). If `prod`, will point to the official ICP ledger.
53+
- `--icp-symbol <symbol>`: Set the ICP token symbol (default: `TESTICP`).
54+
- `--icrc1-ledger <ledger_id>`: Set the ICRC1 Ledger ID (default: `3jkp5-oyaaa-aaaaj-azwqa-cai`).
55+
56+
ATTENTION: The first run might take a few minutes to finish as it'll create the cluster and install the necessary charts in it. After that, all the script will do is re-deploy the rosetta images with different configuration if needed.
57+
58+
## Deploying Local Images
59+
60+
### Build the local containers
61+
In order to build rosetta containers with local changes, you need to do it from inside the dev container:
62+
63+
```bash
64+
$ ./ci/container/container-run.sh
65+
66+
# Build the TAR file target
67+
$ bazel build //rs/rosetta-api/icp:rosetta_image.tar
68+
69+
# Move the resulting TAR file to a place that can be accessed outside the dev container
70+
$ mv bazel-bin/rs/rosetta-api/icp/rosetta_image.tar /tmp
71+
72+
# Same for ICRC1
73+
$ bazel build //rs/rosetta-api/icrc1:icrc_rosetta_image.tar
74+
$ mv bazel-bin/rs/rosetta-api/icrc1/icrc_rosetta_image.tar /tmp
75+
76+
# Exit the dev container
77+
$ exit
78+
```
79+
80+
### Deploy the local containers
81+
82+
You can use the following flags to deploy additional containers with the TAR files generated above:
83+
84+
- `--local-icp-image-tar <path>`: Path to local ICP image tar file.
85+
- `--local-icrc1-image-tar <path>`: Path to local ICRC1 image tar file.
86+
87+
Example that deploys local versions for both:
88+
89+
```bash
90+
./deploy.sh --local-icp-image-tar /tmp/rosetta_image.tar --local-icrc1-image-tar /tmp/icrc_rosetta_image.tar
91+
```
92+
93+
The services and pods deployed with those images will have a `-local` in their names.
94+
95+
96+
### Cleaning up
97+
98+
You can add the `--clean` flag to any usage of `./deploy.sh`. That will wipe out the current cluster and install it from scratch.
99+
100+
For example, the following command installs all prod and local images in a clean cluster:
101+
102+
```bash
103+
./deploy.sh --local-icp-image-tar /tmp/rosetta_image.tar --local-icrc1-image-tar /tmp/icrc_rosetta_image.tar --clean
104+
```
105+
106+
## Monitoring with Grafana
107+
108+
Grafana will run on port 3000. If you're running this in a remote devenv, you'll need to forward your local machine port to your devenv's one in order to access the service from your browser.
109+
110+
The first time you open `http://localhost:3000`, you'll be asked for login credentials. Use `admin` for both username and password. You'll be asked to change the password, you can either do so or just skip, it doesn't matter.
111+
112+
Once in Grafana, import a new dashboard. As an option to import, you'll see a text box to input a json file. Copy and paste the contents of the `rosetta_load_dashboard.json` file in this directory.
113+
114+
Services and pods with suffix `-latest` represent jobs running with the prod images while the ones with suffix `-local` are the ones running with the locally built ones.
115+
116+
117+
## Notes
118+
- The script will automatically install Minikube if they are not found.
119+
- The script uses a dedicated Minikube profile (`local-rosetta`) to avoid conflicts with other Minikube clusters.

0 commit comments

Comments
 (0)