Skip to content
This repository was archived by the owner on Apr 2, 2024. It is now read-only.

Commit 4c666a6

Browse files
ceviancevian
committed
Add functions for selecting labels based on filters
Added functions and operators for working with selectors and filtering on labels. More documentation in sql_schema.md changes in this commit.
1 parent 0d6ffa1 commit 4c666a6

File tree

5 files changed

+442
-15
lines changed

5 files changed

+442
-15
lines changed

‎pkg/docs/sql_schema.md

Lines changed: 93 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,11 @@ We define several views to make working with prometheus data easier.
88

99
Metric views allows access to the full time-series prometheus data for a
1010
given metric. By default, these views are found in the `prom_metric` schema.
11-
Each metric has a view named after the metric name (.e.g. the `cpu_usage`
12-
metric would have a `prom_metric.cpu_usage` view). The view contains a the
13-
following column:
11+
The `prom_metric` view is added to your search_path when you first install
12+
timescale_prometheus and so it is default view you see if you don't
13+
schema-qualify a view name. Each metric has a view named after the metric
14+
name (.e.g. the `cpu_usage` metric would have a `prom_metric.cpu_usage` or
15+
simply `cpu_usage` view). The view contains a the following column:
1416

1517
- time - The timestamp of the measurement
1618
- value - The value of the measurement
@@ -20,7 +22,7 @@ following column:
2022

2123
For example:
2224
```
23-
# \d+ prom_metric.cpu_usage
25+
# \d+ cpu_usage
2426
View "prom_metric.cpu_usage"
2527
Column | Type | Collation | Nullable | Default | Storage | Description
2628
--------------+--------------------------+-----------+----------+---------+----------+-------------
@@ -36,7 +38,7 @@ Example query for single point with their labels:
3638
SELECT
3739
label_array_to_jsonb(labels) as labels,
3840
value
39-
FROM prom_metric.cpu_usage
41+
FROM cpu_usage
4042
WHERE time < now();
4143

4244
```
@@ -53,7 +55,7 @@ Example query for a rollup:
5355
SELECT
5456
get_label_value(node_id) as node,
5557
avg(value)
56-
FROM prom_metric.cpu_usage
58+
FROM cpu_usage
5759
WHERE time < now()
5860
GROUP BY node_id
5961

@@ -95,4 +97,88 @@ Example query to look at all the series:
9597
-----------+---------+------------+-------
9698
4 | {3,4} | dev | pinky
9799
5 | {5,6} | production | brain
98-
```
100+
```
101+
102+
## Series Selectors
103+
104+
We have added simple-to-use series selectors for filtering series in either of the two views above.
105+
106+
### Containment
107+
108+
To test whether a series contains a json fragment you can use the `@>` containment operator.
109+
110+
For example to find all metrics on in the dev namespace and on node pinky, run:
111+
112+
```SQL
113+
SELECT *
114+
FROM prom_series.cpu_usage u
115+
WHERE labels @> jsonb '{"namespace":"dev", "node": "pinky"'}
116+
```
117+
118+
```
119+
series_id | labels | namespace | node | region
120+
-----------+---------+------------+-------+--------
121+
4 | {3,4,5} | dev | pinky | East
122+
5 | {3,4,5} | dev | pinky | West
123+
```
124+
125+
### Label Matchers
126+
127+
You can also match series using label matchers to create predicates on values of particular
128+
label keys. This is very similar to the label matchers available in PromQL.
129+
130+
For example to find all metrics on in the dev namespace and on node brain using
131+
label matcher, you can run:
132+
133+
```SQL
134+
SELECT *
135+
FROM cpu_usage u
136+
WHERE labels ? ('namepace' == 'dev') AND labels ? ('node' == 'brain')
137+
```
138+
139+
Label matchers are formed by using a qualifier of the form `labels ? (<tag_key> <operator> <pattern>)`.
140+
There are four operators,
141+
142+
- `==` match tag values that are equal to the pattern
143+
- `!==` match tag value that are not equal to the pattern
144+
- `==~` match tag values that match the pattern as a regex
145+
- `!=~` match tag values that are not equal to the pattern
146+
147+
These four matchers correspond to each of the four selectors in PromQL but with slightly
148+
different names (to avoid clashing with other PostgreSQL operators). They can
149+
be combined together using any boolean logic with any arbitrary where clauses.
150+
151+
For those coming from PromQL there are a few differences to keep in mind:
152+
- Regexes are not anchored for you. Although, you can of course add anchors (`^$`) yourself.
153+
- The logic for whether series that are missing the tag key pass the qualifier is slightly different:
154+
If the key on the left-hand side is not found `!==` and `!=~` always match, while `==` and `==~` never match.
155+
156+
157+
### Equivalence
158+
159+
The `eq` function tests exact equivalence between labels, without comparing the metric name (`__name__`) label key.
160+
For instance if the labels `a` is `{"__name__":"metric", "foo":"bar", "baz":"frob"}`
161+
then `SELECT eq(a, jsonb {"__name__":"something else", "foo":"bar", "baz":"frob"})` will evaluate to `true`, however, unlike `@>` if `SELECT eq(a, {"__name__":"metric", "foo":"bar"})` will evaluate to `false`.
162+
Thus, it can be used to compare across metrics or within a metric.
163+
164+
For example, to join 2 series that are scraped at the same time:
165+
166+
```SQL
167+
SELECT *
168+
FROM cpu_usage u
169+
INNER JOIN cpu_total t ON (u.time=t.time AND eq(u.labels, t.labels))
170+
WHERE u.labels ? ('namespace' == 'dev') AND u.labels ? ('node' ==~ 'pin*')
171+
```
172+
173+
You can also use eq to compare to a json labels object:
174+
175+
```SQL
176+
SELECT *
177+
FROM cpu_usage
178+
WHERE eq(labels , jsonb '{“namespace”:”prod”,”node”:”pinky”,"zone":"us-a1-east"}')
179+
```
180+
181+
Note the eq function tests equivalence of the entire label object.
182+
Therefore you need to provide the entire json object if using the
183+
function above. For partial matches see the Containment
184+
section above.

‎pkg/internal/testhelpers/containers_test.go

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import (
1111
"io/ioutil"
1212
"os"
1313
"path/filepath"
14+
"runtime"
1415
"testing"
1516

1617
"github.com/jackc/pgx/v4"
@@ -66,7 +67,14 @@ func TestMain(m *testing.M) {
6667
fmt.Println("Error setting up container", err)
6768
os.Exit(1)
6869
}
69-
path, err := ioutil.TempDir("", "prom_test")
70+
71+
tmpDir := ""
72+
if runtime.GOOS == "darwin" {
73+
// Docker on Mac lacks access to default os tmp dir - "/var/folders/random_number"
74+
// so switch to cross-user tmp dir
75+
tmpDir = "/tmp"
76+
}
77+
path, err := ioutil.TempDir(tmpDir, "prom_test")
7078
if err != nil {
7179
fmt.Println("Error getting temp dir for Prometheus storage", err)
7280
os.Exit(1)

‎pkg/pgmodel/migrations/migration_files_generated.go

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/pgmodel/migrations/sql/1_base_schema.up.sql

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -974,3 +974,180 @@ BEGIN
974974
END
975975
$func$
976976
LANGUAGE PLPGSQL VOLATILE;
977+
978+
----------------------------------
979+
-- Label selectors and matchers --
980+
----------------------------------
981+
982+
CREATE DOMAIN SCHEMA_PROM.matcher_positive AS int[] NOT NULL;
983+
CREATE DOMAIN SCHEMA_PROM.matcher_negative AS int[] NOT NULL;
984+
CREATE DOMAIN SCHEMA_PROM.label_key AS TEXT NOT NULL;
985+
CREATE DOMAIN SCHEMA_PROM.pattern AS TEXT NOT NULL;
986+
987+
CREATE OR REPLACE FUNCTION SCHEMA_CATALOG.count_jsonb_keys(j jsonb)
988+
RETURNS INT
989+
AS $func$
990+
SELECT count(*)::int from (SELECT jsonb_object_keys(j)) v;
991+
$func$
992+
LANGUAGE SQL IMMUTABLE PARALLEL SAFE;
993+
994+
CREATE OR REPLACE FUNCTION SCHEMA_PROM.get_matcher_from_json(labels jsonb)
995+
RETURNS SCHEMA_PROM.matcher_positive
996+
AS $func$
997+
SELECT ARRAY(
998+
SELECT coalesce(l.id, -1) -- -1 indicates no such label
999+
FROM SCHEMA_CATALOG.label_jsonb_each_text(labels-'__name__') e
1000+
LEFT JOIN SCHEMA_CATALOG.label l
1001+
ON (l.key = e.key AND l.value = e.value)
1002+
)::SCHEMA_PROM.matcher_positive
1003+
$func$
1004+
LANGUAGE SQL STABLE PARALLEL SAFE;
1005+
COMMENT ON FUNCTION SCHEMA_PROM.get_matcher_from_json(jsonb)
1006+
IS 'returns an array of label ids for the JSONB. This is not a labels array since the order of ids isnt guaranteed. __name__ is ignored.';
1007+
1008+
1009+
1010+
---------------- eq functions ------------------
1011+
1012+
CREATE OR REPLACE FUNCTION SCHEMA_PROM.eq(labels1 SCHEMA_PROM.label_ids, labels2 SCHEMA_PROM.label_ids)
1013+
RETURNS BOOLEAN
1014+
AS $func$
1015+
--assumes labels have metric name in position 1 and have no duplicate entries
1016+
SELECT array_length(labels1, 1) = array_length(labels2, 1) AND labels1 @> labels2[2:]
1017+
$func$
1018+
LANGUAGE SQL IMMUTABLE PARALLEL SAFE;
1019+
COMMENT ON FUNCTION SCHEMA_PROM.eq(SCHEMA_PROM.label_ids, SCHEMA_PROM.label_ids)
1020+
IS 'returns true if two label arrays are equal, ignoring the metric name';
1021+
1022+
1023+
CREATE OR REPLACE FUNCTION SCHEMA_PROM.eq(labels1 SCHEMA_PROM.label_ids, matchers SCHEMA_PROM.matcher_positive)
1024+
RETURNS BOOLEAN
1025+
AS $func$
1026+
--assumes no duplicate entries
1027+
SELECT array_length(labels1, 1) = (array_length(matchers, 1) + 1)
1028+
AND labels1 @> matchers
1029+
$func$
1030+
LANGUAGE SQL IMMUTABLE PARALLEL SAFE;
1031+
COMMENT ON FUNCTION SCHEMA_PROM.eq(SCHEMA_PROM.label_ids, SCHEMA_PROM.matcher_positive)
1032+
IS 'returns true if the label array and matchers are equal, there should not be a matcher for the metric name';
1033+
1034+
1035+
CREATE OR REPLACE FUNCTION SCHEMA_PROM.eq(labels SCHEMA_PROM.label_ids, json_labels jsonb)
1036+
RETURNS BOOLEAN
1037+
AS $func$
1038+
--assumes no duplicate entries
1039+
--do not call eq(label_ids, matchers) to allow inlining
1040+
SELECT array_length(labels, 1) = (SCHEMA_CATALOG.count_jsonb_keys(json_labels-'__name__') + 1)
1041+
AND labels @> SCHEMA_PROM.get_matcher_from_json(json_labels)
1042+
$func$
1043+
LANGUAGE SQL STABLE PARALLEL SAFE;
1044+
COMMENT ON FUNCTION SCHEMA_PROM.eq(SCHEMA_PROM.label_ids, jsonb)
1045+
IS 'returns true if the labels and jsonb are equal, ignoring the metric name';
1046+
1047+
1048+
--------------------- op @> ------------------------
1049+
1050+
CREATE OR REPLACE FUNCTION SCHEMA_CATALOG.label_contains(labels SCHEMA_PROM.label_ids, json_labels jsonb)
1051+
RETURNS BOOLEAN
1052+
AS $func$
1053+
SELECT labels @> SCHEMA_PROM.get_matcher_from_json(json_labels)
1054+
$func$
1055+
LANGUAGE SQL STABLE PARALLEL SAFE;
1056+
1057+
CREATE OPERATOR SCHEMA_PROM.@> (
1058+
LEFTARG = SCHEMA_PROM.label_ids,
1059+
RIGHTARG = jsonb,
1060+
FUNCTION = SCHEMA_CATALOG.label_contains
1061+
);
1062+
1063+
--------------------- op ? ------------------------
1064+
1065+
CREATE OR REPLACE FUNCTION SCHEMA_CATALOG.label_match(labels SCHEMA_PROM.label_ids, matchers SCHEMA_PROM.matcher_positive)
1066+
RETURNS BOOLEAN
1067+
AS $func$
1068+
SELECT labels && matchers
1069+
$func$
1070+
LANGUAGE SQL IMMUTABLE PARALLEL SAFE;
1071+
1072+
CREATE OPERATOR SCHEMA_PROM.? (
1073+
LEFTARG = SCHEMA_PROM.label_ids,
1074+
RIGHTARG = SCHEMA_PROM.matcher_positive,
1075+
FUNCTION = SCHEMA_CATALOG.label_match
1076+
);
1077+
1078+
CREATE OR REPLACE FUNCTION SCHEMA_CATALOG.label_match(labels SCHEMA_PROM.label_ids, matchers SCHEMA_PROM.matcher_negative)
1079+
RETURNS BOOLEAN
1080+
AS $func$
1081+
SELECT NOT (labels && matchers)
1082+
$func$
1083+
LANGUAGE SQL IMMUTABLE PARALLEL SAFE;
1084+
1085+
CREATE OPERATOR SCHEMA_PROM.? (
1086+
LEFTARG = SCHEMA_PROM.label_ids,
1087+
RIGHTARG = SCHEMA_PROM.matcher_negative,
1088+
FUNCTION = SCHEMA_CATALOG.label_match
1089+
);
1090+
1091+
--------------------- op == !== ==~ !=~ ------------------------
1092+
1093+
CREATE OR REPLACE FUNCTION SCHEMA_CATALOG.label_find_key_equal(label_key SCHEMA_PROM.label_key, pattern SCHEMA_PROM.pattern)
1094+
RETURNS SCHEMA_PROM.matcher_positive
1095+
AS $func$
1096+
SELECT COALESCE(array_agg(l.id), array[]::int[])::SCHEMA_PROM.matcher_positive
1097+
FROM _prom_catalog.label l
1098+
WHERE l.key = label_key and l.value = pattern
1099+
$func$
1100+
LANGUAGE SQL STABLE PARALLEL SAFE;
1101+
1102+
CREATE OR REPLACE FUNCTION SCHEMA_CATALOG.label_find_key_not_equal(label_key SCHEMA_PROM.label_key, pattern SCHEMA_PROM.pattern)
1103+
RETURNS SCHEMA_PROM.matcher_negative
1104+
AS $func$
1105+
SELECT COALESCE(array_agg(l.id), array[]::int[])::SCHEMA_PROM.matcher_negative
1106+
FROM _prom_catalog.label l
1107+
WHERE l.key = label_key and l.value = pattern
1108+
$func$
1109+
LANGUAGE SQL STABLE PARALLEL SAFE;
1110+
1111+
CREATE OR REPLACE FUNCTION SCHEMA_CATALOG.label_find_key_regex(label_key SCHEMA_PROM.label_key, pattern SCHEMA_PROM.pattern)
1112+
RETURNS SCHEMA_PROM.matcher_positive
1113+
AS $func$
1114+
SELECT COALESCE(array_agg(l.id), array[]::int[])::SCHEMA_PROM.matcher_positive
1115+
FROM _prom_catalog.label l
1116+
WHERE l.key = label_key and l.value ~ pattern
1117+
$func$
1118+
LANGUAGE SQL STABLE PARALLEL SAFE;
1119+
1120+
CREATE OR REPLACE FUNCTION SCHEMA_CATALOG.label_find_key_not_regex(label_key SCHEMA_PROM.label_key, pattern SCHEMA_PROM.pattern)
1121+
RETURNS SCHEMA_PROM.matcher_negative
1122+
AS $func$
1123+
SELECT COALESCE(array_agg(l.id), array[]::int[])::SCHEMA_PROM.matcher_negative
1124+
FROM _prom_catalog.label l
1125+
WHERE l.key = label_key and l.value ~ pattern
1126+
$func$
1127+
LANGUAGE SQL STABLE PARALLEL SAFE;
1128+
1129+
1130+
CREATE OPERATOR SCHEMA_PROM.== (
1131+
LEFTARG = SCHEMA_PROM.label_key,
1132+
RIGHTARG = SCHEMA_PROM.pattern,
1133+
FUNCTION = SCHEMA_CATALOG.label_find_key_equal
1134+
);
1135+
1136+
CREATE OPERATOR SCHEMA_PROM.!== (
1137+
LEFTARG = SCHEMA_PROM.label_key,
1138+
RIGHTARG = SCHEMA_PROM.pattern,
1139+
FUNCTION = SCHEMA_CATALOG.label_find_key_not_equal
1140+
);
1141+
1142+
CREATE OPERATOR SCHEMA_PROM.==~ (
1143+
LEFTARG = SCHEMA_PROM.label_key,
1144+
RIGHTARG = SCHEMA_PROM.pattern,
1145+
FUNCTION = SCHEMA_CATALOG.label_find_key_regex
1146+
);
1147+
1148+
CREATE OPERATOR SCHEMA_PROM.!=~ (
1149+
LEFTARG = SCHEMA_PROM.label_key,
1150+
RIGHTARG = SCHEMA_PROM.pattern,
1151+
FUNCTION = SCHEMA_CATALOG.label_find_key_not_regex
1152+
);
1153+

0 commit comments

Comments
 (0)