Skip to content

Commit b2d19ec

Browse files
xingyousongcopybara-github
xingyousong
authored and
copybara-github
committed
Documentation on using converters
PiperOrigin-RevId: 607360878
1 parent f3af582 commit b2d19ec

File tree

3 files changed

+359
-3
lines changed

3 files changed

+359
-3
lines changed

‎docs/guides/developer/designers.ipynb‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,7 @@
1010
"\n",
1111
"\n",
1212
"# Designers\n",
13-
"This documentation will allow a developer to use the Designer API for typical algorithm design.\n",
14-
"\n",
15-
"\n"
13+
"This documentation will allow a developer to use the Designer API for typical algorithm design."
1614
]
1715
},
1816
{

‎docs/guides/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ For Users
99

1010
user/running_vizier
1111
user/search_spaces
12+
user/converters
1213
Switching to Vertex <https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/vizier/conversions_vertex_vizier_and_open_source_vizier.ipynb>
1314
user/supported_algorithms
1415

‎docs/guides/user/converters.ipynb‎

Lines changed: 357 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,357 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"id": "c8cEHaZVJXsU"
7+
},
8+
"source": [
9+
"[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/converters.ipynb)\n",
10+
"\n",
11+
"# Converters\n",
12+
"This documentation demonstrates how to use converters for representing PyVizier objects as NumPy arrays and vice-versa."
13+
]
14+
},
15+
{
16+
"cell_type": "markdown",
17+
"metadata": {
18+
"id": "glNyultvKLVg"
19+
},
20+
"source": [
21+
"## Installation and reference imports"
22+
]
23+
},
24+
{
25+
"cell_type": "code",
26+
"execution_count": null,
27+
"metadata": {
28+
"id": "plR-NKqFJOma"
29+
},
30+
"outputs": [],
31+
"source": [
32+
"!pip install google-vizier"
33+
]
34+
},
35+
{
36+
"cell_type": "code",
37+
"execution_count": null,
38+
"metadata": {
39+
"id": "x1BGr_ZvKQoK"
40+
},
41+
"outputs": [],
42+
"source": [
43+
"from vizier import pyvizier as vz\n",
44+
"from vizier.pyvizier import converters"
45+
]
46+
},
47+
{
48+
"cell_type": "markdown",
49+
"metadata": {
50+
"id": "8Uhzgb5yYvkT"
51+
},
52+
"source": [
53+
"Suppose we had a problem statement and some trials associated to the study."
54+
]
55+
},
56+
{
57+
"cell_type": "code",
58+
"execution_count": null,
59+
"metadata": {
60+
"id": "bHm5beYNMXOF"
61+
},
62+
"outputs": [],
63+
"source": [
64+
"# Setup search space\n",
65+
"search_space = vz.SearchSpace()\n",
66+
"root = search_space.root\n",
67+
"root.add_float_param(name='double', min_value=0.0, max_value=1.0)\n",
68+
"root.add_int_param(name='int', min_value=1, max_value=10)\n",
69+
"root.add_discrete_param(name='discrete', feasible_values=[0.1, 0.3, 0.5])\n",
70+
"root.add_categorical_param(name='categorical', feasible_values=['a', 'b', 'c'])\n",
71+
"\n",
72+
"# Setup metric configurations\n",
73+
"m1 = vz.MetricInformation(name='m1', goal=vz.ObjectiveMetricGoal.MAXIMIZE)\n",
74+
"m2 = vz.MetricInformation(name='m2', goal=vz.ObjectiveMetricGoal.MINIMIZE)\n",
75+
"\n",
76+
"# Final problem\n",
77+
"problem = vz.ProblemStatement(search_space, metric_information=[m1, m2])\n",
78+
"\n",
79+
"# Example trials\n",
80+
"trial1 = vz.Trial(\n",
81+
" parameters={'double': 0.6, 'int': 2, 'discrete': 0.1, 'categorical': 'a'},\n",
82+
" final_measurement=vz.Measurement(metrics={'m1': 0.1, 'm2': 0.2}),\n",
83+
")\n",
84+
"trial2 = vz.Trial(\n",
85+
" parameters={'double': 0.1, 'int': 6, 'discrete': 0.3, 'categorical': 'b'},\n",
86+
" final_measurement=vz.Measurement(metrics={'m1': -1.0, 'm2': 0.8}),\n",
87+
")"
88+
]
89+
},
90+
{
91+
"cell_type": "markdown",
92+
"metadata": {
93+
"id": "-xcnx2LQKes_"
94+
},
95+
"source": [
96+
"## Quick Start\n",
97+
"To use numerical models, both our `x` (parameters) and `y` (metrics) need to be formatted as numpy arrays. We can directly do so with `TrialToArrayConverter`:"
98+
]
99+
},
100+
{
101+
"cell_type": "code",
102+
"execution_count": null,
103+
"metadata": {
104+
"id": "CONUnh92Yma3"
105+
},
106+
"outputs": [],
107+
"source": [
108+
"t2a_converter = converters.TrialToArrayConverter.from_study_config(problem)\n",
109+
"xs, ys = t2a_converter.to_xy([trial1, trial2])"
110+
]
111+
},
112+
{
113+
"cell_type": "markdown",
114+
"metadata": {
115+
"id": "IocslOsLa8_i"
116+
},
117+
"source": [
118+
"We can also convert the `xs` back into PyVizier `ParameterDict`s:"
119+
]
120+
},
121+
{
122+
"cell_type": "code",
123+
"execution_count": null,
124+
"metadata": {
125+
"id": "w4o0gM0KaPw3"
126+
},
127+
"outputs": [],
128+
"source": [
129+
"t2a_converter.to_parameters(xs)"
130+
]
131+
},
132+
{
133+
"cell_type": "markdown",
134+
"metadata": {
135+
"id": "MOXYX_oAYm29"
136+
},
137+
"source": [
138+
"Behind the scenes, the `TrialToArrayConverter` actually uses a `DefaultTrialConverter` which first converts both trial parameters and metrics into `dict[str, np.ndarray]` and then concatenates the arrays together."
139+
]
140+
},
141+
{
142+
"cell_type": "code",
143+
"execution_count": null,
144+
"metadata": {
145+
"id": "qiU_rMjfK45Q"
146+
},
147+
"outputs": [],
148+
"source": [
149+
"converter = converters.DefaultTrialConverter.from_study_config(problem)\n",
150+
"xs_dict, ys_dict = converter.to_xy([trial1, trial2])"
151+
]
152+
},
153+
{
154+
"cell_type": "markdown",
155+
"metadata": {
156+
"id": "UlT0t5KDi1OO"
157+
},
158+
"source": [
159+
"Trials can be recovered too:"
160+
]
161+
},
162+
{
163+
"cell_type": "code",
164+
"execution_count": null,
165+
"metadata": {
166+
"id": "wrgpY0ANi73O"
167+
},
168+
"outputs": [],
169+
"source": [
170+
"original_trials = converter.to_trials(xs_dict, ys_dict)"
171+
]
172+
},
173+
{
174+
"cell_type": "markdown",
175+
"metadata": {
176+
"id": "dyjiYSXmddoN"
177+
},
178+
"source": [
179+
"## Customization\n",
180+
"There are multiple ways to convert parameters of specific types. For example,\n",
181+
"some common methods to convert the `'categorical'` parameter (with feasible\n",
182+
"values `['a', 'b', 'c']`) can be:\n",
183+
"\n",
184+
"* Integer Index: `'b' -\u003e 1` since `b` has index 1 among feasible values.\n",
185+
"* One-Hot: `'b' -\u003e [0, 1, 0]` using one-hot encoding.\n",
186+
"\n",
187+
"Additional considerations can for example:\n",
188+
"\n",
189+
"* Whether to scale continuous parameter values into `[0,1]`\n",
190+
"* Whether to always sign-flip metrics to assume maximization only.\n",
191+
"\n",
192+
"These options can be specified when constructing both `TrialToArrayConverter` and `DefaultTrialConverter` ([source code](https://github.com/google/vizier/blob/main/vizier/pyvizier/converters/core.py)):\n",
193+
"\n",
194+
"```python\n",
195+
"@classmethod\n",
196+
"def from_study_config(\n",
197+
" cls,\n",
198+
" study_config: pyvizier.ProblemStatement,\n",
199+
" *,\n",
200+
" scale: bool = True,\n",
201+
" pad_oovs: bool = True,\n",
202+
" max_discrete_indices: int = 0,\n",
203+
" flip_sign_for_minimization_metrics: bool = True,\n",
204+
" dtype=np.float64,\n",
205+
"):\n",
206+
"```"
207+
]
208+
},
209+
{
210+
"cell_type": "markdown",
211+
"metadata": {
212+
"id": "eflsNETlg8VT"
213+
},
214+
"source": [
215+
"For more fine-grained control over specific `ParameterConfig`s and `MetricInformation`s, a user can specify individual arguments to each `DefaultModelInputConverter` and `DefaultModelOutputConverter` respectively."
216+
]
217+
},
218+
{
219+
"cell_type": "code",
220+
"execution_count": null,
221+
"metadata": {
222+
"id": "eDeFdd3Op3f3"
223+
},
224+
"outputs": [],
225+
"source": [
226+
"# Only considers the 'double' parameter values.\n",
227+
"double_pc = search_space.get('double')\n",
228+
"double_converter = converters.DefaultModelInputConverter(double_pc, scale=True)\n",
229+
"double_converter.convert([trial1, trial2])\n",
230+
"\n",
231+
"# Only considers the 'categorical' parameter values.\n",
232+
"categorical_pc = search_space.get('categorical')\n",
233+
"categorial_converter = converters.DefaultModelInputConverter(categorical_pc, onehot_embed=True)\n",
234+
"categorial_converter.convert([trial1, trial2])"
235+
]
236+
},
237+
{
238+
"cell_type": "code",
239+
"execution_count": null,
240+
"metadata": {
241+
"id": "qHbnUXeDqyhb"
242+
},
243+
"outputs": [],
244+
"source": [
245+
"# Only considers the 'm1' metric values.\n",
246+
"m1_converter = converters.DefaultModelOutputConverter(m1)\n",
247+
"m1_converter.convert([trial1.final_measurement, trial2.final_measurement])"
248+
]
249+
},
250+
{
251+
"cell_type": "markdown",
252+
"metadata": {
253+
"id": "SIkZVMAWtdBt"
254+
},
255+
"source": [
256+
"These can be inserted into the `DefaultTrialConverter`:"
257+
]
258+
},
259+
{
260+
"cell_type": "code",
261+
"execution_count": null,
262+
"metadata": {
263+
"id": "_RWxP0Cgthf6"
264+
},
265+
"outputs": [],
266+
"source": [
267+
"parameter_converters = [double_converter, categorial_converter]\n",
268+
"metric_converters = [m1_converter]\n",
269+
"\n",
270+
"custom_converter = converters.DefaultTrialConverter(parameter_converters, metric_converters)\n",
271+
"custom_converter.to_xy([trial1, trial2]) # Same array outputs as above."
272+
]
273+
},
274+
{
275+
"cell_type": "markdown",
276+
"metadata": {
277+
"id": "_r2RuW6oplD5"
278+
},
279+
"source": [
280+
"For full customization, the user may create their own `ModelInputConverter`s and `ModelOutputConverter`s.\n",
281+
"\n",
282+
"```python\n",
283+
"class ModelInputConverter(metaclass=abc.ABCMeta):\n",
284+
" \"\"\"Interface for extracting inputs to the model.\"\"\"\n",
285+
"\n",
286+
" @abc.abstractmethod\n",
287+
" def convert(self, trials: Sequence[vz.TrialSuggestion]) -\u003e np.ndarray:\n",
288+
" \"\"\"Returns an array of shape (number of trials, feature dimension).\"\"\"\n",
289+
"\n",
290+
" @property\n",
291+
" @abc.abstractmethod\n",
292+
" def output_spec(self) -\u003e NumpyArraySpec:\n",
293+
" \"\"\"Provides specification of the output from this converter.\"\"\"\n",
294+
"\n",
295+
" @property\n",
296+
" @abc.abstractmethod\n",
297+
" def parameter_config(self):\n",
298+
" \"\"\"Original ParameterConfig that this converter acts on.\"\"\"\n",
299+
"\n",
300+
" @abc.abstractmethod\n",
301+
" def to_parameter_values(\n",
302+
" self, array: np.ndarray\n",
303+
" ) -\u003e List[Optional[vz.ParameterValue]]:\n",
304+
" \"\"\"Convert and clip to the nearest feasible parameter values.\"\"\"\n",
305+
"```\n",
306+
"\n",
307+
"```python\n",
308+
"class ModelOutputConverter(metaclass=abc.ABCMeta):\n",
309+
" \"\"\"Metric converter interface.\"\"\"\n",
310+
"\n",
311+
" @abc.abstractmethod\n",
312+
" def convert(self, measurements: Sequence[vz.Measurement]) -\u003e np.ndarray:\n",
313+
" \"\"\"Returns N x 1 array.\"\"\"\n",
314+
" pass\n",
315+
"\n",
316+
" @abc.abstractmethod\n",
317+
" def to_metrics(self, labels: np.ndarray) -\u003e Sequence[Optional[vz.Metric]]:\n",
318+
" \"\"\"Returns a list of pyvizier metrics.\"\"\"\n",
319+
"\n",
320+
" @property\n",
321+
" @abc.abstractmethod\n",
322+
" def metric_information(self) -\u003e vz.MetricInformation:\n",
323+
" \"\"\"Describes the semantics of the return value from convert() method.\"\"\"\n",
324+
"\n",
325+
" @property\n",
326+
" def output_shape(self) -\u003e Tuple[None, int]:\n",
327+
" return (None, 1)\n",
328+
"```"
329+
]
330+
}
331+
],
332+
"metadata": {
333+
"colab": {
334+
"last_runtime": {
335+
"build_target": "//learning/deepmind/dm_python:dm_notebook3",
336+
"kind": "private"
337+
},
338+
"name": "Converters.ipynb",
339+
"private_outputs": true,
340+
"provenance": [
341+
{
342+
"file_id": "1nArqkCmNjB9-GwTg3nera6cQ9J4ws1FY",
343+
"timestamp": 1707957172845
344+
}
345+
]
346+
},
347+
"kernelspec": {
348+
"display_name": "Python 3",
349+
"name": "python3"
350+
},
351+
"language_info": {
352+
"name": "python"
353+
}
354+
},
355+
"nbformat": 4,
356+
"nbformat_minor": 0
357+
}

0 commit comments

Comments
 (0)