Skip to content

Commit fc93540

Browse files
vincentsaragovincentsarago
committed
add benchmark
1 parent 891cf23 commit fc93540

File tree

7 files changed

+254
-0
lines changed

7 files changed

+254
-0
lines changed

‎.github/workflows/ci.yml

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,41 @@ jobs:
5252
name: Python${{ matrix.python-version }}
5353
fail_ci_if_error: false
5454

55+
benchmark:
56+
needs: [tests]
57+
runs-on: ubuntu-latest
58+
steps:
59+
- uses: actions/checkout@v4
60+
- name: Set up Python
61+
uses: actions/setup-python@v5
62+
with:
63+
python-version: '3.12'
64+
65+
- name: Install dependencies
66+
run: |
67+
python -m pip install --upgrade pip
68+
python -m pip install -e ".[benchmark]"
69+
70+
- name: Run Benchmark
71+
run: |
72+
python -m pytest tests/benchmarks.py -k morecantile --benchmark-only --benchmark-columns 'min, max, mean, median' --benchmark-sort 'min' --benchmark-json output.json
73+
74+
- name: Store and Compare benchmark result
75+
uses: benchmark-action/github-action-benchmark@v1
76+
with:
77+
name: morecantile Benchmarks
78+
tool: 'pytest'
79+
output-file-path: output.json
80+
alert-threshold: '150%'
81+
comment-on-alert: true
82+
fail-on-alert: false
83+
# GitHub API token to make a commit comment
84+
github-token: ${{ secrets.GITHUB_TOKEN }}
85+
gh-pages-branch: 'gh-benchmarks'
86+
# Make a commit on `gh-pages` only if main
87+
auto-push: ${{ github.ref == 'refs/heads/main' }}
88+
benchmark-data-dir-path: dev/benchmarks
89+
5590
publish:
5691
needs: [tests]
5792
runs-on: ubuntu-latest

‎CONTRIBUTING.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,13 @@ This repo is set to use `pre-commit` to run *isort*, *flake8*, *pydocstring*, *b
2424
$ pre-commit install
2525
```
2626

27+
##### Performance tests
28+
29+
```sh
30+
python -m pip install -e ".[benchmark]"
31+
python -m pytest tests/benchmarks.py --benchmark-only --benchmark-columns 'min, max, mean, median' --benchmark-sort 'min'
32+
```
33+
2734
### Docs
2835

2936
```bash

‎benchmarks/README.md

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
2+
## Benchmark
3+
4+
Compare `mercantile`, `utiles` and `morecantile`
5+
6+
```sh
7+
python -m pip install -r requirements.txt
8+
python -m pytest benchmarks.py --benchmark-only --benchmark-columns 'min, max, mean, median' --benchmark-sort 'min'
9+
```
10+
11+
```
12+
----------------------------------------------------- benchmark 'bounds': 21 tests ----------------------------------------------------
13+
Name (time in ns) Min Max Mean Median
14+
---------------------------------------------------------------------------------------------------------------------------------------
15+
test_bounds[utiles-(486, 332, 30)] 83.0000 (1.0) 52,000.0000 (52.97) 195.3371 (1.48) 208.0000 (1.60)
16+
test_bounds[utiles-(0, 0, 0)] 83.0000 (1.00) 31,167.0000 (31.75) 193.5957 (1.47) 208.0000 (1.60)
17+
test_bounds[utiles-(1, 0, 1)] 125.0000 (1.51) 981.6700 (1.0) 132.0177 (1.0) 130.8300 (1.01)
18+
test_bounds[utiles-(1, 40, 7)] 125.0000 (1.51) 8,606.2500 (8.77) 132.5298 (1.00) 130.0000 (1.0)
19+
test_bounds[utiles-(486, 332, 10)] 128.3300 (1.55) 1,600.8300 (1.63) 133.4046 (1.01) 131.6700 (1.01)
20+
test_bounds[utiles-(1, 1, 1)] 129.5900 (1.56) 16,846.2500 (17.16) 141.9846 (1.08) 134.1700 (1.03)
21+
test_bounds[utiles-(486, 332, 20)] 138.3300 (1.67) 1,123.3300 (1.14) 142.5420 (1.08) 141.2500 (1.09)
22+
test_bounds[mercantile-(1, 40, 7)] 1,000.0000 (12.05) 92,416.0000 (94.14) 1,193.0112 (9.04) 1,167.0000 (8.98)
23+
test_bounds[mercantile-(1, 1, 1)] 1,075.0000 (12.95) 48,066.8000 (48.96) 1,154.8100 (8.75) 1,141.6000 (8.78)
24+
test_bounds[mercantile-(0, 0, 0)] 1,083.0000 (13.05) 21,041.0000 (21.43) 1,214.6049 (9.20) 1,208.0000 (9.29)
25+
test_bounds[mercantile-(1, 0, 1)] 1,083.0000 (13.05) 75,291.0000 (76.70) 1,213.7837 (9.19) 1,208.0000 (9.29)
26+
test_bounds[mercantile-(486, 332, 10)] 1,083.0000 (13.05) 55,750.0000 (56.79) 1,223.7598 (9.27) 1,208.0000 (9.29)
27+
test_bounds[mercantile-(486, 332, 20)] 1,083.0000 (13.05) 89,041.0000 (90.70) 1,248.4256 (9.46) 1,209.0000 (9.30)
28+
test_bounds[mercantile-(486, 332, 30)] 1,083.0000 (13.05) 194,958.0000 (198.60) 1,239.4339 (9.39) 1,209.0000 (9.30)
29+
test_bounds[morecantile-(1, 0, 1)] 20,083.0000 (241.96) 132,625.0000 (135.10) 20,893.1235 (158.26) 20,667.0000 (158.98)
30+
test_bounds[morecantile-(0, 0, 0)] 20,166.0000 (242.96) 120,333.0000 (122.58) 20,974.1961 (158.87) 20,708.0000 (159.29)
31+
test_bounds[morecantile-(1, 1, 1)] 20,334.0000 (244.99) 60,958.0000 (62.10) 21,236.2953 (160.86) 20,750.0000 (159.62)
32+
test_bounds[morecantile-(1, 40, 7)] 21,667.0000 (261.05) 173,458.0000 (176.70) 22,622.6537 (171.36) 22,333.0000 (171.79)
33+
test_bounds[morecantile-(486, 332, 10)] 22,166.0000 (267.06) 205,125.0000 (208.96) 23,512.1607 (178.10) 22,875.0000 (175.96)
34+
test_bounds[morecantile-(486, 332, 20)] 24,708.0000 (297.69) 129,250.0000 (131.66) 25,775.0796 (195.24) 25,375.0000 (195.19)
35+
test_bounds[morecantile-(486, 332, 30)] 71,292.0000 (858.94) 205,625.0000 (209.46) 73,569.4565 (557.27) 72,667.0000 (558.98)
36+
---------------------------------------------------------------------------------------------------------------------------------------
37+
38+
---------------------------------------------------- benchmark 'xy_bounds': 21 tests -----------------------------------------------------
39+
Name (time in ns) Min Max Mean Median
40+
------------------------------------------------------------------------------------------------------------------------------------------
41+
test_xy_bounds[utiles-(0, 0, 0)] 83.0000 (1.0) 57,375.0000 (112.55) 145.9127 (1.57) 125.0000 (1.36)
42+
test_xy_bounds[utiles-(1, 1, 1)] 88.3400 (1.06) 1,335.4200 (2.62) 93.1478 (1.00) 92.5000 (1.00)
43+
test_xy_bounds[utiles-(1, 40, 7)] 88.7500 (1.07) 1,451.2500 (2.85) 93.1258 (1.0) 92.0900 (1.0)
44+
test_xy_bounds[utiles-(1, 0, 1)] 89.5900 (1.08) 861.6600 (1.69) 93.6428 (1.01) 93.3300 (1.01)
45+
test_xy_bounds[utiles-(486, 332, 10)] 90.0000 (1.08) 509.7950 (1.0) 93.2818 (1.00) 92.9150 (1.01)
46+
test_xy_bounds[utiles-(486, 332, 20)] 90.4100 (1.09) 864.1700 (1.70) 94.2974 (1.01) 94.1600 (1.02)
47+
test_xy_bounds[utiles-(486, 332, 30)] 90.4100 (1.09) 1,377.5000 (2.70) 94.7366 (1.02) 94.1600 (1.02)
48+
test_xy_bounds[mercantile-(0, 0, 0)] 708.0000 (8.53) 42,250.0000 (82.88) 843.2292 (9.05) 833.0000 (9.05)
49+
test_xy_bounds[mercantile-(486, 332, 10)] 731.2500 (8.81) 5,393.7500 (10.58) 770.8331 (8.28) 764.6000 (8.30)
50+
test_xy_bounds[mercantile-(486, 332, 30)] 731.2500 (8.81) 7,316.6500 (14.35) 770.2423 (8.27) 764.6000 (8.30)
51+
test_xy_bounds[mercantile-(1, 0, 1)] 732.1429 (8.82) 9,404.8571 (18.45) 791.7721 (8.50) 785.7143 (8.53)
52+
test_xy_bounds[mercantile-(1, 40, 7)] 733.3000 (8.83) 5,950.0000 (11.67) 772.3273 (8.29) 766.7000 (8.33)
53+
test_xy_bounds[mercantile-(486, 332, 20)] 733.3500 (8.84) 7,408.3000 (14.53) 773.4718 (8.31) 768.7500 (8.35)
54+
test_xy_bounds[mercantile-(1, 1, 1)] 735.4000 (8.86) 4,762.5000 (9.34) 783.2100 (8.41) 777.1000 (8.44)
55+
test_xy_bounds[morecantile-(0, 0, 0)] 3,042.0000 (36.65) 108,208.0000 (212.26) 3,287.1089 (35.30) 3,250.0000 (35.29)
56+
test_xy_bounds[morecantile-(1, 0, 1)] 3,291.0000 (39.65) 68,166.0000 (133.71) 3,503.4695 (37.62) 3,459.0000 (37.56)
57+
test_xy_bounds[morecantile-(1, 1, 1)] 3,291.0000 (39.65) 72,875.0000 (142.95) 3,506.8544 (37.66) 3,500.0000 (38.01)
58+
test_xy_bounds[morecantile-(1, 40, 7)] 4,375.0000 (52.71) 76,375.0000 (149.82) 4,630.2893 (49.72) 4,584.0000 (49.78)
59+
test_xy_bounds[morecantile-(486, 332, 10)] 4,875.0000 (58.73) 85,208.0000 (167.14) 5,197.3581 (55.81) 5,125.0000 (55.65)
60+
test_xy_bounds[morecantile-(486, 332, 20)] 6,916.0000 (83.33) 78,458.0000 (153.90) 7,221.0669 (77.54) 7,166.0000 (77.82)
61+
test_xy_bounds[morecantile-(486, 332, 30)] 52,667.0000 (634.54) 404,208.0000 (792.88) 54,749.1232 (587.90) 54,333.0000 (590.00)
62+
------------------------------------------------------------------------------------------------------------------------------------------
63+
```

‎benchmarks/benchmarks.py

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
"""Morecantile/Mercantile/Utiles comparison benchmark
2+
3+
The benchmark suite is adapted from jessekrubin/utiles
4+
https://github.com/jessekrubin/utiles/blob/ea58b9a017a2e3528f03cc20f16ef531737b863f/utiles-pyo3/bench/test_bench.py#L17-L25
5+
"""
6+
# This file is a modified version of https://github.com/jessekrubin/utiles/blob/ea58b9a017a2e3528f03cc20f16ef531737b863f/utiles-pyo3/bench/test_bench.py.
7+
#
8+
# The original license follows.
9+
#
10+
# MIT License
11+
#
12+
# Copyright (c) 2023 jessekrubin
13+
#
14+
# Permission is hereby granted, free of charge, to any person obtaining a copy
15+
# of this software and associated documentation files (the "Software"), to deal
16+
# in the Software without restriction, including without limitation the rights
17+
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
18+
# copies of the Software, and to permit persons to whom the Software is
19+
# furnished to do so, subject to the following conditions:
20+
#
21+
# The above copyright notice and this permission notice shall be included in all
22+
# copies or substantial portions of the Software.
23+
#
24+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
25+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
26+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
27+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
28+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
29+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
30+
# SOFTWARE.
31+
32+
from typing import Callable, Tuple
33+
34+
import mercantile
35+
import pytest
36+
import utiles
37+
38+
import morecantile
39+
40+
tms = morecantile.tms.get("WebMercatorQuad")
41+
42+
TEST_TILES = (
43+
(0, 0, 0),
44+
(1, 0, 1),
45+
(1, 1, 1),
46+
(1, 40, 7),
47+
(486, 332, 10),
48+
# HIGH ZOOM
49+
(486, 332, 20),
50+
# OUTSIDE TMS Range
51+
(486, 332, 30),
52+
)
53+
54+
55+
@pytest.mark.parametrize(
56+
"tile",
57+
[pytest.param(t, id=str(t)) for t in TEST_TILES],
58+
)
59+
@pytest.mark.parametrize(
60+
"func",
61+
[
62+
pytest.param(mercantile.bounds, id="mercantile"),
63+
pytest.param(tms.bounds, id="morecantile"),
64+
pytest.param(utiles.bounds, id="utiles"),
65+
],
66+
)
67+
@pytest.mark.benchmark(group="bounds")
68+
def test_bounds(
69+
tile: Tuple[int, int, int],
70+
func: Callable[[Tuple[int, int, int]], Tuple[float, float]],
71+
benchmark,
72+
) -> None:
73+
"""Benchmark bounds() method."""
74+
_ = benchmark(func, *tile)
75+
76+
77+
@pytest.mark.parametrize(
78+
"tile",
79+
[pytest.param(t, id=str(t)) for t in TEST_TILES],
80+
)
81+
@pytest.mark.parametrize(
82+
"func",
83+
[
84+
pytest.param(mercantile.xy_bounds, id="mercantile"),
85+
pytest.param(tms.xy_bounds, id="morecantile"),
86+
pytest.param(utiles.xy_bounds, id="utiles"),
87+
],
88+
)
89+
@pytest.mark.benchmark(group="xy_bounds")
90+
def test_xy_bounds(
91+
tile: Tuple[int, int, int],
92+
func: Callable[[Tuple[int, int, int]], Tuple[float, float]],
93+
benchmark,
94+
) -> None:
95+
"""Benchmark xy_bounds() method."""
96+
_ = benchmark(func, *tile)

‎benchmarks/requirements.txt

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pytest
2+
pytest-benchmark
3+
4+
morecantile
5+
mercantile
6+
utiles

‎pyproject.toml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,10 @@ test = [
3636
"pytest-cov",
3737
"rasterio>=1.2.1",
3838
]
39+
benchmark = [
40+
"pytest",
41+
"pytest-benchmark",
42+
]
3943
dev = [
4044
"pre-commit",
4145
"bump-my-version",

‎tests/benchmarks.py

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
"""Morecantile benchmark."""
2+
3+
import pytest
4+
5+
import morecantile
6+
from morecantile.commons import BoundingBox
7+
8+
tms = morecantile.tms.get("WebMercatorQuad")
9+
10+
# Test tiles from https://github.com/jessekrubin/utiles/blob/ea58b9a017a2e3528f03cc20f16ef531737b863f/utiles-pyo3/bench/test_bench.py
11+
TEST_TILES = (
12+
(0, 0, 0),
13+
(1, 0, 1),
14+
(1, 1, 1),
15+
(1, 40, 7),
16+
(486, 332, 10),
17+
# HIGH ZOOM
18+
(486, 332, 20),
19+
# OUTSIDE TMS Range
20+
(486, 332, 30),
21+
)
22+
23+
24+
@pytest.mark.parametrize("tile", TEST_TILES)
25+
def test_bounds(tile, benchmark):
26+
str_tile = "Tile(x={},y={},z={})".format(*tile)
27+
benchmark.name = f"morecantile.bounds-{str_tile}"
28+
benchmark.fullname = f"morecantile.bounds-{str_tile}"
29+
benchmark.group = "morecantile.bounds"
30+
31+
r = benchmark(tms.bounds, *tile)
32+
assert isinstance(r, BoundingBox)
33+
34+
35+
@pytest.mark.parametrize("tile", TEST_TILES)
36+
def test_xy_bounds(tile, benchmark) -> None:
37+
str_tile = "Tile(x={},y={},z={})".format(*tile)
38+
benchmark.name = f"morecantile.xy_bounds-{str_tile}"
39+
benchmark.fullname = f"morecantile.xy_bounds-{str_tile}"
40+
benchmark.group = "morecantile.xy_bounds"
41+
42+
r = benchmark(tms.xy_bounds, *tile)
43+
assert isinstance(r, BoundingBox)

0 commit comments

Comments
 (0)