Skip to content

Commit ba00213

Browse files
jp-darkjp-dark
authored
Update TileDB core to 2.28.1 (#4183)
* Update TileDB core version from 2.28.0 to 2.28.1 * Update changelogs and TileDB-R version * Backport a series of small test/linter fixes
1 parent 1902656 commit ba00213

File tree

14 files changed

+123
-47
lines changed

14 files changed

+123
-47
lines changed

‎.github/workflows/libtiledbsoma-asan-ci.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ jobs:
2525
run: |
2626
mkdir -p external
2727
# Please do not edit manually -- let scripts/update-tiledb-version.py update this
28-
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.0/tiledb-linux-x86_64-2.28.0-4764907.tar.gz
28+
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.1/tiledb-linux-x86_64-2.28.1-d648231.tar.gz
2929
tar -C external -xzf tiledb-*.tar.gz
3030
ls external/lib/
3131
- name: Build and install libtiledbsoma

‎.github/workflows/python-ci-packaging.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ jobs:
7777
run: |
7878
mkdir -p external
7979
# Please do not edit manually -- let scripts/update-tiledb-version.py update this
80-
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.0/tiledb-linux-x86_64-2.28.0-4764907.tar.gz
80+
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.1/tiledb-linux-x86_64-2.28.1-d648231.tar.gz
8181
tar -C external -xzf tiledb-linux-x86_64-*.tar.gz
8282
ls external/lib/
8383
echo "LD_LIBRARY_PATH=$(pwd)/external/lib" >> $GITHUB_ENV
@@ -179,10 +179,10 @@ jobs:
179179
mkdir -p external
180180
# Please do not edit manually -- let scripts/update-tiledb-version.py update this
181181
if [ `uname -m` == "arm64" ]; then
182-
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.0/tiledb-macos-arm64-2.28.0-4764907.tar.gz
182+
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.1/tiledb-macos-arm64-2.28.1-d648231.tar.gz
183183
tar -C external -xzf tiledb-macos-arm64-*.tar.gz
184184
else
185-
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.0/tiledb-macos-x86_64-2.28.0-4764907.tar.gz
185+
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.1/tiledb-macos-x86_64-2.28.1-d648231.tar.gz
186186
tar -C external -xzf tiledb-macos-x86_64-*.tar.gz
187187
fi
188188
ls external/lib/
@@ -275,14 +275,14 @@ jobs:
275275
if [ `uname -s` == "Darwin" ]; then
276276
if [ `uname -m` == "arm64" ]; then
277277
# Please do not edit manually -- let scripts/update-tiledb-version.py update this
278-
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.0/tiledb-macos-arm64-2.28.0-4764907.tar.gz
278+
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.1/tiledb-macos-arm64-2.28.1-d648231.tar.gz
279279
else
280280
# Please do not edit manually -- let scripts/update-tiledb-version.py update this
281-
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.0/tiledb-macos-x86_64-2.28.0-4764907.tar.gz
281+
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.1/tiledb-macos-x86_64-2.28.1-d648231.tar.gz
282282
fi
283283
else
284284
# Please do not edit manually -- let scripts/update-tiledb-version.py update this
285-
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.0/tiledb-linux-x86_64-2.28.0-4764907.tar.gz
285+
wget --quiet https://github.com/TileDB-Inc/TileDB/releases/download/2.28.1/tiledb-linux-x86_64-2.28.1-d648231.tar.gz
286286
fi
287287
tar -C external -xzf tiledb-*.tar.gz
288288
ls external/lib/

‎.pre-commit-config.yaml

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,5 @@
11
exclude: ^doc/source/
22
repos:
3-
- repo: https://github.com/psf/black
4-
rev: "25.1.0"
5-
hooks:
6-
- id: black
7-
args: ["--config=apis/python/pyproject.toml"]
83
- repo: https://github.com/astral-sh/ruff-pre-commit
94
rev: v0.11.10
105
hooks:

‎apis/python/HISTORY.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
1010

1111
### Changed
1212

13+
- \[[#4126](https://github.com/single-cell-data/TileDB-SOMA/pull/4126)\] [python] At package import time, validate that the expected TileDB version is installed and used. Raises a RuntimeError exception if the condition is not met. This is an attempt to better warn users who have corrupted conda installations.
14+
- \[[#4137](https://github.com/single-cell-data/TileDB-SOMA/pull/4137)\] [python][BREAKING] Add user-specified obs/var index column to `ExperimentAxisQuery.to_anndata`. This change will also set the index columns based upon metadata hints, following the conventions of `tiledbsoma.io`. See the docstrings for more details. Prior to this change, the index dtype would be set to `string` in all cases - this change removes the forced cast, and leaves the column type as-is.
15+
- Update TileDB version to https://github.com/single-cell-data/TileDB-SOMA/pull/4177
16+
1317
### Deprecated
1418

1519
### Removed
@@ -18,6 +22,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
1822

1923
### Security
2024

25+
## [Release 1.17.0]
26+
27+
Release 1.17.0 upgrades from TileDB 2.28.0 to TileDB 2.28.1.
28+
29+
30+
### Changed
31+
32+
- \[[#4177](https://github.com/single-cell-data/TileDB-SOMA/pull/4177)\] Update [TileDB core to 2.28.1](https://github.com/TileDB-Inc/TileDB/releases/tag/2.28.1).
33+
34+
2135
## [Release 1.17.0]
2236

2337
The primary change in 1.17.0 is the upgrade to TileDB 2.28.

‎apis/python/src/tiledbsoma/io/_registration/ambient_label_mappings.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -495,7 +495,7 @@ def _register_common(
495495
# Step 3: create a joinid map for each axis
496496
#
497497
def _make_joinid_map(
498-
joinids_index: pd.Index, # type:ignore[type-arg]
498+
joinids_index: pd.Index,
499499
prev_joinid_map: pd.DataFrame,
500500
) -> pd.DataFrame:
501501
maps = []
@@ -648,7 +648,7 @@ def reduce(cls, ams: list[Self]) -> AnnDataAxisMetadata:
648648
if len(ams) == 1:
649649
return ams[0]
650650

651-
def _reduce_field_index(indices: list[pd.Index]) -> pd.Index: # type: ignore[type-arg]
651+
def _reduce_field_index(indices: list[pd.Index]) -> pd.Index:
652652
"""Reducer for joinid indices."""
653653
if len(indices) == 0:
654654
return pd.Index([])
@@ -710,10 +710,10 @@ def _merge_categoricals(
710710
return {k: _merge_categoricals(k, v) for k, v in inverted_enum_values.items()}
711711

712712

713-
def _get_dataframe_joinid_index(df: pd.DataFrame, field_name: str) -> pd.Index: # type: ignore[type-arg]
713+
def _get_dataframe_joinid_index(df: pd.DataFrame, field_name: str) -> pd.Index:
714714
"""Given an AnnData obs/var, extract the index for the user-selected join column."""
715715
if field_name in df:
716716
return cast("pd.Index[Any]", pd.Index(df[field_name]))
717717
if df.index.name in (field_name, "index", None):
718-
return df.index
718+
return cast("pd.Index[Any]", df.index)
719719
raise ValueError(f"Could not find field name {field_name} in dataframe.")

‎apis/python/src/tiledbsoma/io/_registration/id_mappings.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -74,14 +74,14 @@ def from_anndata(cls, adata: ad.AnnData, *, measurement_name: str = "RNA") -> Se
7474
return cls(obs_axis=obs_axis, var_axes=var_axes)
7575

7676

77-
def get_dataframe_values(df: pd.DataFrame, field_name: str) -> pd.Series: # type: ignore[type-arg]
77+
def get_dataframe_values(df: pd.DataFrame, field_name: str) -> pd.Series:
7878
"""Extracts the label values (e.g. cell barcode, gene symbol) from an AnnData/H5AD
7979
``obs`` or ``var`` dataframe.
8080
"""
8181
if field_name in df:
82-
values = cast(pd.Series, df[field_name].astype(str)) # type: ignore[type-arg]
82+
values = cast(pd.Series, df[field_name].astype(str))
8383
elif df.index.name in (field_name, "index", None):
84-
values = cast(pd.Series, df.index.to_series().astype(str)) # type: ignore[type-arg]
84+
values = cast(pd.Series, df.index.to_series().astype(str))
8585
else:
8686
raise ValueError(f"Could not find field name {field_name} in dataframe.")
8787

‎apis/python/tests/_util.py

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,27 @@
55
from pathlib import Path
66
from typing import Any, Union
77

8+
import _pytest
89
import numpy as np
910
import pandas as pd
1011
import pytest
1112
from _pytest._code import ExceptionInfo
1213
from _pytest.logging import LogCaptureFixture
13-
from _pytest.python_api import E, RaisesContext
14+
from packaging.version import Version
15+
from typing_extensions import TypeVar
16+
17+
if Version(_pytest.__version__) < Version("8.4.0"):
18+
from _pytest.python_api import RaisesContext
19+
20+
E = TypeVar("E", bound=BaseException, default=BaseException)
21+
MaybeRaisesReturn = Union[RaisesContext[E], ExceptionInfo[E], nullcontext]
22+
else:
23+
from _pytest.raises import RaisesExc
24+
25+
E = TypeVar("E", bound=BaseException, default=BaseException)
26+
MaybeRaisesReturn = Union[RaisesExc[E], ExceptionInfo[E], nullcontext]
27+
28+
1429
from anndata import AnnData
1530
from numpy import array_equal
1631
from pandas._testing import assert_frame_equal, assert_series_equal

‎apis/python/tests/conftest.py

Lines changed: 39 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,12 @@
1+
from __future__ import annotations
2+
3+
import multiprocessing
4+
import os
15
from pathlib import Path
26
from tempfile import TemporaryDirectory
7+
from typing import Any
38

4-
import anndata
9+
import anndata as ad
510
import pytest
611
from anndata import AnnData
712

@@ -12,6 +17,34 @@
1217
from ._util import TESTDATA
1318

1419

20+
@pytest.fixture(scope="session")
21+
def soma_tiledb_config() -> dict[str, Any] | None:
22+
# Configuration primarily focuses on memory usage, as the CI environment
23+
# is memory constrained. The smallest runners have 7GiB of RAM, whereas
24+
# whereas the TileDB core has a default memory budget of 10GiB.
25+
26+
# See https://docs.github.com/en/actions/how-tos/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables
27+
# for default variables in GHA.
28+
29+
tiledb_config: dict | None = None
30+
31+
is_CI = os.getenv("CI", "false") == "true"
32+
if is_CI:
33+
tiledb_config = {
34+
"sm.mem.total_budget": 1 * 1024**3,
35+
"sm.memory_budget": 512 * 1024**2,
36+
"sm.memory_budget_var": 512 * 1024**2,
37+
"soma.init_buffer_bytes": 128 * 1024**2,
38+
}
39+
return tiledb_config
40+
41+
42+
@pytest.fixture(scope="module")
43+
def soma_tiledb_context(soma_tiledb_config: dict[str, Any] | None) -> tiledbsoma.SOMATileDBContext:
44+
"""Fixture which builds a SOMATileDBContext based on a reasonable (i.e., small) default configuration."""
45+
return tiledbsoma.SOMATileDBContext(tiledb_config=soma_tiledb_config)
46+
47+
1548
@pytest.fixture
1649
def conftest_pbmc_small_h5ad_path(request) -> Path:
1750
"""Path to a tiny (80x20) h5ad, useful for unit-test / CI runs."""
@@ -21,7 +54,7 @@ def conftest_pbmc_small_h5ad_path(request) -> Path:
2154
@pytest.fixture
2255
def conftest_pbmc_small(conftest_pbmc_small_h5ad_path) -> AnnData:
2356
"""Tiny (80x20) AnnData, useful for unit-test / CI runs."""
24-
return anndata.read_h5ad(conftest_pbmc_small_h5ad_path)
57+
return ad.read_h5ad(conftest_pbmc_small_h5ad_path)
2558

2659

2760
@pytest.fixture
@@ -57,4 +90,7 @@ def conftest_pbmc3k_h5ad_path(request) -> Path:
5790
@pytest.fixture
5891
def conftest_pbmc3k_adata(conftest_pbmc3k_h5ad_path):
5992
"""Larger (2638x1838) AnnData, which also includes obsm, obsp, and varm arrays."""
60-
return anndata.read_h5ad(conftest_pbmc3k_h5ad_path)
93+
return ad.read_h5ad(conftest_pbmc3k_h5ad_path)
94+
95+
96+
multiprocessing.set_start_method("spawn", force=True)

‎apis/python/tests/ht/_ht_util.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -719,7 +719,8 @@ def arrays_equal(
719719
pa.compute.equal(
720720
combine_chunks(read).dictionary_decode(),
721721
combine_chunks(expected).dictionary_decode(),
722-
)
722+
),
723+
min_count=0,
723724
)
724725
if not is_eq:
725726
note("arrays_equal: dictionary arrays not equal")

‎apis/r/DESCRIPTION

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Description: Interface for working with 'TileDB'-based Stack of Matrices,
66
like those commonly used for single cell data analysis. It is documented at
77
<https://github.com/single-cell-data>; a formal specification available is at
88
<https://github.com/single-cell-data/SOMA/blob/main/abstract_specification.md>.
9-
Version: 1.17.0
9+
Version: 1.17.1
1010
Authors@R: c(
1111
person(given = "Aaron", family = "Wolen",
1212
role = "aut", email = "[email protected]",
@@ -70,6 +70,7 @@ Suggests:
7070
SingleCellExperiment (>= 1.20.0),
7171
SummarizedExperiment (>= 1.28.0),
7272
testthat (>= 3.0.0),
73+
tibble,
7374
withr
7475
Config/testthat/edition: 2
7576
OS_type: unix

0 commit comments

Comments
 (0)