Skip to content

Conversation

@melihmutlu
Copy link
Member

@melihmutlu melihmutlu commented Jun 5, 2025

Introduce attach_chunk and detach_chunk procedures to support attaching
and detaching chunks to/from hypertables.

Detaching a chunk preserves non-dimensional check constraints, indexes,
and foreign keys on the chunk itself. However, any foreign keys
referencing the chunk and dimensional check constraints are dropped.

Attaching a chunk preserves existing constraints, indexes, foreign
keys, etc. Additional constraints inherited from the hypertable are
created, including dimensional constraints. Existing indexes on the
chunk are compared against those on the hypertable; if a matching index
is found, it is reused as the chunk’s index.

Both attach_chunk and detach_chunk acquire a ShareUpdateExclusiveLock
on the hypertable and an AccessExclusiveLock on the chunk, similar to
PostgreSQL partitioning.

Compressed chunks are not currently supported, but it's planned for
future version.

@philkra philkra added this to the v2.21.0 milestone Jun 6, 2025
@philkra philkra requested a review from mkindahl June 16, 2025 14:35
Copy link
Contributor

@mkindahl mkindahl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments below, but otherwise looks fine. As you mentioned, you're working on the tests so waiting with the approval until I've seen those as well. You also need to add upgrade and downgrade code.

@melihmutlu melihmutlu force-pushed the attach-detach-chunks branch 3 times, most recently from 3f2cf62 to 277d184 Compare June 17, 2025 14:34
@codecov
Copy link

codecov bot commented Jun 17, 2025

Codecov Report

Attention: Patch coverage is 82.64463% with 42 lines in your changes missing coverage. Please review.

Project coverage is 82.03%. Comparing base (f0c3113) to head (6d6a4f1).
Report is 36 commits behind head on main.

Files with missing lines Patch % Lines
tsl/src/chunk_api.c 70.37% 5 Missing and 19 partials ⚠️
src/chunk_constraint.c 85.52% 1 Missing and 10 partials ⚠️
src/constraint.c 85.71% 0 Missing and 3 partials ⚠️
src/chunk.c 90.47% 0 Missing and 2 partials ⚠️
src/indexing.c 90.47% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8223      +/-   ##
==========================================
- Coverage   82.23%   82.03%   -0.21%     
==========================================
  Files         257      257              
  Lines       48497    48736     +239     
  Branches    12218    12295      +77     
==========================================
+ Hits        39882    39981      +99     
- Misses       3730     3904     +174     
+ Partials     4885     4851      -34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@melihmutlu melihmutlu force-pushed the attach-detach-chunks branch 7 times, most recently from d78442b to 66ecbfc Compare June 22, 2025 12:30
@melihmutlu melihmutlu marked this pull request as ready for review June 22, 2025 20:48
Copy link
Member

@erimatnor erimatnor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to have some isolation tests as well to see how attach/detach interacts with concurrent:

  • compression
  • queries
  • inserts
  • other attach/detach operations
  • vacuum
    etc.

Copy link
Contributor

@mkindahl mkindahl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly nits, and a few requests for additional tests

@melihmutlu melihmutlu force-pushed the attach-detach-chunks branch 3 times, most recently from 64880c7 to cc8a27b Compare June 25, 2025 01:16
@melihmutlu melihmutlu force-pushed the attach-detach-chunks branch 4 times, most recently from 39fe274 to e405bf0 Compare June 26, 2025 09:01
@melihmutlu melihmutlu force-pushed the attach-detach-chunks branch 8 times, most recently from 1cf413c to 8efe74c Compare June 26, 2025 23:03
Copy link
Member

@erimatnor erimatnor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job with the isolation tests.

I think the PR looks pretty good now but there are some things to fix.

  • The attach test ends prematurely due to a constraint violation so that needs to be fixed.
  • You need to handle partitioning func a bit differently in constraint checking code
  • There were some (potentially) unexpected errors in some isolation test cases that you should double-check

See inline comments.

@erimatnor
Copy link
Member

Perhaps you can also expand on the description for the commit before you merge? It's a bit short right now. Perhaps write something about behavior, locking, etc.

@melihmutlu melihmutlu force-pushed the attach-detach-chunks branch 4 times, most recently from fa9987f to 9fd4ae0 Compare June 27, 2025 13:31
Copy link
Member

@erimatnor erimatnor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Still have a couple of questions. For example, one about attaching a chunk with a different owner than the hypertable.

Comment on lines +492 to +519
if (!object_ownercheck(RelationRelationId, chunk_relid, GetUserId()))
aclcheck_error(ACLCHECK_NOT_OWNER,
get_relkind_objtype(get_rel_relkind(chunk_relid)),
get_rel_name(chunk_relid));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superuser can attach regardless of owners of the hypertable and the chunk.

But can the superuser attach a chunk with owner 'foo' while the hypertable has owner 'bar'? If that works, I am not sure we'd like to support it unless we implicitly change the owner of the chunk to align with the hypertable.

@erimatnor
Copy link
Member

Note typos in commit message. E.g., "procuders"

Introduce attach_chunk and detach_chunk procedures to support attaching
and detaching chunks to/from hypertables.

Detaching a chunk preserves non-dimensional check constraints, indexes,
and foreign keys on the chunk itself. However, any foreign keys
referencing the chunk and dimensional check constraints are dropped.

Attaching a chunk preserves existing constraints, indexes, foreign
keys, etc. Additional constraints inherited from the hypertable are
created, including dimensional constraints. Existing indexes on the
chunk are compared against those on the hypertable; if a matching index
is found, it is reused as the chunk’s index.

Both attach_chunk and detach_chunk acquire a ShareUpdateExclusiveLock
on the hypertable and an AccessExclusiveLock on the chunk, similar to
PostgreSQL partitioning.

Compressed chunks are not currently supported, but it's planned for
future version.
@melihmutlu melihmutlu force-pushed the attach-detach-chunks branch from 9fd4ae0 to 6d6a4f1 Compare June 27, 2025 15:29
@melihmutlu melihmutlu enabled auto-merge (squash) June 27, 2025 15:31
@melihmutlu melihmutlu merged commit 1681acd into timescale:main Jun 27, 2025
49 checks passed
@philkra philkra mentioned this pull request Jul 2, 2025
philkra added a commit that referenced this pull request Jul 8, 2025
## 2.21.0 (2025-07-08)

This release contains performance improvements and bug fixes since the
2.20.3 release. We recommend that you upgrade at the next available
opportunity.

**Highlighted features in TimescaleDB v2.21.0**
* The attach & detach chunks feature allows manually adding or removing
chunks from a hypertable with uncompressed chunks, similar to
PostgreSQL’s partition management.
* Continued improvement of backfilling into the columnstore, achieving
up to 2.5x speedup for constrained tables, by introducing caching logic
that boosts throughput for writes to compressed chunks, bringing
`INSERT` performance close to that of uncompressed chunks.
* Optimized `DELETE` operations on the columstore through batch-level
deletions of non-segmentby keys in the filter condition, greatly
improving performance to up to 42x faster in some cases, as well as
reducing bloat, and lowering resource usage.
* The heavy lock taken in Continuous Aggregate refresh was relaxed,
enabling concurrent refreshes for non-overlapping ranges and eliminating
the need for complex customer workarounds.
* [tech preview] Direct Compress is an innovative TimescaleDB feature
that improves high-volume data ingestion by compressing data in memory
and writing it directly to disk, reducing I/O overhead, eliminating
dependency on background compression jobs, and significantly boosting
insert performance.

**Sunsetting of the hypercore access method**
We made the decision to deprecate hypercore access method (TAM) with the
2.21.0 release. It was an experiment, which did not show the signals we
hoped for and will be sunsetted in TimescaleDB 2.22.0, scheduled for
September 2025. Upgrading to 2.22.0 and higher will be blocked if TAM is
still in use. Since TAM’s inception in
[2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0),
we learned that btrees were not the right architecture. The recent
advancements in the columnstore—such as more performant backfilling,
SkipScan, adding check constraints, and faster point queries—put the
[columnstore](https://www.timescale.com/blog/hypercore-a-hybrid-row-storage-engine-for-real-time-analytics)
close to or on par with TAM without the storage from the additional
index. We apologize for the inconvenience this action potentially causes
and are here to assist you during the migration process.

Migration path

```
do $$
declare   
   relid regclass;
begin
   for relid in
       select cl.oid from pg_class cl
       join pg_am am on (am.oid = cl.relam)
       where am.amname = 'hypercore'
   loop
       raise notice 'converting % to heap', relid::regclass;
       execute format('alter table %s set access method heap', relid);
   end loop;
end
$$;
```

**Features**
* [#8081](#8081) Use JSON
error code for job configuration parsing
* [#8100](#8100) Support
splitting compressed chunks
* [#8131](#8131) Add policy
to process hypertable invalidations
* [#8141](#8141) Add
function to process hypertable invalidations
* [#8165](#8165) Reindex
recompressed chunks in compression policy
* [#8178](#8178) Add
columnstore option to `CREATE TABLE WITH`
* [#8179](#8179) Implement
direct `DELETE` on non-segmentby columns
* [#8182](#8182) Cache
information for repeated upserts into the same compressed chunk
* [#8187](#8187) Allow
concurrent Continuous Aggregate refreshes
* [#8191](#8191) Add option
to not process hypertable invalidations
* [#8196](#8196) Show
deprecation warning for TAM
* [#8208](#8208) Use `NULL`
compression for bool batches with all null values like the other
compression algorithms
* [#8223](#8223) Support
for attach/detach chunk
* [#8265](#8265) Set
incremental Continous Aggregate refresh policy on by default
* [#8274](#8274) Allow
creating concurrent continuous aggregate refresh policies
* [#8314](#8314) Add
support for timescaledb_lake in loader
* [#8209](#8209) Add
experimental support for Direct Compress of `COPY`
* [#8341](#8341) Allow
quick migration from hypercore TAM to (columnstore) heap

**Bugfixes**
* [#8153](#8153) Restoring
a database having NULL compressed data
* [#8164](#8164) Check
columns when creating new chunk from table
* [#8294](#8294) The
"vectorized predicate called for a null value" error for WHERE
conditions like `x = any(null::int[])`.
* [#8307](#8307) Fix
missing catalog entries for bool and null compression in fresh
installations
* [#8323](#8323) Fix DML
issue with expression indexes and BHS

**GUCs**
* `enable_direct_compress_copy`: Enable experimental support for direct
compression during `COPY`, default: off
* `enable_direct_compress_copy_sort_batches`: Enable batch sorting
during direct compress `COPY`, default: on
* `enable_direct_compress_copy_client_sorted`: Correct handling of data
sorting by the user is required for this option, default: off

---------

Signed-off-by: Philip Krauss <[email protected]>
Co-authored-by: philkra <[email protected]>
Co-authored-by: philkra <[email protected]>
Co-authored-by: Fabrízio de Royes Mello <[email protected]>
Co-authored-by: Anastasiia Tovpeko <[email protected]>
@timescale-automation timescale-automation added the released-2.21.0 Released in 2.21.0 label Jul 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

released-2.21.0 Released in 2.21.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants