2.22.0 Release - Continous Aggregate issues

[philkra]

The 2.22.0 release of TimescaleDB uncovered an issue within Continuous Aggregate, leading to gaps in the materialized view as ranges are not correctly refreshed as well as potential segfaults during the refresh policy execution.

We recommend holding off on upgrading to 2.22.0 if you have Continuous Aggregates in your service and to wait for the fix in the next patch. We apologize for the inconvenience this may have caused.

We are working on a fix, which will be shipped with the upcoming 2.22.1 release. Please subscribe to this ticket as updates will be posted here.

[jaskij]

Is there any ETA for 2.22.1?

[smyrman]

We are seeing a similar issue in 2.21.3; do you know when the bug was introduced?

[ssmoss]

@philkra will the fix correct any CAGs if one is already running 2.20.0?

[philkra]

Is there any ETA for 2.22.1?

@jaskij We are validating the fixes

---

We are seeing a similar issue in 2.21.3; do you know when the bug was introduced?

@smyrman can you please elaborate on the issue you are facing? Thank you

---

@philkra will the fix correct any CAGs if one is already running 2.20.0?

@ssmoss the 2.20.x versions and prior are not affected by the issue.

[ssmoss]

@philkra Sorry I meant 2.22.0 not 2.20.0. I am already running the broken version unfortunately and I am curiously what the restoration path will be once the fix is implemented. Thanks.

[smyrman]

Sure. Our case isn't the most typical, and there is definitively a risk that our issues are self-inflicted.

We got a setup where we use several 1min and 1hour rollup continuous aggregate views, updated on an hourly basis. After upgrading to v0.21.x, as well as a switch to using hyper-core (column store), we have been seeing missing segments in some of our 1h and 1min views. They do however seam to get filled eventually, so if this doesn't happen with the issue you describe, then probably our issue is unrelated.

Our case may be self-inflicted as we are using Timescale in a way that is probably relatively uncommon. Earlier, Timescale had relatively poor performance when it came to deleting whole segments in compressed tables. Our understanding is that this is now much better. Either way, to work-around this, we have a very high number of hypertables and associated continuous aggregate views (about 6700 equal schema hyper-tables with 2 continuous aggregate views each). Before the upgrade, there were some minor issues with this, but overall things seamed to work (surprisingly well). However, after the upgrade, we don't only see missing segments; we also see very high CPU & memory usage (for continuous aggregate jobs). We have gone from using less than 1 core, to using about 6-7 CPU cores. Jobs are more often delayed due to out-of-background-worker issues. Most of the continuous aggregate view jobs appear to be doing "nothing"; or at least most of the logs will state "Updated 0 row(s) in ...". It's hard to blame it at the upgrade specifically, because the hypercore / columnstore, does seam to be built up quite differently from the previous system with compression policies and manual chunk-rollup possibilities.

[kamil-certat]

Are you able to elaborate when exactly the issue occurs? Is it only during policy refresh, or also the manually triggered refresh? I have data that has just been migrated to the TimescaleDB 2.22.0, and so far the continues aggregates have been refreshed only by triggering the refreshment manually. I'd love to know if I could just wait (or downgrade TS) before activating live ingestion & the refresh policy, or should I migrate data again using different version. 🥲 Is there a way I could verify if the issue affects my data?

[philkra]

@philkra Sorry I meant 2.22.0 not 2.20.0. I am already running the broken version unfortunately and I am curiously what the restoration path will be once the fix is implemented. Thanks.

@ssmoss, if you are affected, the restoration requires a full refresh of the continuous aggregate, after you upgraded the extension.

[philkra]

@smyrman thank you for providing the details. You are correct that in a recent version, delete optimizations on the columnstore were added, which generate less IO and execute faster.

Can you please reach DM me in TigerData's community Slack channel: https://slack.timescale.com/ ? We can go through your setup because this change in resource usage is odd.

[philkra]

@kamil-certat the missing refreshed buckets are a corner case when minus- or plus infinity ranges are in play. The algorithm merges intersecting time ranges to avoid freshing the same range twice, and if one of those ranges starts with minus or plus infinity the merge does not work correctly.

You can downgrade to 2.21.3 and then upgrade to 2.22.0 and then 2.22.1 after it's shipped.

[kamil-certat]

Thanks for the answer, it's good - it seems not to affect me yet!

[blakemartz]

was it really best practice to downgrade latest today to 2.21.4? Took down our prod server today... I suppose that's what we get for living on docker image tag latest.