missing confstate information after upgrading etcd to v3.6 #20495
Description
Discussed in #20446
Originally posted by DTLP August 6, 2025
Hi,
After upgrading etcd from v3.5.22 to v3.6.0 I'm observing the following in the logs:
{"level":"warn","ts":"2025-08-06T08:33:47.483064Z","caller":"etcdserver/server.go:2304","msg":"Failed to detect schema version","error":"missing confstate information"}
{"level":"error","ts":"2025-08-06T08:33:47.483160Z","caller":"version/monitor.go:120","msg":"failed to update storage version","cluster-version":"3.6.0","error":"cannot detect storage schema version: missing confstate information","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver/version.(*Monitor).UpdateStorageVersionIfNeeded\n\tgo.etcd.io/etcd/server/v3/etcdserver/version/monitor.go:120\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorStorageVersion\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2345\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2526"}
The cluster reports healthy, but these logs are concerning.
$ etcdctl endpoint health --cluster
https://10.22.22.4:2379 is healthy: successfully committed proposal: took = 33.600101ms
https://10.22.22.132:2379 is healthy: successfully committed proposal: took = 36.27905ms
https://10.22.22.68:2379 is healthy: successfully committed proposal: took = 37.965187ms
What I know so far:
- Rolling back to v3.5.22 stops the error
- Upgrading to v3.6.4 brings the error back
- v2store was empty before the upgrade to v3.6
- Restoring etcd to a month old backup and rolling that to v3.6 leads to the same issue
- We have 2 similar clusters that didn't have this issue after the upgrade - one of them required cleaning upv2store
So at this point I'm not sure whether something is going wrong during the upgrade or if the issue was there prior the upgrade, but only surfaced in v3.6.
I would really appreciate some help getting to the bottom of this.
Thank you