Skip to content

Conversation

louiseschmidtgen
Copy link
Contributor

Remove lock on writes

We have discovered that these locks pile up when Dqlite can not keep up with the requests for various reasons.
Instead of spawning more and more goroutines on this lock we should let them proceed to hit Dqlite and retry and fail there.

goroutine profile: total 4588
3158 @ 0x4760ae 0x452c05 0x452bd4 0x4773c5 0x48b45d 0x8f08ed 0x8f08ce 0x8f2375 0xb14e16 0xb0d5d8 0xb07f6c 0xb04845 0xb
075c7 0xaa29e6 0xa63c2b 0xa68beb 0xa6193f 0x47e5a1
#   	0x4773c4    	sync.runtime_SemacquireMutex+0x24                                                   	/snap/
go/10826/src/runtime/sema.go:95
#   	0x48b45c    	sync.(*Mutex).lockSlow+0x15c                                                        	/snap/
go/10826/src/sync/mutex.go:173
#   	0x8f08ec    	sync.(*Mutex).Lock+0x2ec                                                            	/snap/
go/10826/src/sync/mutex.go:92
#   	0x8f08cd    	github.com/canonical/k8s-dqlite/pkg/kine/drivers/generic.(*Generic).execute+0x2cd   	/build
/k8s/parts/k8s-dqlite/build/k8s-dqlite/pkg/kine/drivers/generic/generic.go:374
#   	0x8f2374    	github.com/canonical/k8s-dqlite/pkg/kine/drivers/generic.(*Generic).Update+0x334    	/build
/k8s/parts/k8s-dqlite/build/k8s-dqlite/pkg/kine/drivers/generic/generic.go:477
#   	0xb14e15    	github.com/canonical/k8s-dqlite/pkg/kine/logstructured/sqllog.(*SQLLog).Update+0x55 	/build
/k8s/parts/k8s-dqlite/build/k8s-dqlite/pkg/kine/logstructured/sqllog/sql.go:527
#   	0xb0d5d7    	github.com/canonical/k8s-dqlite/pkg/kine/logstructured.(*LogStructured).Update+0x217	/build
/k8s/parts/k8s-dqlite/build/k8s-dqlite/pkg/kine/logstructured/logstructured.go:199
#   	0xb07f6b    	github.com/canonical/k8s-dqlite/pkg/kine/server.(*LimitedServer).update+0x66b       	/build
/k8s/parts/k8s-dqlite/build/k8s-dqlite/pkg/kine/server/update.go:50
#   	0xb04844    	github.com/canonical/k8s-dqlite/pkg/kine/server.(*LimitedServer).Txn+0x2e4          	/build
/k8s/parts/k8s-dqlite/build/k8s-dqlite/pkg/kine/server/limited.go:37
#   	0xb075c6    	github.com/canonical/k8s-dqlite/pkg/kine/server.(*KVServerBridge).Txn+0x26          	/build
/k8s/parts/k8s-dqlite/build/k8s-dqlite/pkg/kine/server/server.go:137
#   	0xaa29e5    	go.etcd.io/etcd/api/v3/etcdserverpb._KV_Txn_Handler+0x1a5                           	/root/
go/pkg/mod/go.etcd.io/etcd/api/[email protected]/etcdserverpb/rpc.pb.go:6610
#   	0xa63c2a    	google.golang.org/grpc.(*Server).processUnaryRPC+0xe2a                              	/root/
go/pkg/mod/google.golang.org/[email protected]/server.go:1394
#   	0xa68bea    	google.golang.org/grpc.(*Server).handleStream+0xe8a                                 	/root/
go/pkg/mod/google.golang.org/[email protected]/server.go:1805
#   	0xa6193e    	google.golang.org/grpc.(*Server).serveStreams.func2.1+0x7e                          	/root/
go/pkg/mod/google.golang.org/[email protected]/server.go:1029

923 @ 0x4760ae 0x452c05 0x452bd4 0x4773c5 0x48b45d 0xb0e9d5 0xb0e9b2 0x47e5a1
#   	0x4773c4    	sync.runtime_SemacquireMutex+0x24                                                   	/snap/
go/10826/src/runtime/sema.go:95
#   	0x48b45c    	sync.(*Mutex).lockSlow+0x15c                                                        	/snap/
go/10826/src/sync/mutex.go:173
#   	0xb0e9d4    	sync.(*Mutex).Lock+0xd4                                                             	/snap/
go/10826/src/sync/mutex.go:92
#   	0xb0e9b1    	github.com/canonical/k8s-dqlite/pkg/kine/logstructured.(*LogStructured).ttl.func1+0xb1  /build
/k8s/parts/k8s-dqlite/build/k8s-dqlite/pkg/kine/logstructured/logstructured.go:260

@louiseschmidtgen louiseschmidtgen requested a review from a team as a code owner May 30, 2025 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant