Implement trainer init parameters election with etcd #3321
Conversation
helinwang
changed the title
helinwang
force-pushed
the
trainer_etcd
branch
2 times, most recently
from
40965d9 to
8774ce6
Compare
go/master/service_test.go
Outdated
go/pserver/client/etcd_client.go
Outdated
There was a problem hiding this comment.
I'm thinking if the selected trainer fails before calling Done(), another new trainer will start and Select() itself, and init parameters again.
Well this may not harm for the pserver is still in "uninited" state, init twice seems ok.
There was a problem hiding this comment.
That's a good point!
A side note is that we will need to address this related problem: #3331
go/pserver/client/etcd_client.go
Outdated
There was a problem hiding this comment.
Do we still need to use transaction since already got the distributed lock?
There was a problem hiding this comment.
Although extremely unlikely, the program could pause after getting the lock, and resumed without holding the lock, before reaching this statement.
There was a problem hiding this comment.
You mean program pause time exceeding the session timeout, then the lock is released and then program resumes running, then it could get/write the wrong state?
That's quite possible. Curious will etcd client release the lock when session timeouts?
There was a problem hiding this comment.
Yes I mean that.
The etcd client will try it's best to estimate if the lock is expired or not based on the local clock, but it still could have error due to the local clock drift. The most accurate way is to use a transaction conditioned on holding the lock.
|
Oh, and please fix the style check error under CI. |
2 participants
Fixes: #3221