[求助/Help] 主节点损坏，使用的是外部mysql数据库，更换机器连回原数据库 ，重新安装，default-host-health 无法就绪，其它所有服务 都running状态

[sun3book]

os: ubuntu 22.04.05
cloudpods: v3.11.10版

default-host-health 无法就绪，查看日志报错信息如下：

root@zhcx-cloudpods01:/opt# kubectl logs default-host-health-4wcm6 -n onecloud
[info 250625 10:49:17 procutils.WaitZombieLoop(zombie_others.go:36)] My pid is not 1 and no need to wait zombies
[info 250625 10:49:17 options.ParseOptions(options.go:310)] Use configuration file: /etc/yunion/host.conf
[warning 250625 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument enable-health-checker
[warning 250625 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument health-driver
[warning 250625 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument enable-rbac
[warning 250625 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument enable-qmp-monitor
[warning 250625 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument disk-is-ssd
[info 250625 10:49:17 options.ParseOptions(options.go:331)] Set log level to "info"
[info 2025-06-25 10:49:17 options.ParseOptions(options.go:310)] Use configuration file: /etc/yunion/common/common.conf
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument enable-isolated-device-whitelist
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument ext4-hugefile-size-gb
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument default-process-timeout-seconds
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument deploy-concurrent
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument log-with-time-zone
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument no-hpet
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument ext4-largefile-size-gb
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument image-deploy-driver
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument image-cache-expire-days
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument image-cache-cleanup-on-startup
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument enable-default-policy
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument dhcp-renewal-time
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument enable-app-profiling
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument enable-change-owner-auto-rename
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument log-timestamp-format
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument image-cache-cleanup-percentage
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument executor-connect-timeout-seconds
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument dhcp-lease-time
[warning 2025-06-25 10:49:17 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1214)] Cannot find argument image-cache-cleanup-dry-run
[info 2025-06-25 10:49:17 options.ParseOptions(options.go:331)] Set log level to "info"
[info 2025-06-25 10:49:17 main.(*SHostHealthService).InitService(main.go:42)] exec socket path: /var/run/onecloud/exec.sock
[info 2025-06-25 10:49:20 main.(*SHostHealthService).RunService.func1(main.go:55)] Auth complete!!
[fatal 2025-06-25 10:49:23 main.(*SHostHealthService).RunService.func1(main.go:64)] Init host health manager failed init default etcd client: context deadline exceeded

查看etcd 容器日志：kubectl logs default-etcd-b7pfbdgsps -n onecloud

2025-06-25 10:50:48.184549 I | embed: rejected connection from "10.40.175.203:48584" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:48.288926 I | embed: rejected connection from "10.158.155.250:29416" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:48.293628 I | embed: rejected connection from "10.158.155.250:38735" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:48.517165 I | embed: rejected connection from "10.40.175.203:48586" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:48.645475 I | embed: rejected connection from "10.40.175.236:44916" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:48.685110 I | embed: rejected connection from "10.40.175.219:45608" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:48.711510 I | embed: rejected connection from "10.40.175.236:44926" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:48.843433 I | embed: rejected connection from "10.40.175.202:54052" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:49.302068 I | embed: rejected connection from "10.158.155.250:32993" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:49.309920 I | embed: rejected connection from "10.158.155.250:6663" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:49.349389 I | embed: rejected connection from "10.40.175.242:57344" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:49.660983 I | embed: rejected connection from "10.40.175.226:56932" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:49.727535 I | embed: rejected connection from "10.40.175.226:56938" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:50.363164 I | embed: rejected connection from "10.40.175.245:52502" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:50.372084 I | embed: rejected connection from "10.40.175.247:49518" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:50.381436 I | embed: rejected connection from "10.158.155.250:21579" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:50.386546 I | embed: rejected connection from "10.40.175.247:49522" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:50.395513 I | embed: rejected connection from "10.40.175.203:48588" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:50.849522 I | embed: rejected connection from "10.40.175.252:53582" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:50.858701 I | embed: rejected connection from "10.40.175.251:59614" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:50.859751 I | embed: rejected connection from "10.40.175.251:59622" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:50.940406 I | embed: rejected connection from "10.40.175.235:45158" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.031000 I | embed: rejected connection from "10.158.155.250:7208" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.125064 I | embed: rejected connection from "10.40.175.202:54054" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.136335 I | embed: rejected connection from "[::1]:57212" (error "EOF", ServerName "localhost")
2025-06-25 10:50:51.186872 I | embed: rejected connection from "10.158.155.250:16680" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.370387 I | embed: rejected connection from "10.40.175.245:52506" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.382119 I | embed: rejected connection from "10.40.175.247:49526" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.388826 I | embed: rejected connection from "10.158.155.250:17171" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.396584 I | embed: rejected connection from "10.40.175.247:49532" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.858713 I | embed: rejected connection from "10.40.175.252:53586" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.868161 I | embed: rejected connection from "10.40.175.251:59638" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.868342 I | embed: rejected connection from "10.40.175.251:59650" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:51.946039 I | embed: rejected connection from "10.40.175.235:45160" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:52.687141 I | embed: rejected connection from "10.40.175.245:52508" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:52.837009 I | embed: rejected connection from "10.40.175.247:49534" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:52.871918 I | embed: rejected connection from "10.158.155.250:58319" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:53.158600 I | embed: rejected connection from "10.40.175.251:59664" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:53.197802 I | embed: rejected connection from "10.40.175.247:49546" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:53.316993 I | embed: rejected connection from "10.40.175.252:53600" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:53.336526 I | embed: rejected connection from "10.40.175.251:59678" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:53.849820 I | embed: rejected connection from "10.40.175.235:37976" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:55.092513 I | embed: rejected connection from "10.158.155.250:61568" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-25 10:50:55.700054 I | embed: rejected connection from "10.40.175.251:56434" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")

已执行过 ： climc service-cert-delete etcd 重建证书

[sun3book]

default-host-health 无法就绪

[wanyaoqi]

@sun3book
climc service-cert-delete etcd
climc endpoint-list --search etcd 得到两个
climc endpoint-update --delete enable <ETCD_ENDPOINT>
climc endpoint-delete <ETCD_ENDPOINT>

然后等这些重建

如果不行可以禁用 etcd tls:
kubectl edit oc -n onecloud
改 enableTls: false
然后删除etcdpod

[sun3book]

@sun3book climc service-cert-delete etcd climc endpoint-list --search etcd 得到两个 climc endpoint-update --delete enable <ETCD_ENDPOINT> climc endpoint-delete <ETCD_ENDPOINT>

然后等这些重建

如果不行可以禁用 etcd tls: kubectl edit oc -n onecloud 改 enableTls: false 然后删除etcdpod

正常了
重建时间间隔非常短，需要调整一下执行顺序：

climc endpoint-list --search etcd 得到两个，拷贝出来，提前生成好命令，一次性执行如下指令：

climc service-cert-delete etcd
climc endpoint-update --disable d732cb7c6738484686be6d2696eb2ecb
climc endpoint-delete d732cb7c6738484686be6d2696eb2ecb
climc endpoint-update --disable 69a6c88e5f4c48dc87c31f597b2912eb
climc endpoint-delete 69a6c88e5f4c48dc87c31f597b2912eb

执行完之后，等待几十秒，都恢复正常running了

查看日志， default-host-health日志正常，default-etcd 还是有报错信息：

2025-06-26 01:28:08.375366 I | embed: rejected connection from "[::1]:57790" (error "EOF", ServerName "localhost")
2025-06-26 01:28:11.946362 I | embed: rejected connection from "10.40.175.208:43654" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:12.953181 I | embed: rejected connection from "10.40.175.208:43660" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:14.602934 I | embed: rejected connection from "10.40.175.208:59092" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:18.057573 I | embed: rejected connection from "127.0.0.1:60436" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
WARNING: 2025/06/26 01:28:18 grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0 }. Err :connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
2025-06-26 01:28:26.952733 I | embed: rejected connection from "10.40.175.208:51642" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:27.958676 I | embed: rejected connection from "10.40.175.208:51654" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:29.554424 I | embed: rejected connection from "10.40.175.208:51668" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:34.576624 I | embed: rejected connection from "127.0.0.1:60402" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
WARNING: 2025/06/26 01:28:34 grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0 }. Err :connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
2025-06-26 01:28:41.949777 I | embed: rejected connection from "10.40.175.208:42042" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:42.963054 I | embed: rejected connection from "10.40.175.208:42050" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:44.446102 I | embed: rejected connection from "10.40.175.208:56492" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:46.609762 I | embed: rejected connection from "10.40.175.208:56496" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:48.373985 I | embed: rejected connection from "[::1]:48010" (error "EOF", ServerName "localhost")
2025-06-26 01:28:56.945011 I | embed: rejected connection from "10.40.175.208:43928" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:57.959628 I | embed: rejected connection from "10.40.175.208:43938" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:28:59.463787 I | embed: rejected connection from "10.40.175.208:43948" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:29:05.754561 I | embed: rejected connection from "127.0.0.1:41356" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
WARNING: 2025/06/26 01:29:05 grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0 }. Err :connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
2025-06-26 01:29:11.980331 I | embed: rejected connection from "10.40.175.208:34010" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:29:12.990101 I | embed: rejected connection from "10.40.175.208:34016" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:29:14.584167 I | embed: rejected connection from "10.40.175.208:32980" (error "remote error: tls: bad certificate", ServerName "default-etcd-client.onecloud.svc")
2025-06-26 01:29:23.375188 I | embed: rejected connection from "[::1]:58752" (error "EOF", ServerName "localhost")

不知道有没有什么影响。

[wanyaoqi]

@sun3book 应该是其他使用 etcd pod 可能没重启，重启后应该会恢复