Cluster becomes not acessible after 'microk8s stop' and 'microk8s start' #4944
Description
Summary
Cluster becomes not acessible after 'microk8s stop' and 'microk8s start'
Freshly deployed single-node cluster with MicroK8s v1.32.2 revision 7731 operates normally.
All pods are running and the cluster is acessible on its public IP address
Then we perform 'microk8s stop' and 'microk8s start'.
After this operation the server is not accessible on its public IP anymore (127.0.0.1 or localhost do not work either)
Wait for more than 30 minutes - same result
All pods appear up and communicate with each other correctly
operation log
# lets check the public ip
> hostname -I
192.168.111.195 10.217.22.0 10.217.22.1
# test that the application is accessible via public IP
> curl -k https://192.168.111.195 | head
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3608 0 3608 0 0 9535 0 --:--:-- --:--:-- --:--:-- 9544
<!DOCTYPE html>
<!--[if lt IE 7]>
<html lang="en" class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]>
<html lang="en" class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]>
<html lang="en" class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!-->
<html lang="en" class="no-js">
<!--<![endif]-->
> microk8s version
MicroK8s v1.32.2 revision 7731
> microk8s status
microk8s is running
high-availability: no
datastore endpoints:
127.0.0.1:12379
addons:
enabled:
dns # (core) CoreDNS
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
metrics-server # (core) K8s Metrics Server for API access to service metrics
disabled:
cert-manager # (core) Cloud native certificate management
cis-hardening # (core) Apply CIS K8s hardening
community # (core) The community addons repository
dashboard # (core) The Kubernetes dashboard
gpu # (core) Alias to nvidia add-on
ha-cluster # (core) Configure high availability on the current node
host-access # (core) Allow Pods connecting to Host services smoothly
hostpath-storage # (core) Storage class; allocates storage from host directory
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
minio # (core) MinIO object storage
nvidia # (core) NVIDIA hardware (GPU and network) support
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
registry # (core) Private image registry exposed on localhost:32000
rook-ceph # (core) Distributed Ceph storage using Rook
storage # (core) Alias to hostpath-storage add-on, deprecated
> microk8s inspect
Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-flanneld is running
Service snap.microk8s.daemon-etcd is running
Service snap.microk8s.daemon-apiserver-kicker is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy asnycio usage and limits to the final report tarball
Copy inotify max_user_instances and max_user_watches to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Building the report tarball
Report tarball is at /var/snap/microk8s/7731/inspection-report-20250318_142101.tar.gz
> date
Tue Mar 18 14:21:07 UTC 2025
> microk8s stop
Stopped.
Stopped.
> date
Tue Mar 18 14:24:16 UTC 2025
> microk8s status
microk8s is not running, try microk8s start
> microk8s start
> microk8s status
microk8s is running
high-availability: no
datastore endpoints:
127.0.0.1:12379
addons:
enabled:
dns # (core) CoreDNS
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
metrics-server # (core) K8s Metrics Server for API access to service metrics
disabled:
cert-manager # (core) Cloud native certificate management
cis-hardening # (core) Apply CIS K8s hardening
community # (core) The community addons repository
dashboard # (core) The Kubernetes dashboard
gpu # (core) Alias to nvidia add-on
ha-cluster # (core) Configure high availability on the current node
host-access # (core) Allow Pods connecting to Host services smoothly
hostpath-storage # (core) Storage class; allocates storage from host directory
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
minio # (core) MinIO object storage
nvidia # (core) NVIDIA hardware (GPU and network) support
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
registry # (core) Private image registry exposed on localhost:32000
rook-ceph # (core) Distributed Ceph storage using Rook
storage # (core) Alias to hostpath-storage add-on, deprecated
zadmin@debian:~$ date
Tue Mar 18 14:37:22 UTC 2025
zadmin@debian:~$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default configuration-service-79c5cffcc7-sbhrj 1/1 Running 3 125m
default db-management-utility-7bcd5747c9-zwtr4 2/2 Running 6 123m
default db-management-utility-postgresql-744b7b4b4c-r2llv 1/1 Running 3 123m
default fluentd-b9gmp 1/1 Running 10 3h37m
default gldr-agent-7b5d8494fd-82976 1/1 Running 3 125m
default gldr-sync-service-5645975fdd-42vz4 1/1 Running 3 124m
default host-metrics-595c88556c-z85vn 1/1 Running 3 123m
default logcollector-service-54b4c77bd7-jhql2 1/1 Running 3 125m
default management-console-55f797dddf-jcdmz 1/1 Running 3 121m
default pods-metrics-kube-eagle-9f7ccc9cb-w4lvp 1/1 Running 3 123m
default prometheus-server-75c57ff799-4rhwj 1/1 Running 3 123m
default saas-agent-65569b78dc-ch86r 1/1 Running 3 125m
default scripts-service-dc896c6bc-qrq5h 1/1 Running 3 125m
default settings-service-64fb84d8d6-s7qfs 1/1 Running 3 126m
default static-file-system-5cff84c4d9-4249p 1/1 Running 3 125m
default swagger-service-96f75d75d-kjvtt 1/1 Running 3 125m
default tweaks-service-79b4968bbc-smz4p 1/1 Running 3 125m
default upgrade-service-86bfb7c9c8-6wc8j 1/1 Running 3 124m
default zkeycloak-0 1/1 Running 3 129m
default zkeycloak-db-867cbc4866-f484l 1/1 Running 3 129m
default zvm-db-6965b9c8df-kgq4h 1/1 Running 3 125m
default zvm-service-58df5597d7-th5fw 1/1 Running 4 125m
ingress-nginx ingress-nginx-controller-56dd6fc565-82bbg 1/1 Running 4 3h51m
kube-system coredns-dbcd55787-bnp46 1/1 Running 4 3h52m
kube-system metrics-server-metrics-server-fips-6b5669c868-vkp7d 1/1 Running 3 123m
> curl -k https://192.168.111.195 | head
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to 192.168.111.195 port 443 after 13 ms: Couldn't connect to server
> microk8s inspect
Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-flanneld is running
Service snap.microk8s.daemon-etcd is running
Service snap.microk8s.daemon-apiserver-kicker is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy asnycio usage and limits to the final report tarball
Copy inotify max_user_instances and max_user_watches to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Building the report tarball
Report tarball is at /var/snap/microk8s/7731/inspection-report-20250318_143809.tar.gz
Workaround
reboot OS
after full system reboot the cluster is accessible again
Inspection bundles
before 'stop+start'
inspection-report-20250318_142101.tar.gz
after 'stop+start'