katib-mysql pod :Access denied for user 'root'@'localhost' (using password: YES)

[ijackcy]

/kind bug

What steps did you take and what happened:

When I deployed kubeflow1.0, katib-db-manager-849b858bc8-4h8c9 and katib-mysql-7f99dfd774-gc4qp don't work well.

katib-db-manager-849b858bc8-4h8c9 0/1 ImagePullBackOff 0 11h
katib-mysql-7f99dfd774-gc4qp 0/1 Running 1 11h was

when i checked the log of katib-mysql-7f99dfd774-gc4qp :

Events:
Type Reason Age From Message

---

Warning Unhealthy 80s (x4130 over 11h) kubelet, slaver003 Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

when i enter the katib-mysql-7f99dfd774-gc4qp :
root@katib-mysql-7f99dfd774-gc4qp:/# mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
root@katib-mysql-7f99dfd774-gc4qp:/# mysql -D ${MYSQL_DATABASE} -e 'SELECT 1'
ERROR 1049 (42000): Unknown database 'katib'

What did you expect to happen:

all of pod work well

Anything else you would like to add:

and then i try to delete the error pods, and modify the images(mysql 5) ,also the same error.
Access denied for user 'root'@'localhost' (using password: YES)
Environment:

• Kubeflow version:1.0
• Minikube version:
• Kubernetes version: (use kubectl version):1.15.1
• OS (e.g. from /etc/os-release):

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/katib	0.79

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[gaocegege]

May I ask how you install the katib?

[ijackcy]

May I ask how you install the katib?

When I installed kubeflow, automatically started katib pod

[andreyvelich]

@ijackcy Can you check that MYSQL_ROOT_PASSWORD env is set on katib-mysql pod.
Execute on katib-mysql pod and run echo $MYSQL_ROOT_PASSWORD

Also, can you show the logs and describe katib-db-manager-849b858bc8-4h8c9 pod, please.

[crazy-canux]

got the same issue here.

k8s@ubuntu2:~/k8s/kubeflow$ kubectl describe pod katib-db-manager-54b64f99b-nmtzv -n kubeflow
Name: katib-db-manager-54b64f99b-nmtzv
Namespace: kubeflow
Priority: 0
Node: ubuntu4/10.103.238.34
Start Time: Thu, 16 Jul 2020 08:32:27 +0000
Labels: app=katib
app.kubernetes.io/component=katib
app.kubernetes.io/instance=katib-controller-0.8.0
app.kubernetes.io/managed-by=kfctl
app.kubernetes.io/name=katib-controller
app.kubernetes.io/part-of=kubeflow
app.kubernetes.io/version=0.8.0
component=db-manager
pod-template-hash=54b64f99b
Annotations: sidecar.istio.io/inject: false
Status: Running
IP: 10.244.1.23
IPs:
IP: 10.244.1.23
Controlled By: ReplicaSet/katib-db-manager-54b64f99b
Containers:
katib-db-manager:
Container ID: docker://1f5b973902cd2ab197809625fa3e7e7b46c7453d6add485f34d8b7f908828317
Image: gcr.io/kubeflow-images-public/katib/v1alpha3/katib-db-manager:v0.8.0
Image ID: docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha3/katib-db-manager@sha256:60ace3d4dbb66eb346ee19b5ad08a998ecf668e68cf699065aedb623c48fe767
Port: 6789/TCP
Host Port: 0/TCP
Command:
./katib-db-manager
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Fri, 17 Jul 2020 02:03:12 +0000
Finished: Fri, 17 Jul 2020 02:04:13 +0000
Ready: False
Restart Count: 164
Liveness: exec [/bin/grpc_health_probe -addr=:6789] delay=10s timeout=1s period=60s #success=1 #failure=5
Readiness: exec [/bin/grpc_health_probe -addr=:6789] delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
DB_NAME: mysql
DB_PASSWORD: <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'katib-mysql-secrets'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4rrh5 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-4rrh5:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-4rrh5
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

---

Normal Pulled 35m (x157 over 13h) kubelet, ubuntu4 Container image "gcr.io/kubeflow-images-public/katib/v1alpha3/katib-db-manager:v0.8.0" already present on machine
Warning Unhealthy 15m (x798 over 13h) kubelet, ubuntu4 Readiness probe failed: timeout: failed to connect service ":6789" within 1s
Warning BackOff 9s (x3099 over 13h) kubelet, ubuntu4 Back-off restarting failed container

k8s@ubuntu2:~/k8s/kubeflow$ kubectl describe pod katib-mysql-74747879d7-rmlts -n kubeflow
Name: katib-mysql-74747879d7-rmlts
Namespace: kubeflow
Priority: 0
Node: ubuntu3/10.103.238.33
Start Time: Thu, 16 Jul 2020 08:32:43 +0000
Labels: app=katib
app.kubernetes.io/component=katib
app.kubernetes.io/instance=katib-controller-0.8.0
app.kubernetes.io/managed-by=kfctl
app.kubernetes.io/name=katib-controller
app.kubernetes.io/part-of=kubeflow
app.kubernetes.io/version=0.8.0
component=mysql
pod-template-hash=74747879d7
Annotations: sidecar.istio.io/inject: false
Status: Running
IP: 10.244.2.24
IPs:
IP: 10.244.2.24
Controlled By: ReplicaSet/katib-mysql-74747879d7
Containers:
katib-mysql:
Container ID: docker://43d58cf143e185fc48efc2a5f74410cbcd45f50b779c60f728fd404825affa6d
Image: mysql:8
Image ID: docker-pullable://mysql@sha256:fe0a5b418ecf9b450d0e59062312b488d4d4ea98fc81427e3704f85154ee859c
Port: 3306/TCP
Host Port: 0/TCP
Args:
--datadir
/var/lib/mysql/datadir
State: Running
Started: Thu, 16 Jul 2020 10:50:00 +0000
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Thu, 16 Jul 2020 10:48:37 +0000
Finished: Thu, 16 Jul 2020 10:49:59 +0000
Ready: False
Restart Count: 1
Liveness: exec [/bin/bash -c mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}] delay=30s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [/bin/bash -c mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'] delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
MYSQL_ROOT_PASSWORD: <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'katib-mysql-secrets'> Optional: false
MYSQL_ALLOW_EMPTY_PASSWORD: true
MYSQL_DATABASE: katib
Mounts:
/var/lib/mysql from katib-mysql (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4rrh5 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
katib-mysql:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: katib-mysql
ReadOnly: false
default-token-4rrh5:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-4rrh5
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

---

Warning Unhealthy 4m23s (x5478 over 15h) kubelet, ubuntu3 Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

[andreyvelich]

@crazy-canux Can you try to execute on your mysql pod kubectl exec -it <katib-mysql-pod-name> -n kubeflow -- /bin/bash?

[crazy-canux]

@andreyvelich Yes, I can see the password is "test" when execute "echo $MYSQL_ROOT_PASSWORD".

[andreyvelich]

@crazy-canux Can you try to check if mysql DB works.
Run: mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1' on katib-mysql pod.

Also, try to delete /tmp/katib directory if you are using host path for mysql PV on your cluster and restart Katib.

[crazy-canux]

@andreyvelich after restart katib-mysql, error changed.

kubectl describe pod katib-mysql-6c44769759-7w5vq -n kubeflow
Name:         katib-mysql-6c44769759-7w5vq
Namespace:    kubeflow
Priority:     0
Node:         ubuntu3/10.103.238.33
Start Time:   Wed, 22 Jul 2020 02:00:49 +0000
Labels:       app=katib
              app.kubernetes.io/component=katib
              app.kubernetes.io/instance=katib-controller-0.8.0
              app.kubernetes.io/managed-by=kfctl
              app.kubernetes.io/name=katib-controller
              app.kubernetes.io/part-of=kubeflow
              app.kubernetes.io/version=0.8.0
              component=mysql
              pod-template-hash=6c44769759
Annotations:  sidecar.istio.io/inject: false
Status:       Running
IP:           10.244.2.37
IPs:
  IP:           10.244.2.37
Controlled By:  ReplicaSet/katib-mysql-6c44769759
Containers:
  katib-mysql:
    Container ID:  docker://108ef0a18c34fd40c960218be17586a104a354cbb91830cc55f74abd6393f239
    Image:         mysql:8
    Image ID:      docker-pullable://mysql@sha256:fe0a5b418ecf9b450d0e59062312b488d4d4ea98fc81427e3704f85154ee859c
    Port:          3306/TCP
    Host Port:     0/TCP
    Args:
      --datadir
      /var/lib/mysql/datadir
      --skip-grant-tables
    State:          Running
      Started:      Wed, 22 Jul 2020 02:02:15 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Wed, 22 Jul 2020 02:00:53 +0000
      Finished:     Wed, 22 Jul 2020 02:02:13 +0000
    Ready:          False
    Restart Count:  1
    Liveness:       exec [/bin/bash -c mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}] delay=30s timeout=5s period=10s #success=1 #failure=3
    Readiness:      exec [/bin/bash -c mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'] delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MYSQL_ROOT_PASSWORD:         <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'katib-mysql-secrets'>  Optional: false
      MYSQL_ALLOW_EMPTY_PASSWORD:  true
      MYSQL_DATABASE:              katib
    Mounts:
      /var/lib/mysql from katib-mysql (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4rrh5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  katib-mysql:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  katib-mysql
    ReadOnly:   false
  default-token-4rrh5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-4rrh5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m12s                  default-scheduler  Successfully assigned kubeflow/katib-mysql-6c44769759-7w5vq to ubuntu3
  Normal   Killing    3m18s                  kubelet, ubuntu3   Container katib-mysql failed liveness probe, will be restarted
  Normal   Pulled     2m47s (x2 over 4m9s)   kubelet, ubuntu3   Container image "mysql:8" already present on machine
  Normal   Created    2m47s (x2 over 4m9s)   kubelet, ubuntu3   Created container katib-mysql
  Normal   Started    2m46s (x2 over 4m8s)   kubelet, ubuntu3   Started container katib-mysql
  Warning  Unhealthy  2m11s (x12 over 4m1s)  kubelet, ubuntu3   Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
  Warning  Unhealthy  2m8s (x4 over 3m38s)  kubelet, ubuntu3  Liveness probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
  Warning  Unhealthy  111s (x2 over 2m1s)  kubelet, ubuntu3  Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1049 (42000): Unknown database 'katib'

 mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1049 (42000): Unknown database 'katib'

[andreyvelich]

@crazy-canux Did you check that PVC is properly bounded:
kubectl get pvc katib-mysql -n kubeflow ?

Which volume are you using for the mysql PV ?
If it is local hostPath, did you try to delete directory on the k8s cluster node that I mentioned above?

[Utkagr]

@andreyvelich I'm facing the same error. I checked the cluster nodes and there's no such directories /tmp/katib to delete in the first place.