kubernets related logs & configurations
1. Logs
Components level logs
kubelet,kube-apiserver,kube-scheduler,kube-controller,etcd,kube-proxy...
Pod level logs
- generate by container
1.1 Log files
The base path for logs is /var/log/
| Component | Log Path | Description |
| kubelet | /var/log/syslog | Kubelet logs if not using journald. |
| kube-apiserver | /var/log/kube-apiserver.log | API server logs (depends on setup). |
| kube-scheduler | /var/log/kube-scheduler.log | Scheduler logs (depends on setup). |
| kube-controller-manager | /var/log/kube-controller-manager.log | Controller manager logs. |
| Pod logs | /var/log/pods/ /var/log/containers/ | Logs for all containers running on the node. Docker: /var/lib/docker/containers/<container-id>/<container-id>-json.log and containerd: /var/log/pods/<namespace>/<pod-name>/<container-name>.log |
| etcd | /var/log/etcd.log | etcd logs, if configured |
1.2 CMD
kubelet logs:
grep kubelet /var/log/syslog
journalctl -u [kubectl|kube-controller-manager|kube-scheduler|kube-apiserver|etcd|docker|containerd]
# use describe/logs
kubectl describe pod <pod-name> -n kube-system
kubectl logs <pod-name> -c <container-name>
kubectl logs <pod-name> -n kube-system
# use crictl or docker (in case when docker is used) to check the logs and process
crictl ps
crictl logs
docker ps
docker logs
2. Config
2.1 Config Files
The basic path for kubernetes config file store in /etc/kubernetes/ folder, for kubelet normaly stores at /var/lib/kubelet/config.yaml Default Paths:
/etc/kubernetes/: Central directory for most Kubernetes configuration files./var/lib/kubelet/: Directory for kubelet-related data and configuration./var/lib/etcd/: Directory for etcd data./etc/systemd/system/: Directory for systemd service configuration files.
| Component | Config Path | Description | Params |
| kubelet | /var/lib/kubelet/config.yaml | Main kubelet configuration file. | |
/etc/kubernetes/kubelet.conf | Kubeconfig for API server communication. | --config: Points to the kubelet's configuration file (e.g., /var/lib/kubelet/config.yaml). --pod-manifest-path: Specifies the directory for static pod manifests (default: /etc/kubernetes/manifests). --kubeconfig: Path to the kubelet's kubeconfig file for API server communication (default: /etc/kubernetes/kubelet.conf). | |
| kube-apiserver | /etc/kubernetes/manifests/kube-apiserver.yaml | Static pod manifest for API server. | --etcd-servers: Specifies the etcd endpoints. --secure-port: Port for secure communication (default: 6443). --service-cluster-ip-range: Range of IPs for cluster services. --tls-cert-file and --tls-private-key-file: Certificates for secure |
| kube-scheduler | /etc/kubernetes/manifests/kube-scheduler.yaml | Static pod manifest for scheduler. | --kubeconfig: Path to the kubeconfig file for the scheduler. --leader-elect: Ensures only one active scheduler in HA setups. |
| kube-controller-manager | /etc/kubernetes/manifests/kube-controller-manager.yaml | Static pod manifest. | --kubeconfig: Path to the kubeconfig file for the controller manager. --service-account-private-key-file: Key for signing service account tokens. --cluster-signing-cert-file and --cluster-signing-key-file: Certificates for signing. |
| kubeconfig files | /etc/kubernetes/admin.conf | Admin kubeconfig for kubectl. | |
/etc/kubernetes/controller-manager.conf | Kubeconfig for controller manager. | ||
/etc/kubernetes/scheduler.conf | Kubeconfig for scheduler. | ||
| CNI plugins | /etc/cni/net.d/ | Directory for CNI plugin configurations. | |
| Static pods | /etc/kubernetes/manifests/ | Directory for static pod definitions. | |
| etcd | /etc/kubernetes/manifests/etcd.yaml | etcd configuration. | --data-dir: Directory for storing etcd data. --advertise-client-urls: URLs advertised to clients. --listen-client-urls: URLs where etcd listens for client requests. |
| kube-proxy | /var/lib/kube-proxy/config.conf | Often referenced in the kube-proxy pod or daemonset | --cluster-cidr: Range of IPs for the cluster network. --kubeconfig: Path to the kubeconfig file for API server communication. |
2.1.1 /etc/kubernetes/manifests/ file
As kube-controller always monitor the manifests folder files, so we can directly edit in manifests file. But it's better to backup first.
## always make a backup
cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.bk
## make the change
vim /etc/kubernetes/manifests/kube-apiserver.yaml
## wait till container restarts
watch crictl ps
## check for apiserver pod
k -n kube-system get pod
2.2 CMD
# View the active kubeconfig settings.
kubectl config view
# check the corresponding pods
kubectl describe pod <pod-name> -n kube-system
# use systemctl cat
systemctl cat [kubectl|kube-controller-manager|kube-scheduler|kube-apiserver|etcd|docker|containerd]
# can even use find
find / | grep kubeadm
By understanding these logs, configurations and their purposes, we can better manage and troubleshoot our Kubernetes cluster.