Notes Renovation
Kubernets work flow

Kubernets work flow

Cheedge Lee's photo
Cheedge Lee
·

5 min read

Table of contents

1. Workflow of Kubernetes Operations:

  1. kubectl Sends API Request:

    • kubectl sends API requests to the kube-apiserver (usually through the IP address and port specified in the .kube/config or /etc/kubernetes/admin.conf file).

  2. kube-apiserver Processes Requests:

    • The kube-apiserver receives, authenticates, authorizes, and validates the request.

    • The kube-apiserver may update resources or change states in the etcd database, which stores the cluster’s entire state.

    • If the request involves creating or managing resources like Pods, Deployments, or Services, the kube-apiserver triggers other components (e.g., kube-scheduler, kube-controller-manager) to take action.

  3. Interactions Between kube-apiserver and kube-controller-manager:

    • The kube-apiserver does not directly tell the kube-controller-manager to create resources. Instead, the kube-controller-manager constantly watches etcd for changes in the cluster’s desired state.

    • When a change is detected (e.g., a new deployment is created), the appropriate controller in the kube-controller-manager takes action to make the actual state with the desired state.

  4. Creating/Deleting Resources: kube-controller-manager with kubelet

    • kube-controller-manager: It uses various controllers (e.g., DeploymentController, ReplicaSetController, NodeController) to manage different resources. These controllers watch and send instructions to the kube-apiserver to create or delete pods and other resources as needed.

    • kubelet: While the kube-controller-manager instructs the kube-apiserver to create or remove pods, the kubelet is responsible for executing these instructions on the nodes. It reads pod specifications from the kube-apiserver and communicates with the container runtime to create or delete containers on the node.

    • Therefore, the kube-controller-manager ensures the desired state by initiating changes (maintain), while kubelet performs the actual creation and management of containers on nodes (excution).

    • The kube-apiserver then updates etcd with these changes, and kubelet on the assigned nodes performs the actions needed to bring the cluster back to the desired state.

  5. Maintaining Resource State:

    The kube-controller-manager continually monitors the state of resources. If a pod crashes or a desired number of replicas is not met, the appropriate controller detects the issue and instructs the kube-apiserverto make the necessary changes (e.g., creating a new pod or deleting a failed one).

    • The controllers are part of the kube-controller-manager, which itself is a Pod in the kube-system namespace.

    • These controllers do not directly monitor the Pods but watch the kube-apiserver for events and status updates. When they detect a state mismatch (e.g., fewer replicas than desired), they trigger actions (like creating/deleting Pods) through the kube-apiserver, which updates etcd.

  6. kubelet on different Nodes:

    • kubelet runs on both control plane nodes and worker nodes.

      • On control plane nodes, it manages static Pods for core components like kube-apiserver, kube-controller-manager, and etcd by reading manifest files from /etc/kubernetes/manifests.

      • On worker nodes, it focuses on managing user workloads.

    • It communicates with the kube-apiserver to receive Pod specs and uses container runtime (like Docker or containerd) to start and stop containers as instructed.

        /etc/kubernetes/
  |-- manifests
  |   |-- etcd.yaml
  |   |-- kube-apiserver.yaml
  |   |-- kube-controller-manager.yaml
  |   `-- kube-scheduler.yaml

  7. kube-scheduler's Role:

    • Once the kube-apiserver records the creation of new pods, the kube-scheduler is responsible for assigning these pods to appropriate nodes based on resource availability, affinity/anti-affinity rules, and other scheduling constraints.

    • For unscheduled Pods, the kube-scheduler assigns them to specific nodes. It monitors pending Pods by watching the kube-apiserver (not etcd directly) for new unscheduled Pods.

    • After assigning a Pod to a node, the scheduler updates the Pod’s information back to etcd via the kube-apiserver.

  8. kube-proxy and Service Binding:

    • The kube-proxy runs on each node and listens to the kube-apiserver for updates on services and endpoints. When a service is created or updated, kube-proxy configures the node's networking rules (e.g., iptables or IPVS) to route traffic to the appropriate pods.

    • The kube-apiserver informs kube-proxy of changes through the service and endpoint objects stored in etcd. kube-proxy watches for these changes and sets up the networking configuration.

controlplane $ kubectl get pod -n kube-system
NAME                                      READY   STATUS    RESTARTS      AGE
calico-kube-controllers-94fb6bc47-524db   1/1     Running   0             54s
canal-4bqsl                               2/2     Running   2 (47m ago)   4d8h
canal-8px9n                               2/2     Running   2 (47m ago)   4d8h
coredns-57888bfdc7-k8qpf                  1/1     Running   1 (47m ago)   4d8h
coredns-57888bfdc7-ns8f8                  1/1     Running   1 (47m ago)   4d8h
etcd-controlplane                         1/1     Running   2 (47m ago)   4d8h
kube-apiserver-controlplane               1/1     Running   2 (47m ago)   4d8h
kube-controller-manager-controlplane      1/1     Running   2 (47m ago)   4d8h
kube-proxy-ldvw7                          1/1     Running   2 (47m ago)   4d8h
kube-proxy-n7zdb                          1/1     Running   1 (47m ago)   4d8h
kube-scheduler-controlplane               1/1     Running   2 (47m ago)   4d8h

2. Configuration Files and Locations:

2.1 Resource configuration files:

  • kube-apiserver: Configuration typically found in /etc/kubernetes/manifests/kube-apiserver.yaml (static Pod on control plane nodes).

  • kube-controller-manager: /etc/kubernetes/manifests/kube-controller-manager.yaml (static Pod).

  • kube-scheduler: /etc/kubernetes/manifests/kube-scheduler.yaml (static Pod).

  • kubelet: /var/lib/kubelet/config.yaml for kubelet configurations, /etc/systemd/system/kubelet.service.d/10-kubeadm.conf for service options.

  • etcd: /etc/kubernetes/pki/etcd/ for certificates and keys, and /etc/kubernetes/manifests/etcd.yaml for manifest configuration.

  • kube-proxy: Configuration managed via ConfigMap (kube-system namespace), set during cluster setup.

2.2 Some special folders or files

/etc/kubernetes/
|-- admin.conf
|-- controller-manager.conf
|-- kubelet.conf
|-- manifests
|   |-- etcd.yaml
|   |-- kube-apiserver.yaml
|   |-- kube-controller-manager.yaml
|   `-- kube-scheduler.yaml
|-- pki
|   |-- apiserver-etcd-client.crt
|   |-- apiserver-etcd-client.key
|   |-- apiserver-kubelet-client.crt
|   |-- apiserver-kubelet-client.key
|   |-- apiserver.crt
|   |-- apiserver.key
|   |-- ca.crt
|   |-- ca.key
|   |-- etcd
|   |   |-- ca.crt
|   |   |-- ca.key
|   |   |-- healthcheck-client.crt
|   |   |-- healthcheck-client.key
|   |   |-- peer.crt
|   |   |-- peer.key
|   |   |-- server.crt
|   |   `-- server.key
|   |-- front-proxy-ca.crt
|   |-- front-proxy-ca.key
|   |-- front-proxy-client.crt
|   |-- front-proxy-client.key
|   |-- sa.key
|   `-- sa.pub
|-- scheduler.conf
`-- super-admin.conf


.kube/
|-- cache/
`-- config

  • manifests/ Folder:

    • This directory typically contains YAML or JSON manifest files for static Pods. On control plane nodes, this folder is often located at /etc/kubernetes/manifests/.

    • The kubelet watches this directory and ensures that the Pods defined here are running. This mechanism is essential for running critical system components of the cluster (e.g., kube-apiserver.yaml, etcd.yaml).

  • pki/ Folder:

    • The pki/ folder (usually located at /etc/kubernetes/pki/) contains certificates, private keys, and other security-related files used by Kubernetes components for secure communication.

    • Examples include:

      • CA certificate and key (ca.crt, ca.key): Used to sign other certificates within the cluster.

      • etcd certificates: (etcd/ca.crt, etcd/server.crt, etc.) for securing communication with etcd.

      • API server certificates (apiserver.crt, apiserver.key) for the kube-apiserver to communicate securely with other components and clients.

      • Front-proxy certificates: For communication between the kube-apiserver and aggregated API servers.

  • .kube/ Folder

    • The .kube/ folder is a directory that stores configuration files related to kubectl and Kubernetes. It usually contains:

      • ~/.kube/config:

        • config file: This is the main configuration file (~/.kube/config by default) that stores connection details, cluster information, user credentials, and contexts for kubectl. This file allows kubectl to communicate with the kube-apiserver by specifying the cluster’s API server address, authentication method, etc.

        • It is similar to /etc/kubernetes/admin.conf and often contains the same connection details.

    • Developers and administrators can use this file to switch between different Kubernetes clusters and user contexts by modifying or managing multiple configuration files.

  • /etc/kubernetes/

    • /etc/kubernetes/admin.conf:

      • This file is generated by kubeadm during cluster initialization and contains connection credentials for the kube-apiserver.

      • It is often copied to ~/.kube/config for easy kubectl access by the root or admin user.

    • /etc/kubernetes/kubelet.config:

      • /etc/kubernetes/kubelet.config is typically a static configuration file managed by kubeadm, containing default parameters for the kubelet.

      • /var/lib/kubelet/config.yaml is the dynamic runtime configuration file for the kubelet. This file is generated and updated based on the cluster configuration, and kubelet may read additional parameters from here during operation.

Kubernetes