Kubernetes Architecture: Control Plane and Worker Nodes

Kubernetes architecture splits a cluster into two halves: a control plane that stores the desired state and makes decisions, and worker nodes that run your containers. Every component talks through a single API server, and every change follows the same loop. Something records what you want, controllers notice the gap between that and what exists, and agents on the nodes close the gap.

Knowing which component does what explains why a cluster keeps serving traffic when the control plane is down, why losing etcd is a disaster, and where to look when a Pod stays Pending.

The two halves of Kubernetes architecture

CONTROL PLANE
  kubectl --> kube-apiserver <--> etcd
              ^  scheduler
              ^  controller-manager
              ^  cloud-controller-manager (cloud only)
              |
              | watch assigned Pods, report status
              |
WORKER NODES (one set per node)
  kubelet --> container runtime (CRI) --> Pods
  kube-proxy --> Service routing rules

On a managed service such as EKS, GKE or AKS, the provider runs the control plane and you only see its API endpoint. On a kubeadm cluster, the control plane components run as static Pods in kube-system.

Control plane components

kube-apiserver: the only front door

The API server is a REST API, and everything goes through it: kubectl, the controllers, the scheduler and every kubelet. Each request passes the same pipeline:

1. Authentication: who is calling (certificate, token, OIDC).
2. Authorization: is this caller allowed to do this, usually through RBAC.
3. Admission: mutating admission webhooks and plugins can change the object, then validating ones can reject it.
4. Persistence: the object is written to etcd.

The API server also serves watches: components subscribe to changes instead of polling. It is stateless, so you scale it by running several replicas behind a load balancer. If it is unavailable, you can't change anything, but running Pods keep running.

etcd: the cluster's source of truth

etcd is a distributed key-value store that holds every object in the cluster: Deployments, Pods, Services, ConfigMaps, Secrets, node records. Only the API server talks to etcd directly. If etcd is lost without a backup, the cluster loses its memory of what should be running.

etcd uses the Raft consensus algorithm and needs a majority (quorum) of members to accept writes. That is why production clusters run 3 or 5 members: 3 tolerates one failure, 5 tolerates two. A fourth member adds no extra fault tolerance, because quorum rises from 2 to 3.

On a self-managed cluster, take snapshots on a schedule and test restoring them:

etcdctl snapshot save /backup/etcd-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Secrets are stored base64-encoded, not encrypted, unless you enable encryption at rest on the API server. Anyone who can read an etcd backup can read your Secrets.

kube-scheduler: decides where Pods go

The scheduler watches for Pods that have no node assigned. For each one it filters out nodes that can't host it (not enough requested CPU or memory, taints the Pod doesn't tolerate, node selectors and affinity rules that don't match), then scores the remaining nodes and picks the best. It records the decision by writing a binding to the API server. It never starts a container itself. Requests, limits and QoS drive most of this, and they get their own article on Kubernetes requests, limits and scheduling.

kube-controller-manager: the reconcile loops

This single binary runs many controllers: Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, Node lifecycle, EndpointSlice, ServiceAccount and more. Each one runs the same loop:

1. Observe the current state through watches.
2. Compare it with the desired state in the object's spec.
3. Act to close the gap, then write the result to status.

This loop is where self-healing comes from. Delete a Pod that belongs to a ReplicaSet and the ReplicaSet controller creates a replacement.

cloud-controller-manager: the link to your cloud

This component holds the cloud-specific logic: removing node objects when the VM is gone, configuring routes, and creating cloud load balancers for LoadBalancer Services. It only exists on a cloud provider. On bare metal, a LoadBalancer Service stays <pending> until something like MetalLB fills that role, as explained in Kubernetes Service types.

Worker node components

kubelet: the agent that does the work

The kubelet runs on every node. It watches for Pods assigned to its node and makes them real: it pulls images through the container runtime, mounts volumes, starts containers, runs probes and reports status back. When the node runs low on memory or disk, it evicts Pods to protect the node.

Container runtime and CRI

The kubelet doesn't run containers itself. It calls a container runtime through the Container Runtime Interface (CRI). Today that is almost always containerd or CRI-O. Built-in Docker Engine support (dockershim) was removed in Kubernetes 1.24. Images built with Docker still run without changes, because they are standard OCI images.

When a Pod starts, the runtime also calls the cluster's CNI plugin (Calico, Cilium, Flannel and others) to give the Pod its network interface and IP address.

kube-proxy: Service routing on each node

kube-proxy watches Services and EndpointSlices and programs rules on the node, using iptables, IPVS or the newer nftables mode, so that traffic sent to a Service's virtual IP lands on a healthy Pod. In these modes the kernel forwards the packets, and kube-proxy only keeps the rules up to date. Some CNI plugins, Cilium for example, can replace kube-proxy entirely with eBPF.

What happens when you run kubectl apply

Take a Deployment with three replicas and follow it through the cluster:

1. kubectl sends the manifest to the API server.
2. The API server authenticates, authorizes, runs admission, validates the object and stores it in etcd. kubectl prints deployment.apps/web created. At this point the object is stored, but nothing is running yet.
3. The Deployment controller sees the new Deployment and creates a ReplicaSet.
4. The ReplicaSet controller sees that the ReplicaSet wants 3 Pods and has none, so it creates 3 Pod objects with no node assigned.
5. The scheduler sees three unscheduled Pods, picks a node for each and writes the bindings.
6. The kubelet on each chosen node sees a Pod assigned to it. The runtime pulls the image, the CNI plugin assigns an IP, and the containers start.
7. Once the readiness probe passes, the kubelet marks the Pod Ready. The EndpointSlice controller adds its IP to the matching Service, and kube-proxy on every node updates its rules.

No component calls another directly. Each one watches the API server and reacts:

kubectl apply -f deployment.yaml
kubectl get events --watch
kubectl get pods -o wide --watch

That chain also tells you where to look when something is stuck:

Restart loops are their own topic. See debugging CrashLoopBackOff and probes.

Control traffic vs data traffic

Kubernetes architecture keeps two kinds of traffic apart:

• Control traffic manages the cluster: kubectl apply, kubelets reporting status, the scheduler writing bindings. All of it goes through the API server.
• Data traffic is your users' requests. A request goes from the load balancer to a node, where kube-proxy's rules forward it to a Pod. It never touches the API server or etcd.

This split matters for scale and for reliability. If every user request passed through the API server, it would become the bottleneck. And when the control plane goes down, your applications keep serving users.

Serving is not the same as healthy, though. While the control plane is down, nothing is rescheduled if a node dies, you can't deploy or scale, and Service endpoints stop updating, so a Pod that becomes unready stays in rotation. The kubelet can still restart crashed containers locally.

The Kubernetes networking model

The networking model has three basic rules:

• Every Pod gets its own IP address.
• Pods can reach every other Pod, on any node, without NAT.
• Agents on a node, such as the kubelet, can reach all Pods on that node.

The CNI plugin implements them. On top of Pod networking you get:

• Services: stable virtual IPs in front of changing Pod IPs.
• CoreDNS: names such as orders.shop.svc.cluster.local.
• NetworkPolicy: firewall rules between Pods. These only take effect if your CNI plugin supports them.
• Ingress: host- and path-based HTTP routing into the cluster.

Namespaces: organizing a shared cluster

A namespace is a scope for names and policies. Two teams can each have a Service called api in their own namespace. Namespaces are also where you attach RBAC permissions, ResourceQuotas, LimitRanges and NetworkPolicies:

apiVersion: v1
kind: Namespace
metadata:
  name: team-payments
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute
  namespace: team-payments
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.memory: 32Gi
    pods: "50"

Every cluster starts with default, kube-system (control plane and add-ons), kube-public and kube-node-lease (node heartbeats). Some resources are cluster-scoped and belong to no namespace: Nodes, PersistentVolumes, StorageClasses, ClusterRoles, CustomResourceDefinitions and Namespaces themselves. You can list them with kubectl api-resources --namespaced=false.

A namespace is not a security boundary on its own. By default, Pods in different namespaces can reach each other. For real isolation, combine namespaces with RBAC, NetworkPolicies and Pod Security Admission.

Inspecting a cluster's architecture

kubectl get nodes -o wide                  # nodes, versions, runtime
kubectl get pods -n kube-system            # control plane and add-ons
kubectl get --raw='/readyz?verbose'        # API server health checks
kubectl describe node <node-name>          # capacity, allocations, conditions

The same watch-and-reconcile pattern is how Kubernetes is extended: an Operator adds a custom resource type plus a controller for it. See Kubernetes Operators and CRDs.

FAQ

What is the difference between the control plane and worker nodes?

The control plane stores the cluster state and makes decisions: what should run and where. Worker nodes run the actual Pods. The kubelet on each node carries out the decisions the control plane records.

What happens if the Kubernetes control plane goes down?

Running Pods keep running and keep serving traffic. You can't deploy, scale or change anything, Pods on a failed node are not rescheduled, and Service endpoints stop updating until the control plane is back.

Why does etcd need an odd number of members?

etcd needs a majority to accept writes. Three members tolerate one failure and five tolerate two. A fourth member raises the quorum without letting the cluster survive any more failures.

Does Kubernetes still use Docker?

Not as its runtime. Since version 1.24, Kubernetes talks to containerd or CRI-O through CRI. Images built with Docker are OCI images and run unchanged.

Is a namespace a security boundary?

Not by itself. It scopes names, RBAC and quotas, but network traffic between namespaces is allowed until you add NetworkPolicies.

Key takeaways

• The API server is the only entry point, and etcd is the only store. Protect and back up etcd.
• Controllers and the scheduler only write to the API server. The kubelet and the runtime do the actual work on the nodes.
• kubectl apply succeeding means the object is stored, not that it's running. Follow the chain: controller, scheduler, kubelet, endpoints.
• User traffic bypasses the control plane, so an outage there stops changes, not serving.
• Namespaces organize a cluster. Isolation needs RBAC and NetworkPolicy too.

If you want to practice these concepts on a real cluster, Vectorkub's free DevOps courses walk through them step by step.