Kubernetes Operator and CRDs: Build a Reconcile Loop in Go

A Kubernetes operator is a custom controller that manages an application through its own resource type. You describe what you want in a custom resource, for example "a PostgreSQL cluster with three replicas on version 16", and the operator keeps creating, updating and repairing the underlying StatefulSets, Services and backups until reality matches that description. It encodes the knowledge a human operator would otherwise apply by hand.

Operators are built from two Kubernetes extension points: Custom Resource Definitions (CRDs), which add new types to the API, and controllers, which watch those types and act on them. This guide explains both, walks through the reconcile loop, and builds a small database operator in Go with kubebuilder and controller-runtime.

CRDs: adding your own resource types

Kubernetes ships with built-in kinds like Deployment and Service. A CRD registers a new kind with the API server. Once it's applied, the new type behaves like any other resource. You can kubectl get it, apply RBAC to it, and watch it for changes.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: postgresclusters.db.example.com
spec:
  group: db.example.com
  scope: Namespaced
  names:
    kind: PostgresCluster
    plural: postgresclusters
    singular: postgrescluster
    shortNames: [pgc]
  versions:
    - name: v1alpha1
      served: true
      storage: true
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Replicas
          type: integer
          jsonPath: .spec.replicas
        - name: Ready
          type: integer
          jsonPath: .status.readyReplicas
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [replicas, version, storage]
              properties:
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 5
                version:
                  type: string
                  enum: ["15", "16"]
                storage:
                  type: string
                  pattern: '^[0-9]+Gi$'
            status:
              type: object
              properties:
                readyReplicas:
                  type: integer

A few details matter here:

• The schema is validation. The API server rejects a resource with replicas: 9 before your controller ever sees it. Validate as much as you can in the schema.
• subresources.status splits the object into spec (what the user wants) and status (what the controller observed). Users write the spec, the controller writes the status, and they don't overwrite each other.
• additionalPrinterColumns controls what kubectl get shows.

Now a user can create a database with a short manifest:

apiVersion: db.example.com/v1alpha1
kind: PostgresCluster
metadata:
  name: orders-db
  namespace: shop
spec:
  replicas: 3
  version: "16"
  storage: 20Gi

$ kubectl get pgc -n shop
NAME        REPLICAS   READY
orders-db   3          2

On its own, a CRD does nothing. The API server stores the object in etcd and that's it. Something has to act on it.

Controllers and the reconcile loop

Every built-in resource already works this way. The Deployment controller inside kube-controller-manager watches Deployments and makes ReplicaSets match them. A custom controller does the same for your type. The Kubernetes architecture article covers where the built-in controllers run.

The core of any controller is the reconcile loop:

1. Observe. Read the desired state (the custom resource's spec) and the actual state (the objects that exist in the cluster).
2. Diff. Work out what's missing, extra or out of date.
3. Act. Create, update or delete objects to close the gap.
4. Report. Write what you observed into status.
5. Repeat whenever something relevant changes, and periodically as a safety net.

Two properties make this reliable:

• Level-triggered, not edge-triggered. Reconcile doesn't ask "what event just happened?" It asks "what should exist, and what does exist?" If the controller misses an event or restarts halfway through, the next reconcile still reaches the right state.
• Idempotent. Running reconcile ten times in a row with no changes must do nothing harmful. Always check before you create, and update only when something differs.

What makes it an operator

Operator = CRD + controller + operational knowledge. A generic controller might only create a StatefulSet. A database operator also knows how to:

• bootstrap a primary and join replicas to it,
• take scheduled backups and restore from them,
• detect a failed primary and promote a replica,
• perform a minor version upgrade one pod at a time,
• expose connection details as a Secret for applications.

These are the tasks an on-call engineer would otherwise run from a runbook. The Operator Framework describes a maturity model that runs from basic install, through seamless upgrades and full lifecycle management (backup, restore, failover), up to deep insights and auto-pilot tuning. Most in-house operators never need to go past the first few levels.

Choosing a framework

If your team writes Go, start with kubebuilder. The rest of this guide uses it.

Building a database operator with kubebuilder

Scaffold the project and the API:

kubebuilder init --domain example.com --repo github.com/acme/pg-operator
kubebuilder create api --group db --version v1alpha1 --kind PostgresCluster

Define the API as Go types

You don't write the CRD YAML by hand. You write Go structs with markers, and make manifests generates the CRD shown earlier.

// api/v1alpha1/postgrescluster_types.go
package v1alpha1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

type PostgresClusterSpec struct {
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=5
	Replicas int32 `json:"replicas"`

	// +kubebuilder:validation:Enum="15";"16"
	Version string `json:"version"`

	// +kubebuilder:validation:Pattern=`^[0-9]+Gi$`
	Storage string `json:"storage"`
}

type PostgresClusterStatus struct {
	ReadyReplicas int32              `json:"readyReplicas,omitempty"`
	Conditions    []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=pgc
// +kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=`.spec.replicas`
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
type PostgresCluster struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   PostgresClusterSpec   `json:"spec,omitempty"`
	Status PostgresClusterStatus `json:"status,omitempty"`
}

Write the reconciler

// internal/controller/postgrescluster_controller.go
package controller

import (
	"context"
	"fmt"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/meta"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"

	dbv1alpha1 "github.com/acme/pg-operator/api/v1alpha1"
)

type PostgresClusterReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=db.example.com,resources=postgresclusters,verbs=get;list;watch;update;patch
// +kubebuilder:rbac:groups=db.example.com,resources=postgresclusters/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete

func (r *PostgresClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var pg dbv1alpha1.PostgresCluster
	if err := r.Get(ctx, req.NamespacedName, &pg); err != nil {
		// Deleted: owned objects are garbage-collected via owner references.
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	labels := map[string]string{"app.kubernetes.io/instance": pg.Name}
	sts := &appsv1.StatefulSet{
		ObjectMeta: metav1.ObjectMeta{Name: pg.Name, Namespace: pg.Namespace},
	}

	_, err := controllerutil.CreateOrUpdate(ctx, r.Client, sts, func() error {
		if sts.CreationTimestamp.IsZero() {
			// Immutable fields: set only on create.
			sts.Spec.ServiceName = pg.Name
			sts.Spec.Selector = &metav1.LabelSelector{MatchLabels: labels}
			sts.Spec.VolumeClaimTemplates = []corev1.PersistentVolumeClaim{{
				ObjectMeta: metav1.ObjectMeta{Name: "data"},
				Spec: corev1.PersistentVolumeClaimSpec{
					AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce},
					Resources: corev1.VolumeResourceRequirements{
						Requests: corev1.ResourceList{
							corev1.ResourceStorage: resource.MustParse(pg.Spec.Storage),
						},
					},
				},
			}}
		}
		sts.Spec.Replicas = &pg.Spec.Replicas
		sts.Spec.Template.Labels = labels
		sts.Spec.Template.Spec.Containers = []corev1.Container{{
			Name:  "postgres",
			Image: fmt.Sprintf("postgres:%s", pg.Spec.Version),
			Ports: []corev1.ContainerPort{{Name: "pg", ContainerPort: 5432}},
			VolumeMounts: []corev1.VolumeMount{{
				Name: "data", MountPath: "/var/lib/postgresql/data",
			}},
		}}
		return controllerutil.SetControllerReference(&pg, sts, r.Scheme)
	})
	if err != nil {
		return ctrl.Result{}, err // requeued with back-off
	}

	pg.Status.ReadyReplicas = sts.Status.ReadyReplicas
	cond := metav1.Condition{
		Type: "Ready", Status: metav1.ConditionFalse,
		Reason: "Provisioning", ObservedGeneration: pg.Generation,
		Message: fmt.Sprintf("%d/%d replicas ready", sts.Status.ReadyReplicas, pg.Spec.Replicas),
	}
	if sts.Status.ReadyReplicas == pg.Spec.Replicas {
		cond.Status, cond.Reason = metav1.ConditionTrue, "AllReplicasReady"
	}
	meta.SetStatusCondition(&pg.Status.Conditions, cond)

	return ctrl.Result{}, r.Status().Update(ctx, &pg)
}

func (r *PostgresClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&dbv1alpha1.PostgresCluster{}).
		Owns(&appsv1.StatefulSet{}).
		Complete(r)
}

What this code gets right:

• CreateOrUpdate reads the current StatefulSet, applies the mutate function, and sends a create or update only when the result differs from what it read. That keeps reconcile idempotent.
• Immutable fields such as the selector and volume claim templates are set only on create. Changing them on an existing StatefulSet fails validation. The StatefulSet section of the workloads guide explains why.
• Owner references from SetControllerReference mean deleting the PostgresCluster garbage-collects the StatefulSet.
• Owns(&appsv1.StatefulSet{}) triggers a reconcile when the StatefulSet changes, for example when a pod becomes ready. That's how status stays current.
• Returning an error requeues the request with exponential back-off. Update conflicts resolve themselves on the next pass.

Run it against your current kubeconfig while developing:

make manifests   # regenerate CRD and RBAC from markers
make install     # apply the CRD to the cluster
make run         # run the controller locally

A real database operator would go further: create a headless Service and a Secret with credentials, configure replication, and add backups. The loop stays the same, with more objects reconciled in turn.

Cleaning up external resources with finalizers

Owner references only clean up objects inside the cluster. If your operator creates something outside, such as a backup bucket or a DNS record, add a finalizer. Kubernetes then won't remove the object until your controller has cleaned up and removed the finalizer:

const finalizer = "db.example.com/cleanup"

if !pg.DeletionTimestamp.IsZero() {
	if controllerutil.ContainsFinalizer(&pg, finalizer) {
		if err := r.deleteBackups(ctx, &pg); err != nil {
			return ctrl.Result{}, err
		}
		controllerutil.RemoveFinalizer(&pg, finalizer)
		return ctrl.Result{}, r.Update(ctx, &pg)
	}
	return ctrl.Result{}, nil
}
if controllerutil.AddFinalizer(&pg, finalizer) {
	return ctrl.Result{}, r.Update(ctx, &pg)
}

The same idea in Python with kopf

For smaller jobs, kopf lets you write handlers without any scaffolding:

import kopf

@kopf.on.create("db.example.com", "v1alpha1", "postgresclusters")
@kopf.on.update("db.example.com", "v1alpha1", "postgresclusters")
def reconcile(spec, name, namespace, logger, **_):
    replicas = spec["replicas"]
    logger.info(f"ensuring {name} has {replicas} replicas")
    # build and apply the StatefulSet with the kubernetes client here
    return {"desiredReplicas": replicas}

Run it with kopf run operator.py. kopf is handler-based rather than a single reconcile function, so you still need to keep each handler idempotent.

Before you write your own operator

• Check what exists. Mature operators already cover common stateful software: CloudNativePG and Zalando's postgres-operator for PostgreSQL, Strimzi for Kafka (see running Kafka with KRaft on Kubernetes), cert-manager for TLS certificates, and the Prometheus Operator for monitoring.
• Use Helm if there's no ongoing logic. If the job is "install these manifests with some values", a chart is simpler. An operator earns its cost when something must react continuously: failover, backups, scaling, rotation.
• Plan for its failure. The operator is now part of your control path. Run it with leader election, give it resource limits, and alert when reconciles keep failing.

FAQ

What is the difference between a CRD and an operator?

A CRD only defines a new resource type and its schema. An operator is a controller that watches that type and does the work to make the cluster match it. You can have a CRD without an operator, but it just stores data.

Is a Kubernetes operator the same as a controller?

Every operator is a controller, but not every controller is an operator. The term "operator" is used for controllers that manage a specific application's full lifecycle, such as backups, upgrades and failover, through custom resources.

Should I use kubebuilder or Operator SDK?

For Go, they produce nearly the same project, since Operator SDK uses kubebuilder underneath. Pick Operator SDK if you need OLM packaging or want to write the operator in Ansible or Helm. Otherwise kubebuilder is enough.

What happens if the operator crashes?

The workloads it created keep running, because they're ordinary Kubernetes objects. Nothing reconciles them until the operator comes back, so failover or scaling it handles won't happen during that time. When it restarts, level-triggered reconcile brings everything back in line.

Can I run a database on Kubernetes without an operator?

Yes, with a StatefulSet and persistent volumes. But you then handle replication, backups and failover yourself, as covered in running databases in production. For production databases, a mature operator usually removes a lot of manual work.

Takeaways

• A CRD adds a type. A controller reconciles it. An operator adds operational knowledge on top.
• Keep reconcile level-triggered and idempotent, validate in the schema, and report through status conditions.
• Use owner references for in-cluster cleanup and finalizers for anything external.
• Reach for an existing operator or a Helm chart before building your own.

If you'd like hands-on practice with Kubernetes internals like these, Vectorkub's free DevOps courses are a good place to start.