Kubernetes Operator Study Journey

1. Try using existing operators

2. Understand what is Kubernetes operator.

Kubernetes Controller components.
How Kubernetes Controller works.
Custom Resource.

Kubernetes Operator

A Kubernetes operator is an application-specific controller that extends the functionality of the Kubernetes API to create, configure, and manage instances of complex applications on behalf of a Kubernetes user.

From What is a Kubernetes operator?

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

From https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

An Operator is like an automated Site Reliability Engineer for its application.

From Kubernetes Operators ~ Automating the Container Orchestration Platform ~

Operator vs. Controller

Controller (custom controller): a component that manages custom resources by running a control loop (reconciliation loop).

Operator: a CRD and custom controller that automate management of specific software, such as an etcd or MySQL operator.

Translated from A Practical Introduction to Kubernetes Custom Controllers

Controllers can act on core resources such as deployments or services, which are typically part of the Kubernetes controller manager in the control plane, or can watch and manipulate user-defined custom resources.

Operators are controllers that encode some operational knowledge, such as application lifecycle management, along with the custom resources defined in Chapter 4.

From Programming Kubernetes

A controller is a loop that reads desired state ("spec), observed cluster state (others' "status"), and external state, and the reconciles cluster state and external state with the desired state, writing any observations down (to our own "status").

All of Kubernetes functions on this model.

An operator is a controller that encodes human operational knowledge: how do I run and manage a specific piece of complex software.

All operators are controllers, but not all controllers are operators.

From Tutorial: Zero to Operator in 90 Minutes! - Solly Ross, Google (YouTube)

For more detail: - CNCF Operator White Paper - Final Version - CNCF White Paper

3. Create a sample operator following a tutorial

There are several ways to create an operator. You can try any of them:

You can also reference example controllers:

4. Understand more detail about each component

The walkthroughs and diagrams target client-go v0.37.1 and controller-runtime v0.25.1 from go.mod. They include implementation structure, construction paths, and runnable examples.

Simplified:

Detailed:

More Detailed:

from https://github.com/kubernetes/sample-controller/blob/master/docs/images/client-go-controller-interaction.jpeg

client-go:
1. clientset is a client for the built-in API resources.
2. informer: watch the changes of objects and reflect the changes to the in-memory-cache.
  1. factory: informers.NewSharedInformerFactory
  2. watcher
  3. lister
  4. indexer
  5. event handler
  6. reflector
3. lister: Get data from in-memory cache.
4. indexer: in-memory cache
5. workqueue: A queue to store items that the controller will process.
code-generator:
1. Generate codes for clientset for a custom resource.
apimachinery:
1. Scheme: connects Kubernetes API and Go Types, API version conversion
controller-runtime
1. builder
2. cache
3. client
4. cluster
5. controller
6. handler
7. inject
8. log
9. manager
10. reconciler
11. source
12. webhook

Reference:

5. Create your own operator

After creating a sample operator, you should have deeper understanding of Kubernetes operator. Now you can think about what kind of problem that you want to resolve by utilizing operator pattern.

To clarify a problem to resolve with a new operator, you can reference existing operators:

operator	role	language
prometheus-operator	Manage Prometheus, Alertmanager and their configuration	Golang
mysql-operator	Manage MySQL cluster	Python
postgres-operator	Manage PostgreSQL cluster (version upgrade, live volume resize, ...)	Golang
strimzi-kafka-operator	Manage Kafka cluster, user, and topic	Java
...	...	...

Considerations:

Finalizer
docs: Owners and Dependents: example
Reconciliation Loop
- [operator-sdk] Based on the return value of Reconcile() the reconcile Request may be requeued and the loop may be triggered again: (Building a Go-based Memcached Operator using the Operator SDK)
```
// Reconcile successful - don't requeue
return reconcile.Result{}, nil
// Reconcile failed due to error - requeue
return reconcile.Result{}, err
// Reconcile again after a deliberate delay
return reconcile.Result{RequeueAfter: time.Minute}, nil
```
- https://github.com/operator-framework/operator-sdk/issues/4209#issuecomment-729916367
- How can I have separate logic for Create, Update, and Delete events? When reconciling an object can I access its previous state? -> You should not have separate logic. Instead design your reconciler to be idempotent.
  - Q: How do I have different logic in my reconciler for different types of events (e.g. create, update, delete)? in controller-runtime
Testing
- KUbernetes Testing TooL (kuttl): https://kuttl.dev/ KUTTL is built to support some kubernetes integration test scenarios and is most valuable as an end-to-end (e2e) test harness.
- Ginkgo (A Golang BDD Testing Framework): https://onsi.github.io/ginkgo/
- Gomega (Ginkgo's preferred matcher library): https://onsi.github.io/gomega/
- kubetest2: https://github.com/kubernetes-sigs/kubetest2: Kubetest2 is the framework for launching and running end-to-end tests on Kubernetes. It is intended to be the next significant iteration of kubetest.
Managing Errors
- https://cloud.redhat.com/blog/kubernetes-operators-best-practices
- Return the error in the status of the object.
- Generate an event describing the error.
Webhook
- Admission Webhook
  - two types:
    - Mutating Webhook: Make some modifications for a request. e.g. set default value. (Defined with MutatingAdmissionConfiguration)
    - Validating Webhook: Validate a request. (Defined by ValidatingAdmissionConfiguration)
  - Request: AdmissionReview
  - Response: AdmissionReview with response.allowed boolean field.
- Conversion Webhook
  - hub & spoke
  - isConvertible: need to have Hub & all non-Hub types must be able to convert to/from Hub
  - request & response: ConversionReview
Indexing

6. Tools

https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil
https://github.com/spf13/cobra: a library for creating powerful modern CLI applications & a program to generate applications and command files.
- Cobra is used in many Go projects such as Kubernetes, Hugo, and Github CLI to name a few. This list contains a more extensive list of projects using Cobra.

7. Study Golang for better code quality

8. Other Topics

Write Kubernetes Operator in other languages
- kopf for Python
- fabric8io/kubernetes-client for Java
- java-operator-sdk/java-operator-sdk Build Kubernetes Operators in Java Without Hassle
Optimistic Concurrency Control

9. Keep learning

10. Example test inventory

The runnable examples are intentionally split between unit tests and Kubernetes-backed tests:

Example	Live API server	Coverage
`client-go/clientset`	Yes	E2E target (API access)
`client-go/deltafifo`	No	Unit/fake objects
`client-go/indexer`	No	Unit/in-memory indexer
`client-go/informer`	Yes	E2E target; cache sync and Pod add/update/delete
`client-go/lister`	No	Unit/fake indexer
`client-go/listerwatcher`	Yes	E2E target; list/watch delivery
`client-go/reflector`	No runnable binary	Covered through informer/list-watch concepts
`client-go/workqueue`	No runnable binary	Unit-level concept
`controller-runtime/cache`	Yes	E2E target; cache sync and read
`controller-runtime/example-controller`	Yes	E2E target; ReplicaSet `pod-count` reconciliation
`controller-runtime/manager`	Yes	E2E target; manager startup and watches
`controller-runtime/reconciler`	No	Unit test/example
`controller-runtime/source`	Yes, with its example CRD	Documented live-cluster dependency; no remote CRD is applied by CI
`controller-runtime/log`	No	Process/unit example
`controller-runtime/webhook`	Not by the standalone example	Unit/process example; it does not register an API-server webhook

The live-cluster suites are selected independently as client-go and controller-runtime targets. CI creates a temporary kind cluster for each selected target. Developers can run either target against an existing cluster by setting KUBECONFIG; the runner removes only resources created by the test and never deletes a developer-provided cluster.