TLDR: This blog post provides a comprehensive overview of custom resources, custom resource definitions (CRDs), and custom controllers in Kubernetes, explaining their roles, how they extend Kubernetes functionality, and the process of deploying them.
In this blog post, we will explore the concepts of custom resources, custom resource definitions (CRDs), and custom controllers in Kubernetes. These components are essential for extending the functionality of Kubernetes beyond its native resources. We will discuss their roles, how they work together, and the process of deploying them in a Kubernetes cluster.
| resource | custom resource (virtual service) | |
| creation | deployment.yml | virtual service.yml |
| validated | resource definition (RD) | custom resource definition (CRD) |
| extended the feature of K8s | custom resource definition (CRD) |
| devops | user |
| CRD → deployed to K8s (doc) →using Manifest / Helm / operator ——→ validated CR created by user, any issue in CR , then it is not allowed to be created | virtual service.yml → create CR → namespace |
| CC(custom controller) → deployed to namespace → using Manifest / Helm / operator ——→ verify CR created by user ,and controller will perform required action |
| custom controller → initially created using Golang → client-go |
| client-go → watches → custom watchers (CRUD operation) →written in framework( kubernetes controller runtime) |
| custom watchers → configure for CR → whenever a new CR (virtual service) is created/updated/deleted → it is pushed to FIFO worker queue → then process each resource one by one → creating required functionality in K8s |
What are Custom Resources and CRDs?
Kubernetes comes with a set of built-in resources such as deployments, services, pods, config maps, and secrets. However, there are scenarios where you may need to introduce new resources to meet specific requirements. This is where custom resources and CRDs come into play.
Custom Resource Definitions (CRDs)
A Custom Resource Definition (CRD) is a way to extend the Kubernetes API by defining a new type of resource. When you create a CRD, you specify the schema for the new resource, which includes the fields and their types. This allows users to create instances of this new resource, known as custom resources.
For example, if a company like Istio wants to enhance Kubernetes capabilities, they would create a CRD that defines a new resource type, such as a "virtual service." This CRD will specify what fields are available for users to define when they create a custom resource of that type.
Custom Resources
A custom resource is an instance of a CRD. Once a CRD is defined and deployed in a Kubernetes cluster, users can create custom resources based on that definition. For instance, a user might create a custom resource that represents a specific virtual service configuration in Istio.
The Role of Custom Controllers
While CRDs allow you to define new resource types, custom controllers are responsible for managing the lifecycle of these custom resources. A controller watches for changes to custom resources and takes action based on those changes.
How Custom Controllers Work
When a custom resource is created, updated, or deleted, the custom controller detects these changes and performs the necessary actions to ensure the desired state is maintained. For example, if a user creates a virtual service custom resource, the custom controller will read the resource and configure the necessary settings in the Kubernetes cluster to implement the desired behavior.
The Process of Extending Kubernetes with CRDs and Custom Controllers
To effectively use custom resources and controllers, there are three main steps:
Deploy the Custom Resource Definition (CRD): The DevOps engineer deploys the CRD to the Kubernetes cluster. This defines the new resource type and its schema.
Deploy the Custom Controller: After the CRD is in place, the custom controller must be deployed. This controller will watch for changes to the custom resources and manage their lifecycle.
Create Custom Resources: Users can now create instances of the custom resource defined by the CRD. The custom controller will take action based on these resources.
Example: Istio and Virtual Services
To illustrate this process, let’s consider Istio, a popular service mesh solution. When an organization decides to use Istio, the following steps are typically taken:
Deploy the Istio CRD: The DevOps engineer follows the Istio documentation to deploy the CRD for virtual services. This defines the schema for virtual services in the Kubernetes API.
Deploy the Istio Custom Controller: The engineer then deploys the Istio custom controller, which will manage the virtual service resources.
Create Virtual Service Custom Resources: Users can now create virtual service custom resources in their namespaces, specifying the desired configurations.
Writing Custom Controllers
While deploying CRDs and custom controllers is often the responsibility of DevOps engineers, writing custom controllers typically requires programming knowledge, with Go being the most popular language for this purpose. The Kubernetes ecosystem provides libraries like client-go to facilitate the development of custom controllers.
High-Level Overview of Writing a Custom Controller
Set Up Watchers: Custom controllers need to set up watchers to monitor changes to custom resources. This allows the controller to react to events such as creation, updates, or deletions.
Process Events: When an event is detected, the controller processes the event and takes the necessary actions to ensure the desired state of the resource is achieved.
Use Frameworks: Developers can use frameworks like the Kubernetes controller-runtime to simplify the process of writing custom controllers.
istio installation using helm
Conclusion
Custom resources, CRDs, and custom controllers are powerful features in Kubernetes that allow users to extend its capabilities. By understanding how these components work together, DevOps engineers and developers can enhance their Kubernetes clusters to meet specific application needs. Whether you are deploying existing solutions like Istio or developing your own custom controllers, mastering these concepts is essential for effective Kubernetes management.