In the realm of Kubernetes, a multitude of resources are readily accessible and manageable through standard Kubernetes commands. Yet, when the need arises to extend Kubernetes capabilities for custom objects, Custom Resource Definitions (CRDs) emerge as a pivotal solution.
Kubernetes Custom Resource Definitions (CRDs) transform an object into a first-class citizen within the Kubernetes ecosystem. This means you can harness the full potential of standard Kubernetes tools to interact with and oversee these custom objects, just as you would with the built-in Kubernetes resources.
To witness Kubernetes in action, let’s embark on a journey to create our own Custom Resource Definition (CRD). In this endeavor, we’ll craft a CRD that serves as a repository for critical data about your microservice’s external service dependencies. These dependencies span a wide spectrum, including data storage components like databases and in-memory caches, as well as connections to vital cloud services, such as secrets managers like Key Vault in Microsoft Azure.
Our custom resource will be an object containing information about our external dependencies. We will capture the following information for our service:
To do this, we need to create a custom resource definition and apply it to our Kubernetes cluster using the
kubectl apply command. Then, we’ll create an instance of the custom resource and upload it using the
kubectl apply command.
To complete this walkthrough, you’ll need access to a Kubernetes environment. We are using a local environment powered by minikube.
CustomResourceDefinitions are YAML-formatted files. Here’s an example definition for our external dependencies resource structure. Save this as a file named
apiVersion: "apiextensions.k8s.io/v1" kind: CustomResourceDefinition metadata: name: dependencies.example.com spec: group: example.com scope: Namespaced names: plural: dependencies singular: dependency kind: Dependency shortNames: - dep versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: dependency: type: object properties: name: type: string minimum: 3 url: type: string minimum: 10 type: type: string minimum: 3
apiVersion specifies the version for the CRD extension itself. There are older versions (for example,
v1beta1) that uses slightly different YAML formats. Additionally, not all versions may be supported in your deployment of Kubernetes. So make sure the version and syntax specification align. In this example, we’re using CRD version v1.
metadata section scopes our CRD for API purposes. Kubernetes will automatically create API endpoints that enable basic CRUD (Create, Read, Update, Delete) operations on our objects. The
name property will become part of the path of our newly created APIs, which we’ll be able to access at
Next comes the specification for our CRD, defined under
spec. Before defining our object, we include some meta-information:
groupmust match the latter part of the namespace we chose under
scopedetermines whether our CRD is scoped to the cluster or scoped to our namespace.
namescontrols how we’ll refer to our objects in API calls and using the Kubernetes CLI.
We can also define a
shortName to help save on typing.
After this, we specify our CRD under
versions. We can publish multiple versions so that we don’t break backward compatibility with existing tools or CI/CD scripts. The
served flag means this version is enabled.
storage – must be used for the most recent version to designate it’s the version format used for storage in etcd. Kubernetes will convert requests for previous versions to the appropriate version spec on the fly.
schema, we define our custom resource using OpenAPI definition syntax. You’ll recognise this format if you’ve created OpenAPI specifications or used Swagger.
Now, we will discuss how you can use additional OpenAPI definition features to strengthen type safety and runtime parameter checking.
After creating our CustomResourceDefinition, we can upload it to our Kubernetes cluster using the
kubectl apply command:
kubectl apply -f dependencies.yaml
If your CRD is syntactically correct, Kubernetes will create it immediately. The CRD is simply a template for a custom resource. To store resources, you must create them in a separate YAML file and upload them to your cluster. The file below creates a
dependency object that conforms to our custom resource definition. Save this to a file named
apiVersion: "example.com/v1" kind: Dependency metadata: name: api-dependency dependency: name: storageapi url: https://example.com/storageapi/v1/ type: microservice
apiVersionfield is our spec group combined with the version of our custom resource definition.
namewill appear in URL paths, so Kubernetes enforces the use of RFC 1123 syntax. This means you must use only lowercase alphanumeric characters (a-z, 0-9) or the ‘-’ and ‘.’ characters.
dependencyand all of its fields must correspond to the OpenAPI definition we published earlier.
Save this to a file named
dependency-cr.yaml and upload it to Kubernetes:
kubectl apply -f dependency-cr.yaml
Once published, you can use
kubectl commands to manage your dependency objects. To list the object you just created, run the below command:
kubectl get dependencies
Amnics-MacBook-Pro: tmp amnic$ kubectl get dependencies NAME AGE api-dependency 9s
dependencies, we can use the
shortName we defined earlier:
kubectl get dep
You can use other basic
kubectl commands to manage your custom resource. To view the full details of the resource, run the below command:
kubectl get dep api-dependency -o yaml
To delete it, run the below command:
kubectl delete dep api-dependency
We can enhance our CRD by incorporating further restrictions on the values that can be assigned to specific fields. For instance, as previously mentioned, we will only permit three options for the
Add the below snippet to our file
enum: - datastore - cloudservice - microservice
After adding the above snippet to our file, it should look like:
type: string minimum: 3 enum: - datastore - cloudservice - microservice
You can force update your existing CRD like so:
kubectl replace -f dependencies.yaml
Now, try and publish a custom
dependency resource with an unsupported value for
type. Create another file,
dependency-cr-1.yaml, with the below content:
apiVersion: "example.com/v1" kind: Dependency metadata: name: api-dependency-1 dependency: name: storageapi url: https://example.com/storageapi/v1/ type: macroservice
Now, let’s try to apply the newly made changes:
kubectl apply -f dependency-cr-1.yaml
This command will result in an error indicating that
macroservice is not a supported type.
In this blog, we demonstrated how to set up and utilize custom resources in Kubernetes. In our next post, we will delve deeper by constructing a custom Kubernetes operator.