☰

Kubernetes Plugin

With the Cloudify Kubernetes Plugin you can define Kubernetes resources in your blueprints.

Plugin Requirements

Compatibility

Authentication

There are two authentication methods:

Token-Based Authentication

Cloudify Kubernetes Plugin v2.1.0+ includes support for token-based authentication.

Note: Kubernetes client certificates are based on the private IP Address of the cluster node. You must use token-based authentication to manage a remote cluster via the public IP address.

Generate Authentication Token

Note: If you install the example cluster, then this is set up for you .

To generate your authentication token, you must:

  1. Create a Service Account and Cluster Role Binding:

a. Create a sa-crb.yaml file on your Kubernetes Master. 

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: examples-user
      namespace: default
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
      name: examples-user
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
    - kind: ServiceAccount
      name: examples-user
      namespace: default
b. Install the Service Account and Cluster Role Binding: 
    $ kubectl create -f sa-crb.yaml
    ...
3. Now extract the token: 
$ kubectl -n default describe secret $(kubectl -n default get secret | grep examples-user | awk '{print $1}') | grep 'token:' | awk '{print $2}'
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InJlZ3VsYXItdXNlci10b2tlbi1qeHhoNSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJyZWd1bGFyLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiMGE3MzBiOC0yMTM5LTExZTgtODAxZC00MjAxMGEwYjBjMDQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpyZWd1bGFyLXVzZXIifQ.m06FHyC8TbKZ1bcnxIV_JKpKrADIOYDN4BqEcTMR947fzzfTzU8QiVjYJQF4kCgAR1rC3dNYcQI8rtmwLJg3ttmAoFi_myi38Mb6JyW19vMjxUx3BK8xuiXhcReQyEt0X50koSminwQbqFqMNbtGtODqIyjfe-ePfbdbTV57n16YdtKrhpHuifkWhD26Vyskj1BWs7jmfzPmb8Q7ttKHEIsEgxjTjFxhRPMzp-UxeH1pLnd36tnfUxU9v-6dHCzJUIlYpu-IahhQmTvf5sK5eClT2h3bGJzMtDA2oji_0kFWJ0yemeJuOXX4fNNSeRo9lPPCQIlz1gBNPvSHQngwgQ
4. Copy this token and create a secret on your Cloudify Manager with it: 
$ cfy secrets create kubernetes_token -s eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InJlZ3VsYXItdXNlci10b2tlbi1qeHhoNSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJyZWd1bGFyLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiMGE3MzBiOC0yMTM5LTExZTgtODAxZC00MjAxMGEwYjBjMDQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpyZWd1bGFyLXVzZXIifQ.m06FHyC8TbKZ1bcnxIV_JKpKrADIOYDN4BqEcTMR947fzzfTzU8QiVjYJQF4kCgAR1rC3dNYcQI8rtmwLJg3ttmAoFi_myi38Mb6JyW19vMjxUx3BK8xuiXhcReQyEt0X50koSminwQbqFqMNbtGtODqIyjfe-ePfbdbTV57n16YdtKrhpHuifkWhD26Vyskj1BWs7jmfzPmb8Q7ttKHEIsEgxjTjFxhRPMzp-UxeH1pLnd36tnfUxU9v-6dHCzJUIlYpu-IahhQmTvf5sK5eClT2h3bGJzMtDA2oji_0kFWJ0yemeJuOXX4fNNSeRo9lPPCQIlz1gBNPvSHQngwgQ
Secret `kubernetes_token` created

Reference Authentication Token in a Blueprint

The client config-configuration-api_option dictionary supports the following values:

The following is an example blueprint using token-based authentication:

  nginx_deployment:
    type: cloudify.kubernetes.resources.Deployment
    properties:
      client_config:
        configuration:
          api_options:
           host: { get_input: kubernetes_api_server }
           api_key: { get_secret: kubernetes_token }
           debug: false
           verify_ssl: false
      definition:
        apiVersion: extensions/v1beta1
        kind: Deployment
        metadata:
          name: nginx-deployment
        spec:
          selector:
            matchLabels:
              app: nginx
          replicas: 2
          template:
            metadata:
              labels:
                app: nginx
            spec:
              containers:
              - name: nginx
                image: nginx:1.7.9
                ports:
                - containerPort: 80
      options:
        grace_period_seconds: 5
        propagation_policy: 'Foreground'
        namespace: 'default'

The following is an example of secure token based authentication:


  nginx_deployment:
    type: cloudify.kubernetes.resources.Deployment
    properties:
      client_config:
        configuration:
          api_options:
            host: { get_input: kubernetes_api_server }
            api_key: { get_secret: kubernetes_token }
            ssl_ca_cert: { get_input: kubernetes_token_ca_cert }
            debug: true
            verify_ssl: true
      definition:
        apiVersion: extensions/v1beta1
        kind: Deployment
        metadata:
          name: nginx-deployment
        spec:
          selector:
            matchLabels:
              app: nginx
          replicas: 2
          template:
            metadata:
              labels:
                app: nginx
            spec:
              containers:
              - name: nginx
                image: nginx:1.7.9
                ports:
                - containerPort: 80
      options:
        grace_period_seconds: 5
        propagation_policy: 'Foreground'
        namespace: 'default'

You can then provide and input from an inputs file like this:

kubernetes_token_ca_cert: |
    -----BEGIN CERTIFICATE-----
    MIIDKzCCAhOgAwIBAgIRALyDoSRzP4gCM2ni3NhJD/wwDQYJKoZIhvcNAQELBQAw
    ...
    ...
    -----END CERTIFICATE-----

For more information on generating a token and authorizing a service account, please review the following pages in Kubernetes documentation:

Kube Config Authentication

Authentication with the Kubernetes Plugin is via a node that represents the Kubernetes master. The config should be a Kube Config style object.

One of four methods options can be used to provide the configuration:

The token may have expired, in which case it will access the file “/etc/cloudify/.kube/config” if the file exists it will return an error. Example1:

  kubernetes_master:
    type: cloudify.kubernetes.nodes.Master
    properties:
      configuration:
        apiVersion: v1
        kind: Config
        preferences: {}
        current-context: kubernetes-admin@kubernetes
        clusters:
        - name: kubernetes
          cluster:
            certificate-authority-data: { get_input: kubernetes_certificate_authority_data }
            server: { concat: [ 'https://', { get_input: kubernetes_master_ip}, ':', { get_input: kubernetes_master_port } ] }
        contexts:
        - name: kubernetes-admin@kubernetes
          context:
            cluster: kubernetes
            user: kubernetes-admin
        users:
        - name: kubernetes-admin
          user:
            client-certificate-data: { get_input: kubernetes-admin_client_certificate_data }
            client-key-data:{ get_input: kubernetes-admin_client_key_data }

When you deploy Kubernetes Cluster with Cloudify Simple Kubernetes Blueprint or Cloudify Kubernetes Provider, secrets containing the configuration are created.

Example2:

This is an example for authentication with kubeconfig file content:

 inputs:

  configuration_file_content:
    type: string

node_templates:

  sanity_pod:
    type: cloudify.kubernetes.resources.Pod
    properties:
      client_config:
        configuration:
            file_content: { get_input: configuration_file_content }
      definition:
        apiVersion: v1
        kind: Pod
        metadata:
          name: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:stable

From version 2.7.0 of kubernetes plugin, every kubernetes resource can have “client_config” property, with the configuration and authentication credentials. Therefore, it’s recommended to remove the “managed_by_master” relationship in the last example and add the client_config property to the storage class resource. In order to get a sense of how using this is been done you can view our kubernetes examples. On future releases of the kubernetes plugin the “managed_by_master” relationship will be removed.

Release History

The information in this documentation is current for Cloudify Kubernetes Plugin version 2.7.0.

See releases.

Example

This example demonstrates demonstrates a basic node template usage.

  my_application:
    type: cloudify.kubernetes.resources.MultipleFileDefinedResources
    properties:
      files:
        - resource_path: resources/my_app_service.yaml
        - resource_path: resources/my_app_pod.yaml
    relationships:
      - type: cloudify.kubernetes.relationships.managed_by_master
        target: master

  master:
    type: cloudify.kubernetes.nodes.Master
    properties:
      configuration:
        file_content: { get_input: kubernetes_configuration_file_content }

Many more examples are available here.

Types

cloudify.kubernetes.nodes.Master

This node represents an existing Kubernetes master.

Properties:

cloudify.kubernetes.resources.SharedCluster

Derived from cloudify.nodes.SharedResource.

Enables Kubernetes node templates in the current deployment to authenticate with an existing Kubernetes Cluster in another deployment.

The other deployment should meet the following requirements:

Note: Node templates that are connected to this node template for authentication do so with the cloudify.relationships.kubernetes.connected_to_shared_cluster relationship. This will refresh the Kube Config when it is called.

Properties:

Example:

  Pod:
    type: cloudify.kubernetes.resources.FileDefinedResource
    properties:
      file:
        resource_path: resources/pod.yaml
    relationships:
      - type: cloudify.relationships.kubernetes.connected_to_shared_cluster
        target: token

  token:
    type: cloudify.kubernetes.resources.SharedCluster
    properties:
      client_config:
        configuration: { get_capability: [ { get_input: deployment }, connection_details ] }
      resource_config:
        deployment:
          id: { get_input: deployment }

cloudify.kubernetes.resources.ResourceBase

This is the base type of kubernetes resource.

Properties:

Note: Not required if “managed_by_master” relationship is being used. It’s not recommended to use this relationship because on next releases it will be deprecated.

cloudify.kubernetes.resources.ResourceWithValidateStatus

This is the base type of kubernetes resource with validate_resource_status property.

derived_from cloudify.kubernetes.resources.ResourceBase.

Properties:

*validate_resource_status

   If this property set to "true" Cloudify will wait that the resource will be in ready state before moving on.
   If the resouce isnt ready Cloudify will validate its state again(operation retry).


    **type:** boolean

    **default:** false

cloudify.kubernetes.resources.FileDefinedResource

This is a Kubernetes resource, such as a pod, service, deployment, which is defined in a file.

derived_from cloudify.kubernetes.resources.ResourceBase.

Properties:

has these keys:

1. `resource_path`: The path to the file relative to the blueprint.
2. `target_path`: The path the file should be downloaded to. (Don't use this.)
3. `template_variables` If the file has Jinja template variable references, you may provide the variables as a dictionary here.

You can look at example here.

cloudify.kubernetes.resources.MultipleFileDefinedResources

Provide a list of files containing Kubernetes resources.

derived_from cloudify.kubernetes.resources.ResourceBase.

Properties:

cloudify.kubernetes.resources.BlueprintDefinedResource

This is the root type of all Kubernetes resource, such as a pod, service, deployment, which is defined in the blueprint inline. derived_from cloudify.kubernetes.resources.ResourceWithValidateStatus.

Properties:

Derived resource types:

Check the plugin.yaml for latest additions.

More kubernetes resources node types

All the resources in the list are derive from “cloudify.kubernetes.resources.ResourceBase” node type and are “inline style” resources.

For more information, check plugin.yaml.

Deletion Propagation

Some Kubernetes resources create other Kubernetes resources. If you delete them, the default behavior of the Kubernetes Python library is to orphan those resources. To prevent this, create a propagation policy:

  nginx_deployment:
    type: cloudify.kubernetes.resources.Deployment
    properties:
      ...
      options:
        grace_period_seconds: 5
        propagation_policy: 'Foreground'
        namespace: 'default'
    relationships:
      - type: cloudify.kubernetes.relationships.managed_by_master
        target: kubernetes_master

cloudify.kubernetes.resources.CustomBlueprintDefinedResource

This allows you to define a resource type that is supported by the Kubernetes API, but has not yet been incorporated into the Kubernetes plugin.

Properties:

Note: Not required if “managed_by_master” relationship is being used. It`s not recommended to use this relationship because on next releases it will be deprecated.

Example:

The plugin can be easily extended by referencing create, read, and delete api mappings for these supported APIs.

This is an example of a custom blueprint defined resource:

node_types:

  cloudify.kubernetes.resources.PersistentVolumeClaim:
    derived_from: cloudify.kubernetes.resources.CustomBlueprintDefinedResource
    properties:
      api_mapping:
        default:
          create:
            api: CoreV1Api
            method: create_namespaced_persistent_volume_claim
            payload: V1PersistentVolumeClaim
          read:
            api: CoreV1Api
            method: read_namespaced_persistent_volume_claim
          delete:
            api: CoreV1Api
            method: delete_namespaced_persistent_volume_claim
            payload: V1DeleteOptions

Using file defined resources

Kubernetes plugin supports define resources from type : “cloudify.kubernetes.resources.FileDefinedResource”. This option allows the user to write the kubernetes resource definition in a template yaml file and refer to it from the blueprint. like in this example.

We are strongly recommend to define your kubernetes resources in that way and not in the “inline” defenition style(our plugin supports both methods). Here is an inline style resource defenition example.

Workflows

In addition to support for built-in workflows, the Kubernetes Plugin supports the following additional workflows:

update_resource_definition

Updates the resource definition of a cloudify.kubernetes.resources.BlueprintDefinedResource.

Parameters

Update Resource Definition Example

Let’s say that you created an nginx pod with the following blueprint resource definition:

  nginx:
    type: cloudify.kubernetes.resources.Pod
    properties:
      definition:
        apiVersion: v1
        kind: Pod
        metadata:
          name: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:stable
    relationships:
      - type: cloudify.kubernetes.relationships.managed_by_master
        target: master

You specified “stable” as the version. Let’s say that you want to update the version.

You would do so like this:

cfy executions start update_resource_definition -d pod -vv -p resource_definition_changes="
{
  'spec': {
    'containers': [{
      'name': 'nginx',
      'image': 'nginx:latest',
    }],
  }
}" -p node_instance_id=nginx_9pqgdu

refresh_and_store_token

Some Kubernetes clusters, e.g. EKS, may have a TTL on the lifetime of a token. Use this workflow to refresh the token that is stored in a secret.

Parameters

refresh and store token command Example

cfy executions start refresh_and_store_token -d eks -p kubernetes_cluster_node_instance_id=eks_cluster_fa7cbe -p deployment_capability_name=connection_details -p service_account_node_instance_id=new_service_account_qzpvr7 -p secret_token_node_instance_id=secret_akvafl -p store_token_and_kubeconfig_id=store_token_and_kubeconfig_jvnp1h

Using the kubernetes plugin with EKS, AKS, GKE

The kubernetes plugin works with EKS, AKS, GKE.

EKS cluster

On blueprint examples repository, there is an example of deploying an EKS cluster.

this example demonstrates a deployment of eks cluster with one node group. We will explain how we used the AWS plugin alongside kubernetes plugin on this example in order to deploy the cluster.

Firstly, in order to create an eks cluster, the blueprint creates all necessary infrastructure, i.e. VPC, subnets, security groups, etc. After creating the cluster, Cloudify stores the kube config in runtime properties. For more information about those aws instances see AWS plugin documentation.

After we have the eks cluster and the node group deployed, the blueprint defines service account and token node types whose purpose is to generate a token that can be used for simpler plugin authentication. Then, create a sanity_master node instance:

 sanity_master:
    type: cloudify.kubernetes.nodes.Master
    properties:
      configuration: &sanity_master_configuration
        api_options:
          host:  { get_attribute: [eks_cluster, kubeconf, clusters, 0, cluster, server ] }
          api_key: { get_secret: kubernetes_token }
          debug: false
          verify_ssl: false
    relationships:
      - type: cloudify.relationships.depends_on
        target: store_token

The authentication is done with the api options using the service account token.

And now,using the kubernetes plugin it creates resources in the cluster like pods :

  sanity_pod:
    type: cloudify.kubernetes.resources.Pod
    properties:
      client_config:
        configuration: *sanity_master_configuration
      definition:
        apiVersion: v1
        kind: Pod
        metadata:
          name: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:stable
    relationships:
      - type: cloudify.relationships.depends_on
        target: sanity_master

GKE cluster

On examples repository, there is an example of deploying GKE cluster.

This example demonstrates a deployment of kubernetes cluster consists of one node pool(with 2 nodes) and one pod.

In order to deploy the cluster, GCP plugin alongside kubernetes plugin used.

On gcp, the cluster creation is done by using the cloudify.gcp.nodes.KubernetesCluster node type:

This node type defined on gcp plugin, for more details see gcp plugin doumentation.

Here is how it is done on the example blueprint:

  kubernetes-cluster:
    type: cloudify.gcp.nodes.KubernetesCluster
    properties:
      name: { concat: [ { get_input: resource_prefix }, '-cluster']}
      gcp_config: *gcp_config

After the cluster and node pool created, using the kubernetes plugin the blueprint defines a master node and authenticate to gcp using the gcp service account:

  sanity_master:
    type: cloudify.kubernetes.nodes.Master
    properties:
      authentication: &client_config_authentication
        gcp_service_account: { get_input: gcp_credentials }
      configuration: &client_config_configuration
        api_options:
          host: { concat: [ 'https://', { get_attribute: [kubernetes-cluster-attributes, endpoint] }]}
          verify_ssl: false
          debug: false
    relationships:
      - type: cloudify.relationships.depends_on
        target: kubernetes-cluster-attributes
      - type: cloudify.relationships.depends_on
        target: kubernetes-cluster-node-pool

Then, with the same credentials it creates a pod.

AKS cluster

On blueprints examples repository, there is an example of deploying an AKS cluster.

This example demonstrates a deployment of aks cluster with one node pool and three nodes inside the nodepool. In order to deploy the cluster, Azure plugin alongside kubernetes plugin used.

Firstly, in order to create an AKS cluster the blueprint creates resource group on azure. Then, it creates the aks managed cluster and save the kubeconfig content in its runtime properties. you can read more about those Azure instances in our Azure plugin documentation.

After AKS cluster and the node pool deployed, the blueprint defines service account and token node types whose purpose is to generate a token that can be used for simpler plugin authentication. Then it creates a sanity_master node instance:

sanity_master:
type: cloudify.kubernetes.nodes.Master
properties:
  configuration: &sanity_master_configuration
    api_options:
      host:  { get_attribute: [managed_cluster, kubeconf, clusters, 0, cluster, server ] }
      api_key: { get_secret: kubernetes_token }
      debug: false
      verify_ssl: false
relationships:
  - type: cloudify.relationships.depends_on
    target: store_token

The authentication is done with the api options using the service account token.

and now, using the kubernetes plugin it creates resource in the cluster(pod):

  sanity_pod:
    type: cloudify.kubernetes.resources.Pod
    properties:
      client_config:
        configuration: *sanity_master_configuration
      definition:
        apiVersion: v1
        kind: Pod
        metadata:
          name: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:stable
    relationships:
      - type: cloudify.relationships.depends_on
        target: sanity_master

CustomResourceDefinition and Custom Objects

The Cloudify Kubernetes Plugin supports creating and using CustomResourceDefinitions.

To create a CustomResourceDefinition, create a new Kubernetes Resource Node, like you would with any other supported resource type, by providing the path to a Kubernetes resource template in your node template:

  resource:
    type: cloudify.kubernetes.resources.FileDefinedResource
    properties:
      client_config: { get_input: **client_config }
      file:
        resource_path: resources/crd.yaml

In your crd.yaml file, provide the definition for your CustomDefinedResource.

You can also use CustomDefinedResources with the Kubernetes plugin. Usages of CustomDefinedResources are referred to as Custom Objects.

When using Custom Objects, you must add specific annotations to the metadata section of the resource.

Specify the following keys in your metadata:annotations section:

Cloudify needs these keys in order to identify the resource as a usage of a CustomDefinedResource. An alternative method of providing these keys is available when using the cloudify.kubernetes.resources.CustomObject node type. See the cloudify.kubernetes.resources.CustomObject example below.

Define a CustomDefinedResource and use it in the same Kubernetes template:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  # This value is used in the cloudify-crd-group annotation in the usage of the CRD.
  group: stable.example.com
  versions:
  # This value is used in the cloudify-crd-version annotation in the usage of the CRD.
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  scope: Namespaced
  names:
  # This value is used in the cloudify-crd-plural annotation in the usage of the CRD.
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct
---
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
  annotations:
    # See comments in the CRD defintion above for where to find these values.
    cloudify-crd-group: stable.example.com
    cloudify-crd-plural: crontabs
    cloudify-crd-version: v1
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image

Using the cloudify.kubernetes.resources.CustomObject node type:

  resource:
    type: cloudify.kubernetes.resources.CustomObject
    properties:
      client_config: { get_input: **client_config }
      options:
        group: stable.example.com
        plural: crontabs
        version: v1
      definition:
        apiVersion: "stable.example.com/v1"
        kind: CronTab
        metadata:
          name: my-new-cron-object
          annotations:
            # See comments in the CRD defintion above for where to find these values.
            cloudify-crd-group: stable.example.com
            cloudify-crd-plural: crontabs
            cloudify-crd-version: v1
        spec:
          cronSpec: "* * * * */5"
          image: my-awesome-cron-image

Relationship

cloudify.relationships.kubernetes.connected_to_shared_cluster

Connect Kubernetes resource node templates to an existing Kubernetes cluster. Target node type must be cloudify.kubernetes.resources.SharedCluster.