Level 7: Multi-Cluster Kubernetes Orchestration

This example demonstrates defining an environment independent Kubernetes workload and deploying it to development and production clusters. The example runs on Amazon Web Services (AWS) or Microsoft Azure and so requires an account and API credentials. It available on github.

The infrastructure deployment consists of ServiceComponents representing:

The implementation of these components is based on the environment type, which is supplied as an input to the deployment. For example, if a development environment is selected, Minio is used as the object store and Minikube as the Kubernetes provider. If an AWS production environment is selected, S3 and EKS is used for each, respectively, whereas on Azure Blob and AKS is used.


Configuration Indirection

Multi-Kubernetes Cluster orchestration is achieved using a combination of the plugins mentioned above, and the design technique introduced in the Environment as a Service example. The technique uses a structured secret (a JSON object) to map the input environment type to component blueprint names. For example, to retrieve the name of the database blueprint, the following YAML is used in the main blueprint:

id: { get_secret: [ eaas_params, { get_input: cloud_type }, { get_input: env_type }, db, blueprint ] }


Service Composition

The configuration indirection mentioned above helps with mapping simple identifiers (like ‘dev-small’) to complex configuration details (like image names/id, flavors, etc..). This is not sufficient to completely abstract away the different Kubernetes environments required. To do this requires service composition. Service composition allows blueprint nodes to represent entire blueprints themselves, effectively nesting blueprints and enabling a building block approach. To make it possible to use components in the way required by this example, components of a similar kind (for example, blueprints that represent different kinds of database), all have a consistent interface. In Cloudify DSL, this interface is provided by the capabilitites section in the blueprint. This is analogous to the use of interfaces or protocols in an object oriented programming paradigm.

Looking at the multi-Kubernetes cluster example, consider the object storage options minio and S3. Both of these are represented by a blueprint that exposes a single capability: bucket_url. Because of this standard “interface”, the blueprints can be substituted for each other at deploy time. You will find the same pattern for other elements: minikube/EKS (endpoint), psql/RDS (host, master_username, master_password), and so on.

Kubernetes Multi-Cluster Use Case

Running the Example Implementation


This example expects the following prerequisites:

Install the Example

Our Environment-as-a-Service example on GitHub demonstrates a deploy-time selectable Kubernetes-based environment that includes object and relational storage external to the Kubernetes cluster. The three selectable environment types representing small and large development environments, and a production environment.

Path Name File Notes
infra/dev/minikube minikube blueprint.yaml
infra/dev/minio minio blueprint.yaml
infra/dev/multi_node multi_node aws-blueprint.yaml AWS only
infra/dev/multi_node multi_node azure-blueprint.yaml Azure only
infra/dev/psql psql aws-blueprint.yaml AWS only
infra/dev/psql psql azure-blueprint.yaml Azure only
infra/dev/single_node single_node aws-blueprint.yaml AWS only
infra/dev/single_node single_node azure-blueprint.yaml Azure only
infra/dev/vm vm aws-blueprint.yaml AWS only
infra/dev/vm vm azure-blueprint.yaml Azure only
infra/prod/eks eks blueprint.yaml AWS only
infra/prod/aks aks blueprint.yaml Azure only
infra/prod/prod_network prod_network aws-blueprint.yaml AWS only
infra/prod/prod_network prod_network azure-blueprint.yaml Azure only
infra/prod/rds_psql rds_psql aws-blueprint.yaml AWS only
infra/prod/rds_psql rds_psql azure-blueprint.yaml Azure only
infra/prod/s3 s3 aws-blueprint.yaml AWS only
infra/prod/s3 s3 azure-blueprint.yaml Azure only
infra/vpc vpc aws-blueprint.yaml AWS only
infra/rg vpc azure-blueprint.yaml Azure only
cfy deployments create app_dev_small -b app -i env_type=dev-small
cfy executions start install -d app_dev_small
cfy deployments create app_dev_large -b app -i env_type=dev-large
cfy executions start install -d app_dev_large
cfy deployments create app_prod -b app -i env_type=production
cfy executions start install -d app_prod