Level 7: Multi-Cluster Kubernetes Orchestration

This example demonstrates defining an environment independent Kubernetes workload and deploying it to development and production clusters. The example runs on Amazon Web Services (AWS) or Microsoft Azure and so requires an account and API credentials. It available on github.

The infrastructure deployment consists of ServiceComponents representing:

• network
• database
• object store
• kubernetes

The implementation of these components is based on the environment type, which is supplied as an input to the deployment. For example, if a development environment is selected, Minio is used as the object store and Minikube as the Kubernetes provider. If an AWS production environment is selected, S3 and EKS is used for each, respectively, whereas on Azure Blob and AKS is used.

Concepts

Configuration Indirection

Multi-Kubernetes Cluster orchestration is achieved using a combination of the plugins mentioned above, and the design technique introduced in the Environment as a Service example. The technique uses a structured secret (a JSON object) to map the input environment type to component blueprint names. For example, to retrieve the name of the database blueprint, the following YAML is used in the main blueprint:

id: { get_secret: [ eaas_params, { get_input: cloud_type }, { get_input: env_type }, db, blueprint ] }

Where

• eaas_params: the name of the secret containing the JSON object
• cloud_type: a cloud provider (aws, azure)
• env_type: one of the valid environment type names (e.g. dev-small, prod)

Service Composition

The configuration indirection mentioned above helps with mapping simple identifiers (like ‘dev-small’) to complex configuration details (like image names/id, flavors, etc..). This is not sufficient to completely abstract away the different Kubernetes environments required. To do this requires service composition. Service composition allows blueprint nodes to represent entire blueprints themselves, effectively nesting blueprints and enabling a building block approach. To make it possible to use components in the way required by this example, components of a similar kind (for example, blueprints that represent different kinds of database), all have a consistent interface. In Cloudify DSL, this interface is provided by the capabilitites section in the blueprint. This is analogous to the use of interfaces or protocols in an object oriented programming paradigm.

Looking at the multi-Kubernetes cluster example, consider the object storage options minio and S3. Both of these are represented by a blueprint that exposes a single capability: bucket_url. Because of this standard “interface”, the blueprints can be substituted for each other at deploy time. You will find the same pattern for other elements: minikube/EKS (endpoint), psql/RDS (host, master_username, master_password), and so on.

Running the Example Implementation

Prerequisites

This example expects the following prerequisites:

• A Cloudify Manager setup ready. This can be either a Cloudify Hosted service trial account, a Cloudify Premium Manager, or a Cloudify Community Manager.

• The following plugins must be uploaded to the Cloudify Manager

  • If running on AWS:
    • AWS Plugin (version 2.5.6+)
  • If running on Azure:
    • Azure Plugin (version 3.0.10+)
  • Kubernetes Plugin (version 2.9.3+)
  • Terraform Plugin (version 0.15.0+)
  • Fabric Plugin (version 2.0.7+)

• The following secrets must be defined on the Cloudify Manager

  • AWS Specific:
    • aws_access_key_id - The AWS access key
    • aws_secret_access_key - The AWS secret key
  • Azure Specific
    • azure_tenant_id - The Azure Tenant ID
    • azure_subscription_id - The Azure subscription ID
    • azure_client_id - The Azure client ID
    • azure_client_secret - The Azure client secret
  • private_key_content - The SSH private key contents from the keypair
  • public_key_content - The SSH public key contents from the keypair
  • eaas_params - See the example for the structure of this secret.

• Access to the cloud infrastructure you select is required to demonstrate this example. That can mean ability to allocate the required VMs and networking (ECS), and/or access to S3, RDS, and EKS, or the Azure equivalents.

• These instructions use the CLI to run the example. Using the CLI requires an addition installation step unless the example is run on the manager itself, un which case it is pre-installed.

Install the Example

Our Environment-as-a-Service example on GitHub demonstrates a deploy-time selectable Kubernetes-based environment that includes object and relational storage external to the Kubernetes cluster. The three selectable environment types representing small and large development environments, and a production environment.

• Download or clone the example to your local system. If downloaded as an archive, the archive must be extracted.
• Upload each blueprint in the infra directory, and use the names from the table below:

• Upload the application/main blueprint from app/blueprint.yaml.
• Create a small development cluster:

cfy deployments create app_dev_small -b app -i env_type=dev-small
cfy executions start install -d app_dev_small

• Create a large development cluster:

cfy deployments create app_dev_large -b app -i env_type=dev-large
cfy executions start install -d app_dev_large

• Create a production cluster:

cfy deployments create app_prod -b app -i env_type=production
cfy executions start install -d app_prod