Agent Packager

Overview

Cloudify’s Agent is basically a virtualenv with a series of modules installed in it and (optionally) a few configuration files attached.

To use Cloudify with distributions other than the officially supported ones, we’re providing an Agent-Packager tool that will assist you in creating an agent for your distribution.

This tool aims to:

• Solve the problem of compiling module requirements on different distributions, thus bridging the gap of user compiled images, unfamiliar/minor distros and so on.
• Allow users to create their own, personalized Cloudify agents with custom plugins of their choosing.
• Make the agent creation process seamless. One config file. One liner cmd.
• Allow users to override any existing mandatory (or else) modules and provide their own.

You can use Cloudify’s agent-packager to create an agent on the distribution you’re running so that modules that require compilation will use your distribution and compilers to do so.

Creation Process

During the creation process, the agent-packager performs the following:

• Creates a virtualenv using the python binary of your choice.
• Installs mandatory external modules into the virtualenv.
• Installs modules from a provided requirements.txt file.
• Installs mandatory and optional Cloudify plugins and modules into the virtualenv.
• Installs the cloudify-agent module into the virtualenv.
• Installs any additional user chosen Cloudify plugins and Python modules into the virtualenv.
• Validates that all specified modules were installed.
• Generates an included_plugins.py file. This will be used by the cloudify-agent module to automatically load all plugins specified in the file.
• Creates a tar file containing the virtualenv.

Installation

  
pip install cloudify-agent-packager
  

For development purposes:

  
pip install https://github.com/cloudify-cosmo/cloudify-agent-packager/archive/master.tar.gz
  

Usage

IMPORTANT NOTES:

• You must use this tool on the specific version of the distribution you’re intending for your agent to run in as it might require compilation.
• You must have the desired version of python installed on your chosen image.
• You must have the tar binary in your distribution (just run which tar to verify that you have tar installed).

Using from the CLI

  
cfy-ap -h

Script to run Cloudify's Agent Packager via command line

Usage:
    cfy-ap [--config=<path> --force --dryrun --no-validation -v]
    cfy-ap --version

Options:
    -h --help                   Show this screen
    -c --config=<path>          Path to config yaml (defaults to config.yaml)
    -f --force                  Forces deletion and creation of venv and tar file.
    -d --dryrun                 Prints out the modules to be installed without actually installing them.
    -n --no-validation          Does not validate that all modules were installed correctly.
    -v --verbose                verbose level logging
    --version                   Display current version
  

example:

  
cfy-ap -f -c my_config.yaml -v
  

Using from python

To use this from python, do the following:

  
import agent_packager.packager as cfyap

config = {}  # dict containing the configuration as given in the yaml file.

cfyap.create(config=config,
             config_file=None,
             force=False,
             dryrun=False,
             no_validate=False,
             verbose=True)
  

The cloudify-agent module

See here.

Using your agent

After creating the agent you can do one of the following to use your newly created agent:

Using your agent on a per-node basis

You can define the paths to the agent tar file in the blueprint on a per-node basis. See the cloudify-agent documentation for more information.

Install agents in the manager during bootstrap

You can provide urls for agents you’d like to provide during manager bootstrap.

Configuring the Tool

The YAML config file

Here’s an example configuration file.

  
distribution: Ubuntu
release: trusty
python_path: '/usr/bin/python'
requirements_file: path/to/my/requirements/file.txt
cloudify_agent_version: master
cloudify_agent_module: http://github.com/cloudify-cosmo/cloudify-agent/archive/master.tar.gz
core_modules:
    cloudify_plugins_common: http://github.com/cloudify-cosmo/cloudify-plugins-common/archive/master.tar.gz
    cloudify_rest_client: http://github.com/cloudify-cosmo/cloudify-rest-client/archive/master.tar.gz
core_plugins:
    cloudify_script_plugin: http://github.com/cloudify-cosmo/cloudify-script-plugin/archive/master.tar.gz
    cloudify_diamond_plugin: http://github.com/cloudify-cosmo/cloudify-diamond-plugin/archive/master.tar.gz
additional_modules:
    - pyyaml==3.10
additional_plugins:
    cloudify_fabric_plugin: http://github.com/cloudify-cosmo/cloudify-fabric-plugin/archive/master.tar.gz
output_tar: Ubuntu-trusty-agent.tar.gz
keep_virtualenv: true
  

Config YAML Explained

  
python -c "import platform; print platform.dist()"
# e.g. ('Ubuntu', '14.04', 'trusty')
  

• distribution - Which distribution is the agent intended for. If this is omitted, the tool will try to retrieve the distribution by itself. The distribution is then used to name the virtualenv (if not explicitly specified in venv) and to name the output file (if not explicitly specified in output_tar).
• release - Which release (e.g. precise, trusty) of the distribution is the agent intended for. If this is omitted, the tool will try to retrieve the release by itself. The release is then used to name the virtualenv (if not explicitly specified in venv) and to name the output file (if not explicitly specified in `output_tar’).
• python_path - Allows you to set the python binary to be used when creating venv. (Defaults to /usr/bin/python).
• requirements_file - a path to a requirements.txt file containing modules you want installed in the agent.
• cloudify_agent_version - States which version of the cloudify-agent module to install (Is not required if cloudify_agent_module is specified). Note that this can be used to create an agent for a specific Cloudify version.
• cloudify_agent_module - States the url from which the cloudify-agent module should be installed. (Will ignore cloudify_agent_version if specified).
• core_modules - a dict of core modules to install into the virtualenv. (If omitted or with a value of false, the module will be installed as a part of the cloudify-agent dependencies.) See below for a list of current core modules.
• core_plugins - a dict of core plugins to install into the virtualenv. (If omitted or with a value of false, the module will be installed as a part of the cloudify-agent dependencies.) See below for a list of current core plugins. If exclude is set (per module), it will not be installed. Set exclude with extra care!
• additional_modules - a list of additional modules to install into the virtualenv. This is where you can add any additional modules that are not Cloudify plugins.
• additional_plugins - a list of additional Cloudify plugins to install into the virtualenv.
• output_tar - Path to the tar file you’d like to create.
• keep_virtualenv - Whether to keep the virtualenv after creating the tar file or not. Defaults to false.

Agent Modules

Each agent contains a set of Python packages. These modules can be either simple python libraries or they can be plugins.

Core External Modules:

These are modules not developed by Cloudify that are used by the agent.

• Celery (Mandatory)

Core Modules:

These modules are developed by Cloudify and provide core functionality for the agent - thus, the default agents provided with Cloudify come with these pre-installed.

• Cloudify REST Client (Mandatory)
• Cloudify Plugins Common (Mandatory)

Core Plugins:

These plugins are developed by Cloudify and provide core functionality for the agent - thus, the default agents provided with Cloudify come with these pre-installed.

• Cloudify Script Plugin (Optional)
• Cloudify Diamond Plugin (Optional)

The Cloudify Manager actually also runs an instance of an agent, this is called the cloudify_management_agent. This agent is responsible for starting all other agents, and thus requires the following plugin.

• Cloudify Agent