Service Logs
This page briefly explains the different log files that will be available on the Cloudify Manager host.
Downloading the logs
Running cfy logs download will download a tar gzipped file containing the log files discussed in this page. This archive will be vital when requesting support with your Cloudify Manager.
cfy logs download requires SSH access to your Cloudify Manager machine. This means that the SSH key and the SSH username must be set in your CLI profile. You can set them using cfy profiles set --ssh-key <path/to/file.pem> and cfy profiles set --ssh-user <username>.
When working with a cluster of Cloudify Managers, use cfy logs download --all-nodes to download logs from all of the reachable cluster nodes. This will require the SSH key and user be set in the CLI profile for every node. You can set those by using cfy profiles set-cluster <cluster node name> --ssh-user <username> and --ssh-key <path/to/file.pem>.
REST
The REST service runs using the Flask web framework on top of the Gunicorn HTTP server. Each of these layers has its own logs.
Flask Log
The Flask logger is used by the REST service to write custom logs. The information about exceptions raised by the REST service is stored in this log. Additionally, when the logging level is set to DEBUG, it records each request-response pair with their relevant parameters and information.
This log file is located at /var/log/cloudify/rest/cloudify-rest-service.log.
The logging level can be set in config.yaml when installing the Manager, under the restservice.log.level key. The default is INFO, but it can be changed to DEBUG when information about every request-response pair is required. The logging level can be changed on an already-installed manager by editing the REST service config file located at /opt/manager/cloudify-rest.conf and restarting the REST service.
Gunicorn Logs
There are two log files Gunicorn writes to: an access log and a general log.
The access log simply logs a concise entry for each request made to the web server.
This log file is located at /var/log/cloudify/rest/gunicorn-access.log.
The general log mostly has information on Gunicorn workers. It is usually only interesting when there was a problem in starting up the REST service and Flask.
This log file is located at /var/log/cloudify/rest/gunicorn.log.
Security Audit Log
When starting the Cloudify Manager in a secured mode, requests sent to the REST server go through authentication. The security audit log file documents successful and failed REST calls, their origin, method of authentication and, if the authentication failed, the reason for the failure.
The audit log file is located at /var/log/cloudify/rest/rest-security-audit.log.
Management Worker
Starting with Cloudify 3.4, the management worker is responsible for all central deployment operations of all deployments.
Worker Logs
The management worker log contains top level details regarding the tasks it handles, such as task accepted, task succeeded, task failed (with a generic traceback that will always look the same), etc…
The management worker handles all of the workflow tasks, and some operation
tasks that use the central_deployment_agent as their executor. The
workflow/operation execution is done in its own subprocess.
This log file is located at /var/log/cloudify/mgmtworker/mgmtworker.log.
Deployment Logs
In the aformentioned subprocess, which is where the operation/workflow actually runs, the root logger handler is configured to send all logging of all loggers in this subprocess to a deployment specific log file. This is where the “interesting” stuff appears. Specifically, if you are looking for a traceback, this is where you should be looking.
These logs files are located at /var/log/cloudify/mgmtworker/logs/<deployment_id>.log.
System Logs
In addition, the mgmtworker/logs dir will contain an additional log file, that will include all logging from operations/workflows that are
“system wide” (e.g. snapshot, install plugin on REST upload, etc…)
This log file is located at /var/log/cloudify/mgmtworker/logs/__system__.log.
Remote hosts that have a Cloudify Agent running on them will have their log files located at $HOME/<compute_instance_id>/work on the remote host.
Other Components
You can find all log files for Cloudify related services located at /var/log/cloudify.
Log Rotation
All Cloudify related log files on the manager host are managed by logrotate. Log files are configured to rotate when they reach the size of 100MB.
At most, 7 rotated files are kept for each rotated log file.
Log Cleanup
You can remove Cloudify logs and events from the database with the delete_logs_and_events_from_db.py that is located on the Cloudify Manager under /etc/cloudify. When you run the script, the logs and events that exist from after the save period configured in the script are removed.
You can change the DEFAULT_SAVE_PERIOD parameter in the script to set the number of days that are kept in the database (Default: 5). For example, if the DEFAULT_SAVE_PERIOD is 7, logs and events that are older than 7 days are deleted.