Debugging Cloudify Workers
Using the debugger with cloudify
While some parts of cloudify do run locally (eg. the CLI, or the system tests
framework) and can be run under a regular debugger (eg.
pudb <https://pypi.python.org/pypi/pudb>_), most of the interesting parts
run in remote tasks on the Cloudify Manager, so a regular debugger won’t work with that.
Remote debugger for executions
Cloudify does not use anymore celery for running executions but for sake of backward compatibility we still install it in the environment.
We can use remote celery, use rdb, eg. use a text editor to
from celery.contrib import rdb rdb.set_trace()
Then it will start listening on port 6899 (by default), or if that is already in use, 6900, 6901, …
Connect to the debugger using telnet or nc (on centos, you need to install
them using eg.
yum install telnet or
yum install nc), like
telnet 127.0.0.1 6900.
Sometimes it might be hard to guess which port it’s using; you can simply guess or look at the open ports and make a more informed guess. The following command might be useful:
$ netstat -tulpn | grep python
Shows ports that processes named “python” are listening on. (on a cloudify manager usually it’ll be just the REST service, and your debuggers)
Remember that after adding `.set_trace()`, you need to restart the process running that code (usually it'll be mgmtworker - or gunicorn if you're working on the REST service). `sudo systemctl restart cloudify-mgmtworker` `sudo systemctl restart cloudify-restservice`
Also read the
Celery RDB docs <http://docs.celeryproject.org/en/latest/tutorials/debugging.html>_.
Where to put set_trace?
If you’d like to put a
set_trace call inside cloudify code that will be
used by the management worker, edit files inside
Restarting Cloudify workers
After adding a
set_trace, you need to restart Cloudify workers so it loads the new code.
To do it, best to simply do
sudo systemctl restart cloudify-mgmtworker.
Anyway, it is a good idea to tail the mgmtworker logs to verify that it did
restart successfully (
Debugging on agent machines
Celery is also installed on agent machines - the virtualenv is usually located
Restarting agent workers there depends on the process management method, but with the most commonly used initd, you would do
$ systemctl restart cloudify-worker-node_name
rdb is simply pdb using a network socket; all the regular pdb commands can still be used with it. See the pdb docs and this talk to learn how to use pdb.
Attaching remote debugger Pycharm
For attaching remote debugger in Pycharm you need to have Professional Edition license. You can follow this instruction.