PerkinElmer Informatics Support Forum
Decrease font size
Increase font size
Topic Title: What can I do if the Celery service is not responsive?
Topic Summary:
Created On: 12/10/2018 11:27 AM
Status Post and Reply
Linear : Threading : Single : Branch
Topic Tools Topic Tools
View topic in raw text format. Print this topic.
 12/10/2018 11:27 AM
User is offline View Users Profile Print this message


RAH

Posts: 75
Joined: 5/16/2014

Initially you should try starting/stopping the celery service using:

$ /etc/init.d/columbus-celeryd (stop/start/restart/status)

If the celery service is 'hung' and does not respond to either /etc/init.d/columbus stop, or $ /etc/init.d/columbus-celeryd stop then it might be necessary to kill the Celery related processes. Typically you will see more than 1 process. There's a master but depending on the activity there will be several worker processes. They all run under the columbus account e.g:

$ ps f -fU columbus

UID PID PPID C STIME TTY STAT TIME CMD
columbus 12688 12687 0 08:01 ? S 0:01 nginx: worker process
columbus 12782 1 0 08:01 ? Sl 3:11 /usr/local/PerkinElmerCTG/Columbus2.8/webapp/virtualenv/bin/python -m celery.__main__ worker --app=columbus -n c
columbus 9627 12782 2 13:25 ? Sl 0:01 \_ /usr/local/PerkinElmerCTG/Columbus2.8/webapp/virtualenv/bin/python -m celery.__main__ worker --app=columbus
columbus 9731 12782 0 13:25 ? Sl 0:00 \_ /usr/local/PerkinElmerCTG/Columbus2.8/webapp/virtualenv/bin/python -m celery.__main__ worker --app=columbus
columbus 9751 12782 3 13:25 ? Sl 0:01 \_ /usr/local/PerkinElmerCTG/Columbus2.8/webapp/virtualenv/bin/python -m celery.__main__ worker --app=columbus

You'd need to kill the master with PID 12782 but before that the slave processes must be killed, you can use pkill -P with the parent id to do so:

$ sudo kill -9 -P 12782

# Now kill the master

$ sudo kill -9 12782

Now if you start Celery again it should bring it back to life. If this doesn't help then there is potential for something to be broken meaning that the image rendering jobs will fail, if that is the case then check the /var/log/columbus/web/columbus-images-service.log file for errors and forward on to the PKI informatics support team for further troubleshooting advice.

RAH

FuseTalk Basic Edition v4.0 - © 1999-2019 FuseTalk Inc. All rights reserved.