Idea Transcript
SymplecticGames
Getting Rails to fork a process using Capistrano Attempt 1 – detach a process from rake task
# start new one job = fork do exec "command arg(s)" end Process.detach(job)
Works ok if called directly in development env (rake namespace:task). However, attempt to deploy it using Capistrano results in hung Capistrano process.
Attempt 2 – initctl Actually never tried this, bcs by the time I found this advice: “Using a deployment tool like Capistrano to directly start any process is a bad idea. What happens when that process crashes? (Most likely it disappears and now your application is broken.) Who rotates the logs to ensure they don’t fill the disk? (Most likely you after filling the disk the first time.) Instead, integrate your daemon into the init system, configure it to respawn if it crashes and have Capistrano manage the process via init:
task :restart do run "initctl restart sidekiq" end
As always, KISS. Let your operating system handle daemons, respawning and logging while you focus on your application features and users,” Ubuntu already abandoned the approach (“This document compares Upstart and systemd with a view to aiding in the transition to the latter”), over the objections of people who were perfectly happy with initctl (or Upstart, actually). And, btw, the command there is nowadays called “sh” not “run.”
Attempt 3 – systemd OK, so the new initctl is systemd. Several confusing things here. First, the command you use is not even called systemd – it is systemctl. Here is how you use it, except that you need to run it as sudo. See below hot to get around it. First, you create a file daddy.service in /etc/systemd/system, with the content something like
[Unit] Description=Job that does so and so (daddy) [Service] Type=forking ExecStart=/usr/local/bin/daddy [args]
and then run using systemctl,
sudo systemctl start daddy.service
or just
sudo systemctl start daddy
resulting in
bash> sudo systemctl start daddy.service Job for daddy.service failed because the control process exited with error code. See "systemctl status daddy.service" and "journalctl -xe" for details.
bash> systemctl status daddy.service l daddy.service - Job that does so and so (daddy) Loaded: loaded (/lib/systemd/system/daddy.service; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2016-10-24 23:07:17 EDT; 19s ago Process: 11662 ExecStart=/usr/local/bin/daddy [args] (code=exited, status=127) Oct 24 23:07:17 pegasus systemd[1]: Starting Job that does so and so(daddy)... Oct 24 23:07:17 pegasus daddy[11662]: /usr/local/bin/daddy: error while loading shared libraries: libigraph.so.0: c Oct 24 23:07:17 pegasus systemd[1]: daddy.service: Control process exited, code=exited status=127 Oct 24 23:07:17 pegasus systemd[1]: Failed to start Job that does so an so (daddy).
Try to specify the path
[Unit] Description=Job that does so and so (daddy) [Service] Type=forking ExecStart= LD_LIBRARY_PATH=/usr/local/lib [args]
didn’t do the trick either.
Attempt 4 – systemd, Type=simple Finally after a couple of maddening hours, struggling with systemd’s all-but-nonexistent diagnostics, I garnered that the service script should look something like this
[Unit] Description=Job that does so and so(daddy) [Service] Type=simple # specify user so the scoket created by daddy is readable by me # (still need to get rid of the passwd requirement in sudoers) User=username Environment="LD_LIBRARY_PATH=/usr/local/lib" ExecStart=/usr/local/bin/daddy [args] Restart=always [Install] # this should make it run on boot: # note however that sudo systemctl enable daddy # must be run for this to be registered as an on-boot process WantedBy=multi-user.target
note also that the change should be followed by the reload step
bash> systemctl daemon-reload
It took a long search to come across this post, explaining that Type=forking expects a wrapper process that will spawn whatever it is you are actually trying to run, and exit. So, ignore the dire warnings about the dangers of Type=simple, and proceed.
bash> sudo systemctl start daddy.service bash> sudo systemctl status daddy.service l daddy.service - Job that does so and so (daddy) Loaded: loaded (/etc/systemd/system/daddy.service; static; vendor preset: enabled) Active: active (running) since Tue 2016-10-25 12:44:29 EDT; 18s ago Main PID: 6041 (daddy) Tasks: 1 (limit: 4915) CGroup: /system.slice/daddy.service
6041 /usr/local/bin/daddy [args]
Oct 25 12:44:29 pegasus systemd[1]: Started Job that does so and so (daddy).
And now on the minor note, to get rid of the sudo requirement: Add a file in /etc/sudoers.d/daddy
%groupname ALL=NOPASSWD: /bin/systemctl restart daddy
The permissions on this file should be set (after creation/modification) to 0440. Percent sign is used to distinguish between group name and individual name, which uses no prefix. Make sure that you are the member of the group groupname, and reboot. (I could not get it to work without rebooting on one machine running Ubuntu 16.10; on the other running Ubuntu 16.04 LTS it worked without rebooting ). Two things to note: 1) you will still need ‘sudo’ but the system will not prompt you for the password, and 2) sudo takes the commands literally, so if the full path is in sudoers.d, it has to be spelled out when invoking the command; also start and restart are not the same thing. Thus I added an equivalent line there to stop the service – for good measure. [ I still have a problem here: the system socket used by daddy is writeable only by the root]
A note added in print: After a while I realized my deamon was causing the whole system to stall on the reboot, waiting for it to shut down, giving it a whopping minute and a half to do so. I don’t know how and when the system decides dot assign a timeout time to your daemon, but you can check it with:
systemctl show -p TimeoutStopUSec
and indeed, for me the answer was
TimeoutStopUSec=1min 30s
to change it, add another line to your service file in /etc, in [Service] section:
TimeoutSec=1
followed by
sudo systemctl daemon-reload
An attempt to set the timeout time to 0 will result in timeout set to infinity.
Finally, the recipe So finally: we can run daddy as service and without the use of a password. In your RoR application, create lib/tasks/services.rake:
namespace :services do task :restart_daddy do sh "sudo /bin/systemctl restart daddy" end end
This then, can be used in development env, using
rake services:restart_daddy
To make Capistrano aware of it, add the following line to Capfile
# Load custom tasks from `lib/capistrano/tasks` if you have any defined Dir.glob("lib/capistrano/tasks/*.rake").eachort r }
and to lib/capistrano/tasks/sometasks.rake
namespace :custom do desc 'restart daddy daemon' task :restart_daddy do on roles(:app) do within "#{current_path}" do with rails_env: "#{fetch(:stage)}" do execute :rake, "services:restart_daddy" end end end end end
Keep in mind that, of course, in this case you need to redeploy daddy service on your production machine independently.
Report this ad
This entry was posted in harsh life realities on October 25, 2016 [https://symplecticgames.wordpress.com/2016/10/25/getting-rails-to-fork-a-process-usingcapistrano/] .