Section: System Administration tools (1)
Updated: 2014-07-11
Index Return to Main Contents


daemonproxy - a proxy server for daemon/job management  


  # static configuration
  daemonproxy [OPTIONS] -c CONFIG_FILE
  # control via unix socket
  daemonproxy [OPTIONS] -S PATH
  # for interactive experiments
  daemonproxy [OPTIONS] -i



Daemonproxy monitors other processes. You might call it a job server, a daemon supervisor, or variety of similar names. However, unlike most others in this category, daemonproxy doesn't restart services or have any conventions for how services should be run. Instead, its goal is to deliver all supervision events to some external script, and execute actions on behalf of that script. Thus, it is a ``proxy'' for process management.

The purpose of the proxy design is to preserve file descriptors, process hierarchy, and monitoring state in the event that your control script crashes or locks up. Daemonproxy acts as a watchdog for the control script and can restart it, and the control script can then re-sync with the state of daemonproxy.  


The daemonproxy config file is simply a list of commands that should be run on startup. These commands are the same as the protocol used by the control script. The only thing you really need in the config file are commands to create and start your controller script.  


--config FILENAME
Read command stream from FILENAME at startup. The config file is used only once and cannot be ``re-loaded'' later.
Use STDIN+STDOUT as a controller communication pipe. Daemonproxy will terminate at EOF (unless exit-guard is set; then it will keep running in the background).
--socket PATH
Listen on PATH for controller connections. This is not needed for your controller script, but might be helpful for debugging or receiving external events (but your controller script is the better place to receive external events). By default, daemonproxy doesn't listen to anything.
Fork into the background. This prints the new PID on stdout, closes stdin, stdout, and stderr, and calls setsid() to become a session leader.

This option cannot be used when running as PID 1, and is incompatible with --interactive, and suppresses the default logging. (see: log.dest command)

--exit-guard INTEGER
Guard against accidental termination. INTEGER must be a nonzero integer, up to 64 bits in length. With this feature enabled, daemonproxy will refuse to exit for any reason less than a fatal signal. The integer must be supplied in order to disable the feature or ask daemonproxy to exit.
--exit-exec TSV_ARGS
exec() args in any trappable exit scenario.

This causes daemonproxy to exec into another program on any condition which would otherwise cause daemonproxy to exit. This includes anything from normal program termination to fatal signals like SIGSEGV.

--fd-pool N[xM]
Pre-allocate N named handles [of M bytes each]. This sets the total allocation size for file descriptor objects, and prevents further dynamic allocations. It also restricts you to a fixed number of total handle objects, each of a fixed size that might not be large enough for long filenames.
--service-pool N[xM]
Pre-allocate N services [of M bytes each]. This sets the total allocation size for service objects, and prevents further dynamic allocations. It also restricts you to a fixed number of total services, each of a fixed size that might not be large enough for long argument lists.
Call mlockall() after allocating structures. This is primarily intended for use with --fd-pool or --service-pool when running as process 1.
Enable another level of logging output. (see also: log.filter command)
Suppress another level of logging output. (see also: log.filter command)
Quick usage synopsis.
Print complete version information. First line will remain a consistent format, other text is subject to change.


Daemonproxy reads commands in tab-separated-values format, with one command per line. There is no escaping mechanism, and your commands must not contain ASCII control characters. Events are delivered in this same format.

In practice, ASCII control characters shouldn't be needed, and the absence of quoting/escaping makes the protocol easier to implement in your script.

A full protocol reference can be found in the documentation included with daemonproxy. However, here is a quick reference guide:  


Prints all arguments as-is back as an event. This is primarily intended to be used to mark the ends of other commands, by following the other command with an echo and a unique string, then watching for the echo to complete.
Set the timeouts associated with this controller. If the controller's event stream has been blocked for more than RESET_TIMEOUT seconds, daemonproxy will flag the connection as ``overflowed'' and discard further writes until the script resumes reading events. If the pipe has not been cleared by CLOSE_TIMEOUT seconds, daemonproxy will close the pipe, which hopefully sends SIGPIPE to the controller and kills it, after which daemonproxy will restart it if configured to do so.
Close the connection to daemonproxy. 'exit' is a poor name for this command, but people expect to be able to type 'exit' to end a command stream.
Re-emit all events for daemonproxy's current state, to get the controller back into sync. Useful after event overflow, or controller restart.
Create a pipe, with the read-end named NAME_READ and write-end named NAME_WRITE. Re-using an existing name will close the old handle.
fd.open NAME FLAG1,FLAG2,.. PATH
Opens a file at PATH. FLAGS is a comma-sparated list of flags of the set read, write, create, truncate, nonblock, mkdir. Re-using an existing name will close the old handle.
fd.delete NAME
Close (and remove) the named handle.
service.tags NAME TAG_1 TAG_2 ... TAG_N
Set some ad-hoc metadata for this service, for use by the controller script. Beware! if you use a fixed-size service memory pool you won't be able to fit much data here. Also everything you store here will be held in memory, bulking up daemonproxy's footprint. This is mainly intended for small tags and IDs which might be inconvenient to store within the service name.

If you have lots of metadata for a service, consider storing it in a file or database, keyed by the service name.

service.args NAME ARG_1 ARG_2 ... ARG_N
Assign new exec() arguments to the service. NAME will be created if it didn't exist. ARG_1 is both the file to execute and argv[0] to pass to the service. To falsify argv[0], use an external program.
service.fds NAME HANDLE_1 HANDLE_2 ... HANDLE_N
Set the list of file descriptors to pass to the service. Name will be created if it didn't exist. The name 'null' is always available and refers to /dev/null. '-' means to pass the service a closed file descriptor.
service.auto_up NAME MIN_INTERVAL [TRIGGER]...
Start the service (no more rapidly than MIN_INTERVAL seconds apart) if any of the triggers are true.

MIN_INTERVAL counts from the time of the previous start attempt, so if the service has been running longer than MIN_INTERVAL and it exits while a trigger is true, it will be restarted immediately. MIN_INTERVAL cannot be less than 1 second. A MIN_INTERVAL of '-' disables auto-up.

Currently, triggers are 'always', SIGINT, SIGHUP, SIGTERM, SIGUSR1, SIGUSR2, SIGQUIT.

'always' means the service will always start if it is not already running. Using 'always' with a large MIN_INTERVAL can give you a cron-like effect, if you want a periodicaly-run service and don't care what specific time it runs.

Signal triggers cause the service to start if the pending count of that signal is nonzero. (and the service is expected to issue the command ``signal.clear'' to reset the count to zero, to prevent being started again)

Start the service, optionally at a future time (or cancel a previous request).

FUTURE_TIMESTAMP is an integer of seconds according to CLOCK_MONOTONIC. (This might be extended to allow fractional seconds in the future.) The service.start will take place as soon as that time is past, but if the timestamp is more than 100000 seconds in the past it is considered an error. The service.state will become 'start'.

If FUTURE_TIMESTAMP is the string ``-'', a pending start event will be canceled. However, because commands are asynchronous the service might get started anyway. Watch for the service.state event to find out whether it started or was canceled.

Starting a service can reveal configuration errors, such as invalid FD names, or an argv that can't be executed. Error messages are events, so they are asynchronous and might be received in any order.

service.signal NAME SIGNAL [FLAGS]
Send SIGNAL to the named service's pid, if it is running. Optional flag may be ``group'', in which case (if the service leads a process group) the pprocess group is sent the signal.
service.delete NAME
Delete a service. The service can only be deleted if it is not currently running. Once deleted, there is no trace of the service's state.
log.filter [+|-|none|LEVELNAME]
Change the logging filter level of daemonproxy. A value of none causes all log messages to be printed. A value of + or - increases or decreases the filter level. A level of 'info' would suppress 'info', 'debug', and 'trace' messages. Note that trace messages are only available when compiled in debug mode.
log.dest fd FD_NAME
Redirect daaemonproxy's logging to named file descriptor. FD_NAME must be a valid name, but it does not need to exist yet. The logging system will check this name until it is available, and then resume logging. Likewise if the descriptor by that name is deleted or re-used.

WARNING: the file descriptor will be put into non-blocking mode, so it is best not to share this descriptor with other processes, especially processes that might reset it to a blocking state and cause daemonproxy to hang on a blocked logging pipe. (However, if you're one of those types who preferrs your daemons freeze up when the logging is interrupted, then here's your workaround.)

signal.clear SIGNAL COUNT
Decrements the count of one signal. Daemonproxy increments the count each time a signal is received, and generates an event for any listening controllers. Statedumps will continue to show the signal while it has a nonzero count. This command decrements the count, allowing a controller to know that the signal has been dealt with.
socket.create OPTIONS PATH
Create the controller socket at the designated path. Options must be empty or the literal string ``-''. (options will be added in the future)

Only one controller socket may exist. If create is called a second time, the previous socket will be unlinked.

Takes no arguments. Cleans up the previously created socket. If no socket exists, this is a no-op.
Terminate daemonproxy immediately. No cleanup is performed, and all handles and child processes will be lost. Graceful shutdown should be part of the controller script, and this should be the final step.

If the terminate-guard feature is enabled, then you need an additional argument of the correct code in order for the command to happen.

If the exec-on-exit feature is enabled, daemonproxy will exec() instead of exit(). If daemonproxy is process 1, terminate will fail unless exec-on-exit is enabled.

terminate.exec_args [ARG_1] .. [ARG_N]
Set argument list for daemonproxy's exec-on-exit feature. This feature causes daemonproxy to exec(ARGS) instead of exiting in any trappable scenario. An empty argument list disables the feature.
terminate.guard [+|-] CODE
Enable or disable the terminate-guard feature. '+' enables the feature and sets the guard code to CODE. '-' disables the feature only if CODE matches the previously set value.


NAME is the C constant like SIGINT. TS is a timestamp from CLOCK_MONOTONIC when the signal was last received. COUNT is the number of times it was received since last cleared (however you can't actually know the exact count due to the nature of signals)
The state of service has changed. STATE is 'start', 'up', 'down', or 'deleted'. TS is a timestamp from CLOCK_MONOTONIC. PID is the process ID if relevant, and '-' otherwise. EXITREASON is '-', 'exit', or 'signal'. EXITVALUE is an integer or signal name. UPTIME and DOWNTIME are in seconds, and '-' if not relevant.
service.tags NAME TAG_1 TAG_2 ... TAG_N
Tags for the service have changed.
service.args NAME ARG_1 ARG_2 ... ARG_N
Arguments for the service have changed.
service.fds NAME HANDLE_1 HANDLE_2 ... HANDLE_N
File handles for the service have changed.
service.triggers NAME [TRIGGER_1], [TRIGER_2] ...
Triggers for auto-starting the service have changed.
TYPE is 'file', 'pipe', 'special', or 'deleted'. Deleted means the file handle has just been removed and no longer exists. Type 'file' has FLAGS that match the flags used to open it (though possibly in a different order). Type 'pipe' has flags of 'to' or 'from'. DESCRIPTION is the filename (possibly truncated), the pipe-peer handle name, or a free-form string describing the handle.
Error events are reported free-form, with ``error'' as the first tab delimited token, but MESSAGE being arbitrary ascii text.




Here is how you run a basic service. Remember that the whitespace between fields must be TABs.

  service.args  mysql   runuid  -s      mysql   mysqld
  service.fds   mysql   null    stdout  stdout
  service.auto_up       mysql   2       always

That example leaves error messages going to daemonproxy's STDOUT. If you want to set up a logger, try this:

  fd.pipe       mysql-log.r     mysql-log.w
  service.args  mysql-log       sissylog        mysql
  service.fds   mysql-log       mysql-log.r     stderr  stderr
  service.auto_up       mysql-log       2       always
  service.args  mysql   runuid  -s      mysql   mysqld
  service.fds   mysql   null    mysql-log.w     mysql-log.w
  service.auto_up       mysql   2       always

That example requires 'sissylog' from the perp package. Also, you might want to configure mysql to use /dev/fd/1 as its log file.

To down a service, use this sequence:

  service.auto_up       mysql   -
  service.signal        mysql   SIGKILL

and then wait for the event that it went down.

These examples use auto_up, which asks daemonproxy to auto-restart the service. If you're using a controller script (which was the main use-case for the design of daemonproxy) then use service.start instead, and when you get the service.state event saying the service died, you can decide to restart it on your own.  

Using a Controller Script

If you are actually using a controller script, here is all you need in a config file:

  service.args  my_controller   ./my-controller.pl
  service.fds   my_controller   control.event   control.cmd     stderr
  service.auto_up       my_controller   1       always

This starts a controller service which reads events on stdin and writes daemonproxy commands on stdout. Its stderr goes to daemonproxy's stderr. The controller can do whatever you like, such as listening on a unix socket, or even a TCP socket, or even a UDP socket, or using INotify to watch a directory, or anything else you can think of.  

Other Examples

See the examples in the doc directory that come with daemonproxy. If they aren't installed by your package manager, you can find them on github or the home page.  


Please report bugs in the issue tracker at <http://github.com/silverdirk/daemonproxy>  


The documentation distributed with the source code contains an informative README and a ``api.ltw'' which contains more detailed documentation for the command and event protocol.
GitHub Page


Michael Conrad <mike@nrdvana.net>



Using a Controller Script
Other Examples

This document was created by man2html, using the manual pages.
Time: 23:35:33 GMT, July 11, 2014