libpipeline is a C library for manipulating pipelines of subprocesses in a flexible and convenient way. It is available in at least the following operating systems:
For macOS, you can install libpipeline using Homebrew with this "tap".
When I took over man-db in 2001,
one of the major problems that became evident after maintaining it for a
while was the way it handled subprocesses. The nature of man and friends
means that it spends a lot of time calling sequences of programs such as
zsoelim < input-file | tbl | nroff -mandoc -Tutf8. Back then,
it was using C library facilities such as system and
popen for all this, and I had to deal with several bugs where
those functions were being called with untrusted input as arguments without
properly escaping metacharacters. Of course it was possible to chase around
every such call inserting appropriate escaping functions, but this was
always bound to be error-prone and one of the tasks that rapidly became
important to me was arranging to start subprocesses in a way that was
fundamentally immune to this kind of bug.
In higher-level languages, there are usually standard constructs which
are safer than just passing a command line to the shell. For example, in
Perl you can use system([$command, $arg1, $arg2, ...]) to
invoke a program with arguments without the interference of the shell, and
perlipc(1) describes various facilities for connecting them
together. In Python, the
subprocess
module allows you to create pipelines easily and safely (as long as you
remember the
SIGPIPE
gotcha). C has the fork and execve
primitives, but assembling these to construct full-blown pipelines correctly
is difficult and error-prone, so many programmers don't bother and use the
simple but unsafe library facilities instead.
libpipeline solves this problem. In the following examples, function
names starting with pipecmd_ or pipeline_ are real
functions in the library, while any other function names are pseudocode.
Constructing the simplified example pipeline from my first paragraph using this library looks like this:
pipeline *p; int status; p = pipeline_new (); pipeline_want_infile (p, "input-file"); pipeline_command_args (p, "zsoelim", NULL); pipeline_command_args (p, "tbl", NULL); pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", NULL); status = pipeline_run (p);
You might want to construct a command more dynamically:
pipecmd *manconv = pipecmd_new_args ("manconv", "-f", from_code,
"-t", "UTF-8", NULL);
if (quiet)
pipecmd_arg (manconv, "-q");
pipeline_command (p, manconv);
Perhaps you want an environment variable set only while running a certain command:
pipecmd *less = pipecmd_new ("less");
pipecmd_setenv (less, "LESSCHARSET", lesscharset);
You might find yourself needing to pass the output of one pipeline to several other pipelines, in a "tee" arrangement:
pipeline *source, *sink1, *sink2; source = make_source (); sink1 = make_sink1 (); sink2 = make_sink2 (); pipeline_connect (source, sink1, sink2, NULL); /* Pump data among these pipelines until there's nothing left. */ pipeline_pump (source, sink1, sink2, NULL); pipeline_free (sink2); pipeline_free (sink1); pipeline_free (source);
Maybe one of your commands is actually an in-process function, rather than an external program:
pipecmd *inproc = pipecmd_new_function ("in-process", &func, NULL, NULL);
pipeline_command (p, inproc);
Sometimes your program needs to consume the output of a pipeline, rather than sending it all to some other subprocess:
pipeline *p = make_pipeline ();
const char *line;
pipeline_want_out (p, -1);
pipeline_start (p);
line = pipeline_peekline (p);
if (!strstr (line, "coding: UTF-8"))
printf ("Unicode text follows:\n");
while (line = pipeline_readline (p))
printf (" %s", line);
pipeline_free (p);
See the Savannah project or the GitLab repository for more information. The latest release is 1.5.3, made on 2020-08-13.
The libpipeline(3) manual page is available in HTML and text.
If your distribution includes a package of libpipeline, it's usually best to install that. However, if you need to install it starting from source code, then you will need these separate packages installed before configuring libpipeline in order to run its test suite: