[buug] A bit more[1] on named pipes

Michael Paoli Michael.Paoli at cal.berkeley.edu
Thu Aug 5 18:27:17 PDT 2010


Some more practical named pipe examples.

Here's scenario I used to semi-regularly encounter.  A large database.
It's desired to do a full dump of the database, but the particular
database can only do that by writing to a named file - it has no
provision to write to stdout.  Due to the size of the database, it's
desired (or required) that the output be compressed (e.g. pack(1),
compress(1), gzip(1), bzip2(1), or what have you).  It's also desired to
avoid all the double I/O of writing to uncompressed file on disk, then
reading that whole file to then write to compressed file.  There may
also be insufficient room to have both the uncompressed and compressed
file around at the same time or there may even be insufficient room to
have the uncompressed file around at all on disk.
So, dumping such a database via named pipe to compressed file, something
like this:
$ mknod p p
$ < p > dbdump.bz2 bzip2 -9 &
Launch database dump, telling it to write to file p (our named pipe) we
created.  The named pipe (p) can be removed after our compression
(bzip2) process has completed.
A highly similar procedure can be used to reload database that has same
type of limitation - where the database only knows how to read an
uncompressed file, and cannot do the load via stdin:
$ mknod p p
Launch the database load, telling it to read from file p (our named
pipe).  This will need to be left running until we're done doing our
uncompress (bzip2 -d) and the load has completed.  Note also that it
will block on I/O until something starts writing to the pipe, then it
will proceed.  We then do:
$ < dbdump.bz2 > p bzip2 -d
Once the uncompress (bzip2 -d) has completed and the database load has
completed, we can then remove our named pipe (p).  (Actually, if we
remove it earlier, as long as we do so after the processes reading and
writing it already have it open, we're actually okay.  It would just
persist as an unlinked open file[2] until it's no longer open by any
processes).

Here's another example, relatively similar to before.  I'd just created
debian-505-amd64-netinst.iso by using jigdo[3][4] - specifically
jigdo-lite(1) and mounted image debian-505-amd64-CD-1.iso which had also
itself been earlier created with jigdo-lite(1) and likewise been
validated.  I wanted to validate the debian-505-amd64-netinst.iso image
against the hashes it should match, and I also only want to only read
that file once to do so.  So, I proceed as follows ... note also that
the lines below starting with > are PS2 (secondary prompt), not I/O
redirection:
$ mknod p p
$ mknod p2 p
$ mknod p3 p
$ < p > md5 md5sum &
[1] 2996
$ < p2 > sha1 sha1sum &
[2] 2997
$ < p3 > sha256 sha256sum &
[3] 2999
$ < debian-505-amd64-netinst.iso tee p |
> tee p2 |
> tee p3 |
> > sha512 sha512sum; wait; cat md5 sha1 sha256 sha512
[1]   Done                    md5sum < p > md5
[2]-  Done                    sha1sum < p2 > sha1
[3]+  Done                    sha256sum < p3 > sha256
2de6321d0e9ab431f59b0c95eb84cc2c  -
03f97c6284d50dc799dbcc5d873c2ab9aaa5c404  -
111039bd95fb68a0426c8dcb1f4ae2d147ade092d01f9f2a3bfe790849b0d02a  -
ba6663987664833b026dd149172c690b5fdc5cf637c9039391571dd40ba046d805b704ffa572974b0fff4643042c0e05112d50d57bf96fdd388b40c961c1f673   
-
$ rm p p2 p3
$
I then compare those hashes above with hashes in files I validated via
gpg signatures, and I find that they indeed match.

footnotes/references:
1. continuing from:
    http://www.weak.org/pipermail/buug/2010-August/003639.html
2. unlink(2)
    news:56d4652c-8f86-45e6-90b4-0b7e212a484a at v25g2000yqk.googlegroups.com
3. http://en.wikipedia.org/wiki/Jigdo
4. Downloading Debian CD images with jigdo
    http://www.debian.org/CD/jigdo-cd/




More information about the buug mailing list