1 ddc03123 2020-03-28 op In my [previous article](/2019-10-16-poor-mans-unbound-dashboard.html)
2 ddc03123 2020-03-28 op I used a bash(1) construct to send the same data to two different places.
4 ddc03123 2020-03-28 op The fact that I was using a bash-specific (more or less) construct really
5 ddc03123 2020-03-28 op bothered me this time. So I thought if there is a way in sh(1) to take
6 ddc03123 2020-03-28 op a piece of data from a pipeline and send it to two different places.
8 ddc03123 2020-03-28 op The canonical way to do this is `tee(1)`, and it's what I've used it the linked article:
11 a3ab6f61 2020-09-22 op something | tee >(cmd) | something-else
14 ddc03123 2020-03-28 op (the `>(cmd)` construct is bash, and zsh I think, specific: it executes
15 ddc03123 2020-03-28 op `cmd` in a subshell and execute `tee` with something like `/dev/fd/63`
16 ddc03123 2020-03-28 op -- at least on linux and OpenBSD.)
18 ddc03123 2020-03-28 op Now I'll try to describe three different approaches to emulate the same
19 ddc03123 2020-03-28 op behavior of `>(...)` in pure sh(1), trying to outline both the advantages
22 ddc03123 2020-03-28 op ## temporary files
27 a3ab6f61 2020-09-22 op something | tee a-file | something-else
30 a3ab6f61 2020-09-22 op # and then, probably
34 ddc03123 2020-03-28 op (or even better with `mktemp(1)`)
36 ddc03123 2020-03-28 op This is the most simple approach I can think of. The obvious disadvantage
37 ddc03123 2020-03-28 op is the need to create and delete files quickly. This shouldn't be a huge
38 ddc03123 2020-03-28 op problem in most cases, but for script do this constantly and (possibly)
39 ddc03123 2020-03-28 op quickly it can be. Plus, at least for me, it doesn't seem very elegant,
40 ddc03123 2020-03-28 op but it's subjective.
42 ddc03123 2020-03-28 op ## fifo to the rescue
44 ddc03123 2020-03-28 op An almost similar approach is to use fifo. A fifo, aka a named pipe,
45 ddc03123 2020-03-28 op is a special file that you can write to and read from, in a FIFO
50 a3ab6f61 2020-09-22 op something | tee a-fifo | something-else
51 a3ab6f61 2020-09-22 op while read line < a-fifo; do echo $line; done | cmd
54 ddc03123 2020-03-28 op This is especially useful if you need to have a long-running script that
55 ddc03123 2020-03-28 op needs to duplicate a stream from one source. You create the fifos at the
56 ddc03123 2020-03-28 op startup and then you're done, instead of constantly creating temporary
57 ddc03123 2020-03-28 op files over and over again. An annoying thing is that you need to wrap
58 ddc03123 2020-03-28 op the read from the fifo in some sort of loop: AFAIK to read the next item
59 ddc03123 2020-03-28 op in a fifo you need to close and re-open it.
61 ddc03123 2020-03-28 op ## An unexpected sed
63 ddc03123 2020-03-28 op Let's add another requisite: other than dumbly copy the stream, suppose
64 ddc03123 2020-03-28 op that you want also to apply some sort of transformation to one branch
65 ddc03123 2020-03-28 op of the duplication, like filter out something.
67 ddc03123 2020-03-28 op Well, you can adapt the last two examples by adding a `| sed '...'`.
68 ddc03123 2020-03-28 op Or you can use the `w` command of sed.
70 ddc03123 2020-03-28 op I'll be a bit more specific: let's say that you need to *dispatch*
71 ddc03123 2020-03-28 op something taken from a single stream and feed *n* subprocess. It's
72 ddc03123 2020-03-28 op exactly what I've tried to achieve in my last post: I've used `>(cmd)`
73 ddc03123 2020-03-28 op to dispatch the unbound statistics to different fifo, and then I had
74 ddc03123 2020-03-28 op different processes pulling out those stats from the fifo to draw graphs.
76 ddc03123 2020-03-28 op The code was like
79 a3ab6f61 2020-09-22 op unbound-control stats \
80 a3ab6f61 2020-09-22 op | tee >(grep something > a-fifo) \
81 a3ab6f61 2020-09-22 op | tee >(grep something-else > b-fifo) \
85 ddc03123 2020-03-28 op Well, that can be rewritten as
88 a3ab6f61 2020-09-22 op unbound-control stats \
89 a3ab6f61 2020-09-22 op | sed 's/something/&/w a-fifo' \
90 a3ab6f61 2020-09-22 op | sed 's/something-else/&/w b-fifo \
94 a3ab6f61 2020-09-22 op by leveraging some sed behaviors:
96 ddc03123 2020-03-28 op 1. every line read is printed to stdout (eventually transformed)
97 ddc03123 2020-03-28 op 2. after the `s` command you can use `w` to write the transformed lines
98 ddc03123 2020-03-28 op to a local file
100 ddc03123 2020-03-28 op I've got this *illumination* by reading [Sed - An introduction and
101 ddc03123 2020-03-28 op Tutorial by Bruce Barnett](http://www.grymoire.com/Unix/Sed.html).
103 ddc03123 2020-03-28 op I tried to use `sed 'g/something/w a-fifo'` but `sed` complained
104 ddc03123 2020-03-28 op about "extra characters after command", so I assume that the no-op
105 ddc03123 2020-03-28 op `s/something/&/` is needed.
109 ddc03123 2020-03-28 op Edit: I tried with `g/...` because I thought that `sed` had a `g` command
110 ddc03123 2020-03-28 op that behaves like the `vi(1)` or `sam(1)` `g` command. `sed` has a `g`
111 ddc03123 2020-03-28 op command, but does something different. The correct form would be `sed
112 ddc03123 2020-03-28 op '/something/w a-fifo'` because `sed`, like `awk(1)`, has the ability to
113 ddc03123 2020-03-28 op perform commands only on lines that match a regular expression.
117 ddc03123 2020-03-28 op I find this last technique pretty, even prettier than the `>(grep ... >)`
118 ddc03123 2020-03-28 op idiom I used previously since it avoids the subshell to do the filtering
119 ddc03123 2020-03-28 op and because it does not make assumption on the shell used -- it only
120 ddc03123 2020-03-28 op requires a POSIX shell.
122 ddc03123 2020-03-28 op I hope you found this interesting. At least to me it really is.
