1 I have a tiny i686 at home with OpenBSD where I run, amongst some other
2 things, an instance of unbound.
4 Last night I decided that I wanted a dashboard to collect some statistics
7 My first thought was the ELK stack. The only problem is the ram.
8 The little i686 has 1GB of ram, I don't know if it's enough to run
9 logstash, let alone the whole ELK.
11 A simple solution would be to collect the logs elsewhere, but I'm not
12 going to do this for various reason (lazyness being the first, followed
13 by the fact that having statistics about my dns queries isn't that
14 useful in my opinion, even if it's nice-to-have.)
16 Instead, my solution involves a bit of bash (don't hate me on this),
17 some fifos, tmux and ttyplot.
19 The primarly source of inspiration is [this
20 post](https://dataswamp.org/~solene/2019-07-29-ttyplot.html) that I red
21 some time ago: it's about plotting various system statistics with ttyplot.
25 
27 (note that I usually disable colors in xterm)
33 / various -------> multiple
34 unbound stats ------- fifos -------> ttyplot
35 \ -------> per tmux pane
39 The idea is to run `unbound-control stats` every once in a while,
40 multiplexing its output and draw each (interesting) stats with ttyplot
43 Why the fifos? Well, if I'm not wrong, every time you call
44 `unbound-control stats` it will clear the statistics, so you can't run
45 it *n* times to collect *n* different stats. And since the whole setup
46 requires only a couple of script, the easiest way was to use some fifos.
48 The whole setup requires three script:
54 ### `gen-dashboard.sh`
56 This is the startup script. I run it on my crontab as `@reboot
57 /path/to/gen-dashboard.sh`. It will create the required fifos, then
58 spawn a tmux session and create two windows and some panes.
64 for f in netstat queries hit miss time; do
70 tmux new-session -d -s $session
73 tmux new-window -t $session:1 -n 'logs'
74 tmux send-keys "/path/to/mystatd.sh" C-m
76 # create the dashboard
77 tmux select-window -t $session:0
79 # setup the layout of the panes
84 tmux split-window -v -p 66
85 tmux split-window -v -p 50
87 # load the correct ttyplot in the panes
89 tmux send-keys "/path/to/dashboard.sh netstat" C-m
92 tmux send-keys "/path/to/dashboard.sh queries" C-m
95 tmux send-keys "/path/to/dashboard.sh hit" C-m
98 tmux send-keys "/path/to/dashboard.sh miss" C-m
100 tmux select-pane -t 4
101 tmux send-keys "/path/to/dashboard.sh time" C-m
104 (A possible improvement may be to tell tmux which command to run when
105 creating a pane instead of sending the keys to the shell, but it works
108 There's nothing special about this script, so let's move to the next.
112 This script also isn't interesting, all it does is pull the data out of
113 the correct fifo and start ttyplot with the correct labels and units.
119 echo "missing dashboard type"
120 echo "usage: $0 <dashboard-name>"
128 done) | ttyplot -t "IN Bandwidth in KB/s" \
135 done) | ttyplot -t "DNS Queries/5s" \
142 done) | ttyplot -t "DNS cache hit/5s" \
149 done) | ttyplot -t "DNS cache miss/5s" \
156 done) | ttyplot -t "DNS query time avg/5s" \
161 printf "%s\n" "$1 is not a valid dashboard"
169 This is the (only?) interesting script. It's also the only one that
170 requires bash, because I'm lazy, it was already installed as dependecy of
171 something, and because of the `>(cmd)` construct. Rewriting the script
172 using only pure sh(1) constructs is left as an exercise to the reader
173 (hint: you need some extra fifo.)
179 grep "$1" | awk -F= '{print $2}' > /tmp/my-$2
184 unbound-control stats \
186 | tee >(filter queries= queries) \
187 | tee >(filter hit hit) \
188 | tee >(filter miss miss) \
189 | filter time.avg time
194 # netstat - ty solene@ for the awk
204 if(!index($4,":") && old>=0) {
212 }' | tee -a /tmp/my-netstat
218 The first piece collects the stat from unbound. Let's break it in pieces.
220 - `unbound-control stats` outputs the stats. Keep in mind that this
221 requires some privileges. I've solved this by creating a script
222 in /usr/local/bin that executes the command and allowed my user to
223 launch that script via `doas(1)`. Or you can run `mystatd.sh` as root.
225 - `grep -v thread0` removes the per-thread stats (since my unbound
226 uses only one thread). A more solid approach like `egrep -v ^thread`
228 - `tee >(filter queries= queries) |` duplicates the stream: one copy
229 goes to the subshell with `filter` and another copy goes on the pipes.
230 - `filter` is just a small function to grep the desired entry and send
231 it to `/tmp/my-$something`
233 The netstat bit filters the output of netstat (the awk is copied-pasted
234 from the previously linked post by solene@). You may want to change the
235 `^em0` to match your network device.
239 ## Possible improvements
241 - if you `SIGINT` `mystatd.sh` some of its subprocess still run. Maybe a
242 `trap` is needed. Since it is the only bash running on that system,
243 `pkill bash` is, albeit a bit aggressive, a working solution.
244 - replace bash. It's not difficult, but requires more fifos.
247 ## Final considerations
249 This was fun. Now I have a tmux session I remotely attach with cool
250 graphs. This doesn't cover the archiviation of the statistics tho.
251 I think it should be trivial to add (just one more `|tee -a` to a local
252 file, maybe a cronjob to do rotation, ...) but for the moment I'm happy
