1 ddc03123 2020-03-28 op I have a tiny i686 at home with OpenBSD where I run, amongst some other
2 ddc03123 2020-03-28 op things, an instance of unbound.
4 ddc03123 2020-03-28 op Last night I decided that I wanted a dashboard to collect some statistics
7 ddc03123 2020-03-28 op My first thought was the ELK stack. The only problem is the ram.
8 ddc03123 2020-03-28 op The little i686 has 1GB of ram, I don't know if it's enough to run
9 ddc03123 2020-03-28 op logstash, let alone the whole ELK.
11 ddc03123 2020-03-28 op A simple solution would be to collect the logs elsewhere, but I'm not
12 ddc03123 2020-03-28 op going to do this for various reason (lazyness being the first, followed
13 ddc03123 2020-03-28 op by the fact that having statistics about my dns queries isn't that
14 ddc03123 2020-03-28 op useful in my opinion, even if it's nice-to-have.)
16 ddc03123 2020-03-28 op Instead, my solution involves a bit of bash (don't hate me on this),
17 ddc03123 2020-03-28 op some fifos, tmux and ttyplot.
19 ddc03123 2020-03-28 op The primarly source of inspiration is [this
20 ddc03123 2020-03-28 op post](https://dataswamp.org/~solene/2019-07-29-ttyplot.html) that I red
21 ddc03123 2020-03-28 op some time ago: it's about plotting various system statistics with ttyplot.
23 ddc03123 2020-03-28 op The result is this
25 ddc03123 2020-03-28 op 
27 ddc03123 2020-03-28 op (note that I usually disable colors in xterm)
33 a3ab6f61 2020-09-22 op / various -------> multiple
34 a3ab6f61 2020-09-22 op unbound stats ------- fifos -------> ttyplot
35 a3ab6f61 2020-09-22 op \ -------> per tmux pane
39 ddc03123 2020-03-28 op The idea is to run `unbound-control stats` every once in a while,
40 ddc03123 2020-03-28 op multiplexing its output and draw each (interesting) stats with ttyplot
41 ddc03123 2020-03-28 op in a tmux pane.
43 ddc03123 2020-03-28 op Why the fifos? Well, if I'm not wrong, every time you call
44 ddc03123 2020-03-28 op `unbound-control stats` it will clear the statistics, so you can't run
45 ddc03123 2020-03-28 op it *n* times to collect *n* different stats. And since the whole setup
46 ddc03123 2020-03-28 op requires only a couple of script, the easiest way was to use some fifos.
48 ddc03123 2020-03-28 op The whole setup requires three script:
50 ddc03123 2020-03-28 op - `gen-dashboard.sh`
51 ddc03123 2020-03-28 op - `dashboard.sh`
52 ddc03123 2020-03-28 op - `mystatd.sh`
54 ddc03123 2020-03-28 op ### `gen-dashboard.sh`
56 ddc03123 2020-03-28 op This is the startup script. I run it on my crontab as `@reboot
57 ddc03123 2020-03-28 op /path/to/gen-dashboard.sh`. It will create the required fifos, then
58 ddc03123 2020-03-28 op spawn a tmux session and create two windows and some panes.
63 ddc03123 2020-03-28 op # create the fifos
64 ddc03123 2020-03-28 op for f in netstat queries hit miss time; do
65 ddc03123 2020-03-28 op mkfifo /tmp/my-$f
68 ddc03123 2020-03-28 op session=dashboard
70 ddc03123 2020-03-28 op tmux new-session -d -s $session
72 ddc03123 2020-03-28 op # start mystatd.sh
73 ddc03123 2020-03-28 op tmux new-window -t $session:1 -n 'logs'
74 ddc03123 2020-03-28 op tmux send-keys "/path/to/mystatd.sh" C-m
76 ddc03123 2020-03-28 op # create the dashboard
77 ddc03123 2020-03-28 op tmux select-window -t $session:0
79 ddc03123 2020-03-28 op # setup the layout of the panes
80 ddc03123 2020-03-28 op tmux split-window -h
81 ddc03123 2020-03-28 op tmux select-pane -L
82 ddc03123 2020-03-28 op tmux split-window -v
83 ddc03123 2020-03-28 op tmux select-pane -R
84 ddc03123 2020-03-28 op tmux split-window -v -p 66
85 ddc03123 2020-03-28 op tmux split-window -v -p 50
87 ddc03123 2020-03-28 op # load the correct ttyplot in the panes
88 ddc03123 2020-03-28 op tmux select-pane -t 0
89 ddc03123 2020-03-28 op tmux send-keys "/path/to/dashboard.sh netstat" C-m
91 ddc03123 2020-03-28 op tmux select-pane -t 1
92 ddc03123 2020-03-28 op tmux send-keys "/path/to/dashboard.sh queries" C-m
94 ddc03123 2020-03-28 op tmux select-pane -t 2
95 ddc03123 2020-03-28 op tmux send-keys "/path/to/dashboard.sh hit" C-m
97 ddc03123 2020-03-28 op tmux select-pane -t 3
98 ddc03123 2020-03-28 op tmux send-keys "/path/to/dashboard.sh miss" C-m
100 ddc03123 2020-03-28 op tmux select-pane -t 4
101 ddc03123 2020-03-28 op tmux send-keys "/path/to/dashboard.sh time" C-m
104 ddc03123 2020-03-28 op (A possible improvement may be to tell tmux which command to run when
105 ddc03123 2020-03-28 op creating a pane instead of sending the keys to the shell, but it works
106 ddc03123 2020-03-28 op neverthless.)
108 ddc03123 2020-03-28 op There's nothing special about this script, so let's move to the next.
110 ddc03123 2020-03-28 op ### `dashboard.sh`
112 ddc03123 2020-03-28 op This script also isn't interesting, all it does is pull the data out of
113 ddc03123 2020-03-28 op the correct fifo and start ttyplot with the correct labels and units.
118 ddc03123 2020-03-28 op if [ -z "$1" ]; then
119 ddc03123 2020-03-28 op echo "missing dashboard type"
120 ddc03123 2020-03-28 op echo "usage: $0 <dashboard-name>"
127 ddc03123 2020-03-28 op cat /tmp/my-netstat
128 ddc03123 2020-03-28 op done) | ttyplot -t "IN Bandwidth in KB/s" \
129 ddc03123 2020-03-28 op -u "KB/s" -c "#"
134 ddc03123 2020-03-28 op cat /tmp/my-queries
135 ddc03123 2020-03-28 op done) | ttyplot -t "DNS Queries/5s" \
136 ddc03123 2020-03-28 op -u "q/5s" -c "#"
141 ddc03123 2020-03-28 op cat /tmp/my-hit
142 ddc03123 2020-03-28 op done) | ttyplot -t "DNS cache hit/5s" \
143 ddc03123 2020-03-28 op -u "ch/5s" -c "#"
148 ddc03123 2020-03-28 op cat /tmp/my-miss
149 ddc03123 2020-03-28 op done) | ttyplot -t "DNS cache miss/5s" \
150 ddc03123 2020-03-28 op -u "cm/5s" -c "#"
155 ddc03123 2020-03-28 op cat /tmp/my-time
156 ddc03123 2020-03-28 op done) | ttyplot -t "DNS query time avg/5s" \
161 ddc03123 2020-03-28 op printf "%s\n" "$1 is not a valid dashboard"
167 ddc03123 2020-03-28 op ### `mystatd.sh`
169 ddc03123 2020-03-28 op This is the (only?) interesting script. It's also the only one that
170 ddc03123 2020-03-28 op requires bash, because I'm lazy, it was already installed as dependecy of
171 ddc03123 2020-03-28 op something, and because of the `>(cmd)` construct. Rewriting the script
172 ddc03123 2020-03-28 op using only pure sh(1) constructs is left as an exercise to the reader
173 ddc03123 2020-03-28 op (hint: you need some extra fifo.)
176 ddc03123 2020-03-28 op #!/usr/bin/env bash
179 ddc03123 2020-03-28 op grep "$1" | awk -F= '{print $2}' > /tmp/my-$2
182 ddc03123 2020-03-28 op # unbound stats
183 ddc03123 2020-03-28 op ( while :; do
184 ddc03123 2020-03-28 op unbound-control stats \
185 ddc03123 2020-03-28 op | grep -v thread0 \
186 ddc03123 2020-03-28 op | tee >(filter queries= queries) \
187 ddc03123 2020-03-28 op | tee >(filter hit hit) \
188 ddc03123 2020-03-28 op | tee >(filter miss miss) \
189 ddc03123 2020-03-28 op | filter time.avg time
194 ddc03123 2020-03-28 op # netstat - ty solene@ for the awk
199 ddc03123 2020-03-28 op done) | awk '
204 ddc03123 2020-03-28 op if(!index($4,":") && old>=0) {
205 ddc03123 2020-03-28 op print ($5-old)/1024
209 ddc03123 2020-03-28 op if(old==-1) {
212 ddc03123 2020-03-28 op }' | tee -a /tmp/my-netstat
218 ddc03123 2020-03-28 op The first piece collects the stat from unbound. Let's break it in pieces.
220 ddc03123 2020-03-28 op - `unbound-control stats` outputs the stats. Keep in mind that this
221 ddc03123 2020-03-28 op requires some privileges. I've solved this by creating a script
222 ddc03123 2020-03-28 op in /usr/local/bin that executes the command and allowed my user to
223 ddc03123 2020-03-28 op launch that script via `doas(1)`. Or you can run `mystatd.sh` as root.
224 ddc03123 2020-03-28 op Do as you please.
225 ddc03123 2020-03-28 op - `grep -v thread0` removes the per-thread stats (since my unbound
226 ddc03123 2020-03-28 op uses only one thread). A more solid approach like `egrep -v ^thread`
227 ddc03123 2020-03-28 op is probably better.
228 ddc03123 2020-03-28 op - `tee >(filter queries= queries) |` duplicates the stream: one copy
229 ddc03123 2020-03-28 op goes to the subshell with `filter` and another copy goes on the pipes.
230 ddc03123 2020-03-28 op - `filter` is just a small function to grep the desired entry and send
231 ddc03123 2020-03-28 op it to `/tmp/my-$something`
233 ddc03123 2020-03-28 op The netstat bit filters the output of netstat (the awk is copied-pasted
234 ddc03123 2020-03-28 op from the previously linked post by solene@). You may want to change the
235 ddc03123 2020-03-28 op `^em0` to match your network device.
237 ddc03123 2020-03-28 op And that's all!
239 ddc03123 2020-03-28 op ## Possible improvements
241 ddc03123 2020-03-28 op - if you `SIGINT` `mystatd.sh` some of its subprocess still run. Maybe a
242 ddc03123 2020-03-28 op `trap` is needed. Since it is the only bash running on that system,
243 ddc03123 2020-03-28 op `pkill bash` is, albeit a bit aggressive, a working solution.
244 ddc03123 2020-03-28 op - replace bash. It's not difficult, but requires more fifos.
247 ddc03123 2020-03-28 op ## Final considerations
249 ddc03123 2020-03-28 op This was fun. Now I have a tmux session I remotely attach with cool
250 ddc03123 2020-03-28 op graphs. This doesn't cover the archiviation of the statistics tho.
251 ddc03123 2020-03-28 op I think it should be trivial to add (just one more `|tee -a` to a local
252 ddc03123 2020-03-28 op file, maybe a cronjob to do rotation, ...) but for the moment I'm happy
253 ddc03123 2020-03-28 op with this result.
