1 .\" smarc.7 was written by Omar Polo <op@openbsd.org> and is placed in
2 .\" the public domain. The author hereby disclaims copyright to this
9 .Nd mailing list web archive generation system
12 is a system to incrementally generate a web archive for a mailing list
13 and optionally provide search capabilities.
14 The generated archive is a set of HTML files that can be served as-is,
15 while searching requires the use of a FastCGI server.
18 is made of three components:
20 .Bl -tag -width msearchd_8_ -compact -offset indent
22 to generate the static HTML files from a maildir.
24 to import emails into a sqlite3 database.
26 to provide search results.
29 There are several step necessary to initialize the web archive:
31 .Bl -enum -compact -offset indent
33 Create and populate the output directory.
35 Customize the templates.
39 Generate the web archive.
41 Set up the database for searching.
43 Configure the web server.
46 It is reccommended to use a dedicate user.
47 Commands to be run as a unpriviledged user are preceded by a dollar sign
49 while commands requiring superuser privileges by a hash mark
51 Hereafter, it will be assumed that the local user is called
53 .Ss 1. Create and populate the output directory
54 The web archive is made of several static files, mostly HTML, that needs
55 to be served by a web server like
58 is the default location, but a different path can be used.
60 .Bd -literal -offset indent
61 # mkdir -p /var/www/smarc
62 # chown smarc /var/www/smarc
65 Then copy the CSS file, optionally tweaking it.
67 .Dl $ cp /usr/local/share/examples/smarc/style.css /var/www/smarc
70 .Pq e.g.\& logo images
71 need to be copied here as well.
72 .Ss 2. Customize the templates
73 The default templates are installed at
75 Since these are anonymous, they need to be tweaked to include
76 information about the mailing list.
78 Care should be taken when editing these files after generating the
79 archive since existing pages won't be automatically updated.
82 needs to be deleted and the web archive generated again.
84 has to be stopped and restarted as well.
85 .Ss 3. Prepare the maildir
86 The maildir with the mailing list entries needs to be prepared.
87 It is assumed to be at
89 by default, but a different path can be used.
90 .Ss 4. Generate the web archive
92 can be finally used to generate the web archive.
93 The first run may take a while, depending on the size of the maildir,
94 while subsequent runs will be incremental and take less time.
96 .Dl $ smarc -m path/to/maildir -o path/to/outdir
98 On multi-processor machines multiple processes may be used to save some
100 .Xr smarc 1 Fl j No flag.
102 The generated files may be compressed to save bandwidth:
104 .Dl $ gzip -krq /var/www/smarc </dev/null 2>/dev/null
105 .Ss 5. Set up the database for searching
106 This is an suggested yet optional step.
109 offers full text search capabilities using a sqlite3 database that has to
112 First, create a directory in the
116 .Bd -literal -offset indent
117 # mkdir -p /var/www/msearchd
118 # chown smarc /var/www/msearchd
121 Then, populate the database with all emails in the maildir:
122 .Bd -literal -offset indent
123 $ sqlite3 /var/www/msearchd/mails.sqlite3 \e
124 </usr/local/share/examples/smarc/schema.sql
125 $ mlist ~/Mail/smarc | smingest /var/www/msearchd/mails.sqlite3
131 .Ss 6. Configure the web server
132 The web server needs to serve the contents of the outdir as-is and
133 handle the requests for
140 configuration is provided here for reference:
141 .Bd -literal -offset indent
142 server "marc.example.com" {
147 # leave out when not using msearchd(8)
149 fastcgi socket "/run/msearchd.sock"
153 .Sh HANDLING NEW MESSAGES
154 New messages should be fetched periodically using tools like
158 the database updated with
160 and the web archive refreshed using
163 It is reccommended to create a script like the following and schedule
164 its execution periodically with
166 .Bd -literal -offset indent
171 minc ~/Mail/smarc | smingest /var/www/msearchd/mails.sqlite3
173 yes | gzip -krq /var/www/smarc/ 2>/dev/null || true
179 new messages still needs to be incorporated
180 .Po i.e.\& moved from
185 but no database has to be updated.
186 In that case simplify the
190 .Dl minc -q ~/Mail/smarc
195 .Sh HANDLING MULTIPLE MAILING LISTS
196 If the archive for multiple mailing lists needs to be served from the
197 same box, care must be taken to use different directories and database
198 files to avoid mixing messages.
201 handles only one database at a time, so multiple instances need to be
202 run, each pointing at the database for only one mailing list.
203 Different FastCGI socket path needs to be used per-instance.
206 outdir, maildir and cachedir must be unique per-mailing list, i.e.\& the
207 .Fl c , Fl m No and Fl o
208 flag must always be provided.
210 Very likely, each mailing list will needs its own set of templates, so
211 those needs to be prepared and both
215 have to be pointed at the right template directory.
