commit 8b4d5615764cd32d20fd81c458f8c82859a6e1b5 from: Omar Polo date: Fri May 05 19:46:57 2023 UTC add gotmarc.7: an introduction to the gotmarc system commit - 1c7e97664992c1a96487c2dcf425c52f72ce9f90 commit + 8b4d5615764cd32d20fd81c458f8c82859a6e1b5 blob - /dev/null blob + 5789f2bc239734af7b010c1aa310396808313a7e (mode 644) --- /dev/null +++ gotmarc.7 @@ -0,0 +1,222 @@ +.\" gotmarc.7 was written by Omar Polo and is placed in +.\" the public domain. The author hereby disclaims copyright to this +.\" source code. +.Dd May 5, 2023 +.Dt GOTMARC 7 +.Os +.Sh NAME +.Nm gotmarc +.Nd mailing list web archive generation system +.Sh DESCRIPTION +.Nm +is a system to incrementally generate a web archive for a mailing list +and optionally provide search capabilities. +The generated archive is a set of HTML files that can be served as-is, +while searching requires the use of a FastCGI server. +At a higher level, +.Nm +is comprised of three main components: +.Pp +.Bl -tag -width msearchd_8_ -compact -offset indent +.It Xr gotmarc 1 +to generate the static HTML files from a maildir. +.It Xr gmimport 1 +to import emails into a sqlite3 database. +.It Xr msearchd 8 +to provide search results. +.El +.Sh INITIAL SETUP +There are several step necessary to initialize the web archive: +.Pp +.Bl -enum -compact -offset indent +.It +Create and populate the output directory. +.It +Customize the templates. +.It +Prepare the maildir. +.It +Generate the web archive. +.It +Set up the database for searching. +.It +Configure the web server. +.El +.Pp +It is reccommended to use a dedicate user. +Commands to be run as a unpriviledged user are preceded by a dollar sign +.Sq $ , +while commands requiring superuser privileges by a hash mark +.Sq # . +Hereafter, it will be assumed that the local user is called +.Sq gotmarc . +.Ss 1. Create and populate the output directory +The web archive is made of several static files, mostly HTML, that needs +to be served by a web server like +.Xr httpd 8 . +.Pa /var/www/gotmarc +is the default location, but a different path can be used. +To prepare it, issue: +.Bd -literal -offset indent +# mkdir -p /var/www/gotmarc +# chown gotmarc /var/www/gotmarc +.Ed +.Pp +Then copy the CSS file, optionally tweaking it. +.Pp +.Dl $ cp /usr/local/share/examples/gotmarc/style.css /var/www/gotmarc +.Pp +Other eventual assets +.Pq e.g.\& logo images +need to be copied here as well. +.Ss 2. Customize the templates +The default templates are installed at +.Pa /etc/gotmarc . +Since these are anonymous, they need to be tweaked to include +information about the mailing list. +.Pp +Care should be taken when editing these files after generating the +archive since existing pages won't be re-generated. +The outdir and cachedir +.Pq see Xr gotmarc 1 +needs to be deleted and the web archive generated again. +.Xr msearchd 8 +has to be stopped and restarted as well. +.Ss 3. Prepare the maildir +The maildir with the mailing list entries needs to be prepared. +It is assumed to be at +.Pa ~/Mail/gotmarc +by default, but a different path can be used. +.Ss 4. Generate the web archive +.Xr gotmarc 1 +can be finally used to generate the web archive. +The first run may take a while, depending on the size of the maildir, +while subsequent runs will be incremental and take less time. +.Pp +.Dl $ gotmarc -m path/to/maildir -o path/to/outdir +.Pp +On multi-processor machines multiple processes may be used to save some +time with the +.Xr gotmarc 1 Fl j No flag. +.Pp +The generated files may be compressed to save bandwidth: +.Pp +.Dl $ gzip -krq /var/www/gotmarc /dev/null +.Ss 5. Set up the database for searching +This is an suggested yet optional step. +.Pp +.Xr msearchd 8 +offers full text search capabilities using a sqlite3 database that has to +be populated with +.Xr gmimport 1 . +First, create a directory in the +.Pa /var/www +.Xr chroot 8 +jail: +.Bd -literal -offset indent +# mkdir -p /var/www/msearchd +# chown gotmarc /var/www/msearchd +.Ed +.Pp +Then, populate the database with all emails in the maildir: +.Bd -literal -offset indent +$ sqlite3 /var/www/msearchd/mails.sqlite3 \e + /dev/null || true +.Ed +.Pp +If +.Xr msearchd 8 +is not used, +new messages still needs to be incorporated +.Po i.e.\& moved from +.Pa new/ +to +.Pa cur/ +.Pc +but no database has to be updated. +In that case simplify the +.Xr minc 1 +invocation as: +.Pp +.Dl minc -q ~/Mail/gotmarc +.Pp +and don't call +.Xr gmimport 1 +at all. +.Sh HANDLING MULTIPLE MAILING LISTS +If the archive for multiple mailing lists needs to be served from the +same box, care must be taken to use different directories and database +files to avoid mixing messages. +.Pp +.Xr msearchd 8 +handles only one database at a time, so multiple instances need to be +run, each pointing at the database for only one mailing list. +Different FastCGI socket path needs to be used per-instance. +.Pp +.Xr gotmarc 1 +outdir, maildir and cachedir must be unique per-mailing list, i.e.\& the +.Fl c , Fl m No and Fl o +flag must always be provided. +.Pp +Very likely, each mailing list will needs its own set of templates, so +those needs to be prepared and both +.Xr gotmarc 1 +and +.Xr msearchd 8 +have to be pointed at the right template directory. +.Sh SEE ALSO +.Xr gmimport 1 , +.Xr gotmarc 1 , +.Xr minc 1 , +.Xr sqlite3 1 , +.Xr httpd 8 , +.Xr msearchd 8