commit - 1c7e97664992c1a96487c2dcf425c52f72ce9f90
commit + 8b4d5615764cd32d20fd81c458f8c82859a6e1b5
blob - /dev/null
blob + 5789f2bc239734af7b010c1aa310396808313a7e (mode 644)
--- /dev/null
+++ gotmarc.7
+.\" gotmarc.7 was written by Omar Polo <op@openbsd.org> and is placed in
+.\" the public domain. The author hereby disclaims copyright to this
+.\" source code.
+.Dd May 5, 2023
+.Dt GOTMARC 7
+.Os
+.Sh NAME
+.Nm gotmarc
+.Nd mailing list web archive generation system
+.Sh DESCRIPTION
+.Nm
+is a system to incrementally generate a web archive for a mailing list
+and optionally provide search capabilities.
+The generated archive is a set of HTML files that can be served as-is,
+while searching requires the use of a FastCGI server.
+At a higher level,
+.Nm
+is comprised of three main components:
+.Pp
+.Bl -tag -width msearchd_8_ -compact -offset indent
+.It Xr gotmarc 1
+to generate the static HTML files from a maildir.
+.It Xr gmimport 1
+to import emails into a sqlite3 database.
+.It Xr msearchd 8
+to provide search results.
+.El
+.Sh INITIAL SETUP
+There are several step necessary to initialize the web archive:
+.Pp
+.Bl -enum -compact -offset indent
+.It
+Create and populate the output directory.
+.It
+Customize the templates.
+.It
+Prepare the maildir.
+.It
+Generate the web archive.
+.It
+Set up the database for searching.
+.It
+Configure the web server.
+.El
+.Pp
+It is reccommended to use a dedicate user.
+Commands to be run as a unpriviledged user are preceded by a dollar sign
+.Sq $ ,
+while commands requiring superuser privileges by a hash mark
+.Sq # .
+Hereafter, it will be assumed that the local user is called
+.Sq gotmarc .
+.Ss 1. Create and populate the output directory
+The web archive is made of several static files, mostly HTML, that needs
+to be served by a web server like
+.Xr httpd 8 .
+.Pa /var/www/gotmarc
+is the default location, but a different path can be used.
+To prepare it, issue:
+.Bd -literal -offset indent
+# mkdir -p /var/www/gotmarc
+# chown gotmarc /var/www/gotmarc
+.Ed
+.Pp
+Then copy the CSS file, optionally tweaking it.
+.Pp
+.Dl $ cp /usr/local/share/examples/gotmarc/style.css /var/www/gotmarc
+.Pp
+Other eventual assets
+.Pq e.g.\& logo images
+need to be copied here as well.
+.Ss 2. Customize the templates
+The default templates are installed at
+.Pa /etc/gotmarc .
+Since these are anonymous, they need to be tweaked to include
+information about the mailing list.
+.Pp
+Care should be taken when editing these files after generating the
+archive since existing pages won't be re-generated.
+The outdir and cachedir
+.Pq see Xr gotmarc 1
+needs to be deleted and the web archive generated again.
+.Xr msearchd 8
+has to be stopped and restarted as well.
+.Ss 3. Prepare the maildir
+The maildir with the mailing list entries needs to be prepared.
+It is assumed to be at
+.Pa ~/Mail/gotmarc
+by default, but a different path can be used.
+.Ss 4. Generate the web archive
+.Xr gotmarc 1
+can be finally used to generate the web archive.
+The first run may take a while, depending on the size of the maildir,
+while subsequent runs will be incremental and take less time.
+.Pp
+.Dl $ gotmarc -m path/to/maildir -o path/to/outdir
+.Pp
+On multi-processor machines multiple processes may be used to save some
+time with the
+.Xr gotmarc 1 Fl j No flag.
+.Pp
+The generated files may be compressed to save bandwidth:
+.Pp
+.Dl $ gzip -krq /var/www/gotmarc </dev/null 2>/dev/null
+.Ss 5. Set up the database for searching
+This is an suggested yet optional step.
+.Pp
+.Xr msearchd 8
+offers full text search capabilities using a sqlite3 database that has to
+be populated with
+.Xr gmimport 1 .
+First, create a directory in the
+.Pa /var/www
+.Xr chroot 8
+jail:
+.Bd -literal -offset indent
+# mkdir -p /var/www/msearchd
+# chown gotmarc /var/www/msearchd
+.Ed
+.Pp
+Then, populate the database with all emails in the maildir:
+.Bd -literal -offset indent
+$ sqlite3 /var/www/msearchd/mails.sqlite3 \e
+ </usr/local/share/examples/gotmarc/schema.sql
+$ mlist ~/Mail/gotmarc | gmimport /var/www/msearchd/mails.sqlite3
+.Ed
+.Pp
+At this point,
+.Xr msearchd 8
+can be started.
+.Ss 6. Configure the web server
+The web server needs to serve the contents of the outdir as-is and
+handle the requests for
+.Pa /search
+via the
+.Xr msearchd 8
+FastCGI server.
+A sample
+.Xr httpd 8
+configuration is provided here for reference:
+.Bd -literal -offset indent
+server "marc.example.com" {
+ listen on * port 80
+ root "/gotmarc"
+ gzip-static
+
+ # leave out when not using msearchd(8)
+ location "/search" {
+ fastcgi socket "/run/msearchd.sock"
+ }
+}
+.Ed
+.Sh HANDLING NEW MESSAGES
+New messages should be fetched periodically using tools like
+.Xr fdm 1
+or
+.Xr mbsync 1 ,
+the database updated with
+.Xr gmimport 1
+and the web archive refreshed using
+.Xr gotmarc 1 .
+.Pp
+It is reccommended to create a script like the following and schedule
+its execution periodically with
+.Xr cron 8 :
+.Bd -literal -offset indent
+#!/bin/sh
+
+set -e
+fdm -l fetch
+minc ~/Mail/gotmarc | gmimport /var/www/msearchd/mails.sqlite3
+gotmarc
+gzip -krq /var/www/gotmarc/ </dev/null 2>/dev/null || true
+.Ed
+.Pp
+If
+.Xr msearchd 8
+is not used,
+new messages still needs to be incorporated
+.Po i.e.\& moved from
+.Pa new/
+to
+.Pa cur/
+.Pc
+but no database has to be updated.
+In that case simplify the
+.Xr minc 1
+invocation as:
+.Pp
+.Dl minc -q ~/Mail/gotmarc
+.Pp
+and don't call
+.Xr gmimport 1
+at all.
+.Sh HANDLING MULTIPLE MAILING LISTS
+If the archive for multiple mailing lists needs to be served from the
+same box, care must be taken to use different directories and database
+files to avoid mixing messages.
+.Pp
+.Xr msearchd 8
+handles only one database at a time, so multiple instances need to be
+run, each pointing at the database for only one mailing list.
+Different FastCGI socket path needs to be used per-instance.
+.Pp
+.Xr gotmarc 1
+outdir, maildir and cachedir must be unique per-mailing list, i.e.\& the
+.Fl c , Fl m No and Fl o
+flag must always be provided.
+.Pp
+Very likely, each mailing list will needs its own set of templates, so
+those needs to be prepared and both
+.Xr gotmarc 1
+and
+.Xr msearchd 8
+have to be pointed at the right template directory.
+.Sh SEE ALSO
+.Xr gmimport 1 ,
+.Xr gotmarc 1 ,
+.Xr minc 1 ,
+.Xr sqlite3 1 ,
+.Xr httpd 8 ,
+.Xr msearchd 8
