1 021d5bb6 2021-08-12 op As part of the regression suite for a project I’m working on, I designed a simple scripting language (which is not even Turing-complete by the way) to create specific situations and test how the program respond. I’ve almost finished the interpreter for it, so it’s the time to start writing tests. How do you edit a file if you don’t have a proper major mode available? You write one!
3 021d5bb6 2021-08-12 op A major mode is a lisp program that manage how the user interacts with the content of a buffer. (Friendly remainder that a buffer may or may not be an actual file; things like dired or elpher are major modes after all, but they’re not the kind of modes I’m interested in now.)
5 021d5bb6 2021-08-12 op Major modes for text files usually do at least three things:
6 021d5bb6 2021-08-12 op * font-lock (i.e. syntax highlighting)
7 021d5bb6 2021-08-12 op * setup the syntax-table
8 021d5bb6 2021-08-12 op * manage indentation (the hardest part)
9 021d5bb6 2021-08-12 op and probably more, like providing useful keybindings and interactions with other packages.
11 021d5bb6 2021-08-12 op I’ve never had to deal with the fontification or syntax tables, nor realised how difficult the indentation can be, so it’s been lots of fun.
13 021d5bb6 2021-08-12 op The difficulty of writing a major mode seems to be at least proportional to the “complexness” of the target language. In my case, the grammar of the language is dead-simple and so the major mode is simple too. cc-mode on the other hand is probably at the other side of the spectrum (well, after all it manages C, C++, Java, AWK and more…)
15 021d5bb6 2021-08-12 op Before describing the elisp implementation, here’s a look at the custom DLS, “nps”:
18 021d5bb6 2021-08-12 op include "lib.nps"
20 021d5bb6 2021-08-12 op # consts comes in two flavors
25 021d5bb6 2021-08-12 op const foo = "hello there"
27 021d5bb6 2021-08-12 op # procedures works as expected, … is for the rest argument
28 021d5bb6 2021-08-12 op proc message(type, ...) {
29 021d5bb6 2021-08-12 op send(type:u8, ...) # type casts
32 021d5bb6 2021-08-12 op # it’s a DSL for regression tests after all
33 021d5bb6 2021-08-12 op testing "cooking skills" {
34 021d5bb6 2021-08-12 op message(Make, "me", "a", "sandwitch")
37 021d5bb6 2021-08-12 op # asserts comes in two flavors too
39 021d5bb6 2021-08-12 op m.type == What
40 021d5bb6 2021-08-12 op m.content == "Make it yourself."
42 021d5bb6 2021-08-12 op assert m.id = 5
46 021d5bb6 2021-08-12 op Now let’s jump in to the mode implementation.
48 021d5bb6 2021-08-12 op The elisp file starts with the usual header. I’m enabling the lexical-binding even if it’s the default from emacs 27
50 021d5bb6 2021-08-12 op ```elisp header
51 021d5bb6 2021-08-12 op ;;; nps-mode.el --- major mode for nps -*- lexical-binding: t; -*-
54 021d5bb6 2021-08-12 op I’ll also make use of the rx library to write regexps, so
56 021d5bb6 2021-08-12 op (eval-when-compile
57 021d5bb6 2021-08-12 op (require 'rx))
60 021d5bb6 2021-08-12 op ## fontification
62 021d5bb6 2021-08-12 op i.e. syntax highlighting. There are probably different ways of doing this, but I’ll stick with the simplest one: a bunch of regexps.
64 021d5bb6 2021-08-12 op ```defining the font lock regexps
65 021d5bb6 2021-08-12 op (defconst nps--font-lock-defaults
66 021d5bb6 2021-08-12 op (let ((keywords '("assert" "const" "include" "proc" "testing"))
67 021d5bb6 2021-08-12 op (types '("str" "u8" "u16" "u32")))
68 021d5bb6 2021-08-12 op `(((,(rx-to-string `(: (or ,@keywords))) 0 font-lock-keyword-face)
69 021d5bb6 2021-08-12 op ("\\([[:word:]]+\\)\s*(" 1 font-lock-function-name-face)
70 021d5bb6 2021-08-12 op (,(rx-to-string `(: (or ,@types))) 0 font-lock-type-face)))))
73 021d5bb6 2021-08-12 op Yes, I got the number of parenthesis wrong (multiple times) at first.
75 021d5bb6 2021-08-12 op This value will be later set to the buffer-local font-lock-defaults variable. I’ve not yet wrapped my head around the different levels mentioned in the documentation, but the code seems to work. We’re using rx to build a regexp that matches the keywords and using the face ‘font-lock-keyword-face’ for the matches. The zero is there because the regexp doesn’t have any sub-groups.
77 021d5bb6 2021-08-12 op The second entry is slightly more complex and interesting. It matches a symbol followed by an open paren and applies the face ‘font-lock-function-name-face’ to it. The regexp has a sub-group (the \\( and \\) bit) that matches only the symbol, and the number 1 tells font-lock to highlight only the first match and not the whole regexp.
79 021d5bb6 2021-08-12 op The third one is like the first, it highlights the “types”.
81 021d5bb6 2021-08-12 op ## syntax-table
83 021d5bb6 2021-08-12 op This is pure black magic, I can assure you. Nah, just kidding. But it looks like.
85 021d5bb6 2021-08-12 op It’s a very important piece of the major-mode. Various lisps function will inspect the current syntax-table to query over what kind of text the point is. It also interacts with the font-lock and various other parts of Emacs.
87 021d5bb6 2021-08-12 op This is also the part I’m less confident with. Some major-modes I’ve seen add explicit entries for the braces and the quotes, other doesn’t. I’ve decided to be explicit and list all the characters I’m using, just to be sure.
89 021d5bb6 2021-08-12 op The idea is to specify for each character (or range of characters) some properties. These properties are expressed in a very terse notation using a string. To add entries to the syntax table you need to use ‘modify-syntax-entry’: it takes the character (or range), the string description of the properties and the syntax table.
91 021d5bb6 2021-08-12 op The format of the specification is better explained in the elisp manual, but the gist is that is a sequence of character with a special interpretation. The first character identifies the “class” (punctuation, word component, comment delimeter, parenthesis, …), the second if not a space specifies the matching character, and then there are further fields that I won’t use.
93 021d5bb6 2021-08-12 op Just to provide an example before showing the code, in a programming language the syntax entry for the character ‘(’ probably looks like "()":
94 021d5bb6 2021-08-12 op * it’s a parethesis, as the first character is an open paren and
95 021d5bb6 2021-08-12 op * the ‘)’ character is its matching character.
96 021d5bb6 2021-08-12 op The syntax table for ‘)’ instead will look like "((" because
97 021d5bb6 2021-08-12 op * it’s a parenthesis
98 021d5bb6 2021-08-12 op * its matching character is ‘(’
100 021d5bb6 2021-08-12 op So, here’s the syntax table for nps in its all glory:
103 021d5bb6 2021-08-12 op (defvar nps-mode-syntax-table
104 021d5bb6 2021-08-12 op (let ((st (make-syntax-table)))
105 021d5bb6 2021-08-12 op (modify-syntax-entry ?\{ "(}" st)
106 021d5bb6 2021-08-12 op (modify-syntax-entry ?\} "){" st)
107 021d5bb6 2021-08-12 op (modify-syntax-entry ?\( "()" st)
109 021d5bb6 2021-08-12 op ;; - and _ are word constituents
110 021d5bb6 2021-08-12 op (modify-syntax-entry ?_ "w" st)
111 021d5bb6 2021-08-12 op (modify-syntax-entry ?- "w" st)
113 021d5bb6 2021-08-12 op ;; both single and double quotes makes strings
114 021d5bb6 2021-08-12 op (modify-syntax-entry ?\" "\"" st)
115 021d5bb6 2021-08-12 op (modify-syntax-entry ?' "'" st)
117 021d5bb6 2021-08-12 op ;; add comments. lua-mode does something similar, so it shouldn't
118 021d5bb6 2021-08-12 op ;; bee *too* wrong.
119 021d5bb6 2021-08-12 op (modify-syntax-entry ?# "<" st)
120 021d5bb6 2021-08-12 op (modify-syntax-entry ?\n ">" st)
122 021d5bb6 2021-08-12 op ;; '==' as punctuation
123 021d5bb6 2021-08-12 op (modify-syntax-entry ?= ".")
127 021d5bb6 2021-08-12 op ## indentation
129 021d5bb6 2021-08-12 op Indentation at first doesn’t seem like a difficult thing. After all, when we’re staring at code we don’t have the slightest doubt on how a certain line needs to be indented. Turns out, like most other “obvious” things, that coming up with a program that decides how to indent is not that straightforward.
131 021d5bb6 2021-08-12 op In my case fortunately the logic is pretty simple. The level of the indentation is how nested we are in parenthesis multiplied by the tab-width (because yes, nps uses hard tabs), with the exception of a closing parenthesis which gets indented one level less. Take this snippet for instance:
133 021d5bb6 2021-08-12 op ```snippet of nps to show how indentation works
134 021d5bb6 2021-08-12 op proc foo(x) {
135 021d5bb6 2021-08-12 op y = bar(x.id)
142 021d5bb6 2021-08-12 op The first line, the ‘proc’ declaration, is indented at the zeroth column because we aren’t inside a nested pair of parenthesis. The ‘y’ variable is indented one tab level because it’s inside the curly braces. The body of the assert is inside two nested pairs of parenthesis, so it’s indented twice. The closing parenthesis of the assert is indented by only one level because of the special case: it should be two, but since it’s a closing we drop one indentation level.
144 021d5bb6 2021-08-12 op The code for ‘nps-indent-line’ is probably not the prettiest, but seems to work nonetheless:
147 021d5bb6 2021-08-12 op (defun nps-indent-line ()
148 021d5bb6 2021-08-12 op "Indent current line."
150 021d5bb6 2021-08-12 op boi-p ;begin of indent
152 021d5bb6 2021-08-12 op (point (point))) ;lisps-2 are truly wonderful
153 021d5bb6 2021-08-12 op (save-excursion
154 021d5bb6 2021-08-12 op (back-to-indentation)
155 021d5bb6 2021-08-12 op (setq indent (car (syntax-ppss))
156 021d5bb6 2021-08-12 op boi-p (= point (point)))
157 021d5bb6 2021-08-12 op ;; don't indent empty lines if they don't have the in it
158 021d5bb6 2021-08-12 op (when (and (eq (char-after) ?\n)
160 021d5bb6 2021-08-12 op (setq indent 0))
161 021d5bb6 2021-08-12 op ;; check whether we want to move to the end of line
163 021d5bb6 2021-08-12 op (setq move-eol-p t))
164 021d5bb6 2021-08-12 op ;; decrement the indent if the first character on the line is a
166 021d5bb6 2021-08-12 op (when (or (eq (char-after) ?\))
167 021d5bb6 2021-08-12 op (eq (char-after) ?\}))
168 021d5bb6 2021-08-12 op (setq indent (1- indent)))
169 021d5bb6 2021-08-12 op ;; indent the line
170 021d5bb6 2021-08-12 op (delete-region (line-beginning-position)
172 021d5bb6 2021-08-12 op (indent-to (* tab-width indent)))
173 021d5bb6 2021-08-12 op (when move-eol-p
174 021d5bb6 2021-08-12 op (move-end-of-line nil))))
177 021d5bb6 2021-08-12 op The real workhorse is ‘syntax-ppss’ that tells us how deep in parens we are. A better real-world example is probably the indent-line of the go-mode: it’s obviously more complex, but it’s still manageable.
179 021d5bb6 2021-08-12 op ## abbrev table
181 021d5bb6 2021-08-12 op This is not strictly needed, but it’s nice to have. I’m using abbrev tables for various languages to automatically correct some small typos (like ‘inculde’ instead of ‘include’).
184 021d5bb6 2021-08-12 op (defvar nps-mode-abbrev-table nil
185 021d5bb6 2021-08-12 op "Abbreviation table used in `nps-mode' buffers.")
187 021d5bb6 2021-08-12 op (define-abbrev-table 'nps-mode-abbrev-table
191 021d5bb6 2021-08-12 op ## Completing the mode
193 021d5bb6 2021-08-12 op Now that we have all the pieces, let’s define the mode:
196 021d5bb6 2021-08-12 op ;;;###autoload
197 021d5bb6 2021-08-12 op (define-derived-mode nps-mode prog-mode "nps"
198 021d5bb6 2021-08-12 op "Major mode for nps files."
199 021d5bb6 2021-08-12 op :abbrev-table nps-mode-abbrev-table
200 021d5bb6 2021-08-12 op (setq font-lock-defaults nps--font-lock-defaults)
201 021d5bb6 2021-08-12 op (setq-local comment-start "#")
202 021d5bb6 2021-08-12 op (setq-local comment-start-skip "#+[\t ]*")
203 021d5bb6 2021-08-12 op (setq-local indent-line-function #'nps-indent-line)
204 021d5bb6 2021-08-12 op (setq-local indent-tabs-mode t))
207 021d5bb6 2021-08-12 op nps mode derives from prog-mode, a generic mode used for programming language. This way, users can easily define keybindings and options only for programming-related buffers and have a consistent experience. The body of the ‘define-derived-mode’ macro is just some code that gets executed when the mode is activated. There, we set the font-lock-defaults that was computed previously, define comment-start and comment-start-skip so functions like ‘comment-dwim’ (M-;) works as expected and setup the ‘indent-line-function’. Then, also enable indent tabs mode because nps uses real hard tabs. That’s it.
209 021d5bb6 2021-08-12 op Registering this mode to the ‘nps’ file extension ensures that Emacs will enable nps-mode automatically:
212 021d5bb6 2021-08-12 op ;;;###autoload
213 021d5bb6 2021-08-12 op (add-to-list 'auto-mode-alist '("\\.nps" . nps-mode))
216 021d5bb6 2021-08-12 op Sidebar: what are those ‘autoload’ comments? It’s a trick used by Emacs to cheat and not load all the code in a file until it’s needed. Emacs will only evaluate the ‘add-to-list’ and register a ‘nps-mode’ autoload, but won’t evaluate anything else until ‘nps-mode’ is called. The first time that ‘nps-mode’ is called, it’ll make Emacs load the whole ‘nps-mode.el’ file and then call again ‘nps-mode’. This is how Emacs can starts so quickly and still load TONS of emacs-lisp files.
218 021d5bb6 2021-08-12 op Major modes usually defines also some keys and/or integration with other packages (flymake for example). I’m not going do to neither, but it’s still pretty easy. To provide some keys all you have to do is to declare a ‘$mode-map’ variable that holds a keymap, then ‘define-derived-mode’ will take care of enabling it:
220 021d5bb6 2021-08-12 op ```example of a mode-map
221 021d5bb6 2021-08-12 op (defvar nps-mode-map
222 021d5bb6 2021-08-12 op (let ((map (make-sparse-keymap)))
223 021d5bb6 2021-08-12 op (define-key map "C-c c" #'do-stuff)
225 021d5bb6 2021-08-12 op map)) ; don’t forget to return the map here!
228 021d5bb6 2021-08-12 op ## Wrapping up
230 021d5bb6 2021-08-12 op Writing a major-mode from scratch this way was really interesting in my opinion. The knowledge on how major-mode works and how to write one will probably come in handy in the future, either to write more major-mode for (hopefully) real programming languages or to tweak existing ones.
232 021d5bb6 2021-08-12 op In retrospect, I ended up choosing the hardest possible way to build a major mode. For a project like this, where I’m only interested in basic font-locking, there was at least two other options to choose from:
233 021d5bb6 2021-08-12 op * use generic-mode
234 021d5bb6 2021-08-12 op * derive from cc-mode
236 021d5bb6 2021-08-12 op generic-mode is provide an easy, but limited, way to write major-modes.
238 021d5bb6 2021-08-12 op cc-mode it’s the mode that powers C, C++, Java and (at least) AWK. It’s pretty flexible and it was designed to handle “all” C-like programming languages.
240 021d5bb6 2021-08-12 op However, writing nps-mode from scratch was a pleasant experience and I had some fun hacking in emacs lisp. The implementation is also not too bad and still pretty simple, so it has been worth the time.
242 021d5bb6 2021-08-12 op I’m not sharing the code in this post because it’s part of the aforementioned project that it’s still heavily worked on. The code in this post is everything I wrote in nps-mode.el anyway.
244 021d5bb6 2021-08-12 op Some useful links:
246 021d5bb6 2021-08-12 op => https://www.emacswiki.org/emacs/GenericMode [https] generic-mode
247 021d5bb6 2021-08-12 op => https://nullprogram.com/blog/2020/01/22/ [https] A makefile for Emacs Packages
