1 EDIT April 2021: I have revisited this parser and published it as a library. The version presented here is not only overly-complex, but also overly-verbose.
3 => /post/gemtext-clojure.gmi Parsing text/gemini with Clojure revisited
6 I've written various parsers in the past. From simplistic stuff in Korn shell, to hand written recursive descendant parsers in Go and or C, to yacc-based parsers in C. I even played with Alex and Happy in Haskell, but that was ages ago and I don't recall anything.
8 One thing that I never tried was to write a parser in Clojure. Until now.
10 This post is an experiment: I'm trying to do some literate programming in gemtext (the Gemini protocol "native" response format, text/gemini). If you extract all the block codes you should end up with the same gemtext parser I'm using.
12 => https://git.omarpolo.com/blog/tree/src/blog/gemtext.clj?id=965388145931751bf314276404816f631c27d01d gemtext.clj (as at the time of writing)
13 => gemini://gemini.circumlunar.space/docs/specification.gmi Gemini specification (over Gemini)
14 => https://gemini.circumlunar.space/docs/specification.html Gemini specification (over HTTP)
16 One last note before the implementation: while reading the code, please keep in mind that I really wanted to play with the transducers. There are probably more efficient or neater way to parse stuff in clojure, but I didn't want to write an efficient or fast parser. I wanted to have some fun with the transducers!
18 Given that gemtext is a line-oriented protocol, I thought to split the input in into lines and use some transducers magic to end up with a neat hiccup-like data structure.
20 Now we can begin. The aren't third parties dependencies here:
25 [clojure.string :as str]
26 [clojure.walk :as walk]))
29 We'll use only clojure.string and clojure.walk, as well as the standard library.
31 We also need one helper function, starts-with?: it's a wrapper around clojure.string/starts-with?. The need for such function will be clear later.
35 "check if `s` starts with `substr`. Return `false` if `s` is not a
39 (str/starts-with? s substr)))
42 Every syntactical element of gemtext will be parsed by its own little transducer. The transducer chain will be fed with a stream of lines, and will return a stream of hiccup-like data structure. Let's begin with the most elaborate one: the match-code-blocks transducer:
45 (defn- match-code-blocks []
47 (let [acc (volatile! [])
48 state (volatile! :normal)]
51 ([result] (rf result))
53 (let [in-verbatim? (= :verbatim @state)
54 marker? (starts-with? line "```")]
56 (and (not in-verbatim?) marker?) ;go to verbatim
57 (do (vreset! state :verbatim)
60 ;; return what we've got and go to :normal
61 (and in-verbatim? marker?)
62 (let [res [:verbatim (str/join "\n" @acc)]]
63 (vreset! state :normal)
68 (do (vswap! acc conj line)
72 (rf result line))))))))
75 Woah, that was a lot. We defined a function, that returns a function that takes a reducing-function rf. This sets up some local state (acc and state variables), and returns another function!
77 That's a lot of functions.
79 In the innermost function, only the 2-arity branch is interesting, the other two are just scaffolding. There we check if we've got a line with the ``` marker, and if so we switch between the "capturing state", where we capture all the lines in a code block, and an "identity state", where we pass what we've read unconditionally.
81 When we switch from capturing to identity state, we also return a vector of `[:verbatim "matched lines"]`.
83 The important thing here is to return the line we got as-is if it's not a marker or within the two markers.
85 The rest is similar to this, but maybe simpler. In retrospect, I should have wrote some macro to reduce the boilerplate.
87 Now, let's parse the headings. gemtext has three types of headings, with a syntax (and meaning) similar to markdown.
90 (defn- match-headings []
94 ([result] (rf result))
98 ;; the space character after the # is madatory
99 (starts-with? line "# ") [:h1 (subs line 2)]
100 (starts-with? line "## ") [:h2 (subs line 3)]
101 (starts-with? line "### ") [:h3 (subs line 4)]
105 There are definitely similarities with the previous example, but here you'll understand why I defined starts-with? instead of using directly clojure.string/starts-with?. The "line" we get here can be a string. Or can be a hiccup form. In fact, every transducer will se as input the output of the transducers that run before him. The helper function saves us from checking every time if the input is a string or not.
107 Now, some of the other syntactical elements are so similar to parse that I wrote a generic matcher:
110 (defn- generic-matcher
111 "Return a generic matcher transducer. Will wrap line that starts with
112 `start` within `[type line]`."
117 ([result] (rf result))
120 (if (starts-with? line start)
121 [type (subs line (count start))]
125 and I've used it to match the lists and the quotes:
128 (defn- match-lists [] (generic-matcher "* " :li))
129 (defn- match-blockquotes [] (generic-matcher "> " :blockquote))
132 In case you didn't know, lists in gemtext starts with a star * followed by at least one space, followed by arbitrary text. The quote are similar, except that they begins with >.
134 Matching the links is a little bit difficult:
137 (defn- match-links []
141 ([result] (rf result))
143 (let [spaces? #{\space \tab}
144 nonblank? (complement spaces?)]
146 (if-not (starts-with? line "=>")
149 (drop 2) ; drop the marker
150 (drop-while spaces?) ; drop also the optional spaces
151 (split-with nonblank?) ; separate URL from (optional) label
152 (apply #(vector :link
154 (apply str (drop-while spaces? %2))))))))))))
157 First of all, unlike in HTML, links on gemini aren't inline entities. Second, their syntax is an "arrow" `=>` eventually followed by spaces, followed by the URL, and an optional description.
159 In the previous function, if we match a line that starts with "=>" we start split it into the URL and the (optional) description.
161 (seq line) will return a sequence of character, from which we remove the marker. Then we split this in the URL and description. To keep the implementation easy an URL is just a sequence of bytes other than a space or tab. We also drop any blanks from the description.
163 How cool are the threading macros? (also note that we used a set as a function for even more style points!)
165 Anyway, the only missing piece is matching the paragraphs. Given that we match on every syntactical entities present in the specification (as of now at least), every non-matched line is a paragraph.
168 (defn match-paragraphs [] (generic-matcher "" :paragraph))
173 Now we only need to chain these transducer together. Keeping in mind that the order is important, here's the chain:
177 (comp (match-code-blocks)
185 Lines will flow from the first transducer (match-code-blocks) towards the end, being enriched at every step. Beautiful!
187 Parsing is thus dead-simple now that we've got every piece:
191 "Given a string representing a gemtext document, parse it into an
192 hiccup-like data structure."
194 (transduce parser conj [] (str/split-lines str)))
197 We use our chain (the parser), fed with the lines from str, and conj everything into []. The empty vector is important, because if you use a list or nil, due to how conj works, you'll get the parsed document in reverse order.
199 Aaaand that's all! As a final bonus, if you reached this point, here's an implementation of unparse, a function that takes our hiccup-like format and returns a string, and to-hiccup to translate our hiccup to HTML-hiccup.
202 (defn unparse [thing]
203 (let [sw (StringBuilder.)]
211 (if-not (keyword? (first t))
216 :verbatim (str "```\n" a "\n```")
221 :blockquote (str "> " a)
222 :link (str "=> " a " " b)
230 I've used clojure.walk/prewalk to traverse the input, as it makes easy to navigate inside a nested data structure. The idea is that if we get a sequence whose first element is a keyword, that's a "tag", otherwise we recursively inspect its content. If it's a tag, we convert it to a string, pushing it into a StringBuilder.
232 The to-hiccup function is practically the same
235 (defn to-hiccup [doc]
244 (if-not (keyword? (first t))
249 :verbatim [:pre [:code a]]
253 :li [:ul [:li a]] ;; TODO!
254 :blockquote [:blockquote a]
255 :link [:p.link [:a {:href a} b]]
262 The only thing that's missing from to-hiccup is the (html/gemini) list handling. That is, while on gemini you only have "list item", in HTML the lists item must be wrapped inside a container tag, ul or ol. Since I'm not using lists that much, I don't care at the moment. One can probably improve it by doing some post-processing on the content of @l and grouping every :li.
264 But that's really all.
266 I'm using this code to parse the posts (so that they can be rendered also in HTML), and to build every gemini page.
