Pandoc: A Tool I Use and Like
Pandoc is a fantastic tool for converting between markup formats. Here are a few of the ways I use it.
Today I want to talk to you about one of my favorite command-line tools, Pandoc. From the project website:
If you need to convert files from one markup format into another, pandoc is your swiss-army knife.
I spend a lot of time writing, and I love Vim, Markdown, and the command line (and avoid browser-based WYSIWYG editors when I can), so that’s where a lot of my Pandoc use comes in, but it has a ton of utility outside of that – really, anywhere you need to move between different text-based formats, Pandoc can probably help. A few examples from recent memory:
Markdown ➞ Craft Blog Post
This website you’re reading presently uses Craft CMS, a flexible and powerful content management system that doesn’t perfectly match my writing process1. Rather than composing directly in Craft, I prefer to write locally, pipe the output through Pandoc, and put the resulting HTML into a text block in the CMS. This gets me a few things I really like:
- Curly quotes in place of straight ones and en-dashes in place of
- Daring Fireball-style footnotes with return links
By default, Pandoc uses Pandoc Markdown when converting Markdown docs to other formats, an “extended and slightly revised version” of the original syntax, which is how footnotes and a bunch of other things work.
Markdown ➞ Rich Text (Basecamp)
I also sometimes find myself writing decently long Basecamp posts. Basecamp 3 has a fine WYSIWYG editor (🪦 Textile), but again, I’d rather be in Vim. Pasting HTML into Basecamp doesn’t work (just shows the code verbatim), but I’ve found that if I convert my Markdown notes to HTML and open the HTML in a browser, I can copy and paste that directly into Basecamp with good results. Leveraging MacOS’
open command, this one-liner does the trick2:
cat [filename.md] \ | pandoc -t html \ > /tmp/output.html \ && open /tmp/output.html \ && read -n 1 \ && rm /tmp/output.html
This will convert the contents to HTML, save that to a file, open the file in a browser, wait for the user to hit enter, and the remove the file. Without that
read -n 1, it’ll remove the file before the browser has a chance to open it.
HTML ➞ Text
We built an app for one of our clients that takes in news articles (in HTML) via an API and sends them as emails to their clients (think big brands) if certain criteria are met. Recently, we were making improvements to the plain text version of the emails, and we noticed that some of the articles were coming in without any linebreaks in the content. When we removed the HTML (via Rails’
strip_tags helper), the resulting content was all on one line, which wasn’t very readable. So imagine an article like this:
<h1>Headline</h1> <p>A paragraph.</p> <ul><li>List item #1</li> <li>List item #2</li></ul>
Our initial approach (with
strip_tags) gives us this:
Headline A paragraph. List item #1 List item #2
Not great! But fortunately, some bright fellow had the idea to pull in Pandoc, and some even brighter person packaged up some Ruby bindings for it. Taking that same content and running it through
PandocRuby.html(content).to_plain gives us:
Headline A paragraph. - List item #1 - List item #2
Much better, and though you can’t tell from this basic example, Pandoc does a great job with spacing and wrapping to generate really nice-looking plain text from HTML.
HTML Element ➞ Text
A few months ago, we were doing Pointless Weekend and needed a domain for our Thrillr app. A few of us were looking through lists of fun top-level domains, but we realized that AWS Route 53 only supports a limited set of them. In order to get everyone the actual list, I needed a way to get all the content out of an HTML
<select> element, and you’ll never guess what I did (unless you guessed “use Pandoc”). In Firefox:
- Right click the select element, then click “Inspect”
- Find the
<select>in the DOM view that pops up
- Right click it, then go to “Copy”, then “Inner HTML”
- You’ll now have all of the
<option>elements on your clipboard
- In your terminal, run
pbpaste | pandoc -t plain
The result is something like this:
.ac - $76.00 .academy - $12.00 .accountants - $94.00 .agency - $19.00 .apartments - $47.00 .associates - $29.00 .au - $15.00 .auction - $29.00 ...
Preview Mermaid/Markdown (
A different client recently asked for an architecture diagram of a complex system that Andrew and I were working on, and we opted to use Mermaid (which is rad BTW) to create sequence diagrams to illustrate all of the interactions. Both GitHub and GitLab support Mermaid natively, which is really neat, but we wanted a way to quickly iterate on our diagrams without having to push changes to the remote repo.
We devised a simple build chain (demo version available here) that watches for changes to a Markdown file, converts the Mermaid blocks to SVG, and then uses Pandoc to take the resulting document and convert it to a styled HTML page using the
--standalone option (here’s the key line). Then we could simply make our changes and refresh the page to see our progress.
Generate a PDF
Finally, and this is not something I need to do very often, but Pandoc also includes several ways to create PDF documents. The simplest (IMO) is to install
wkhtmltopdf, then instruct Pandoc to convert its input to HTML but use
echo "# Hello\n\nIs it me you're looking for?" | pandoc -t html -o hello.pdf
I think that’s about all I have to say about Pandoc for today. A couple final thoughts:
- Pandoc is incredibly powerful – I’ve really only scratched the surface here. Look at the man page to get a sense of everything it can do.
- Pandoc is written in Haskell, and the source is pretty fun to look through if you’re a certain kind of person.
So install Pandoc with your package manager of choice and give it a shot. I think you’ll find it unexpectedly useful.
- My writing process is (generally):
- Write down an idea in my notebook
- Gradually add a series of bullet points (this can sometimes take awhile)
- Once I feel like I have a solid outline, copy that into a Markdown file
- Start collecting links (in the
- Write a intro
- Convert the bullet points to headers, edit + rearrange
- Fill in all the sections, write jokes, etc.
- Write a conclusion
- Create a Gist, get feedback from the team
- Convert Markdown to HTML, copy to clipboard
cat [file] | pandoc -t html | pbcopy)
- Create a new post in Craft, add a text section, flip to code view, paste clipboard contents
- Fill in the rest of the post metadata
- 🚢 ↩︎
- I’ve actually got this wired up as a Vim command in
command Mdpreview ! cat % \ | pandoc -t html \ > /tmp/output.html \ && open /tmp/output.html \ && read -n 1 \ && rm /tmp/output.html