Twenty Years of Pandoc

"The choice of Haskell has also led to a high quality and low volume of contributors"

I feel this influence of choosing a tech stack and its impact on self selected and auto-reenforced culture is most often underestimated.

From my own experience, at a time I was (involuntarily) working in Java, and when .Net was released, from a pure technical point of view it was like a breath of fresh air. Java was suffering from overengineering, archtecture astronauts galore and no sensible UX framework. .Net, the new kid, came in lean and clean with a UX library that 'just worked'.

Problem later was that for all its flaws and being overly 'academic', in teams (the real thing, not the awfull app), you could have indepth discussions about non trivial aspects of SWE topics in the Java world, whereas for all its technical prowess, in .Net land you were mostly dwelling amongst the 2 week CRUD app bootcamp folks. This ofc is a gross oversimplication. You had brilliant engineers and challanged codemonkeys on both sides. But the skew was more than a little biased.

> by writing N parsers (“readers”) and M renderers (“writers”), one could support N × M conversions.

Beautiful writeup for a wonderful project. In an age of vibe-coding hype it's also so nice to see how things can be extended and snowball in usefulness when things are built correctly, by hand, from basic principles.

> Perhaps, then, in the future, people will no longer have a need for tools like pandoc.

I think we will need wonderful things like pandoc more and more. As mentioned there is a huge ecological and practical difference. Even if LLMs could get infintisamally close to deterministic-level reliability, it's still so many more orders of magnitude better in efficiency, especially with big batch jobs etc.

That a professor of philosophy [1] made tools [2] that are used by millions around the world... that's just mind boggling. In a good (great!) way.

Pandoc is my go-to tool. Thank you, Sir!

[1] https://johnmacfarlane.net/index.html

[2] https://johnmacfarlane.net/tools.html

Pandoc is great. I use it multiple times per day to move content between Outlook emails and a coding harness:

https://gist.github.com/rahimnathwani/210b1f9cb6ce731a304322...

  cat email-to-xyz.md | md2clip

  clip2md > email-from-xyz.md

Love pandoc, I use it for all sorts of stuff…here is my minimal static site generator:

find . -name '*.md' -type f -exec sh -c '

  for file do  
    out="docs/${file#./}"  
    out="${out%.md}.html"
    mkdir -p "$(dirname "$out")"  
    pandoc --quiet --template template.html "$file" -o "$out"  
  done  

My most used pandoc snippet:

tidyhtml () { pandoc -f html-native_divs-native_spans -t markdown-raw_html-raw_attribute | pandoc -f markdown -t html }

which strips out styling, wrapper divs, spans, inline attributes, etc from (for instance) HTML copied from a google or word doc. Just the semantic goodness!

Pandoc is awesome. My fav usage is configuring git to use it to normalize binary docs to markdown (like a .docx) so they can be diffed. Works amazing for redlining contacts.

To top it all off Pandoc has a great experience for contributors. Over the past few years I've opened several bug reports related to Typst and docx and all of them got responses that were kind and helpful. I even had a few PRs merged in despite knowing almost nothing of Haskell.

Thanks so much for pandoc!

I never heard about djot [0], is anyone using it?

[0]: https://djot.net/

Thank you for pandoc. I made the (at the time perhaps not transparently wise) choice to go all in on it when I started my PhD, and that decision has paid nothing but dividends since. I owe my career as a scientist to it.

Pandoc is a fantastic piece of software. I have never had any issues with it, and it's the tool I reach for anytime I need to covert documents. I'm super grateful to John MacFarlane for creating it and maintaining it for all these years!

One thing I like with pandoc is that it produces clean html or latex code (it can also produce clean markdown with the right option).

If you compare this with the HTML produced by typst or hevea, this is super useful, as we can then roll out our own styles.

TIL pandoc for the people, courtesy of pandoc compiled to WASM. cool

https://pandoc.org/app/

I have at times mused that writing the internal state of pandoc to disk (yes 13th standard etc) would be the most interoperable file format

> From what I have seen, Rust appears to have many of the good features of Haskell, while producing faster, more memory-efficient, and more compact code. But Haskell still strikes me as more “ergonomic,” better suited to express abstractions, and just closer to the ideal of a language that helps the developer think.

For similar reasons I would rather pick up Swift, Scala, Kotlin or F#, instead of Rust, in the areas I work on.

Pandoc is amazing. I have HTML and want a PDF? One command and it just works painlessly. That was just my last use of pandoc this weekend. I do not use it that often, but when I do its perfect.

I discovered pandoc in college, in 2013 or 2014 and it quickly became one of my favourite pieces of software. It was like magic; hassle free and incredibly powerful. Writing problem sets in Markdown with embeded LaTeX saved me hours and my sanity.

I havent used it much after graduation, but I will always remember if fondly. This was an amazing read, both to learn about its history and as an update on what's been happening lately. Thank you for this great tool and all the work put into it over the past 20 years!

Thank you for Pandoc! I used it for many things over the years, and it was always there as a good option. I mostly use it for Markdown->Typst these days and it does this job very well.

A few years ago, I gobbled together some bash scripts around pandoc to build a site generator for my personal website. Works great. I use html templates, markdown for the content, etc. Mostly the bash scripts just serve to list files and process them one by one. I actually process them concurrently by forking processes so it's reasonably fast. A bit wonky but it works fine for my use case.

As for Haskell, I guess tree transformations and parsing are the perfect use case for functional programming. I studied in Utrecht in the nineties when Erik Meijer was still teaching there (later went to work at Microsoft Research where he contributed to things like F# and Linq). In short, my compiler course was taught using functional programming. We were toying around with writing our own parser generators to implement a subset of Modula 3 or our own toy languages. Lots of monads and other esoteric abstractions.

I haven't really done much professionally with any of that since except having a really easy time when languages like Kotlin, Javascript, etc. started borrowing liberally from functional programming. These days, if you have a list, calling map or forEach on it with another function is perfectly normal in many languages. Very nice alternative to a for or while loop.

Thanks also that new reader/writer are also integrated like typst.

I use it to create pdfs for my blog posts with that code :-)

  python3 "$SCRIPT_DIR/preprocess.py" > blog-post.md && \
  /datadisk/Downloads/pandoc-3.9/bin/pandoc \
    blog-post.md \
    --pdf-engine=typst \
    --template="templates/blog.typ" \
    -o "out.pdf" 2>/tmp/pdf-pandoc-err.txt;

I don't need Pandoc often, but when I do there's nothing like it.

Twenty years of quiet, consistent maintenance. No hype, no VC money, just solid work. Rare and precious

I remember noting "djot" when John M. created it but hadn't really looked into it before now. After reading through the design goals and differences from Markdown, I can see we'd have been better off if somehow djot had been able to come first. Gruber has said he thinks Markdown being underspecified was a feature, but as a user trying to mix and match different tools it drove me nuts.

pandoc was (and still is) critical to some of my flows. I hope it never dies!

It's not easy to maintain so many parsers and renderers, and at the same time to keep the qualities of them.

unending thanks, as much for pandoc as for the example of a clear design+implementation that lasts.

TIL of this wonderful tool, how did I never used this before, boggles the mind

Effort worth admiration. Thank you for creating pandoc! It allowed me to write my CV in more readable markdown, but also get nice pdf with latex.

I can only imagine the hell it must be when the AST is changed to update all inbound and outbound code.

Happy birthday and thank you so much!

Thank you for pandoc!

I've always taken particular note of how wisely scope-limited Pandoc is. Markup that aligns with natural language convention[1], is tightly converted, but the further from natural language, the less fidelity Pandoc can promise. Until, at the DITA or S1000D stage of "this ain't natlang, brah", Pandoc says "forget it" and just won't even pretend that such markup is even convertible.

Because, spoiler, it's not.

Constructs like tables and bibliographies challenge natural language markup - resulting in an explosion of different formalisms - but component content system artifacts for transclusion and conditionals shatter any pretense that these file types are "documents" at all. Both of those artifacts must draw formal structure from outside of language, i.e., from their own product / domain. They're parts of a system that make documents, but are not themselves documents or natural language. They are meaningless - or, worse, full of wrong meaning - outside of their runtime environment inside an explicit knowledge domain. Something that newer component content formats like Typst recognize explicitly.

The proof of all this is, as they say, in the pudding. What do people write documents in today? Well, they stick to natural language formats, sometimes they let the document system handle tables in some bespoke way, but conditionals are viewed with justified suspicion. DITA and S1000D projects, and the cursed migrations that lead to them, are sparse and driven almost exclusively by regulatory requirements, or, more often, program offices misreading regulatory requirements[0].

And here we all are in the LLM age, where natural language is being vindicated in ways both awe-inspiring and devastating. While component content systems force an LLM to expand its context window to the entire repository to make sense of any single sentence.

The crap of all this is, this is stuff that computer / information science has known since at least the 1980s. There are papers written about it. But high-complexity component content systems are sellable to non-technical writer groups because they don't see the tripwires in the fundamentals, or they think[2] that their product domain is so structured that the tripwires can be rigged as structure.

[0] No, converting to a pile of S1000D 040As doesn't magically fix your MTAs or your ILS or anything else.

[1] I do realize that proximity to natural language is correlate, not cause. Markdown converts well as a low-power notation whose instances denote values; it reads like natural language because that's what low-power does. The operative variable is whether the artifact denotes a document or a function from configuration to documents. AsciiDoc with `ifdef::[]` and `include::[]` converts every bit as badly as DITA, although without the fundamental nonsense of XSD and Horn's Information Mapping.

[2] Almost always wrongly