Frustrations with Markdown: Part 1

I've been using Markdown for a few years now, and I really want to like it. There's something elegant about having a system in which you can easily write prose and have it readable in plain text and then rendered into HTML or another format. That said, I've found myself quite frustrated with the process of building my own publication system using Markdown.

When I was building this publication system, I originally drafted copy in a text editor. When I needed to put it into the publication system I would copy from that editor into NeoVim and format the text as markdown. This worked fine, but I eventually noticed that the text editor used soft wrapping for the lines, meaning they became one large line when pasted into NeoVim. This was fine for the initial rendering of the websites, but when I needed to copyedit I found that these long lines were frustrating. No problem, I thought, I would just run a few quick commands in NeoVim and have all of those paragraphs turned into hard wrapped lines around the 100 character mark. What I hadn't noticed is that doing this actually resulted in the HTML that was output by Goldmark would also contain these hard wraps in the form of newline characters (\n).

After I had already been most of the way through finding a solution to this problem I checked what the browser actually did. I found that browsers don't actually pay any attention to these newlines and strip them out when they're rendering. This was a problem that I did not need to fix. Even so, I found the structure of the HTML to be unpleasing, and even if it wasn't rendered in the browser as such, I wanted the HTML to look good. Yes, even if no one actually ever looks at it. During this process I discovered a few things about Goldmark.

1. It doesn't appear that I can just insert arbitrary text during the conversion process, at least not easily. I would need to do that in the renderer, which I did not want to mess with. I could have potentially preprocessed the markdown as well, but that would add even more complexity to an already complex system.

2. There is a way remove those newlines. The ast.Text node type contains a property called SoftLineBreak and setting this property to false would strip away those generated newline characters. This worked well, but it didn't put a space between the newly concatenated lines. Since I couldn't change the text to just insert spaces I needed to find another solution.¹ Luckily, the text.Segment type has a padding property. So what I did was inserting an AST node between each of the text nodes that contained no content and a padding value of 1. This worked, although I found places where there were too many spaces and other places where there weren't enough. I fixed that problem, but I still need to go through another iteration to fix some other spacing issues.

Ultimately, I'm not pleased with how this entire process has been going. I decided early on that I want to write directly in HTML and then process that HTML into full pages. Something similar to what the Wattsi or Bikeshed. There are a few reasons why I want to do this, with the biggest being the ability to fully customize each article that I write with whatever HTML I see fit, which opens the door to easily add equations with MathML and figures with SVG without having to tinker with configuration to ensure the Markdown renderer doesn't see these elements as unsafe and strip them out. But this frustration certainly adds to that desire to use something other than Markdown.