Custom Markdown Parsing

Question

I write a weekly digest email for my university which appears in the following email format:

############################
#          HEADER          #
# ======================== #
#   IMAGE | Title          #
#         | Description    #
# ------------------------ #
#           ...            #
# ------------------------ #
#   IMAGE | Title          #
#         | Description    #
# ======================== #
#          FOOTER          #
############################

Currently these are built each week from a horrible HTML template via Copy-Paste-Tweak.

I blog with and regularly use Markdown and I'm wondering if there is some way I can use it easily to parse a file which contains the main details for the digest to produce the above output?

The basic structure of a 'digest email' (including other fields) is

- header
  - main title
  - subtitle
  - date

- entry
  - image
  - title
  - body
   ...
- entry
  - image
  - title
  - body

- footer text

And I would like to render it out in the format above.

Some initial ideas I have had and would appreciate some input on:

YAML + Custom Processor in Ruby/Python - but then how would markdown formatted entry bodies work?
Redcarpet seems to allow custom rendering but could I work it into the format above?
Python Markdown - I saw some posts talking about extensions to markdown using python-markdown but I'm not sure how applicable it is here?

Doubtless I'm overlooking some beautifully simple implementation.

** UPDATE **

I apologise for the vagueness of this question. I couldn't really formulate an elegant description of what I was trying to do :(

In the end, I used a YAML file with multi line strings which were later parsed as Markdown using Maruku.

For those who are interested how it works and what the end result looks like, the project is on github here

Can't you just run the 'body' parts through markdown, after you read the above structure as a YAML or similar file? So something like: `read YAML -> step through all parts -> render 'body' through Markdown -> construct HTML -> send it off`. I don't directly know of a library that does this all for you at once though. — , Nov 12 '12 at 17:18
So that was my initial idea (point 1 above) but I'm not sure how putting markdown in a YAML file would work — Pete Hamilton, Nov 12 '12 at 17:29
Seems it's not possible... :( http://stackoverflow.com/questions/3517968/markdown-within-yaml-yaml-multi-line-escape-sequence — Pete Hamilton, Nov 12 '12 at 17:29
Have a markdown template using a template library like [mako](http://www.makotemplates.org/) or [jinja2](http://jinja.pocoo.org/docs/)? — Pedro Romano, Nov 12 '12 at 17:48
Ah; hadn't realised this was more a problem with YAML not being able to accept Markdown. But, with the simple syntax you give above, it doesn't seem too hard to write a quick'n'dirty parser for the various section, instead of using YAML (of course, extensibility then may become an issue later on). — , Nov 12 '12 at 18:03
I've added an answer which provides an object that will split YAML frontmatter from a Markdown file. Hope that helps, but I'd recommend using Jekyll if possible because it supports this AND handles layouts for you. — Gregory Brown, Nov 12 '12 at 19:26

Gregory Brown · Answer 1 · 2012-11-12T19:31:13.417

This sounds like a great usecase for Jekyll: https://github.com/mojombo/jekyll

Alternatively, you can use any templating system (ERb, Haml, etc) + any Markdown system ( I like RedCarpet ), and then a simple Ruby object for extracting YAML frontmatter from a Markdown file:

  class Page
    def initialize(source_file)
      self.filename = File.basename(source_file)

      read_page(source_file)
    end

    attr_reader :filename, :metadata, :contents

    private

    attr_writer :filename, :metadata, :contents

    def read_page(filename)
      self.contents = File.read(filename)

      begin
        if (md = contents.match(/^(?<headers>---\s*\n.*?\n?)^(---\s*$\n?)/m))
          self.contents = md.post_match
          self.metadata = YAML.load(md[:headers])
        end
      rescue => e
        puts "YAML Exception reading #{filename}: #{e.message}"
      end
    end
  end

Using this object, suppose you had a file called first_post.md with the following content:

---
title: First Post
category: essays
---

# My First Post!

It's *really* awesome

You would be able to process it using the Page object in the following way:

page = Page.new("first_post.md")
p page.metadata[:title]    #=> "First Post"
p page.metadata[:category] #=> "essays"
p page.contents            #=> "# My First Post!\n\nIt's *really* awesome"

markdown = Redcarpet::Markdown.new(Redcarpet::Render::HTML)
formatted_content = markdown.render(page.contents)
   #=> "<h1>My First Post!</h1>\n\n<p>It&#39;s <em>really</em> awesome</p>\n"

From here, all you need to do is wire up the templating engine. If you're already working with HTML templates, ERb may be easiest.

You'd have a template called something like layout.erb, which would be an HTML file with a bit of Ruby mixed in:

<html>
  <body>
     <h1><%= page.metadata[:title] %></h1>
     <%= formatted_content %>
  </body>
<html>

Then you'd do the following:

puts ERB.new(File.read("layout.erb")).result(binding)

From there, I think you'd have what you need, but you'll obviously need to stitch this all together in a bit more of an organized fashion first :-)

score 1 · Answer 2 · answered Nov 13 '12 at 02:06

Here is a python version for html output. Output

import markdown
import pprint
import yaml
from mako.template import Template

pp = pprint.PrettyPrinter(indent=4)

MAX_WIDTH=50
raw = """
header:
    main_title: title text
    subtitle: subtitle text
    date: 2012-11-13

entries:
    - image: https://upload.wikimedia.org/wikipedia/en/c/cb/Placeholder_logo.png
      title: title1
      body: >
        Lorem ipsum dolor sit amet,
         consectetuer adipiscing elit,
         sed diam **nonummy** nibh euismod
         tincidunt ut laoreet dolore
         magna aliquam erat volutpat.

    - image: https://upload.wikimedia.org/wikipedia/en/c/cb/Placeholder_logo.png
      title: title2
      body: >
        * Lorem ipsum dolor sit amet,

        * consectetuer adipiscing elit,

        * sed diam **nonummy** nibh euismod

        * tincidunt ut laoreet dolore

        * magna aliquam erat volutpat.

footer:
    text: i'm footer
"""
stream = yaml.load(raw)

for i in range(len(stream['entries'])):
    stream['entries'][i]['html'] = markdown.markdown(stream['entries'][i]['body'])

md_template = """
<table>
    <tr><td colspan=2><h2>${stream['header']['main_title']}</h2></td></tr>
    <tr><td colspan=2><h3>${stream['header']['subtitle']}</h3></td></tr>
    <tr><td colspan=2><h3>${str(stream['header']['date'])}</h3></td></tr>

    % for x in stream['entries']:
        <tr>
            <td>
            <img src=${x['image']}>
            </td>
            <td>
            <h3>${x['title']}</h3>
            </br>
            ${x['html']}
            </td>
        </tr>
    % endfor
</table>
"""

print Template(md_template).render(stream = stream)

mmyjona · Answer 3 · 2012-11-12T18:48:47.727

I think what you need is a lib called texttable to format multi-line text. And then you can use regex to replace the decorations with what ever you need.

__author__ = 'mmyjona'
#coding=utf-8

import pprint
from texttable import Texttable
import yaml

pp = pprint.PrettyPrinter(indent=4)

#set the width here
MAX_WIDTH=50
raw = """
header:
    main_title: title text
    subtitle: subtitle text
    date: 2012-11-13

entries:
    - image: 1.jpg
      title: title1
      body: Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.

    - image: 2.jpg
      title: title2
      body: Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.

footer:
    text: i'm footer
"""
stream = yaml.load(raw)

pp.pprint(stream)

tar = """
############################
#          HEADER          #
# ======================== #
#   IMAGE | Title          #
#         | Description    #
# ------------------------ #
#           ...            #
# ------------------------ #
#   IMAGE | Title          #
#         | Description    #
# ======================== #
#          FOOTER          #
############################
"""
table1 = Texttable()
table1.__init__(max_width=MAX_WIDTH)
table1.set_chars(['=', '|', '+', '-'])
table1.set_cols_align(["r", "l"])
table1.set_cols_valign(["t", "t"])
table1.add_rows([["foo","bar"]
    ,["Main Title",stream['header']['main_title']+"\n"]
    ,["Subtitle",stream['header']['subtitle']+"\n"]
    ,["Date",str(stream['header']['date'])+"\n"]
])

for x in stream['entries']:
    table1.add_row([x['image'],x['title'] + "\n" + x['body']])

table1.add_row(['Footer', stream['footer']['text']])
print table1.draw() + "\n"

output:

{   'entries': [   {   'body': 'Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.',
                       'image': '1.jpg',
                       'title': 'title1'},
                   {   'body': 'Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.',
                       'image': '2.jpg',
                       'title': 'title2'}],
    'footer': {   'text': "i'm footer"},
    'header': {   'date': datetime.date(2012, 11, 13),
                  'main_title': 'title text',
                  'subtitle': 'subtitle text'}}
+=======================+=======================+
|          foo          |          bar          |
+-----------------------+-----------------------+
|            Main Title | title text            |
+=======================+=======================+
|              Subtitle | subtitle text         |
+=======================+=======================+
|                  Date | 2012-11-13            |
+=======================+=======================+
|                 1.jpg | title1                |
|                       | Lorem ipsum dolor sit |
|                       | amet, consectetuer    |
|                       | adipiscing elit, sed  |
|                       | diam nonummy nibh     |
|                       | euismod tincidunt ut  |
|                       | laoreet dolore magna  |
|                       | aliquam erat          |
|                       | volutpat. Ut wisi     |
|                       | enim ad minim veniam, |
|                       | quis nostrud exerci   |
|                       | tation ullamcorper    |
|                       | suscipit lobortis     |
|                       | nisl ut aliquip ex ea |
|                       | commodo consequat.    |
+=======================+=======================+
|                 2.jpg | title2                |
|                       | Lorem ipsum dolor sit |
|                       | amet, consectetuer    |
|                       | adipiscing elit, sed  |
|                       | diam nonummy nibh     |
|                       | euismod tincidunt ut  |
|                       | laoreet dolore magna  |
|                       | aliquam erat          |
|                       | volutpat. Ut wisi     |
|                       | enim ad minim veniam, |
|                       | quis nostrud exerci   |
|                       | tation ullamcorper    |
|                       | suscipit lobortis     |
|                       | nisl ut aliquip ex ea |
|                       | commodo consequat.    |
+=======================+=======================+
|                Footer | i'm footer            |
+=======================+=======================+

If what you want HTML output don't use this. Use moko or jinjia2. — mmyjona, Nov 12 '12 at 18:53

Custom Markdown Parsing

3 Answers3