Benchmarking with Ruby on Rails

One of the things I really enjoy about Ruby on Rails is that it has a lot of tiny conveniences that can speed up your workflow a lot.

For example, I recently needed to quickly convert Markdown to HTML, but I wasn't sure which gem to use: a quick search revealed the venerable Redcarpet, which is mostly written in C, but also Commonmarker, which wraps a Rust library and provides some additional features.

But which one to choose?

Luckily, Rails makes it really simple to create a benchmark via a generator:

bin/rails g benchmark markdown_parsing

It should generate a markdown_parsing.rb file in the script/benchmarks folder with the following contents:

# frozen_string_literal: true

require_relative "../../config/environment"

# Any benchmarking setup goes here...

Benchmark.ips do |x|
  x.report("before") { }
  x.report("after") { }

  x.compare!
end

Then add the Redcarpet, Commonmarker and Kramdown (as a baseline) gems to your Gemfile (plus benchmark-ips if it's not already in the development group), use something like Lorem Markdown to generate a sample Markdown file, and then add the benchmark code, which would look something like this:

# frozen_string_literal: true

require_relative "../../config/environment"

md = File.read(Rails.root.join("script", "benchmarks", "sample.md"))

Benchmark.ips do |x|
  x.report("Commonmarker.to_html") { Commonmarker.to_html(md) }
  x.report("Kramdown.to_html")     { Kramdown::Document.new(md).to_html }
  x.report("Redcarpet.render")     { Redcarpet::Markdown.new(Redcarpet::Render::HTML).render(md) }
  x.compare!
end

Finally, run the benchmark using the Rails runner:

bin/rails runner script/benchmarks/markdown_parsing.rb

On my development machine, this was the output:

ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +YJIT +PRISM [x86_64-linux]
Warming up --------------------------------------
Commonmarker.to_html 65.000 i/100ms
Kramdown.to_html 155.000 i/100ms
Redcarpet.render 7.151k i/100ms
Calculating -------------------------------------
Commonmarker.to_html 648.281 (± 4.0%) i/s (1.54 ms/i) - 3.250k in 5.021352s
Kramdown.to_html 1.525k (± 3.5%) i/s (655.91 μs/i) - 7.750k in 5.089232s
Redcarpet.render 102.672k (±17.1%) i/s (9.74 μs/i) - 486.268k in 5.011136s

Comparison:
Redcarpet.render: 102672.4 i/s
Kramdown.to_html: 1524.6 i/s - 67.34x slower
Commonmarker.to_html: 648.3 i/s - 158.38x slower

The results are a bit surprising, especially since Kramdown is written in pure Ruby and seems to be much faster than Commonmarker, which is basically just a thin Ruby wrapper over Rust code. So I'm suspecting that Commonmarker is actually doing a lot more.

Let's run the code from the benchmark in the REPL and see what's the output for each renderer.

For example, the <h1> heading from the original Markdown file looks like this:

# Monitis velle

Here's a heading generated by Commonmarker is quite complex:

<h1>
  <a href="#monitis-velle" aria-hidden="true" class="anchor" id="monitis-velle">
  </a>
  Monitis velle
  <h1></h1>
</h1>

Kramdown only adds an ID to the heading:

<h1 id="monitis-velle">Monitis velle</h1>

The output from Redcarpet is very basic, just the HTML and nothing more:

<h1>Monitis velle</h1>

Since the output is different, this means that the comparison wasn't exactly fair, so let's improve the benchmark by making sure all renderers behave the same (as much as possible) by disabling any extra features since we're only interested in converting from Markdown to plain HTML:

# frozen_string_literal: true

require_relative "../../config/environment"

md = File.read(Rails.root.join("script", "benchmarks", "sample.md"))

PLAIN_CM_OPTS = {
  extension: {
    strikethrough: false, table: false, autolink: false, tasklist: false,
    footnotes: false, description_lists: false, shortcodes: false,
    header_ids: nil
  },
  render: {
    github_pre_lang: false, escaped_char_spans: false, sourcepos: false,
    full_info_string: false, unsafe: false
  },
  parse: { smart: false }
}

Benchmark.ips do |x|
  x.report("Commonmarker.to_html") do
    Commonmarker.to_html(
      md,
      options: PLAIN_CM_OPTS,
      plugins: { syntax_highlighter: nil }
    )
  end

  x.report("Kramdown.to_html") do
    Kramdown::Document.new(md, auto_ids: false, syntax_highlighter: nil, math_engine: nil)
      .to_html
  end

  x.report("Redcarpet.render") do
    Redcarpet::Markdown.new(Redcarpet::Render::HTML).render(md)
  end

  x.compare!
end

Now let's run the benchmark again:

bin/rails runner script/benchmarks/markdown_parsing.rb

... and here's the output:

ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +YJIT +PRISM [x86_64-linux]
Warming up --------------------------------------
Commonmarker.to_html 2.677k i/100ms
Kramdown.to_html 139.000 i/100ms
Redcarpet.render 7.175k i/100ms
Calculating -------------------------------------
Commonmarker.to_html 27.100k (± 2.3%) i/s (36.90 μs/i) - 136.527k in 5.040564s
Kramdown.to_html 1.561k (± 3.0%) i/s (640.50 μs/i) - 7.923k in 5.079224s
Redcarpet.render 102.521k (±19.2%) i/s (9.75 μs/i) - 487.900k in 5.102568s

Comparison:
Redcarpet.render: 102520.6 i/s
Commonmarker.to_html: 27100.1 i/s - 3.78x slower
Kramdown.to_html: 1561.3 i/s - 65.66x slower

Redcarpet is still much faster than Commonmarker, but the difference is two orders of magnitude smaller, while Kramdown is unsurprisingly the slowest.

While this is a tiny example, the main advantage of the benchmark scripts is that they load your entire application, meaning that you can easily A/B test different chunks of code from your application to check for performance regressions or improvements!