Slides for RUM Presentation

Ought to be available here

Also, everyone but me has a beer.

Someone must think Lisp’s macros are a big deal. They’re discussed frequently on Hacker News, they’re probably one of the most-cited examples of ‘what makes Lisp special’, and copies of the definitive book on macros are currently selling for $177 – used. A book available for free in PDF form.

What’s the big deal?

Lots of good programmers don’t use macros, and write code in productive, Smalltalk-styled languages like Ruby, where libraries and community are plentiful. More to the point, especially in Ruby, they use those libraries mostly via Domain-Specific Languages, marvels of clarity and concision that make moving mountains appear effortless. Life is good. Right?

Hey, I love Ruby too. But I love her with a pang, and I am not alone.

Those who have once supped the sweet sup of macros are never fully satisfied without them. We see them in every shovelful of functions and blocks we heave onto the scorching roadway, under the watchful gaze of the Parser With No Eyes.

We pine. But with Ruby to hand, it’s hard to pin down exactly what we’re pining for.

The answer crystallized for me recently, when I wrote a dead-simple implementation of Lisp-style macros for CoffeeScript. I’d like to explain the difference I feel between macro-style and Ruby-style metaprogramming, and why I think macros are coming back even without Lisp.

Macro meta is different. Here’s a tease: how would you write code that can reliably turn the synchronous form into the asynchronous?

1
2
3
4

if x = asyncGet docname, cc()
  p x
else
  p "whoa"

1
2
3
4
5
6
7
8

asyncGet(docname, function(_thang) {
  var x;
  if (x = _thang) {
    p(x);
  } else {
    p("whoa");
  }
}

There are at least two frameworks for doing this in Javascript … with macros, all it takes is 20 lines.

What are ‘Lisp-Style’ Macros?

Most languages can be extended towards a problem. In Ruby, for example, people use :symbols, hashes => 'and' of course {|blocks| …} to make little languages (and by overriding method_missing enough, you can make it do nearly anything).

Macros are different. Macros are like functions that change your code before it runs, tiny programs that can deeply modify your code’s structure.

To understand how they do this, you first should know a few things about compilers.

The source code of a program starts out as a string that you and I can read, like

1

2 + 2

But to the computer, that’s just 32 20 2B 20 32. It then gets turned into individual tokens:

1
2
3
4

[ ['NUMBER',2,0],
  ['+', '+',0],
  [NUMBER, 2,0],
  [TERMINATOR,,0] ]

Still just a list of stuff the computer doesn’t understand. Finally, it is transformed into a tree of nodes, e.g.:

1
2
3
4
5
6
7
8
9
10
11
12
13

{ expressions: [
    operator: +,
    first: {
      base: { value: 2},
      properties: [ ]
    },
    second: {
      base: { value: 2},
      properties: [ ]
    },
    flip: false
  ]
}

This is the AST, or Abstract Syntax Tree. The AST has enough detail, structure, and associated behavior to compile into a language the computer does understand. For example, CoffeeScript compiles (almost) seamlessly into Javascript.

This tree of nodes models everything that happens in your source code; it’s not just a meaningless sequence of letters or tokens. Lisp-style macros let you mess with that tree right before it is compiled.

This is why macros are powerful. If you want to write a legal expression in the language, and you want it to mean some other legal expression, with macros, it really will become that other expression, not just act_like it. You can extend the language with the same fearlessness that you extend objects.

How Macro Meta Differs (From, Say, Ruby)

The best programming (and the reason we reach for meta-programming) is because we understand a problem well enough to confidently say, “I want to write this, and I want it to do that,” and then we see what tools the language has that can get us there.

Ruby meta-programming happens at run-time. We do whatever it takes (hooks on module inclusion, instance_exec, monkey-patch) to create context within classes or blocks, and use it to build up structures (Markaby), or declaratively assign behaviors (“acts-as” methods), or both (DataMapper).

Macros happen at compile-time. They actually replace x with y via the AST – an esoteric operation with great trade-offs, and one that requires us to think in a way that may be new.

What might that look like? Well, imagine that Ruby had a compilation phase:

1

obj = foo { acts_as... }

1
2
3

(obj = Foo.new).instance_exec{
  def behavior; ... end
}

Holy cows, the macro call was compiled away! It will behave correctly, but we got the “meta” out of our system during compilation.

Macros are not Ruby DSLs, though. They don’t rely on a limited set of allowed methods for extending the language (method_missing, instance_exec et al) – rather, they can use the entire language as they see fit, because they can expand into any valid code.

Difference, Illustrated: Control Flow

Macros good, run-time bad.

Imagine that the if then else operators in Ruby were named AscertainingThat, BeItResolved and Contrariwise. Would rewriting if be a good fit, even if it saved all those keystrokes?:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

def iif(cond, ok, notok)
  AscertainingThat cond
    ok.call()            # d'oh!
  Contrariwise
    notok.call()         # d'oh!
  end
end

iif 2 > 1,
  ->{
    p "success!"
  },
  ->{
    p "humbuggery!"
  }

That would behave correctly. Problems?

• Performance. We use ‘if’ everywhere. Run-time lambda calling is too ‘heavy’.
• Evaluation. It may not be obvious, but you can never add a short-circuiting elsif without incurring two lambda calls per elsif.

Contrariwise, a macro version would work reliably, compiling down to the wordy, IfItBeAscertainedThat… version, with no run-time penalty.

Does It Matter For DSLs?

Okay, so macros can do some things that Ruby DSLs aren’t suited for. Still, it’s not as though Ruby is sorely lacking for control flow options, especially with the welcome addition of Fibers. In terms of day-to-day coding, one rarely develops new methods of control flow, whereas a DSL that presents an attractive interface for commonly used functionality is a frequent goal in Lisps and in Ruby.

The conventional wisdom about macros is probably this:

“If you aren’t writing control structures, DSL techniques in dynamic languages are as good as macros.”

Are DSL techniques similar to macros?

Yes, in that they let us write exactly what we want to write, most of the time. Idiomatic Ruby libraries are amazingly easy to use, and the resulting code, even easier to read. Taken as a whole, they are an unsurpassed marvel of library design.

But is the production of a DSL similar to the production of a macro?

This question is rarely asked, but I think it’s crucial to explaining why Lispers and former Lispers, including current Rubyists, harbor such deep feelings for macros. The answer is: No.

DSL-first programming

A common phrase associated with Lisp macros is “bottom-up programming.” But a fine way of conceptualizing the benefits for Rubyists would be to replace it with the phrase “DSL-first programming.”

It’s a tantalizing phrase because we all know that starting by designing the DSL you wish existed is:

• a great way to gain insights into the nature of a new problem
• nonetheless, not something you do day-to-day.

Sure, we all write DSLs … sometimes.

Ask yourself: why aren’t we writing them, like, all the freaking time?

One answer is that we are, in fact. If we’re under contract to do a run-of-the-mill Rails app without any extensive custom pieces, then all of your idiomatic code is in the Rails DSL, because the biggest win is to write code that meshes well with the existing Rails idioms. Rails is like a DSL you share with future maintainers.

But when you open a new file yourself, when you require 'sinatra' and start hacking together something via some APIs, refactoring as necessary into functions, modules and classes, as you’re getting a handle on the problem …

How often do you build a DSL with metaprogramming techniques when you’re in that problem-focused mode?

Rarely. Honestly, it’d be a distraction, a break in your concentration, even if only a minor one. (By DSL, I mean ‘meta’ techniques that create context, not just yielding to a block). You’re thinking about the problem, the APIs that you’re calling and the task that needs to be accomplished, and the resulting models of the processes and data that are reflected in your code. But if you add the challenge of metaprogramming by modifying run-time state, well, it’s no wonder that when we’re in the throes of a problem we don’t often reach for instance_exec.

Building a DSL would be a distraction at the messiest and most productive part of the problem-solving process. I think that’s often true, and that if it is, it’s a weakness, because it’s different when you’ve got macros.

And this may be the most well-hidden benefit of macros. My theory is that the macro is easier to develop than a run-time-based DSL, for two reasons:

1. Directness. The macro is a tool that exists for exactly this purpose, and can generate any code that is legal in the language, whereas Ruby DSLs use a limited toolkit of runtime class and object modification techniques to provide desired behavior.

   The benefit of this directness? It makes DSL-style syntactic refactoring ‘cheaper’ mentally.

   With macros, you can refactor language at the same time as you refactor your functions. You can see that there’s extra work being done in your code that a DSL could remove – so you remove it, because you have an idea of what you want to say and you know what code it should generate, because you’re typing it right now. It’s very direct, and it doesn’t feel like a detour.

2. Transparency. The DSL increases run-time complexity. This taxes your brain, your runtime, and your buglist. Compilation is a surprisingly transparent abstraction compared to noodling around in ObjectSpace.In addition, macros can be less ‘magick-y’ and more transparent.

   The ‘magic’ all happens at compile-time. Only a very few macros of this type will do anything complicated structurally. Most will in fact just be straight templates for a pattern in code. And you can visually inspect the compiled output and know that the abstraction isn’t leaking, which compares well to noodling around in ObjectSpace.

Arguments Against Macros: Sound Familiar?

I don’t offer these assertions as proof of macros’ merits – and I certainly don’t hope to convince you that you should run out and write macros, that DSLs in Ruby are anything but great, etc. I just want to try to explain, in Ruby DSL terms, the weird attraction that Ruby macros have.

Still, given what you may have heard about macros, those might seem like surprisingly positive assertions.

Macros have a somewhat mixed reputation - exalted in hyperbolic style by their proponents, derided as unsustainable, unmaintainable, team-killing kludges by their opponents.

But there’s a reason Rubyists should suspect the complaints about macros.

“Use [meta/macros], and people who use your code aren’t writing [Ruby/Lisp] any more. They’re writing your language.”

“Stuff written with [meta/macros] doesn’t follow the rules of the language. You just have to know the library and its expected behavior.”

“Anaphoric [meta/macro] techniques break encapsulation. They just pop names into scope. Where did they come from? You just have to know.”

Ruby is, itself, a language and community guilty of the supposed sins of macros.

To take just one of the most common, and basic, differences of opinion: Ruby’s require breaks encapsulation. (I’ve heard that one a lot, and I have to say that I can’t disagree strongly enough with anyone who would propose we start doing the CommonJS thing and have to say things = require('thing'); things.thing_i_want … in order to pull a single name into scope).

So, as a Rubyist, I have to consider the arguments against macros with a grain of salt. Lots of programmers have different views than I do about how restrictive/’safe’ a programming language should be.

Ruby is generally acknowledged to be a pretty permissive language, and so it’s fitting if arguments against macros leave a Rubyist unmoved.

So, they’re direct(DSL->expansion), don’t increase run-time complexity, and you can read the expansions.

Aaaaaand they’re nonexistant in Ruby.

Well, that’s not quite true. There’s the admirable RubyMacros project, which is a couple of years old and uses RedParse – but works, and has an appropriately terse syntax reminiscent of MetaLua’s quasiquote.

And let’s not forget Rubinius.

Rubinius already codifies and supports the kind of AST transformations that Lisp-style macros rely on. All that’s missing is a single, ambitious hack that would go beyond one-off transforms, and add support for writing such transforms.

The question of macros would look like in Ruby – what would a ‘typical project’ look like if the community adopted them? – is one that I can’t answer. They seem an odd fit for Ruby, a compile-time abstraction in a message-passing world. That’s why I’m more immediately interested in writing them for CoffeeScript, and even more interested in using someone else’ implementation if it’s production-ready.

But macros have been adopted into a number of freewheeling languages besides Lisp. Like Factor, for instance. And Julia. And Metalua. Those languages have their own strong idioms for metaprogramming, their own characteristic look and feel. Macros may not have been ese