Scala Macros Use Cases

Use case: Aspect-oriented programming

dev@scalamacros.org — Tue, 15 Nov 2011 00:00:00 PST

Object-oriented and functional programming provide quite different and mostly complementing approaches to decomposition, but there are certain aspects (pun intended) of composition that cannot be readily conveniently in either school of thought. These traditionally include such crosscutting concerns as logging, authentication/authorization, caching and so on.

Here is an Emacs Lisp snippet that exemplifies introduction of an advice that gets executed before a certain function (the snippet is courtesy of GNU Emacs Manual, Chapter 17.1 A Simple Advice Example):

(defadvice previous-line (before next-line-at-end
                                 (&optional arg try-vscroll))
  "Insert an empty line when moving up from the top line."
  (if (and next-line-add-newlines (= arg 1)
           (save-excursion (beginning-of-line) (bobp)))
      (progn
        (beginning-of-line)
        (newline))))

Similar functionality can be implemented either by embedding the advice in a lambda parameter of a macro annotation (@before(...) def previousLine = ...) or by creating a custom macro annotation that would host the desired domain logic). However, this comes with a restriction. Unfortunately, due to statically typed nature of Scala it'd be impossible to advice methods from a different compilation unit.

Also, with the help of annotations on package objects, it might be possible to support, so to say, wildcards for advices, called pointcuts (e.g. here's a pointcut from AspectJ: pointcut set() : execution(* set*(..)) && this(Point)). However, this possibility is unclear at the moment and is subject to additional brainstorming.

Use case: Type-safe bindings

dev@scalamacros.org — Sat, 12 Nov 2011 00:00:00 PST

The advent of compile-time metaprogramming in C# gave birth to several interesting techniques that add extra expressivity to the type system. In this installation, we examine one of these techniques that provides a type-safe way to refer to object/class members.

One of the natural scenarios that involves type-safe bindings is programming web applications in MVC style. The following example is taken from a blog post Strongly Typed ASP.Net MVC Helpers by Sean McAlinden (take a look at the post if you wonder about the internals of this neat trick):

<% using (Html.BeginForm("Update", "Home")) { %>
  <%= Html.RenderTextBoxFor(Model, c => c.FirstName) %>
  <%= Html.RenderValidationMessageFor(Model, c => c.FirstName) %>
  <%= Html.RenderTextBoxFor(Model, c => c.LastName) %>
  <%= Html.RenderValidationMessageFor(Model, c => c.LastName) %>
<% } %>

This code snippet leverages surrogate lambdas hack to describe model properties that are bound to form widgets. This works because Render methods specify the type of the second parameter as an Expression<Func<T, U>>. Whenever C# compiler sees a function type in the context when an expression type is required, it performs automatic code lifting (i.e. transforms the body of the function into an AST).

The same effect would be achieved if Render methods were macro defs. Since macros are expanded during the compile-time, they always get ASTs that correspond to their arguments, which provides code analysis capabilities. This topic is discussed in more details in "Advanced domain-specific languages" and "Language integrated queries" case studies.

If you're inspired by this possibility, you might be delighted to know that similar functionality is already present in Scala since version 2.8 (though it has experimental status). To see how it works, let's compile and run the following code:

import scala.reflect._

object Lift extends App {
  def bind[T, U](field: Code[T => U]) =
    println(field.tree.toString)

  class Foo(val bar: Int, val baz: Int)
  bind((foo: Foo) => foo.bar)
}

Function(
  List(LocalValue(NoSymbol,foo,PrefixedType(ThisType(Class(Lift)),Class(Lift.Foo)))),
  Select(
    Ident(LocalValue(NoSymbol,foo,PrefixedType(ThisType(Class(Lift)),Class(Lift.Foo)))),
    Method(Lift.Foo.bar,NullaryMethodType(PrefixedType(ThisType(Class(scala)),Class(scala.Int))))))

For more information about Scala reflection API, take a look at the following resources: Reflecting Scala, Embedded Domain-Specific Languages Using Libraries and Dynamic Metaprogramming and reflection API in the trunk of Scala 2.10, or write directly to our mailing list at http://groups.google.com/group/scala-user.

Use case: Advanced domain-specific languages

dev@scalamacros.org — Fri, 11 Nov 2011 00:00:00 PST

One of the most important use cases of compile-time metaprogramming involves increasing the area of applicability of embedded domain-specific languages (eDSLs).

As it was nicely summarized in Embedded Domain-Specific Languages using Libraries and Dynamic Metaprogramming by Gilles Dubochet, sooner or later DSLs hit the limitations of even very advanced programming languages. Here's an illustrative example from Squeryl, a Scala ORM for SQL databases:

var avg: Option[Float] =
  from(grades)(g =>
    where(g.subjectId === mathId)
    compute(avg(g.scoreInPercentage))
  )

This query looks almost like a series of high-order functions applied to a collection, and, in fact, it is: everything is strongly-typed and usual stuff (e.g. closing over the mathId variable) works as expected.

However, this comes at a price. First of all, the amount of syntactic boilerplate is truly gigantic. Moreover, even Scala's type system is still not enough to serve the DSL. In order to capture the subjectId field access, Squeryl builds an auxiliary instance of type Grade, wraps it into a proxy and sifts that proxy through the query during the runtime, recording accesses to fields and building the corresponding AST.

And still, all this effort is not enough to provide seamless embedding into the host language. See that ===? This is a special surrogate operator that stands for ==, because the equality operator itself cannot be overloaded to produce desired results.

If from were a macro def, then it would get the g => ... function argument in the form of a Tree object that represents that query at the compile-time. As a consequence, one would then be able to use any syntax in the query (including non-overloadable constructs, such as the equality operator). Also, the lifting (i.e. transformation of domain-specific codes into corresponding domain ASTs) would come for free, without the need to carefully arrange the web of boilerplate to capture the intent of the user.

This brings sweet memories of coding up Conflux, a framework for programming CUDA in pure C#. Since the computational architecture of massively parallel devices is incompatible with von Neumann architecture, I didn't even have a luxury of symbolic evaluation that is employed in Squeryl. Instead I went for writing a decompiler for .NET bytecode. If I were programming in a language with macros, that effort would be completely unnecessary. For example, take a look how a similar goal is achieved by the NUDA library written in Nemerle.

Use case: Language integrated queries

dev@scalamacros.org — Fri, 11 Nov 2011 00:00:00 PST

LINQ is a brilliant concept. Being released five years ago, even now it remains state of the art. However, it has its weaknesses: 1) relying on a clever but rigid compiler hardcode, 2) lack of composability, 3) unclear semantics of calls to external code.

The trick employed by LINQ is, actually, very neat. Whenever a compiler witnesses an expression of some functional type, say, Func<T, Boolean>, in the context that requires an Expression<Func<T, Boolean>>, it automatically lifts the provided function into an AST that is accessible during the run-time. This and this alone immediately solves the problem that plagued data access frameworks for a long time (we discuss that problem in detail in the "Advanced domain-specific languages" use case).

Being a step forward in comparison with traditional language tools for building eDSLs, LINQ is still not very extensible solution. For example, LINQ only supports lifting expressions, but not statements, which rules out many useful language constructs, and the programmer cannot do anything about that.

Also, this particular lifting scheme brings an unfortunate composability restriction. Consider the following code (the example is taken from the blog post "Calling functions in LINQ queries" by Tomas Petricek, if you want to know more about the composability problem - be sure to take a look at that post):

static bool MyPriceFunc(Nwind.Product p) {
  return p.ProductName.StartsWith("B");
}

var q =
  from p in db.Products
  where MyPriceFunc(p.UnitPrice) > 30m
  select p

This code compiles with no errors, but when you execute it DLINQ throws an exception saying: "Static method System.Boolean MyTest(LINQTest.Nwind.Product) has no supported translation to SQL".

This happens because the lifting scheme is hardcoded to be shallow - lifting only processes the expression it sees and does not recursively generate ASTs for the functions used in the query. There exist several solutions to this problem (one of those is described in the aforementioned blog post), but they are not especially elegant.

In our proposal LINQ can be implemented as follows (below we omit the discussion of LINQ infrastructure, and focus on the lifting part of the implementation):

class Queryable[T, Repr](query: Query) {
  def filter(p: T => Boolean): Repr = Impl.filter[T, Repr]
}

object Impl {
  def filter[T: c.TypeTag, Repr: c.TypeTag](c: Context)(p: c.Expr[T => Boolean]) = reify {
    val b = c.prefix.value.newBuilder
    b.query = Filter(c.prefix.value.query, c.reifyTree(p))
    b.result
  }
}

Implemented as a macro def, the filter high-order function receives its predicate in the form of an AST. Having done that, the macro is free to perform arbitrary manipulations (e.g. to call the reify function that builds a domain-specific AST from the compiler AST that is provided in p) to accumulate and, eventually, process the query.

Also, there's a solution to the composability problem: if we declare MyPriceFunc as a macro def as well, then it will expand into the call site and, consequently, will be processed by the domain-specific query translator. Among alternative approaches is the macro that requests ASTs for all annotated functions in the compilation unit and stores those ASTs for future uses (that macro could be implemented as a method-level macro annotation or even as a program-wide package annotation). The domain of possible solutions is quite big, thanks to macros being a core language feature, not an ad-hoc extension to the compiler.

Use case: Type providers

dev@scalamacros.org — Fri, 11 Nov 2011 00:00:00 PST

Code generation is a common solution to the interoperability challenge. It is employed not only for data access, but also for dealing with web services and in RPC technologies in general (ASN.1, protobuf, to name only a few).

Despite having traditionally good support from integrated development environments, this approach leaves much to be desired. Generated code is often unintelligible, is hard to be customized by hand (since the customizations might be lost after regeneration) and, what's also very important, needs to be fostered by the programmer. Finally, it's virtually impossible to implement any sufficiently complex code generator without turning it into a spaghetti of textual templates and control flow that expresses generation logic (the proliferation of various template engines somewhat supports the stated fact).

Compile-time metaprogramming techniques come to the rescue. By leveraging these techniques, the developer can programmatically generate required classes into the compiler instead of having to provide them in the source code form. One of the popular approaches that involves compile-time codegen is represented by F# type providers (take a read of an excellent blog post "F# type providers - as if by magic..." that outlines the underlying principles behind type providers):

[<TypeProvider>]
type VectorProvider() =
  interface ITypeProvider with // ...
  interface IProvidedNamespace with // ...

// Cheating for simplicity
type Vector() = inherit obj()

[<TypeProvider>]
type VectorProvider() =
  interface IProvidedNamespace with
    member this.ResolveTypeName(typeName) = // ...
    member this.NamespaceName with get() = // ...
    member this.GetNestedNamespaces() = // ...
    member this.GetTypes() = // ...

We propose an approach of even greater power that leverages the foundations of a macro system to generate the code during the compile-time. Instead of being a dedicated ad-hoc syntactic construct, macro types are smoothly incorporated into Scala.

On the one hand, macro types are full-fledged macros, i.e. they can accept parameters in AST form, they can generate resulting ASTs, which provides necessary flexibility to address code generation tasks. On the other hand, macro types are interchangeable with regular types: they can be inherited, they can be mixed in, they can participate in instantiation of generic types and, finally, they can be used as first-class modules (objects in Scala). The latter is a topic of a separate installation (see the "Generation of boilerplate" case study).

type MySqlDb(connString: String) = macro ...
type MyDb = Base with MySqlDb("Server=127.0.0.1;Database=Foo;")

val products = new MyDb().products
products.filter(p => p.name.startsWith("foo")).toList

All in all, macro types solve common problems with code generation by providing flexible extension points into the generation process and keeping the generator itself readable and maintainable. This introduces robust solutions to the problems that inevitably require writing boilerplate in modern mainstream languages.

Use case: Generation of boilerplate

dev@scalamacros.org — Fri, 11 Nov 2011 00:00:00 PST

The following excerpt from a development branch of LMS project illustrates a typical class of problems. Sometimes neither object-oriented, nor functional programming can overcome certain kinds of boilerplate:

trait TupledFunctions extends Functions with TupleOps {
  implicit def fun[A1:Manifest,A2:Manifest,B:Manifest](f: (Rep[A1], Rep[A2]) => Rep[B])
    : Rep[((A1,A2))=>B] = fun((t: Rep[(A1,A2)]) => f(tuple2_get1(t), tuple2_get2(t)))
  implicit def fun[A1:Manifest,A2:Manifest,A3:Manifest,B:Manifest](f: (Rep[A1], Rep[A2], Rep[A3]) => Rep[B])
    : Rep[((A1,A2,A3))=>B] = fun((t: Rep[(A1,A2,A3)]) => f(tuple3_get1(t), tuple3_get2(t), tuple3_get3(t)))
  ...
}

As outlined in the "Type providers" case study, one of the solutions to the problem is textual code generation. Often, it's the most obvious and the only accessible one. The aforementioned case study dwells upon the shortcomings of this approach: 1) unintelligibility of generated code, 2) poor extensibility, 3) sub-par maintainability of code generators - and proposes a solution that involves macro types.

However, there are situations when generating a single type will not save the day. Sometimes the way to go is to create multiple types and/or functions at once. We considered to introduce macro packages, but decided that macro types will be enough, since Scala objects already represent an abstraction equivalent to packages (and even supercede them, because objects can be nested and, as of such, can capture values from the enclosing lexical scope).

There are plenty of other situations when convenient facilities for boilerplate generation may come in handy. Just off the top of my head: enums, serialization/deserialization, lenses, zippers and so on.

Use case: Data types "a la carte"

dev@scalamacros.org — Fri, 11 Nov 2011 00:00:00 PST

Besides being useful for type providers and large-scale code generation, macro types provide another interesting opportunity for both a language designer and a programmer.

Recall that macro types can be mixed in into abritrary classes, and let's think this idea over. We've already seen how mixing in independently generated logic can be useful for real-world programming tasks, but what if macro types would generate the code and be aware of their context?

An augmentation of macro types lets them read the contents (i.e. the members) of the host class. This is immediately useful for building so called data types "a la carte". Take case classes. Write the following code and inspect the output:

echo 'case class Foo(bar: String, baz: Int)' >> Hello.scala
scalac -Xprint:typer Hello.scala

We see that the compiler has generated a lot of stuff for a small case class declaration, i.e. among the others implementations of Product, Serializable, equals/hashCode, toString, as well as, the entire companion object. Certain auto-implementations are frequently useful on their own, but, currently, it's all or nothing - either you declare your class as a case class and live with all restrictions being imposed, or you don't get any of the auto-generated goodies.

A more balanced API can be provided by augmented macro types or, alternatively, macro annotations. Such macros can discover members of the main class, so they can provide auto-generated functionality in piecemeal fashion. This restaurant metaphor gives the technique its fancy name.

Use case: Integration of external DSLs

dev@scalamacros.org — Fri, 11 Nov 2011 00:00:00 PST

The "Why It's Nice to be Quoted: Quasiquoting for Haskell" paper by Geoffrey B. Mainland describes an astonishing concept of embedding and integrating arbitrary languages into a sufficiently capable host language.

One of the most impressive examples in the paper involves a peephole optimizer for x86 assembly language. The example exhibits such prominent features of quasiquoting as: 1) expressing ASTs in native syntax, 2) easy composition, 3) enjoyable pattern matching.

peep :: [Asm] -> [Asm]
peep [:asm | mov&s $&r1, &r2
             cmp $&r3, &r4
             je &lbl |] : rest
     | r3 == r1 && r4 == r2

   = [:asm | mov&s $&r1, &r2
             jmp &lbl |] : rest
...

The point here is that a language with quasiquotations can subsume and embed arbitrary external domain-specific languages, granted someone implements quasiquoting services (parsing, splicing, pattern matching) for that language.

Beauty of this concept from the pragmatic standpoint comes from the fact that these external DSLs can be processed (for example, validated) by macros, which are evaluated during the compile time. This means that format strings, regexes, XML (less relevant for Scala), HTML, JavaScript - all those can be to certain extent validated during the compilation of the application, bringing additional static safety to the program.

With the syntax introduced by (an unrelated) SIP: String interpolation and formatting, it becomes possible to write quasiquotations in Scala, and with macros it becomes possible to process these quasiquotations during the compile-time.

Another useful application of this notion is convenient notation for new kinds of literals. For example, using the syntax from the aforementioned string interpolation SIP one could write b"10001" to express an integer literal 17. The interpolation would be processed by a macro during the compile-time, which means that the programmer wouldn't pay any runtime penalty for the convenience.

Use case: Units of measure

dev@scalamacros.org — Fri, 11 Nov 2011 00:00:00 PST

One of the interesting requests in the comments to my recent post was units of measure for Scala. Units of measure represent an augmentation of the type system of F# (you might wish up to read a blog series for more details: first post, second post, third post, final post).

Despite the fact that compile-time metaprogramming lies on a different (lower!) level than type systems, this very use case can be implemented with macros. This brings an interesting question of when macros are appropriate, and when it's better to do away with them in favor of other language mechanisms. We will elaborate on this topic in future installations.

In the following example F# compiler is capable of inferring the unit of measure for a derived value speedOfImpact (the dimensionality will be m/s):

let gravityOnEarth = 9.81<m/s^2>
let heightOfMyOfficeWindow = 3.5<m>
let speedOfImpact = sqrt(2.0 * gravityOnEarth * heightOfMyOfficeWindow)

With macro types it becomes possible to write the code like that:

val gravityOnEarth = u(9.81, "m/s^2")
val heightOfMyOfficeWindow = u(3.5, "m")
val speedOfImpact = sqrt(2.0 * gravityOnEarth * heightOfMyOfficeWindow)

u is a macro def that takes a number and wraps it in a macro type that depends on the unit of measure passed. That macro type hosts all necessary stuff that overloads relevant operations (arithmetic, math._ stuff, so on). These operations upon units of measure would themselves be macros that produce new types as necessary.

An alternative approach would be to implement u as a macro annotation that would operate on expression-level and ascribe numeric values with units of measure. Such a macro would be able to do everything a macro def can do, but it would look in, arguably, a nicer way:

val gravityOnEarth = 9.81: @u("m/s^2")
val heightOfMyOfficeWindow = 3.5: @u("m")
val speedOfImpact = sqrt(2.0 * gravityOnEarth * heightOfMyOfficeWindow)

At a glance, this looks like a workable solution, but, of course, it will be possible to fully assess the feasibility of this idea only when we have an implementation of the macro system that can be used for tests.