Macros and reflection in Scala 2.11.0

Eugene Burmako
05.03.2014

Year 2013 was a very busy year for the Scala reflection/macro team. We've experimented with several new macro flavors, significantly improved the experimental reflection API, expanded the docs, and fixed a huge number of bugs submitted to JIRA. In 2014 we'll be working towards making macros non-experimental - cleaning up the API, improving tool integration and addressing long-standing issues. Below you can find a brief summary of our work and plans for the future, and the full list of changes can be found in the official changelog.

What are macros good for?

We've given a number of talks about macros, of which we would like to highlight "What Are Macros Good For?" as a great introduction to Scala macros. The talk outlines what Scala macros are, and in what capacity the notion of compile-time metaprogramming can be useful, on a series of concrete use cases from research and industry. The talk shows how Slick, Play, Shapeless, Akka, Pickling, Async, Specs, and others use macros and highlights key aspects of associated techniques. In May, we plan to follow up with a workshop at flatMap that will elaborate on implementation details of typical macros.

Quasiquotes

The biggest thing that happened to macros last year is undoubtedly quasiquotes. First prototyped and rejected in 2011, quasiquotes made a glorious comeback in 2013 thanks to the efforts of Denys Shabalin. Having proven to be incredibly useful for working with abstract syntax trees in macros and beyond, quasiquotes were included in Scala 2.11.0-M4 and are going to be shipped with the upcoming Scala 2.11.0 release. For Scala 2.10.x, quasiquotes are available in the macro paradise plugin. Denys has thrown together an epic quasiquote guide that explains everything that you need to know in order to effectively work with abstract syntax trees in your macros. In June, Denys is going to present at ScalaDays, outlining inner workings of quasiquotes and showing how quasiquotes simplify integration of external languages.

Reshaping the reflection API

The 2.11.0-RC1 release of Scala is going to include a major rehash of the reflection API that introduces a number of much needed day-to-day functionality (such as Type.typeArgs, symbolOf[T] or Symbol.isConstructor) and provides first-class support for complex macros. We're by far not done with shaping the reflection API, but the changes in Scala 2.11.0 are going to provide the foundation for the refactoring to come in the future (read more about that below in the "Future work" section). The full list of changes and recommendations on source compatibility with Scala 2.10 are provided in the changelog.

Thread safety of runtime reflection

The most pressing problem in reflection for Scala 2.10.x was its thread unsafety. Attempts to use runtime reflection (e.g. type tags) from multiple threads resulted in weird crashes documented above. We believe to have fixed this problem in Scala 2.11.0-RC1 by introducing a number of locks in critical places of our implementation. On the one hand, the strategy we are using at the moment is sub-optimal in the sense that certain frequently used operations (such as Symbol.typeSignature or TypeSymbol.typeParams) are hidden behind a global lock, but we plan to optimize that in the future. On the other hand, most of the typical APIs (e.g. Type.members or Type.<:<) either use thread-local state or don't require synchronization at all, so it's definitely worth a try.

Macro annotations

Last year, we've made several attempts at making macros capable of introducing publicly visible definitions. Our first stab at it was type macros, which made public appearance as part of the first version of macro paradise. Due to reasons outlined elsewhere, type macros were rejected for inclusion into Scala, and we shifted our attention to macro annotations, first implemented in August in the second version of macro paradise. The jury's still out on whether macro annotations are going to be included in some future version of Scala, but they are definitely not going to happen in Scala 2.11. Nevertheless, you can still get macro annotations with the macro paradise plugin, working for both Scala 2.10.x and Scala 2.11.0. In April, Travis Brown and Eugene Burmako are going to give a talk at Scalar that will elaborate on how macro annotations can be used to emulate type providers and how one can work around the absense of macro annotations in the official distribution using a clever trick.

Fundep materialization

Since Scala 2.10.2, implicit whitebox macros can be used to materialize instances of type classes, however such materialized instances can't guide type inference. In Scala 2.11.0, materializers can also affect type inference, helping scalac to infer type arguments for enclosing method applications, something that's used with great success in Shapeless. Follow Eugene Burmako's talk at a BASE meetup to learn more about this interesting technique.

Blackbox and whitebox macros

In Scala 2.11.0-RC1, macro defs are split into blackbox macros and whitebox macros, distinguished by the type of Context used in their macro implementations. Blackbox macros are currently better supported in IDEs and are more likely to be included into Scala in non-experimental capacity, whereas whitebox macros are more powerful. This is the current state of things however, and it can change in the future as we're planning to improve tool support of whitebox macros. You can read more about the differences between blackbox and whitebox macros in our documentation.

Future work

At this point it is clear that reflection and macros are useful. However there's still quite some work to be done in order to make them non-experimental. The main issues that remain to be addressed are verbosity and complexity of the underlying API, difficulties with tool integration, and potential for confusion caused by leaking compiler internals and cryptic error messages. This is something that we've been working on in Project Palladium, which aims at becoming the new foundation for metaprogramming in Scala, keeping existing functionality in place and at the same time fixing fundamental problems intrinsic to the original implementation. At ScalaDays, Eugene Burmako will present our results, showing how easy it is to metaprogram with Palladium, and outlining the place of our developments in the future of Scala.