Use Cases for Implicit Conversion Blackbox Macros

Language Design
1

This is a follow up for @odersky’s Proposed Changes and Restrictions For Implicit Conversions. I think in general most people agree that unrestricted implicit conversions are not good. The proposed ~/into parameters help one use case - “magnet pattern” implicit constructors - but there are still some use cases that are not already satisfied by the proposal. In particular, implicit conversions macros are something that despite the scary language features end up having pretty useful, pedestrian use cases.

@soronpo already brought up one use case in its own thread where implicit whitebox macros are useful (Before deprecating old-style implicit conversions, we need this) and is worth a discussion, but I’d like to bring up a few more concrete use cases of implicit blackbox macros that already have broad adoption today:

Compile-Time Refined Literal Constructors

@Iltotore brought this up in Proposed Changes and Restrictions For Implicit Conversions - #177 by Iltotore, a bit late, and I think it deserves more discussion than it ended up having:

Refined:

// This refines Int with the Positive predicate and checks via an
// implicit macro that the assigned value satisfies it:
 val i1: Int Refined Positive = 5

Iron 2:

//An inline implicit conversion requiring a given instance of `Constraint[Int, Positive]` is called.
val x: Int :| Positive = 5

In this case, the implicit conversion is necessary to convert from an Int to a Refined[Int, Positive] or :|[Int, Positive] in a statically-checked way. They cannot be a normal implicit conversion, nor can they be into parameters because:

  1. They need to occur at any point in which someone expects a Refined[Int, Positive]. This may be in a parameter list that supports into params, but could also be in local variables, in collections (e.g. Seq[Refined[Int, Positive]](1, 2, 3), and other places.

  2. They need to perform compile-time analysis of the literals being give to them, in order to provide compile-time error messages if the literal cannot be converted.

The alternative is to do these checks at runtime, which opens us up to either (a) runtime errors or (b) returning an Option[Refined[Int, Positive]] (or equivalent). These approaches are available and widely used, but are respectively less safe or less ergonomic than providing precise compile-time checks and errors where possible.

Other related use cases could include e.g. a sql("...") function or html("...") that allows literal strings to be used as SQL or HTML strings (respectively), performing any necessary validation at compile time, while also disallowing sql(myStr) or html(myStr) to work with variables of type String OR allowing arbitrary myStr: Strings to be passed and performing the validation at runtime (and either throwing exceptions or wrapping the result in Options/Eithers

It is possible that some compiler support around working with singleton types could make some of these use cases doable at compile time as part of the normal type system, but it seems unlikely we’ll ever support the full Scala language when working with compile-time singleton types. Implicit conversion macros provide an easy way to do so, letting us easily drop into “full Scala” to perform assertions on literal primitive expressions to provide assertions

Compile-time refined types aren’t super ubiquitous, but I think we can agree that it’s a cool approach that provides both more safety and more ergonomics than the alternatives of doing such validation at runtime and throwing exceptions or returning Option/Eithers, and libraries like Refined have a pretty long history in the Scala ecosystem. Searching “eu.timepit” “refined” language:Scala on Github pulls up a ton of usage out in the wild. Although I haven’t used these heavily myself, I think they are a use case worth preserving.

sourcecode.Text

def debug[V](value: sourcecode.Text[V])(implicit enclosing: sourcecode.Enclosing) = {
  println(enclosing.value + " [" + value.source + "]: " + value.value)
}

class Foo(arg: Int){
  debug(arg) // sourcecode.DebugRun.main Foo [arg]: 123
  def bar(param: String) = {
    debug(arg -> param)
  }
}
new Foo(123).bar("lol")  // sourcecode.DebugRun.main Foo#bar [arg -> param]: (123,lol)

sourcecode.Text uses an implicit conversion macro to capture the source code of an expression. This is super useful for debugging helpers, and is used e.g. in pprint.log from the com.lihaoyi::pprint library, and replaces the very common println("foo = $foo) pattern.

Python has embedded this functionality into the language as print(f"{foo=}"), in the context of debug-logging/string-interpolation, but Scala’s ability to do it to arbitrary expressions while preserving the value itself is super useful and extends its use case in Scala beyond what you can do in Python . For example, I have also used it in other contexts to automatically extract code snippets from a test suite in order to embed them in documentation.

The implicit conversions occasionally cause type inference to trip up, and the macro has some parsing edge cases, but overall this works surprisingly well. You can search "sourcecode.Text" language:Scala on Github to see this macro used pretty widely across the ecosystem. I think this is also a use case we want to preserve.

This use case cannot be satisfied with singleton type ops. It could be done with special compiler support, and perhaps it’s specific enough that compiler support is a valid way of approaching this. Unlike Compile-Time Refined Literal Constructors above, this use case is basically a single implementation, and does not have the different variants and flavors that refined types do.

Mill Task Definitions

The Mill build tool uses mill.T{...} as an implicit macro that performs source code transformations, similar to the SBT := macro, to lift “direct-style” code into a free applicative. This allows us to write code in a “direct style” (with some limitations) while still getting the benefits of introspectability, parallelizability, e.g. that the free applicative gives us

As an example, given this source code:

trait FansiModule extends Module with ... { 
  def artifactName = "fansi"

  def pomSettings = PomSettings(
    description = artifactName(),
    organization = "com.lihaoyi",
    ...
  )
}

expanding via an implicit conversion into

trait FansiModule extends Module with ... { 
  def artifactName = T { "fansi" }

  def pomSettings = T{
    PomSettings(
      description = artifactName(),
      organization = "com.lihaoyi",
      ...
    )
  }
}

Where T{...} is a macro, that (approximately) expands into

trait FansiModule extends Module with ... { 
  def artifactName = T.zipMap(){ "fansi" }

  def pomSettings = T.zipMap(artifactName){ artifactName2 =>
    PomSettings(
      description = artifactName2,
      organization = "com.lihaoyi",
      ...
    )
  }
}

This T{...} implicit macro is pretty core to the whole Mill developer experience. While the implicit conversion does occasionally trip up type inference, in general it is pretty seamless and results in a huge reduction of boilerplate in Mill builds. Mill is widely used across the ecosystem, and a github search for "import mill" language:Scala pulls up tons of build.sc files where this macro would be used almost ever def

While it is possible we could force people to call the T{...} macro explicitly, doing so would be a significant amount of boilerplate for Mill builds: you would need to add it to every def, and a Mill build file is basically all defs, so this would basically add a bunch of line-noise to every single line of code.

Unlike the earlier use cases of Compile-Time Refined Literal Constructors and sourcecode.Text, I do not see an easy alternative here: the macro transformation is performed on a big AST, and so cannot be reduced to singleton type-level logic, and is also too ad-hoc to bake it directly into the compiler. Furthermore, the Applicative-focused transform is sufficiently different from the Monadic transforms that are common throughout the Scala ecosystem (Scala-Async, ZIO-Direct, etc.) that it’s unlikely that any “standard” direct-style transformation built around Monads would be an alternative for Mill.

It’s possible that some baked-in compiler support for Applicative Idiom Brackets could make this implementable in user-land without an implicit macro, but no such thing exists today. I think this is a use case we need to keep given the growing ubiquity of Mill builds across the ecosystem.

These are three concrete use cases I could think of off the top of my head that the proposal to limit implicit conversions does not support. IMO finding some solution for these is a blocking requirement before we can limit implicits conversions:

  1. These use cases are both useful and pedestrian: they aren’t esoteric DSLs built for the sake of it, but are straightforward ergonomic improvements to common workflows. The implementation and implementor may be advanced, but the code and end-users that use these implementations are not. Removing these libraries makes the Scala experience worse for these not-advanced users, not better

  2. These use cases are pretty widespread throughout the ecosystem. Lots of projects use refined, sourcecode.Text, or Mill today. Killing implicit conversion macros without a proper alternative is giving a bunch of projects have the option of making their code worse (less safe, more boilerplatey) or remaining on an old version of Scala. It’s not a good position for any individual project to be in, and it’s not a good position for the ecosystem as a whole if different projects make different choices around this decision

8 Likes