SIP: Curried varargs

Scala Improvement Process
68

I don’t think macros actually convert ASTs to Tasty. They just use the Tasty-reflect interface into the compiler’s AST, as far as I know (needs confirmation).

They now are (see @soronpo’s link).

True, but if the macro ends up working well, it could be integrated into the standard library.

That said, I also think the overall experience (and type-based documentation) would be better with a first-class mechanism. The tradeoff with respect to making the language more complex is not that obvious, though. Indeed, allowing it as a plugin could change the parameters of the tradeoff, but then again giving this kind of power to people is a sensitive matter.

1 Like
69

@yangbo

It is slower than analytic plugin or compiler built-in implementation because it type-checks twice and needs additional conversion between AST and Tasty in Scala 3.

This remains to be verified. I am not sure it will be much slower, nor that it would matter in the big picture.

  • You cannot determine return type from parameters unless using white box macros, which are not supported in Scala 3.

Whitebox macros are supported in Scala 3 by now!

  • A 3rd party implementation cannot be a dependency of core libraries
    [/quote]

Sure. But, if this version is successful, then it might well migrate into the core, and then core libraries could depend on it.

1 Like
70

Unfortunately currently Dotty does not support macro with varargs. The current type of 'vararg is Expr[Seq[_]], while we need Seq[Expr[_]]. Also there is no AST of each parameter when unseal the expr, instead it returns a Inlined(Nil, _, Indent("vararg"))

71

Check out how it was done here: https://infoscience.epfl.ch/record/267528

Not sure whether the code still works or not.

72

You probably need to call underlyingArgument, for up-to-date macro examples see https://github.com/lampepfl/dotty/blob/master/library/src-bootstrapped/dotty/internal/StringContextMacro.scala, GitHub - lampepfl/xml-interpolator: XML String Interpolator for Dotty, https://github.com/dotty-staging/scalatest/tree/dotty-community-build/scalactic.dotty/src/main/scala/org/scalactic

3 Likes
73

It’s concerning that we’re being told both: “you should do this via metaprogramming” and “you’re going to need this method, which is not mentioned in the docs”

This is particularly troublesome as the only supported IDE does not support jump to definition outside the current project, which really hurts discoverability.

Are there any up-to-date resources geared towards teaching this? Dotty is still experimental, so if I’m going to sacrifice my evenings and weekends to dig through this, I’d appreciate something more than, “here are some examples, subject to change without notice”.

2 Likes
74

For the general case, this would would require specifying a return type of <: Any, which seems to work against the point of having the limitation of only specializing a return type. I don’t have a particular problem with that, other than it feels like the sort of thing that the compiler is going to complain about.

75

Current Dotty macros are still not enough for implementing this proposal because it lacks of the ability to produce untyped AST. For example, in Scala 2, a macro author can create the untyped select Select(callee, TermName("applyBegin")) in spite of the definition of applyBegin. However it will crash in Dotty if applyBegin requires an additional type parameter.

4 Likes
76

Well, that was throughly demoralizing.

I’ve gone through all of the source files in those examples, re-read the docs multiple times, and I’m no closer to implementing a “show” interpolator which could serve as a POC for Curried Varargs than I was when I started.

I can go from Expr[Seq[Any]] to the individual Terms, but there doesn’t seem to be a path from Term to Type, which I’d need to summon the needed implicits. Found a whole bunch of examples of widening the type to Any, and a whole bunch of examples doing type tests against a hierarchy of known types, but nothing even hinting that going the other way is possible.

I even tried grepping my way around the Dotty source code, but 42 pages of results for “QuoteContext” is beyond my fortitude at the moment. As “jump to definition” isn’t currently supported, I’m pretty much stuck.

Finding the docs for underlyingArgument were two sections removed from “Macros” (both of those two sections dealing with compilation details), after a disclaimer that basically says, “Don’t use this unless you know what you’re doing”, was particularly disheartening.

Between this, and the results of my experiment with significant whitespace (I built a tool I’ve been wanting for a while, and it was every bit as painful as I remember from Python), I’m really losing enthusiasm for the switch over.

1 Like
77

Maybe my suggestion is silly, but what if you developed the macro within the dotty project, so you can jump to definitions?

78

Here’s a small interpolator I threw together around a week ago that does plenty of type inspection.

3 Likes
79

The ExprSeq extractor is there for that. I added some documentation and an extra underlyingArgument directly on expressions to make it simpler to use in https://github.com/lampepfl/dotty/pull/7697

2 Likes
80

I think it is OT to ask if it’s possible to get a dotty-themed spongepowered t-shirt.

Usually that is just the regular t-shirt, but with dots.

81

Thanks for the example, but this has the same problem as the other examples WRT being relevant to what I’m trying to do: it’s checking against a known list of types.

I’ll try and explain a little more clearly I’m looking to do, as the provided examples seem to indicate I have not explained it well.

Raw Code

show"Tried to foo the $bar, and got a $baz instead"

Scala 2 expansion

new StringContext("Tried to foo the ", ", and got a ", " instead")
  .show(
    Shown.mat(bar)(implicitly[Show[Bar]]),
    Shown.mat(baz)(implicitly[Show[Baz]])
  )

This works because it’s driven by implicit conversions, which for Scala 2 only need to be enabled at the definition site. This approach would be extremely problematic because, at least according to my experiments, in Dotty (and thus presumably Scala 3) you need to import the feature flag at both the definition site and the use site. It makes zero sense to require enabling a feature known to make code less comprehensible on a near global level to opt-in to a feature intended to make code less error prone.

Ideal Scala 3 expansion

new StringContext("Tried to foo the ", ", and got a ", " instead")
  .s(
    bar.show(given summon[Show[Bar]]),
    baz.show(given summon[Show[Baz]])
  )

Unfortunately, this means that I need to be able to summon a known typeclass for each passed argument. While the compiler apparently knows these types, I can’t seem to find any documentation or example hinting at how to access so I can splice together the calls. I’ve found plenty of examples of how to check if it’s one of a handful of types, so I accept that the compiler has this information, but as far as I can tell, it’s completely inaccessible if you don’t know it ahead of time.

While the above is interesting because I’d like Curried Varargs so I can do this reasonably, it’s also a much simpler problem and if we can’t do this it casts signifiant doubt on the plausibility of implementing Curried Varargs via a macro (as was suggested).

82

It is possible to find implicits within a macro using searchImplicitExpr http://dotty.epfl.ch/docs/reference/metaprogramming/macros.html#find-implicits-within-a-macro.
I will extend the documentation to add an example where it is used to find implicits based on types of some arguments. Note that I am updating the API to use summonExpr instead in https://github.com/lampepfl/dotty/pull/7687/files, I will probably add the documentation in that PR.

It might also be possible to find implicits using only inline and summonFrom http://dotty.epfl.ch/docs/reference/metaprogramming/inline.html#summoning-implicits-selectively. Though I’m not sure if it would work in this particular case.

83

I’m aware of searchImplicitExpr and summonFrom, I’ve gone over what’s in the docs multiple times. Once I get the type, summoning the instance isn’t a problem.

Getting the type is the problem, so if your example could include a case where the type isn’t known ahead of time (specifically, going from Expr[Any] or Term to Type) it would be extremely helpful.

84

But with curried varargs I would imagine we would just need to write something like

class StringContext(s: String*, args: Any*) extends AnyVal {
      def applyNext[T](next: T)(given Show[T]): StringContext = StringContext(s, (args :+ next.show): _*)
      def applyEnd: String = this.s(args)
    }

or probably as extension methods.
Where each applyNext knows the type of the element and can find the necessary implicit.

85

I will try it out and have a complete example. The trick will be to use a pattern arg match { case '{ $arg: $t } => to recover the type of the argument. I’ll have a look at the details.

86

Yes, with curried varargs.

We’re currently without them, and have been told to do what we need to with current metaprogramming capabilities (which means in this case, mostly macros), or implement curried varargs using the same.

My primary usecase for curried varargs is exactly what you’ve proposed (I’ve included in a previous comment an extremely similar sample implementation using curried varargs). It’s also considerably simpler than what we’d need to do to implement curried varargs with a macro, so it provides a useful canary.

87

That’s an interesting bit of syntax, I had to go back through the docs to see if I’d missed that. I could only find examples where that was part of the match, not a value bound by the match. If it works, that would be awesome.

Something needs to be done about the lack of documentation for this, so much is missing that right now macros seem very much like something reserved for the High Priesthood.

Thanks for giving it a shot, good luck :+1: