I just wrote a blog post about a new proposed feature for Scala: Implicit function types. Happy to discuss here if there are questions or comments.
6 Likes
This looks like a great step in a very promising direction! It is a little peculiar, though, that it’s solved by muddying the distinction between arguments and return values. Of course currying already does exactly that, so it’s not like it establishes a wholly new isomorphism. But there are advantages to having arguments stay in argument position including making it easier to reason about co/contravariance (“it’s just another argument”), ease of telling what the return type actually is, and knowing where to put type parameters and such.
So I think the idea of implicit functions is great, but I wonder whether the syntax is more different than it needs to be, and the logic is more sutble than it needs to be. These two forms are equivalently compact:
def foo(i: Int): Bar[Int]
def foo(i: Int)(Bar): Int
which would require only that we say that we can list implicit types without a name in a final parameter block, and otherwise would work more or less the same as the proposal (with adjustments to precedence based on nesting).
It feels like there is an important difference, but I haven’t yet been able to clearly describe it, and since I’m necessarily not very familiar with this proposal I don’t know which is better. But my sense is certainly that the form above would be a lot more familiar.
Additionally, it doesn’t seem to fully address a second problem with implicits: sometimes you need them to do something, and it is frustrating to try to deal with them internally by calling implicitly[SomeHugeTypeName] over and over again. Implicit functions help this a little by making it easier to write methods to help you, but actually being able to name the implicit is very often important, which is why, for instance, I almost never use [A: Baz] notation instead of [A](...)(implicit baz: Baz[A]) even with all the extra boilerplate: it’s too important to have access to baz internally in the method.
The end result is code that looks extremely similar to the Reader monad - thisTransaction is just Reader(identity [Transaction]) - the only difference is the implicit is threaded through completely invisibly rather than in a for/yield.
I think the <- syntax of for/yield composition is probably a better way to represent this kind of thing than making the implicit completely invisible - Scala has constantly walked a tightrope between reducing boilerplate and making things too magic, and I fear this is the wrong side of the line. Compare e.g. how many coding standards now prefer JavaConverters over JavaConversions.
But assuming we do want this functionality, is a special case for Reader only really worth the cost in the language complexity budget? Wouldn’t it be better to standardise a general-purpose solution along the lines of https://github.com/pelotom/effectful that would work for any kind of for/yield-compatible construct, not just Reader but also e.g. Writer (which can be seen as an implicit return value, the counterpart to an implicit parameter), Try, Future and the like?
It would be great if implicit parameters could go from being a language-level special case to being a library construct that makes use of general-purpose functionality in the language, just as has been done for e.g. XML literals and string interpolation.
Additionally, it doesn’t seem to fully address a second problem with implicits: sometimes you need them to do something, and it is frustrating to try to deal with them internally by calling implicitly[SomeHugeTypeName] over and over again.
The example in the blog shows how to get around this: I can write a method thisTransaction once, which picks up the implicit for everyone:
def thisTransaction: Transactional[Transaction] = implicitly[Transaction]
If I take the dotty compiler as a use case, this costs me 1 line, but saves me 2641 implicit parameter declarations.
the only difference is the implicit is threaded through completely invisibly rather than in a for/yield.
This is actually a rather big difference.
-
You gain (I estimate) a factor of 10 in performance. And that does not even include trampolining. Would be great to see real numbers, for sure.
-
You gain in composability. If you have a
Reader[X, Reader[Y, A]]but need aReader[Y, Reader[X, A]], you have to use boilerplate code. For implicit functions, this is no problem:val x: implicit X => implicit Y => A
val y: implicit Y => implicit X => A = x
typechecks as is.
For me, both of these points are huge deals. I could not imagine writing the dotty compiler with the reader monad, and would invite anybody who does not believe me to try this.
2 Likes
def thisTransaction: Transactional[Transaction] = implicitly[Transaction]
Depends, the workaround seems to come at the expense of having to define a globally unique name, so at use site you may be referring to the somewhat cumbersome thisTransaction many times, whereas with current Scala you can define short form implicit ev: ..., implicit ctx: ..., etc. and have terse local usage.
It’s a tradeoff, better would be if there was syntax for named implicit functions as @Ichoran suggests; then we could have our cake and eat it too.
1 Like
I would certainly like Reader to perform better. But I view it as more important (for the Scala community) to get the language semantics right first. If the compiler can’t compile the Reader-based for/yield version as efficiently even when it “obviously” does the same thing, that’s a compiler issue, but that shouldn’t stop us from offering the general-purpose syntax for the common case where consistency is more important than performance. Compare SI-1338.
Composability is indeed a concern, but again one I’d rather see solved generally. E.g. with language-level coproducts, and compiler knowledge that the type X:+:Y is the same as Y:+:X, one could use a Freek-like type (something like Effect [Reader [X] :+: Reader [Y], A]) .
These problems are real and important. I just think they’re real and important in all the cases, and Reader isn’t special enough to be worth language-level special support.
I would happily write the dotty compiler in terms of Reader - it’s just a mechanical transformation, no? In my own code I have already reached a position of avoiding implicit parameters like this in the general case, using implicits only for typeclasses, magnets, pimping (extension methods), or where required by a third-party interface.
2 Likes
def thisTransaction saves 2641 implicit parameter declarations, but the construction as a whole comes at the cost of 2641 places where the actual return type of the function is no longer as clear because the implicit is entangled with the return type.
Also, the def helps some but it doesn’t help nearly so much when you have, many different implicits that are used, say, three times each. Maybe it’s enough–the many-repeats case is arguably the bulk of the work–but since you have to state the type anyway, you lose some of the benefit. On the other hand, we already have syntax for binding variable names at unexpected locations: @. This could be allowed also, or even re-used to mean “this is implicit”.
def foo[A: tr @ Transactional](i: Int): Int
def foo(i: Int)(tr @ Transactional): Int
Aside from the overall feature, there are a variety of boilerplate-reducing alternatives that don’t push the implicit into the return type:
def foo[:Transactional](i: Int): Int
def foo(i: Int)(Transactional): Int
def foo(i: Int) with Transactional: Int
and I don’t immediately see that this is a barrier to the increased composability and flexibility one gets from a first-class notion of implicit function.
def thisTransaction[: Transactional]: Transaction
def thisTransaction(Transactional): Transaction
def thisTransaction with Transactional: Transaction
would be the forms with this alternate syntax, and they should all work just as in the example. Here we haven’t even needed to curry the function, but if we did, we’d get the same implicit function as in the proposal.
2 Likes
If I have type Implicitly1[T] = implicit Implicit1 => T and type Implicitly2[T] = implicit Implicit2 => T, is there a way to easily compose the two to get implicit Implicit1 => implicit Implicit2 => T? Or will I have to write a new type alias for that?
The point is when you’re passing these functions around. For instance in
Play you might need to take in an implicit RequestHeader => A.
Also this lays the groundwork for an effects tracking system. He touched on
it in the scaladays talks. I think there was another talk where he expanded
it but I can’t find it. But essentially the idea is that you could say that
for instance in order to be allowed to mutate a variable you would need an
implicit value representing the permission. Thus effects become a part of a
function’s type signature. Making the implicitness a part of it too (and
being able to alias it) makes this less painful.
Come to think of it, presumably the "action X needs an implicit Y in scope"
rule could be done by a compiler plugin, so maybe someone from the community could try that out
3 Likes
Suppose they weren’t implicit or aliased, just ordinary functions, one of type Implicit1 => T and one of type Implicit2 => T. How would you compose them? If you wanted to write a function of type Implicit1 => Implicit2 => T using just those two functions how would you go about it? You’d of course start with
def f = { (a: Implicit1) => (b: Implicit2) =>
val result1: T = f(a)
val result2: T = g(a)
val returnValue: T = ??? // uh oh, now what??
returnValue
}
Implicit functions are only adding a drop of metadata to functions (is it of type FunctionN or of type ImplicitFunctionN). And implicit parameters themselves are only the compiler writing a bit of code for you, that you could have written yourself. So if something is not possible with ordinary functions, implicits and implicit functions can’t make it possible. They can only make the already-possible easier.
Unless your question meant, is there a way to express in the type level, essentially a type lambda X[A, B] such that X[implicit Implicit1 => T, implicit Implicit2 => T] =:= implicit Implicit1 => Implicit2 => T. Well the arrows are just syntactic sugar for a trait with type parameters, so it shouldn’t be any more possible than with any type constructor F[A, B]. So can you write a type alias (lambda) X[A, B] such that X[F[A, T], F[B, T]] =:= F[A, F[B, T]]?
BTW, of course it’s possible to do these computations with more parameters (which can be implicit, maybe even typeclasses). So in the first interpretation, you’d need to take in a function (T, T) => T so you could combine the results of the two functions. In the second, you’d essentially need X to be a typeclass.
1 Like
I guessed as much, but it just seems to me that this would become impractical fast. Especially when you would encode an effect system with implicit parameters. For every combination of multiple effects and typeclasses and contexts you would need to write a new type alias. So I think, while this proposal helps a lot for cases like the dotty compiler, I don’t see how it would necessarily make an effect system more practical.
In all these examples, what is “Transactional”? Is it a type? Which one?
1 Like
Tried some things.
Would be really cool if in Main_works3.scala the return type would be inferred as implicit Show[T] => List[String] without the explicit implicit parameter.
Here it would just be
type Transactional = implicit Transaction
That is, these methods would be a shortcut for implicitly[Transaction]. I’m simply uncurrying the type signatures you wrote in your blog.
I guess the key is allowing implicit to be a property of the type rather than a property of the use site. This to me seems orthogonal to allowing implicit functions (which are helpful in transporting implicit capability around).
1 Like
But that would change the rules of the game completely. It would not resemble implicits in Scala at all, it would be a completely different language design. And you would have to answer a host of other questions, like how these things behave under generics.
1 Like
Depending on what restrictions were placed, the design could be rather different, yes. But it’s the logical extension of the direction implicit functions are already going. Maybe it is too much extension.
The current proposal allows implicits to be part of the type only for functions and provides the equivalent of some automatic currying/uncurrying to jump between a view of the implicit as part of the return value and a “plain” return value. I am convinced that this saves a good deal of boilerplate, but I’m not convinced that it’s the only extension that saves a good deal of boilerplate or that there aren’t alternate or complementary changes that would save more.
I’m also not sure that this as it is doesn’t “change the rules of the game completely”. I can now say that a method is returning a Foo[A], and it doesn’t give me a Foo[A] at all when I use it. That’s a pretty new game.
In the surface syntax, yes. But the crucial point here is that implicit A => B is just syntactic sugar for ImplicitFunction1[A, B], and we know what that is. By contrast, we have no idea what adding implicit to every type would mean, nor if it would make sense in the end to do so (my hunch would be it wouldn’t).
1 Like
I guess it depends how and whether one attempts to abstract the idea.
After thinking it through a bit more, I think that there is a natural interpretation of implicit that applies to anything but it is not identical to the idea in implicit functions, so it is an awkward fit to extend it in the way I was thinking. In particular, conflating an implicit (function from arguments to result) and a function from (implicit arguments) to result seems, on further reflection, like a dangerous idea.
If you use an implicit-argument function as a return value, you can consider it either of two ways:
- It is a formalism that expresses your intent to require an implicit argument, or
- It is an actual object that is created that evaluates when an implicit argument is available
If the latter, then I worry about the performance hit; this is definitely not a zero-cost abstraction if you’re adding the creation of an ImplicitFunction object to every method call. If the former, then I don’t think we’re necessarily prevented from coming up with other low-boilerplate syntax to express the intent when what you really mean is “I don’t care how you do it, just carry along these implicit values” not “I really want a function that takes implicit arguments”. Especially if you tackle the group-of-implicits problem and have an ImplicitTuple3 etc. that is isomorphic to a parameter block of three implicits (and then ImplicitFunction2[A, B, Z] would have a natural relationship with ImplicitTuple2[A, B] => Z).
It seems like implicit tupling is slightly more general than implicit functions in that implicit tuples allow one to talk in a generic way about groups of implicit parameters distinct from what you might use them for, whereas implicit functions require you to have a transformation in mind.
Beware: This change is dropping an arbitrary restriction from the language in comparison to implicits in many other systems (like Agda, Coq, …). If a method can take some arguments, a closure should be able to abstract over those. Another such restriction forbids polymorphic function values.
I agree the blog post is showing a single (though potentially important) use case, and it’s not 100% clear whether you’d want to use that. In particular, instead of seeing when this pattern works, I’m curious to know when this pattern fails, to see if there are pitfalls, use cases which can’t be expressed, cases where you get the wrong value inferred, and so on.
I’m much more concerned about unintended side effects for this change:
We fix the problem by introducing a new disambiguation rule which makes nested occurrences of an implicit take precedence over outer ones.
1 Like