"Unpacking" classes into method argument lists

Language Design
44

Yes, exactly. I was going to comment earlier, but the keyword idea didn’t seem to be getting enough traction for me to bikeshed it.

But, yes, calling it unpack is weirdly backwards.

I still, despite using it extremely heavily, put given and using backwards, because that’s how they’re normally used in language. I hope someday to get it right, but it’s been over a year of heavy usage and I still reflexively get it wrong unless I pay attention. (It is worse because my mathematics background.)

Definitely pack vs. unpack.

However, use-site disambiguation is important, and I think the symbols do better as a parallel there.

1 Like
45

unpack is what Python uses. I think it makes sense. The elements of the parameter type appear in unpacked form as possible arguments to the function.

1 Like
46

Aha, if you think about it in that direction it works - the definition says how the caller will see it. Hmm. I think I can get used thinking that other way around (and not thinking that it is boxed in the definition).

47

Yes so we could see this:

def f(xs: Int*) = ???

as sugar for

def f(unpack xs: Seq[Int]) = ???

wich actually could allow us to use any sequence, not just ArraySeq, with a repeated argument apply as vararargs:

def f(unpack xs: Vector{Int]) = ???

which is indeed more general.

48

I think the general feeling behind that proposition is really extremelly good and motivating. That kind of structured parameter is something that I missed a lot, and the proposal (the general direction) is really nice to see.

I would be an immense simplification in everything looking like “initializing parameters for services from config files”. Read the config in case class, pass them to init methods. In the method, directly use the named param without the boilerplate.
In test, directly call the methods without the case class.

It will likely need some work for tooling to understand the naming correspondance, but it looks like it worth it. Thanks for you and @lihaoyi to bring these kind of enhancements to the lang.

1 Like
49

I see enthusiasm on the proposed solution, however I think the problem could also be solved through the prism of structural types, which I would like to describe here.

The support of structural typing merits a debate on its own, but I think it is better to have it in mind when designing a solution for arguments unpacking. Indeed, I believe better support for structural typing could “naturally” solve the stated problem in addition to solving many other problems. However, if the current proposal was implemented, and assuming at some point in the future the support for structural typing would be improved, then the situation would be confusing because we would have two different unrelated ways to implement arguments unpacking.

That being said, the solution I see via structural typing would have a limitation compared to the current proposal: unpacking arguments would be supported only in argument lists containing exactly one record type (see the example below). But maybe that limitation could be relaxed, I am not sure.

Let’s pretend that Scala supports a short syntax for structurally typed records. So, instead of the following:

type Movie = Selectable { val name: String; val year: Int }

We can write:

type Movie = (name: String, year: Int)

And we can write record literals with the following syntax:

val theMeaningOfLife: Movie = (name = "The Meaning of Life", year = 1983)

Now, assuming there is a method foo that takes a movie as a parameter:

def foo(movie: (name: String, year: Int)): Unit = ()
// or, with the type alias defined above
def foo(movie: Movie): Unit = ()

We can call the method foo as follows:

foo(name = "The Meaning of Life", year = 1983)

Which, in effect, is equivalent to arguments unpacking.

The way this works is similar to the existing (though controversial) auto-tupling mechanism. The call expands to the following:

foo((name = "The Meaning of Life", year = 1983))

Which is a regular method call with a record literal passed as a parameter.

I know the topic of structurally typed records is a can of worms, but I wanted to mention it in this discussion because it would also bring a solution the arguments unpacking problem.

10 Likes
50

The auto-tupling-structural-type proposal misses one important property of the original proposal: that you can unpack part of a parameter list. That means a group of methods can share a common set of arguments, while still defining their own unique to each one. That is what make the equivalent feature so useful in Python and FSharp (if it lands)

In particular, this limitation makes the structural type proposal unusable for ~all the example use cases i brought up, in uPickle/OS-Lib/Requests-Scala. Those all are collections of similar methods that share many-but-not-all of their arguments.

It also would not be help reduce the boilerplate when declaring a large number of related case classes with common fields, which is another big motivator for this and supported by @odersky’s strawman proposal

I’m in support of lightweight named tuples as much as anyone else, but it seems an orthogonal feature to this one and not a substitute

51

Trying to get my head around that, I have a fear : my parameter case classes are almost never flat, they are a composition of smaller case classes :slight_smile:

final case class Id(value : String) 
final case Smthg1(id1: Id, p1: Int)
final case Smthg2(id2: Id, p2: String) 

final case class Smthg3(s1: Smthg1, s2:Smthg2) 

def m0(x1: X1, x2: X2, id: Id**) 
def m1(x1: X1, x2: X2, s1: Smthg1**) 
def m2(s1: Smthg1**, s2: Smthg2**)
def m3(s3: Smthg3**)

(modulo the syntax subject)

1/ Would it be possible to call m0 with flat parameters? (I guess yes, it’s the whole point)

val res = m0(x1, x2, "something")

2/ same question with m1 (ie with recursive flattening of ID?)

val res = m1(x1, x2, "something", 42)

I guess it would work given @odersky comment on recursive unpacking.

3/ same question with m2?

I guess no since @odersky message on multiple unpacking.

4/ same question with m3?

I guess yes given recursive unpacking?

If so, it’s really not a problem to not have multiple unpacking, since we will be able to create an englobing case class and work around the limit.

Did I correctly get the state of things?

52

Yes, that reflects what I proposed.

1 Like
53

How common is this problem for others? This clearly depends on the kind of codebases you work on, and I haven’t encountered this problem myself, that I can recall. And I would expect this to be addressable using the builder pattern. There’s a lot that we have going on in typing a method application (and teaching it to users), so I would be concerned in adding something like this for a very niche and non-blocked use case like the requests ones mentioned.

1 Like
54

This can be considered a fluent syntax for building structurally typed records from others.

55

This is a long thread, so I’m not sure if I’ve missed a clarification somewhere, but there seems to be some ambiguity about what we’re talking about, exactly. I’ve seen three concepts discussed:

1:

def f(unpack foo: Foo) = ??? //Or pick your favorite symbol suggestion

f(a = "bar", b = "baz") 
//Aside: could this just be f("bar", "baz")? Being forced to name things when
// unambiguous and complete seems less than ideal.

This is what I believe a few people have referenced as being better named “packing” (or pack in the keyword form, I suppose).

2:

def f(unpack foo: Foo) = ???

f(myFoo) //Maybe 'myFoo' has some symbol before or after?

This is … nothing? Just a normal argument like we’ve always had.

3:

def f(a: String, b: String) = ???

f(myFoo**) //Or whatever symbol here

This is more what I would think of as “unpacking”.

Version 1 seems to need a keyword or symbol at the def site to clarify and enable more flexible interaction patterns. Though, if we are going the keyword route, I think it should be pack not unpack. And, in particular, NO keyword or symbol seems necessary at the call site.

Version 2 doesn’t strictly need a keyword or symbol, but having the pack at the def site wouldn’t hurt anything and provides options for the caller. No symbol or keyword at the call site here either.

Version 3 doesn’t seem to need a keyword at the def site, but DOES need one at the call site.

So, I have two questions, I guess:

  1. This seems awfully reminiscent of implicit, no? Just like given and using clarified purpose, I think different symbols or keywords disambiguates between the call site feature and the def site feature (which, as above, are not the same). I’m struggling to come up with a scenario that would need to combine them that isn’t just scenario 2 above, but even if there is a use case, clarity of purpose of keywords/symbols at the call and def sites wouldn’t harm that at all.

  2. Does scenario 1 above even need a keyword / symbol at all? Could that not just be added as a feature of the language for all case class arguments? I suppose there could be some ambiguity problems there, but that would be the case with a keyword too, right? Or are we suggesting that use of the keyword also adds additional restrictions on parameter lists at the def that authors wouldn’t have to deal with otherwise?

2 Likes
56

I am also unsure how widespread the applicability of this is.

On the one hand, I don’t have immediate use cases either in my code. But my code is far from the typical application.

On the other hand, I have a couple of observations that somewhat speak for the proposal, even if the arguments are circumstantial:

  • It’s similar to varargs and varargs certainly are useful.
  • A similar proposal was proposed and accepted for Python. And Python is a language with generally good APIs. On the other hand, people complain that the APIs of many Scala libraries are too convoluted. Fluent APIs and builder patterns have their place, but arguably people reach too quickly for them when a simpler, flatter API could have done the job as well. So this addition might help redress the balance.

Maybe people with more experience in relevant domains could weigh in on the question of applicability.

2 Likes
57

I can only speak for my case: we are building Rudder, a devops & compliance assessment tool, in Scala since 2010. It is the typical a-main-web-app-for-complex-business-logic-with-API-and-front-end of the java style (plus a ton of remote server management stuff, but that is done in c/rust/python/F# and is managed-OS dependant and irrelevant here).

So, the config of the app is trivial but there is tons (hundreds) of parameters depending of “waves at topologies, bases connection params, charge target, SSO, admin preferences, on what OS it is deployed, auto-clean-up habits, etc”.
Parameter are coming from config files (other kind of params are API-provided, but again, irrelevant here).

So we have an init sequence that looks like:

  • read params from the config file
  • translate to validated domain model of parameters (ie typed values/case classes after parsing/typing, logging problems for ops, giving default values if the property is not in the config file, etc)
  • then, with all these typed properties/case classes, we init services, ie we instanciate long running classes like the PostgresSQL connection manager, the user management service, etc

With the proposed feature, we could:

  • use unpackable case classes for parameters of our services, and in the init use and reuse the smaller case class to build these input,
  • still use direct arg calling for tests, where we want to vary just some args, or avoid the ceremony of building up a case classes

Is it a must have feature for us ? Not at all, we lived more than a decade without. And the use case is lass emphatic than the one provided by @lihaoyi were you want to provide a clean API for your users (here, the users are ourselves, we are tolerant and able to understand the balance).

1 Like
58

Solutions similar to introducing the unpack modifier introduce additional complexity. For example, from the Toolkit’s perspective, the user must understand the new unpack feature. He will see snippets using different ways to invoke the API method. There won’t be an easy way to see that a given parameter comes from an unpacked param in the definition (unless you know how unpack works). It’s also a new keyword and a language feature; it adds to the complexity by being a new element to consider - especially since it is planned as a new building block for library APIs. I highly recommend that we make sure that it does help simplify a lot of APIs before going forward.

I also have a question: What is the main advantage of unpack over passing the Config object? Passing the object seems sometimes to provide more structure to the invocation.

5 Likes
59

I agree with @szymon-rd that its a real burden to newcomers for every new thing we add. So I really like the idea by @julienrf to reuse the existing feature of Selectable + structural types and extend its applicability to cater for the understandable urge by @lihaoyi to scrap boilerplate.

Can we combine Selectable with default params?

type Config = 
  (name: String = "some app", year: Int, prompt: String = ">")

And allow selecting only a few when adapting the Selectable to param names?

def foo(year: Int, prompt: String) = ???

def bar(cfg: Config) = foo(cfg)  // cfg.year and cfg.prompt is selected because cfg is Selectable

bar(Config(year=1984)) // utilize default args given in the type def of the Selectable

With this you now can…

So I mean: the liberal rule is that if you pass a Selectable it can be unpacked if param names match unambiguously, without any extra syntax. Or is that too liberal @odersky ?

1 Like
60

The advantage is the same as many other language features Scala provides:

  1. What’s the advantage of lambdas over anonymous inner classes, or explicit named classes?
  2. Or the advantage of _.foo syntax over bar => bar.foo?
  3. Or named parameters foo(bar=qux) over builder pattern new FooBuilder().bar(qux).result()?

The main thing that unpack provides is that it lets the user at the callsite say what they want - pass foo = bar to some function qux() - without needing to figure out they need a QuxConfig() object from import com.mycompany.foo.QuxConfig and instantiate it. And without the boilerplate of defining mostly-the-same argument lists across all the methods which may share it, along with overloads taking the config object and code to pack and unpack the flat parameter lists into the config objects (I have linked to example code that does so earlier in this thread)

As many have said, this feature would not let you do anything you couldn’t do without it. On the other hand, if we look at the direction modern languages are going - not just Scala or Python, but Kotlin, Swift, and others - it’s definitely trending towards “call functions with named parameters” rather than “construct an object implementing the Command/Builder/Fluent-API pattern to configure the logic you want to call”. (e.g. Swift doesn’t have unpack/kwargs, but it has leading dot syntax which serves a similar purpose in removing boilerplate from method callsites, as an alternative to fluent builders and other patterns)

Scala has been how it is for 10-15 years now. It doesn’t need to change. OTOH, for much of those 10-15 years, Scala has had a reputation for being unfriendly to beginners: as a language, as libraries, as an ecosystem. If we want that to change, then questioning some of our long-held assumptions and taking notes from other languages which are known to be beginner-friendly does seem like a reasonable thing to do.

6 Likes
61

@lihaoyi Do you think reusing and extending Selectable as a marker trait for allowing case classes to be unpacked without extra syntax would fit your purposes and be possible? If it can be done then we might be able to have the cake and eat it (i.e. scrap boilerplate and without inventing new syntax).

case class Config(
  name: String = "some app", 
  year: Int, 
  prompt: String = ">"
) extends Selectable  // marks its ability to be selectively unpacked

def foo(year: Int, prompt: String) = ???

def bar(cfg: Config) = foo(cfg)  // partially apply selected fields of cfg

bar(Config(year=1984))  // normal call

bar(year = 1)  // cfg is packed with selected default args
62

I agree; that is certainly a value from this proposal.

Python has a very different context than Scala. They had the kwargs dict for a long time. It is somewhat of a standard approach at this point. It doesn’t call any constructors. It is just a syntactic sugar for passing a dict. In Scala, the unpack, however, will appear in a kind of void. Maybe if we had something similar to the kwargs-like dicts, we could work with that.

I know that it is similar to what some people here were proposing, but I think we could try to combine it by implementing these steps:

Named tuples

Named tuples could be a de facto type-safe substitute for dynamic dicts in Python. So let’s say we have a named tuple like this:

type Foo = (a: Int, b: String)

Unpacking case classes

To make it possible to use case classes like Config in this solution, we can introduce a parameterized type alias special-cased by the compiler - Unpack. Unpack would take a T <: Product and, when possible, return a named tuple with its elements.

case class Config(a: Int, b: String)

type Foo = Unpack[Config] // Foo = (a: Int, b: String)

That way, we don’t have to use the Selectable that I’d argue is meant to serve a bit different purpose (cc @prolativ)

Spreading named tuples in parameter lists

After one has obtained the named tuple, he would be able to spread it using a spread operator (like ..)

def foo(bar: Int, ..conf: Unpack[Config]) = ???

Spreading named tuples as values in the call site

To accommodate for easy transformation back to the Config object, we could spread the named tuple at the call site:

def foo(bar: Int, ..conf: Unpack[Config]) = 
 val conf = Config(..conf)
 ???

I think that this is an approach that solves the problem that Python solved with their PEP. It doesn’t call any constructors, and would serve just as a syntactic sugar to skip the unnecessary parentheses and sometimes to be able to pass the params of an unpacked Product without specifying them one by one.

2 Likes
63

Why is the … syntax even needed? Isn’t it enough that it is marked Unpack ?