Proposal to retain private[this] in Scala 3

Language Design
1

This is a proposal to retain private[this] in Scala 3.

This proposal based on this discussion.

The PR to remove private[this] from Scala 3 is here. Some of the threads that lead to the current removal of private[this] from Scala 3 are here and here.

This proposal is not based on an ideological desire to reduce the number of changes between Scala 2 and 3. (Although we will happily accept support from this camp :slight_smile:).

This proposal is motivated by the fact that private[this] is a very useful design tool.

private[this] is a mechanism to designate a variable as a secret within an object instance. When we use private[this], future developers are prevented from using the variable in a non-secret context. There’s a mechanical connection between our use of private[this] and the future developers, ie. they will encounter a compile error if they attempt to use the secret variable outside of the object instance. This mechanical connection is what makes private[this] useful.

This mechanism has a valuable second-order effect: future programmers can observe that the original designer intended for the variable to be a secret. (This is an instance of what MIT’s Jimmy Koppel calls the Embedded Design Principle.)

Scala’s concept of companion objects makes private[this] even more useful than it might otherwise be:

For example, in Java a variable marked private can still be accessed by another instance of that type. afaik there’s no way to mark a variable as a secret to be accessed only within its object instance. Java is missing this tool.

If Scala 3 omits private[this] it will be missing this tool like Java. However, since Scala has companion objects, this omission seems more egregious because the gap between the set of potential accessors of a variable under private[this] vs private is larger due to the use of private variables in companion objects. Ie., in Java a private variable might be used in its own instance or another instance of its type. Scala extends this set with potential access in the companion object.

So – this proposal asks that Scala 3 retain the ability to mark a variable as usable only within an object instance and not other instances of its type or companion. This is a very useful thing.

Martin remarks on the reasoning to remove private[this]:

  1. it is syntactically an irregular case. A pair of brackets usually encloses a type, but this is a value.
  2. it leads to bike shedding: should I use private or private[this] ? One is shorter but the other might be more efficient.
  3. its effect over private is purely local and can be easily inferred

Responding to each point –

(1) this proposal does not consider syntax. Personally I think the syntax is satisfactory. The syntax may be technically irregular but it’s visibly similar to access qualifiers like private[MyObject], a feature which I believe is retained in Scala 3.

(2) this proposal agrees that any efficiency of private[this] vs private may be subject to bike shedding. However this proposal argues for the value of private[this] as a design tool. Its use from this standpoint is not at all bike shedding.

(3) discussions of efficiency and inference are orthogonal to the value of private[this] as a design tool. Inference is done at compile time, after the code has been written. So the inference has nothing to do with private[this] being a mechanism for the programmer to express the design intention of the variable being a secret within a class. That’s why dropping private[this] is a loss to the language.

In conclusion,

We have argued that private[this] is a valuable design tool, orthogonal to concerns of inference or efficiency. We hope that Scala 3 retains private[this].

As a philosophical note, I feel that private[this] is an exemplar of a kind of rich ability to embed one’s design intent within the code that is a hallmark of Scala. It helps make Scala special.

10 Likes
2

Another point Martin made in favor of removal is that it’s easy to visually verify how the private member is used.

By coincidence, I just touched a class of only 60 lines, removed two uses of private[this] as obsolete style, and moved the two methods to local defs, so I could better understand the entry points.

In this case, mutability of private vars was tricky to reason about, adding to the cognitive load.

So I think private[this] is useful for its stated purpose. It’s not easy to verify how private members are used, although it is possible to do so with pencil and paper.

However, I also think private should express this most restricted level of access. It is a more natural default. “I need to manipulate the internals of another instance” is unusual and deserves the special syntax private[C].

As a further caveat, however, this morning I was reviewing the tickets related to “can we please reach some kind of decision on access?”, the umbrella issue.

I noticed that some irregular cases of qualified access were fixed in Scala 3 by ignoring the qualification, rather than by improving correctness.

The irregularities and unspecified behaviors around qualified access really demand a review and some difficult design choices.

The related question is whether sealed/final should be default for classes. What is the unit of modularity? Or of comprehension?

The compilation unit is still under-leveraged in this respect. Currently, it means a file. But it seems natural to me to submit a directory of files to the compiler as a single compilation unit. to satisfy constraints on companion objects, sealedness, and name binding.

Maybe there are also alternative approaches to addressing the use cases for qualified access.

9 Likes
3

I really like this:

  1. Moving to express the semantics of private with private[C] fixes the syntactic irregularity by removing the special case of a value where we’d normally expect a type.

  2. The current behavior of private meaning private[C] is rather counter-intuitive to how the rest of the modifier work. In every other situation, adding a modifier to private expands access instead of restricting it further, so it’ll make access more semantically regular as well.

  3. Lastly, I’m not actually sure how often people actually rely on direct cross-object access to non-public fields.

    Quick tangent: I couldn’t think of a better word for this than “incestuous”, which seems a bit loaded. Is there a standard term for this - or at least something less awkward than “cross-object access to non-public fields”?

    In the code I’ve been exposed to it seems like most people treat private as if it already meant private[this], so it seems like a good default behavior.

4 Likes
4

In my mid-sized, mostly non-functional Scala 2 codebase, I have found all of Scala 2’s flavors of private to be highly useful:

  • private has been useful to refer to a private variable in a companion object or other instance of the same type
  • private[this] has been useful as stated in the proposal
  • private[MyFoo] has been useful to share a variable among a local group of modules without exposing it publicly

It is the argument of this proposal that these three flavors of private are each fundamentally useful and should be retained in Scala 3.

This proposal is syntax-agonistic and as such is open to proposals to change the syntax for any/all of these capabilities.

The goal of this proposal is for Scala 3 to retain all of these flavors of private as distinct capabilities.

As a related matter, protected with access qualifiers has been less useful to me:

  • protected[this] is apparently unsound. I never needed it.
  • protected[MyFoo] might be useful, but I never needed it.
5

protected[this] itself is sound, of course, it’s just the compiler allowing stuff under protected[this] to break variance is not, see: Infer private[this] by odersky · Pull Request #7411 · lampepfl/dotty · GitHub

2 Likes
6

Agreed, they’re all useful. What they all probably aren’t is equally common.

To preserve the effect of private[this], it may be useful to propose tweaks that would fix some of the warts that are helping to drive the impulse to remove it.

1 Like
7

Having private[this] does lead to bike shedding, in my experience. I have seen a lot of review comments on PRs that suggested changing private to private[this]. This is a real cost in terms of time lost and churn.

For any property we state explicitly in the language we have to decide whether the cost of doing so justifies the benefits. Possible costs are: language complexity, churn because of bikeshedding, cost for the compiler to verify the property. For private[this] verifying the property is easy but the other two costs are real. By contrast, the benefits are unclear.

Normally, we use types and other annotations to define a contract between two parties. I.e. a type defines a contract between definition and use sites, or a @tailrec annotation defines a contract between the programmer and the runtime, ensuring that a function will not blow up with a stackoverflow. What’s the contract ensured by private[this]? Who are the parties? What examples exist to show us that any of this matters? I.e. what would go wrong if the contract was violated? It would be good to see real examples taken from production code, since the costs of private[this] are also very real.

6 Likes
8

Couldn’t you make the same argument about the possibility of inferring val for local variables, so that programs could use local var only? It wouldn’t be a good idea because val documents more specific intents.

I agree with @som-snytt that the best tradeoff is to make private mean private[this] and use private[ThClass] for incestuous cross-instance usages, which I wager are very rare.

9 Likes
9

In this thread, and a few other related ones, things like this are said several times, without evidence. When I first read the threads on this topic a while back a few critical and somewhat common cases came to mind immediately, and I am shocked all the experienced developers here have not mentioned them yet.

The claims above need to be demonstrated. Perhaps try making the change to have private mean private[this] on the community build and see what breaks.

Yes, other instances accessing fiddly mutable state is dangerous – for example in concurrency control primatives, data aggregations, or buffer-like classes.

But there is another type of class where I would have to use private[MyClass] on almost every single field. Every class that has private inner structure that requires a custom equals or compare method. These use cases jumped out to me immediately: when do we desire access to another instance’s fields? When checking equivalence or comparing; we have no other choice.

Maybe, there can be an implicit allowance of accessing other instance members when inside equals, compare, and perhaps jvm serialization methods, even if they are private[this].

Now, in Scala case classes mean that overriding equals happens less often, as most classes without state that represent data tend to expose their fields publicly.
But I often need to hide the inner members from public view when writing data structures or high performance serialization / java interop. Case classes are ill-fit for several reasons here, and custom code is often required to write proper equals/hashCode/compare anyway, especially since Arrays are often present for internal structure (not necessarily state) that you would never expose publicly.

Anyhow, maybe I just do these tasks more than others, but before I accept the claim that this is rare I would want to see evidence.

I have also never personally witnessed a bug caused by the wrong instance accessing another instance’s private member by accident. This sort of thing tends to be easy to notice: generally accessing the other instance’s field must be qualified. Does anyone have any examples? Bugs are more likely with path-dependent types / nested classes and companion objects where private access is less obvious than between two instances of the same class. I tend to avoid too much nesting precisely because of the additional intimacy and lack of encapsulation it implies, especially if any mutable state is involved.

Various compromises seem possible and some could reduce bike shedding on whether something should be private[this] or private. Perhaps private[this] is restricted to disallowing access from the companion or nested classes but allows access from other instances? IMO this prevents the major abuses and leaves the useful and easy to audit one, and the bike shedding will be reduced because now in most cases its about hiding something from the companion (or the companion hiding something from instances). Or the inverse – by default a private field is not visible to companions or nested classes but is visible to other instances of the same class.

EDIT: IMO taking inventory of the community build and looking at what breaks if private suddenly meant what private[this] does would help decision making on the topic. Each breakage could be categorized:

  • other instance access (general)
  • other instance access (equals)
  • other instance access (compare)
  • other instance access (jvm serialization)
  • nested class access
  • companion accessing instance
  • instance accessing companion
  • other
3 Likes
11

This is probably going to be our best bet for getting actual data on this.

At least for the reviews I’ve been part of, I tend to see a pattern in the bikeshedding: it comes in waves as a dev first realizes that private didn’t mean what they thought it did, then peaks and wanes as they realize that almost nobody is going to go to bother annotating all their private accessors private[this]. They get tired of being “that guy” and give up on it, and everyone just sort of treats it like private means private[this].

I’ve noticed a similar behavior when a dev first realizes that scala.Seq is the one collection with default visibility that isn’t immutable.

3 Likes
12

Doesn’t even have to be a custom equals/compare. All case classes with private fields will also break immediately:

case class Plus(private val a: Int, private val b: Int) {
  val computed = a + b
}

There’s probably a couple of those in every library, so not much of the community build will survive if any!
I think the only hope for this change is to either make private forever mean private[C] in case classes (private[this] is already disallowed for case class fields), or have a transitionary period with private allowing cross-instance uses with a deprecation warning and a rewrite.

1 Like
13

My personal preference is that we should use private to mean “private to this instance” and private[MyClassName] to mean “private to this instance and its companion”.

I do not want to have the private/private[this] distinction inferred by the compiler. The whole point of private isn’t as some crutch for a not-smart-enough compiler; the point of private is as guardrails for overworked developers to ensure that they do not accidentally do the wrong thing.

“this member should be private to this instance” is very valuable information for any developer reading the code, and is a different and stricter guardrail than “this member should be private to this all instances of this class and their companion”. Both are valuable to have, but empirically “this member should be private to this instance” is what I want the vast majority of the time, so it should be given the shorter name

11 Likes
14

And how would a nested class, C.D, gain the visibility it now has into C? This affects more than companions and other instances of the same class.

16

There is a typical situation when neither ‘private’ nor ‘protected’ fits well.

abstract class A

open class B extend A 
// user class
class C extend B

There are no ways to declare methods in abstract class which are visible to open class and which are not visible to user class.

So the only way to make saftey abstraction with open class is to use aggregation.

abstract class A

open class B{
  private val a:A=???
}
// user class
class C extend B

It is not always comfortable.

17

Li writes,

My personal preference is that we should use private to mean “private to this instance” and private[MyClassName] to mean “private to this instance and its companion”.

This is a good suggestion. I prefer this now, too.

If I understood correctly, Li’s observation is that Scala 2’s private is equivalent to Scala 2’s private[MyClassName], where MyClassName is the class in which the variable is defined. So we are presented with a convenient opportunity, as follows:

Amended, specific proposal

  1. The syntax private[this] is dropped in Scala 3
  2. Scala 2’s private[this] becomes private in Scala 3
  3. Scala 2’s private[C] remains the same in Scala 3. i.e. the rules determining which C may be chosen are the same in Scala 2 as in Scala 3
  4. The effect of Scala 2’s private is now achievable with private[MyClassName] in Scala 3, where MyClassName is the class in which the variable is defined.

In this way Scala 3 is able to match all of Scala 2’s flavors of private access. But, Scala 3 gains the benefits of dropping the irregular private[this] syntax and private now meaning “private to this instance” (which seems to be what developers expect from the word “private”).

This amended proposal seems compatible with tooling to automatically upgrade Scala 2 code to Scala 3. The tool would simply rename private -> private[MyClassName] and private[this] -> private.

4 Likes
18

The previous thread: Change `private` to mean `private[this]`?

1 Like
19

Second this, the default private should honestly be private[this] and if you want other class instances to access the value than something like private[ClassName] makes perfect sense

2 Likes
20

Not to miss another feature of private[this] that I, particularly, rely on quite often: byte-code optimization for using field-level access rather than via a “get” method.

21

One could easily cure the syntactic anomaly by requiring private[this.type].

22

The cargo cult tells me that HotSpot always inlines accessor methods, but https://github.com/scala/scala/pull/8286 means that even private members will get to enjoy that particular optimization soon.