Why the current Export is problematic for Import

[benhutchison]

In a previous thread Idiomatic Imports, I raised the problem that we lack a language-level mechanism to factor out repeated structure in the import section of Scala code.

I am foremost a developer of end-user applications in Scala. Building on top of ever richer and more complex library ecosystems, my experience is that managing the size and complexity of import clauses is becoming increasingly onerous. Imports are bloated with repetitive boilerplate that is highly correlated across source files in my apps.

A clear sign that imports are boilerplate is that the leading Scala IDE Intellij hides imports by default. The IDE assumes there’s little worth seeing in them, and it’s right!

Most of us write Scala not because our employer mandates it, but as an active choice to select a language that is concise, expressive, and equipped with good tools to factor out common structure. But in the import section, the available tools are falling short.

In the previous thread, my read was that Scala 3’s export feature is proposed as the tool for collating groups of imports together into higher-level units. Currently export has some significant limitations around wildcard exporting packages, and some bugs, but no “deal-breakers”, nothing that can’t be worked around temporarily.

So I went away and experimented and tried to make export work. By “work” I mean, in an application codebase, the baseline imports are brought in with just a handful of imports of exporter objects, which themselves are full of exports of the underlying libraries.

Example exporter for the Coulomb library:

object CoulombSpire:
  export coulomb.{/, `*`, ^}
  export coulomb.syntax.{withUnit, withDeltaUnit}

  object allalgebra extends algebra.instances.AllInstances
  export allalgebra.{*, given}

  export coulomb.ops.algebra.spire.all.{*, given}
  export coulomb.policy.spire.standard.given

  export coulomb.units.si.{*, given}
  export coulomb.units.time.{Second as _, ctx_unit_Second as _, *, given}
  export coulomb.units.si.prefixes.{*, given}
  export coulomb.units.javatime.{*, given}
  export coulomb.units.javatime.conversions.all.{*, given}
  export coulomb.conversion.{UnitConversion, TruncatingUnitConversion}

  //workaround https://github.com/lampepfl/dotty/issues/17151
  type Quantity[V, U] = coulomb.Quantity[V, U]
  type DeltaQuantity[V, B, U] = coulomb.DeltaQuantity[V, B, U]

Example application source file:

import CoulombSpire.{*, given}

BigDecimal("3.14").withUnit[Meter]

And it basically works. But I see a looming problem with how this scales. Poorly.

All those types and symbols that are being exported are generating large quantities of forwarding bytecode in CoulombSpire (I’m seeing about 500kb per class in this example).

And what makes this worse, is we’ve only captured one particular import configuration. Coulomb has other combinations of imports for different situations, that would need to be collated into a different exporter, for more bytecode. And once we add in other libraries, Http4s for example, we’ll have more exporters for them, and the beginning of a combinatorial explosion in bytecode.

The Scala compiler is extremely adept at weaving bytecode. But the amount it emits is already problematic for runtime memory usage. And using export to do import will only push things further in the wrong direction.

An open Spire issue illustrates the practical impact of bytecode bloat. A user reported that when their JVM first hits the following line, the heap usage jumps by 450Mb (!), and intermittently trips an OutOfMemory error:

 def identity[N:Fractional](a:N): N = a

The proposed explanation is simply that this line triggers the loading of a whole lot of Spire machinery & bytecode associated with the Fractional typeclass & friends

So what’s an alternative to the current export behavior? I’ve considered a bunch of approaches including project/SBT-level compiler directives, and some def-like syntax to declare aliases for groups of imports.

But I’ve come back to favoring @rssh earlier suggestion of a transparent export mode:

@rssh

How about a variant, to reintroduce non-proxy exports as ‘transparent export’?
I.e.

package myLayer
object exports {
    transparent export myDependency1.*
    transparent export myDependency2.*
}    

and then use

import myLayer.exports.*

Essentially, the transparent modifier would create alias exports that have no runtime forwarding. They are purely a compile-time concept that changes what the compiler looks at when compiling. This solves the bytecode bloat problem and is also probably the most incremental enhancement from Scala today.

[lorandszakacs]

I very much echo this write-up. I have hit the problem of bytecode explosion when trying to use export to manage imports. And myself support a simple compile-time-only redirection of imports as a workeable solution.

[Sporarum]

I don’t know if this would work, but what about

inline export ...

Since an export is a def forwarder, an inline export would be an inline def forwarder

If I’m not mistaken, it solves this issue, and has the benefit of being more general

Edit: I realized after it wasn’t clear: This is not part of the language yet, it was a proposal

[hypatian]

Could this cause any issues around access level of symbols? (i.e. if package X exports something from package Y which accessible to X, and then package W imports X but doesn’t have access to Y.)

That seems like a place where actual forwarding would work but inline might not (although IMO it should).

It’s something that could be dealt with, even if it were a problem.

[smarter]

The current logic behind inline def already generates synthetic accessors to work around accessibility issues (there’s also an in-progress SIP about making this more explicit via annotations: Add @binaryAPI by nicolasstucki · Pull Request #16992 · lampepfl/dotty · GitHub)

[benhutchison]

I do like the name inline here, it’s probably a better name than transparent (although mostly I just want this capability, whatever it’s called).

I was also thinking about the current limitation on export, whereby packages cannot be wildcard exported. When trying to apply export in practice, this proves quite a nuisance and impedes using export to tidy up sprawling imports.

Would a compile-time export help resolve this limitation?

There are 3 parties to the problem:

• A ‘exportee’ package whose symbols are being exported
• An compile-time ‘exporter’, who has created an alias to the exportee’s symbols
• A ‘client’ who has the exporter imported or otherwise in scope, and sees the exportee through the exports.

Compile-time export would apply at the time code is compiled, just as imports do today. I think the compiler just needs to mark the client as depending on the exportee via the export. Any time the exportee changes, the client ought to be recompiled. During compilation of the client, symbols get resolved and changes will be picked up. But there’s no need to keep a bunch of forwarder defs in the exporter synchronized with changes in the exportee package. That seems to make the problem easier, but Im not sure…?