Is the platform transitively closed?

This came up in gitter with no obvious resolution.

My question is, can a platform module have a runtime dependency on something that’s not in the platform? It the goal is predictability then it seems like the answer should be no in general. But it certainly would have to be ok for at least some dependencies, such as

  • scala-library, scala-reflect, macro-paradie
  • the JDK (which version? seems important)

But perhaps depending on other non-platform Scala libraries would be disallowed? If so then what about dependencies on Java libs like Netty or Asm? Or scala-js platform libs that depend on Javascript libraries?

I don’t have a solution to offer but it seems like something that needs to be discussed.

2 Likes

I think one should carefully control external dependencies of he platform. Something like a global whitelist of allowed dependencies could work. And I expect that that whitelist would include scala-library and scala-reflect.

1 Like

Any external run-time dependency would be de-facto part of the platform.

Whether a run-time dependency is included in the platform by how the platform is defined, or is included in the platform as a technical consequence of dependencies doesn’t seem to matter much. At the point a library is only an import away, one might as well call it as part of the platform.

It gets weird when the platform provides stuff that is not part of it.

That makes a dependency white-list effectively a set of privileged libraries that are part of the platform, but don’t have to follow its code of conduct and development guidelines, and at the same time, are part of the platform not by their own choice.

I have no value judgement about that, but although that may be necessary, it does seem a bit odd.

This is an important question, thanks for asking!

I think that the Scala Platform cannot be completely closed and the whitelist proposed by you and @odersky could work. I would accept scala-library, scala-reflect, and the lowest maintained JDK version supported by the latest Scala major version. As the SPP proposes, modules need to be cross-compiled with the latest two major Scala versions. As of now, this means 2.11. and 2.12.x. In this case, although 2.11.x is compatible with JDK6, JDK7 and JDK8, the latest version 2.12.x requires JDK8. Therefore, the Scala Platform needs to keep up with 2.12.x and only allow JDK8 libraries. This is unfortunate for 2.11 users that may want to run the Platform, but when these breaking changes happen in the compiler the Scala Platform has to honor them.

macro-paradise is going to stay for a while since Scala Meta is its official replacement but it’s not ready yet. I think it’s reasonable to whitelist it for the first Scala Platform version (which will be 0.1.x) and even more if Meta doesn’t stabilize before that; and then ask modules to use Scala Meta instead for the 1.0.0 release. We would do a similar migration path for scala-library when the split core-platform is materialized in 2.13.

If we have these whitelisted versions, it’s not necessary to have a tool to check collision of dependencies (different versions of the same artifact). Otherwise, the Scala Platform should have a tool that checks this and notifies mantainers.

I’m not sure how people feel about these rules, they may be a little bit strict. This strictness in my opinion is a feature of the Scala Platform rather than a disadvantage: we want our users to know that we honor stability. Bringing in uncontrolled external dependencies could cause severe problems to both users and module contributors/maintainers. Because of this, I’m not sure that @martijnhoekstra’s observation would lead to a good solution. Considering runtime dependencies part of the Platform not only is odd, but could turn out to be dangerous…

1 Like

As a quick note, I didn’t really make a proposal, just an observation: if a library has an external dependency, and if the library is made part of the platform, then dependency also is a de-facto part of the platform.

Sorry, I meant observation. I have just updated the text.

In Typelevel, we don’t enforce any rules on transitive dependencies. I personally don’t think the platform should be strict about this, except for requiring dependencies to be FLOSS. This guarantees that if a project becomes outdated, dormant, or otherwise untenable, it can be easily forked and republished.

How do you ensure that Typelevel libraries do not have conflicting dependencies? Imagine, for instance, that one transitively uses v0.3.1 of foo via an external dependency bar and other one depends directly on v0.4.0. Doesn’t this become a problem?

In the Scala Platform it would since it would be an artifact depending on concrete module versions. I’ve already proposed a way to solve this if people want the Platform to be open, but I’m just curious. This problem could become much bigger (and subtler) if “far” transitive dependencies collide.

We don’t, currently. But it’s a good point. With that in mind, I would argue that the proposed “whitelist” should exist, but module maintainers can freely add libraries as long as the versions are consistent.

@larsrh I agree that we should let the whitelist evolve naturally. I am bit hesitant to have this being managed as a “free for all” so to speak. The point is, external dependencies are important because they affect all your users. So, if we want to encourage inner-platform dependencies (in order to avoid duplications and inconsistencies) it becomes all the more important to control external dependencies, since they become a shared liability of many platform modules.

1 Like

I really think there needs to be a clearer description of the goals of the platform, in particular how it is consumed by users.

If the goal it to create a more extensive library so people don’t need to choose an http library, for instance, than the platform is really one logical entity and it should be consistent. That means there should be no external dependencies (in practice, it means the white list should be kept to a minimum).

Another question is whether there should be two libraries that solve the same problem. How many JSON libraries should there be in the platform? Any limit here is very tricky to impose, so naturally there will be many. How would they interoperate? Other libraries will make a decision on what JSON library they use, and now you’re suddenly having subsets of the platform that don’t interoperate.

2 Likes

I’ve discussed these issues here and here.

In short, your definition of the Scala Platform as a consistent logical entity is pretty appropriate. This is also why I think that being close with a whitelist is a benefit. I wouldn’t enjoy having problems with unmaintained external dependencies that cannot be taken up by anyone in the Community because of lack of time/resources.

Thanks for reminding me. You should ammend the platform proposal, it’s a very important point and most people won’t be up to date with all forum discussions.

1 Like

I think module maintainers have responsibility for third-party dependencies (except for Scala itself), even if they are on the whitelist. Forking is time-consuming, yes, but it’s not that difficult. I have forked numerous libraries in the process of maintaining libisabelle, mostly to make sure I can get a timely 2.12 release out the door. I wouldn’t want to see a situation where the platform can’t move to 2.13 because some external dependency becomes unmaintained.

Regarding runtime dependencies, there is the issue with libraries like https://github.com/typesafehub/scala-logging. This is a wrapper over slf4j, which is often important as a runtime dependency (and there are also different types of runtime dependencies as there are different logging implementations, including ones that go to logstash, i.e. https://github.com/logstash/logstash-logback-encoder

Also is this what we mean when we are talking about runtime dependencies?

Right, my concern is stability of the platform, which we can only guarantee for things in the platform. I think in practice it’s probably less of an issue for Java libs because you don’t have to maintain/release N versions, where N increments every 18 months or so. So I’m not too worried personally. I just think it’s something that needs to be documented: When is it ok for a platform module to link non-platform code? seems like a likely FAQ.