Adding better-files to the Scala Platform

To the best of my knowledge, there is no IO library that meets this requirement. Itā€™s good to have, but not a hard requirement that should be taken into consideration for the acceptance of this library ā€“ itā€™s hard to get right and requires lots of time and maintenance. We could figure it out later on :smiley:.

I think that async support is not necessary and can be easily done by third parties. If people do show interest in this feature, why donā€™t we work it out in the incubation period or later versions of this module alongside @pathikrit?

better-files has already a lot of value by what it provides to JVM users. Asking for more is good, especially if itā€™s followed by PRs and interesting discussions, but the mentioned features are not essential to solve the IO problem and can wait for later. What better-files buys us is a battle-tested and sane library for doing IO and interoperating with Java files (a bad evil for those in the JVM world). As data shows, it enjoys a good usage among Scala developers that look for an alternative of the IO utilities in the standard library.

Have you thought about an alternative name?

I totally agree.

We need to turn our hopes/requirements into actions that help us improve what we have right now. Thereā€™s always room for improvement. I personally think that some better-files features could be improved and more typesafety could be added. Iā€™m happy to share these insights in the future and follow them with a PR. But these opinions are not conceptually incompatible with this library proposal. The incubation period exists to achieve consensus via patches and it was added because it proved to work very well in Apache projects. I think itā€™s the right solution for this kind of disagreements and invites us all to collaborate together.

I think the thin wrapper over NIO is the correct approach because it can be used as the basis for higher-level abstraction if others decide this is necessary. It also fills a clear need in a ā€œmainstream scalaā€ style, which I think is realistic for the platform. As much as I would love to see pure mondic IO, thatā€™s not where most of the community is right now, there is insufficient default support for this style of programming, and itā€™s not the direction Martin wants to go.

So this seems like a great contribution to me. :thumbsup:

9 Likes

Java NIO is a very low level library, and it doesnā€™t really make sense on some platforms (Scala.js) and could be very different on other libraries (scala-native). If Scala.js would add suppor this library via platforms like node.js, I suspect it would be very hard to make a proper interface to Java NIO within Scala.js and for scala-native I suspect that the use case in scala-native (low level accesss to C/C++ libraries) would mean that implementing Java NIO would be less than ideal

The collections analogy isnā€™t apt, its an apples to oranges comparison. Collections API is designed to be high level, IO access unless its a very high level API isnā€™t (unless we deliberately design the scope of this SPP to be very small which gets to my earlier point)

java.io.File is deliberately a high level API, java.nio is not (at least in its entirety of its design, i.e. memory mapping of files and channels). It provides very low level access to filesystem operations, and also its idea of ā€œasyncā€ (or non blocking) is quite different to what typical non blacking actions look like in Scala.

I for example do not see how channels would work with Scala.js, and memory mapped files isnā€™t something thats also official supported with node.js

To be clear, if we wanā€™t this SPP to just be very high level without async or any other features like that, than the better abstraction would actually be Javaā€™s original java.file and not java.nio.

1 Like

The thing is, if we hardcode our scala IO interface to java.nio then we have to deal with with the ramifications in the future which would make it really hard to revert. If you we are using java.nio, then we providing the expectation that it works similarly across all platforms, if not then it causes issues that Scala.js has issues with right now (regex is a very good example).

I would much rather prefer if the types that this IO library implements are Scala defined types (which on JVM can just wrap over java.nio) and on other platforms do whatever they need to do. This way you can actually document that ā€œthis behavior is platform specificā€ and that other behaviour is ā€œplatform independentā€.

Point is, and in general, I think we should only be using java types if we can say the behaviour is similar in all platforms, else we should create our own types.

1 Like

I agree, this would be very nice, and itā€™s something that we could all work on during the incubation period. But to be honest, putting this as a requirement raises the barrier too much, not for this library but for others to come. Cross-platform is a feature that should be prioritized, but in cases where no other libraries have it thereā€™s nothing to do.

Could you open an issue in better-files and we discuss it there? Iā€™m happy to help on this one, and perhaps @sjrd can also voice his opinion there. This effort could also help future modules to become cross-platform. It would be great to explore ways to bring together Java compatibility and cross-platform at the same time! But my point is that this requirement should not be deciding for its acceptance given that no other library provides this support now.

Since I was summoned ā€¦

IMO cross-platformness of libraries should be a significant concern of the Scala Platform, insofar as it is straightforward to design for any particular library.

For example, letā€™s say we have a math library. In all likelihood, the API provided by this library can be implemented regardless of the platform. Better: thereā€™s a high probability that the entire source code of that library would cross-compile without effort. In that case, IMO the process should favor and encourage a cross-platform library. Even more, it should probably question why such a library would be proposed without cross-platform support.

On the other hand, I do not think we should demand cross-platformness from a library addressing a concern that is not ā€œtriviallyā€ portable. I/O might just be the ultimate example of this problem. Yes, some things can be made sort-of cross-platform. But in general, any I/O library design will make trade-offs when supporting multiple platforms. In that case, I would avoid holding it against the library the fact that it is not cross-platform. If it complies with the SP requirements for at least one platform, and is not provided for other platforms, itā€™s fine.

3 Likes

The thing, its very easy for library to create more issues (indrectly) than it solves if it sets a precedent that is very hard to change in the future. If everyone ends up coding against better-files (highly common since it is an IO library) then the library is forced to maintain a very long term compatibility.

And honestly, at least for the initial release, there doesnā€™t need to be any support for the other libraries. The first release of the library can just support JVM by wrapping over the java.nio type and not exposing it (or any other types that are needed for that matter). For Scala.js, a library can provide opt in support for file operations (since its not native to Javascript, only to stuff like node.js) and scala-native can implement the file support as it sees fit.

Although having something is usually better than having nothing, if its something thats very central we shouldnā€™t just include it because it doesnā€™t exist yet, I donā€™t believe that is very prudent.

Furthermore there are alternatives that exist, in fact quite a few. sbt-io is an example, and is probably (indirectly) one of the most used file implementations out there, considering how SBT is also used on many platforms and in many different situations

Sure thing, the issue is here (Wrap over java.nio types instead of exposing them directly Ā· Issue #97 Ā· pathikrit/better-files Ā· GitHub). Will also try to help out to get it moving

Re Java.NIO Iā€™d distinguish two issues:

  • java.nio APIs like java.nio.file.* which could be replicated like @lihaoyi says and like it has been done.
  • low-level APIs (like memory-mapped I/O) which might be platform-specific (currently, work everywhere but on Scala.JS) ā€” in the proposal, this seems to be newFileChannel. Maybe users should have a way to opt-in to those features, possibly standardized by the platform?

Finally: do you want the Scala Platform to use scala.*, not something like scalax.*?

6 posts were split to a new topic: Should the Scala Platform modules use the Scala namespace?

This library has been officially incubated in the Scala Platform, congratulations! This means the following:

  • Library authors have complete access to Scala Platformā€™s infrastructure.
    • Automatic release process.
    • Drone integration with caching and several customization features.
    • Official group ids are at maintainersā€™ disposal. They can release under them if they desire so.
  • Library authors will take part into future decisions regarding Scala Platform Processā€™s rules.
  • There will be a final vote to accept this proposal into the Scala Platform. This final vote will be done whenever library maintainers feel itā€™s the right moment to end the incubation.

More information in the official Scala Platform Process.

Incubation period

The incubation period is the perfect moment for gathering developers around your library, creating a community, cleaning up APIs (note that changes in public APIs cause binary incompatibility and are done every year and a half), accepting PRs, creating well-documented issues that people can work on, et cetera.

Next steps:

  • Library maintainers accept Scala Centerā€™s Code of Conduct and use it in their projects.
  • Library maintainers decide the license they will use (they can stay with the same they have).
  • Library maintainers decide whether they endorse C4 or not.
  • Libraries have Gitter channels and pertinent CONTRIBUTION guidelines for people to submit paches/PRs!

Remember that taking decisions on these issues is extremely important for creating a community around the modules ā€“ our end goal! You can also participate in the current open debates to abstain from recommending C4 / MPLv2 or changing the major cycle to one year instead of 18 months; your opinion is highly valuable, so please comment.

At the Scala Center, weā€™re planning to run a series of hackathon in well-known Scala conferences to encourage people to hack on open issues of Scala Platform modules and join their community. Our goal is to boost the success of the Platform and help us get to a point where we can all benefit from a high-quality collection of modules. This is why having CONTRIBUTION guidelines, tips to developers and a getting started guide is important ā€“ consider writing them.

Better files

In the case of this proposal, there have been several recommendations by the Scala Platform Committee:

  1. Change name (@pathikrit has already agreed this is a good idea). Does someone have any idea for alternative names to Better files?
  2. Clean up the API, especifically the extreme use of symbols. /cc Eugene Yokota and @Ichoran can probably be more specific in an issue about what theyā€™d like to be polished.

Also, in this conversation weā€™ve been discussing new ideas that would be worth to be implemented in the Incubation period. @mdedetrich has already open an issue to discuss ways to make Better files cross-compatible with Scalajs. Please, if youā€™re interested in this, get in touch with him!

@larsrh or @tpolecat: Do you know of something that would be interested in creating a pure API based on better-files?

Regarding infrastructure, Iā€™ll contact library maintainers to make the transition in the next days. Thank you very much for getting involved in this collaborative effort to improve the experience of Scala developers all around the world.

4 Likes

IMO, better-file belongs scala.nio._ with the understanding that these are wrappers around Javaā€™s NIO. A future pure library should go in scala.io._?

Please comment here: Move all special symbols (`~`, `>>`) and Unix DSL to File.dsl package Ā· Issue #102 Ā· pathikrit/better-files Ā· GitHub
Basically plan to move all the symbolic DSL to an optional import.

Please file issues here: Issues Ā· pathikrit/better-files Ā· GitHub
I created a new tag related to SPP here.
Please tag any issues related to SPP with above tag.

I donā€™t think platform modules should be in the scala namespace, which implies that they are part of the language core.

2 Likes

You are right. Well I am open to better name for better-files (I am terrible at naming)

That could be its name though: ā€œscala-nioā€, plain and simple.

7 Likes

I am fine with scala-nio. Some questions:

  1. Do I rename just the git repo or the package too?
  2. What would be the new package and group-id be? scala.nio.File? I imagine that is an acceptable inconvenience for users?
  3. Since we would publish under new group-id after renaming, do I restart semvar (i.e. start from v1.0.0) - currently better-files is at 2.17.x?
  4. What are the next steps?

@pathikrit
A package id of scala.nio will get us back to the problem of using a scala core namespace.

Is is mandatory that the group-id for platform libs should be changed?

Ideas that wonā€™t have scala in the name:
smart-files
x-files

1 Like

alasca.files :wink:

Is there any update to the progress of this? Is better-files (renamed somehow) going to be the default file IO library in Scala, or are there different plans? I canā€™t see anything post Feb 2017.

1 Like

@kunalkapoor from what I can tell, the Scala Platform has been cancelled (see this 7 month old thread where I asked for an update and still have received no answer).