Adding better-files to the Scala Platform

[pathikrit]

Will the library possibly be renamed? If we are doing a IO library, something that is very central to languages in general, it should probably be called scala.io and be in a scala.io package or something along those lines

Yes better-files is a terribly unfortunate name. It started its life as:

implicit class BetterFile(javaFile: java.io.File) {
   // utils !
}

I am not a fan of just being a very light wrapper around Java NIO, mainly because I think we should be caring for multiple platforms and hence stuff like scala-native shouldn't have any concepts of java-nio (so this library wouldn't be a nice design for a platform agnostic IO)

I suggest this:

1. A better designed, pure, monadic, side-effect free, purely-Scala IO library that is compatible with Scala native (and Scala JS?) belongs to scala.io._. No such thing currently exists.

2. better-files belongs to something like scala.io.unsafe._ or scala.io.JavaConverters._ package. The understanding here is this library is a simple impure wrapper around Java. Interacting with Java's IO stuff is an unfortunate truth that many developers have to deal with.

Alternatively, I am fine with keeping better-files out of the Scala Platform and we can wait for 1) to happen and people can use better-files as a non-platform library.

[lihaoyi]

I am not a fan of just being a very light wrapper around Java NIO, mainly because I think we should be caring for multiple platforms and hence stuff like scala-native shouldn't have any concepts of java-nio

There is no reason we can't implement a facade for java.nio in scala-native, or for that matter in Scala.js for all the synchronous APIs that exist (in-memory FS, node's sync FS, ...). That's what we've done with Java collections, for example, and the effort required is relatively trivial compared to designing a whole new IO library from scratch.

There are plenty of reasons to want something better than better files (betterer files?) but scala-native/scala-js compatibility is not one of them. I've reimplemented a large chunk of java.io.File in Scala.js in-memory to port some Scala-JVM code; took less than an hour to get quite good coverage, and I see no reason that java.nio would be different.

No such thing currently exists.

IMO this is a showstopper for the idea of writing a new IO library satisfying whatever monadic/async/portable/etc. criterion and putting it in the platform. The platform should be about stability and consensus, not experimentation and research. But I'm of the opinion that any candidate library should already exist and have a reasonably large following before being considered for inclusion in the "scala platform", whichever interpretation of that term you want to take.

That's not to say we can't go and write a whole new Scala IO library and explore; it's just that an add-to-platform-proposal is not the place to brainstorm ideas for that. "no library is satisfactory, including this one, is satisfactory to include in the platform" would also be a fine answer here, but let's not kid ourselves that there's a whole new idiomatic, good, safe, monadic, async, cross-platform IO library just around the corner...

[jvican]

It probably reflects the JVM platform to closely (matters for stuff like Scala.js and possibly scala-native).

To the best of my knowledge, there is no IO library that meets this requirement. It's good to have, but not a hard requirement that should be taken into consideration for the acceptance of this library -- it's hard to get right and requires lots of time and maintenance. We could figure it out later on .

async (i.e. providing an async implementation that returns Future's)

I think that async support is not necessary and can be easily done by third parties. If people do show interest in this feature, why don't we work it out in the incubation period or later versions of this module alongside @pathikrit?

better-files has already a lot of value by what it provides to JVM users. Asking for more is good, especially if it's followed by PRs and interesting discussions, but the mentioned features are not essential to solve the IO problem and can wait for later. What better-files buys us is a battle-tested and sane library for doing IO and interoperating with Java files (a bad evil for those in the JVM world). As data shows, it enjoys a good usage among Scala developers that look for an alternative of the IO utilities in the standard library.

Yes better-files is a terribly unfortunate name.

Have you thought about an alternative name?

That's not to say we can't go and write a whole new Scala IO library and explore; it's just that an add-to-platform-proposal is not the place to brainstorm ideas for that.

I totally agree.

We need to turn our hopes/requirements into actions that help us improve what we have right now. There's always room for improvement. I personally think that some better-files features could be improved and more typesafety could be added. I'm happy to share these insights in the future and follow them with a PR. But these opinions are not conceptually incompatible with this library proposal. The incubation period exists to achieve consensus via patches and it was added because it proved to work very well in Apache projects. I think it's the right solution for this kind of disagreements and invites us all to collaborate together.

[tpolecat]

I think the thin wrapper over NIO is the correct approach because it can be used as the basis for higher-level abstraction if others decide this is necessary. It also fills a clear need in a "mainstream scala" style, which I think is realistic for the platform. As much as I would love to see pure mondic IO, that's not where most of the community is right now, there is insufficient default support for this style of programming, and it's not the direction Martin wants to go.

So this seems like a great contribution to me.

[mdedetrich]

There is no reason we can't implement a facade for java.nio in scala-native, or for that matter in Scala.js for all the synchronous APIs that exist (in-memory FS, node's sync FS, ...). That's what we've done with Java collections, for example, and the effort required is relatively trivial compared to designing a whole new IO library from scratch.

Java NIO is a very low level library, and it doesn't really make sense on some platforms (Scala.js) and could be very different on other libraries (scala-native). If Scala.js would add suppor this library via platforms like node.js, I suspect it would be very hard to make a proper interface to Java NIO within Scala.js and for scala-native I suspect that the use case in scala-native (low level accesss to C/C++ libraries) would mean that implementing Java NIO would be less than ideal

The collections analogy isn't apt, its an apples to oranges comparison. Collections API is designed to be high level, IO access unless its a very high level API isn't (unless we deliberately design the scope of this SPP to be very small which gets to my earlier point)

There are plenty of reasons to want something better than better files (betterer files?) but scala-native/scala-js compatibility is not one of them. I've reimplemented a large chunk of java.io.File in Scala.js in-memory to port some Scala-JVM code; took less than an hour to get quite good coverage, and I see no reason that java.nio would be different.

java.io.File is deliberately a high level API, java.nio is not (at least in its entirety of its design, i.e. memory mapping of files and channels). It provides very low level access to filesystem operations, and also its idea of "async" (or non blocking) is quite different to what typical non blacking actions look like in Scala.

I for example do not see how channels would work with Scala.js, and memory mapped files isn't something thats also official supported with node.js

To be clear, if we wan't this SPP to just be very high level without async or any other features like that, than the better abstraction would actually be Java's original java.file and not java.nio.

[mdedetrich]

To the best of my knowledge, there is no IO library that meets this requirement. It's good to have, but not a hard requirement that should be taken into consideration for the acceptance of this library -- it's hard to get right and requires lots of time and maintenance. We could figure it out later on .

The thing is, if we hardcode our scala IO interface to java.nio then we have to deal with with the ramifications in the future which would make it really hard to revert. If you we are using java.nio, then we providing the expectation that it works similarly across all platforms, if not then it causes issues that Scala.js has issues with right now (regex is a very good example).

I would much rather prefer if the types that this IO library implements are Scala defined types (which on JVM can just wrap over java.nio) and on other platforms do whatever they need to do. This way you can actually document that "this behavior is platform specific" and that other behaviour is "platform independent".

Point is, and in general, I think we should only be using java types if we can say the behaviour is similar in all platforms, else we should create our own types.

[jvican]

I would much rather prefer if the types that this IO library implements are Scala defined types (which on JVM can just wrap over java.nio) and on other platforms do whatever they need to do. This way you can actually document that "this behavior is platform specific" and that other behaviour is "platform independant"

I agree, this would be very nice, and it's something that we could all work on during the incubation period. But to be honest, putting this as a requirement raises the barrier too much, not for this library but for others to come. Cross-platform is a feature that should be prioritized, but in cases where no other libraries have it there's nothing to do.

Could you open an issue in better-files and we discuss it there? I'm happy to help on this one, and perhaps @sjrd can also voice his opinion there. This effort could also help future modules to become cross-platform. It would be great to explore ways to bring together Java compatibility and cross-platform at the same time! But my point is that this requirement should not be deciding for its acceptance given that no other library provides this support now.

[sjrd]

Since I was summoned ...

IMO cross-platformness of libraries should be a significant concern of the Scala Platform, insofar as it is straightforward to design for any particular library.

For example, let's say we have a math library. In all likelihood, the API provided by this library can be implemented regardless of the platform. Better: there's a high probability that the entire source code of that library would cross-compile without effort. In that case, IMO the process should favor and encourage a cross-platform library. Even more, it should probably question why such a library would be proposed without cross-platform support.

On the other hand, I do not think we should demand cross-platformness from a library addressing a concern that is not "trivially" portable. I/O might just be the ultimate example of this problem. Yes, some things can be made sort-of cross-platform. But in general, any I/O library design will make trade-offs when supporting multiple platforms. In that case, I would avoid holding it against the library the fact that it is not cross-platform. If it complies with the SP requirements for at least one platform, and is not provided for other platforms, it's fine.

[mdedetrich]

On the other hand, I do not think we should demand cross-platformness from a library addressing a concern that is not "trivially" portable. I/O might just be the ultimate example of this problem. Yes, some things can be made sort-of cross-platform.

The contention I have is over having the direct usage and exposure of the java.nio type, since this is something that is quite tied to the JVM or how stuff is done the "Java" way (which is very ugly compared to other languages). I don't really disagree with anything else being mentioned, I just don't want to see a situation where java.nio gets implemented improperly/differently on different platforms

But in general, any I/O library design will make trade-offs when supporting multiple platforms. In that case, I would avoid holding it against the library the fact that it is not cross-platform. If it complies with the SP requirements for at least one platform, and is not provided for other platforms, it's fine.

I agree, this would be very nice, and it's something that we could all work on during the incubation period. But to be honest, putting this as a requirement raises the barrier too much, not for this library but for others to come. Cross-platform is a feature that should be prioritized, but in cases where no other libraries have it there's nothing to do.

If IO wasn't so central to using the language (in general) then I would agree, but since its pretty much a requirement on all platforms then I think the problem should be taken more seriously. I believe we should be clear about what is and what can't be supported.

But my point is that this requirement should not be deciding for its acceptance given that no other library provides this support now.

The thing, its very easy for library to create more issues (indrectly) than it solves if it sets a precedent that is very hard to change in the future. If everyone ends up coding against better-files (highly common since it is an IO library) then the library is forced to maintain a very long term compatibility.

And honestly, at least for the initial release, there doesn't need to be any support for the other libraries. The first release of the library can just support JVM by wrapping over the java.nio type and not exposing it (or any other types that are needed for that matter). For Scala.js, a library can provide opt in support for file operations (since its not native to Javascript, only to stuff like node.js) and scala-native can implement the file support as it sees fit.

Although having something is usually better than having nothing, if its something thats very central we shouldn't just include it because it doesn't exist yet, I don't believe that is very prudent.

Furthermore there are alternatives that exist, in fact quite a few. sbt-io is an example, and is probably (indirectly) one of the most used file implementations out there, considering how SBT is also used on many platforms and in many different situations

Could you open an issue in better-files and we discuss it there? I'm happy to help on this one, and perhaps @sjrd can also voice his opinion there. This effort could also help future modules to become cross-platform. It would be great to explore ways to bring together Java compatibility and cross-platform at the same time! But my point is that this requirement should not be deciding for its acceptance given that no other library provides this support now.

Sure thing, the issue is here (https://github.com/pathikrit/better-files/issues/97). Will also try to help out to get it moving

[Blaisorblade]

Re Java.NIO I'd distinguish two issues:
- java.nio APIs like java.nio.file.* which could be replicated like @lihaoyi says and like it has been done.
- low-level APIs (like memory-mapped I/O) which might be platform-specific (currently, work everywhere but on Scala.JS) — in the proposal, this seems to be newFileChannel. Maybe users should have a way to opt-in to those features, possibly standardized by the platform?

Finally: do you want the Scala Platform to use scala.*, not something like scalax.*?

[jvican]

6 posts were split to a new topic: Should the Scala Platform modules use the Scala namespace?

[jvican]

This library has been officially incubated in the Scala Platform, congratulations! This means the following:

• Library authors have complete access to Scala Platform's infrastructure.
  • Automatic release process.
  • Drone integration with caching and several customization features.
  • Official group ids are at maintainers' disposal. They can release under them if they desire so.
• Library authors will take part into future decisions regarding Scala Platform Process's rules.
• There will be a final vote to accept this proposal into the Scala Platform. This final vote will be done whenever library maintainers feel it's the right moment to end the incubation.

More information in the official Scala Platform Process.

Incubation period

The incubation period is the perfect moment for gathering developers around your library, creating a community, cleaning up APIs (note that changes in public APIs cause binary incompatibility and are done every year and a half), accepting PRs, creating well-documented issues that people can work on, et cetera.

Next steps:

• Library maintainers accept Scala Center's Code of Conduct and use it in their projects.
• Library maintainers decide the license they will use (they can stay with the same they have).
• Library maintainers decide whether they endorse C4 or not.
• Libraries have Gitter channels and pertinent CONTRIBUTION guidelines for people to submit paches/PRs!

Remember that taking decisions on these issues is extremely important for creating a community around the modules -- our end goal! You can also participate in the current open debates to abstain from recommending C4 / MPLv2 or changing the major cycle to one year instead of 18 months; your opinion is highly valuable, so please comment.

At the Scala Center, we're planning to run a series of hackathon in well-known Scala conferences to encourage people to hack on open issues of Scala Platform modules and join their community. Our goal is to boost the success of the Platform and help us get to a point where we can all benefit from a high-quality collection of modules. This is why having CONTRIBUTION guidelines, tips to developers and a getting started guide is important -- consider writing them.

Better files

In the case of this proposal, there have been several recommendations by the Scala Platform Committee:

1. Change name (@pathikrit has already agreed this is a good idea). Does someone have any idea for alternative names to Better files?
2. Clean up the API, especifically the extreme use of symbols. /cc Eugene Yokota and @Ichoran can probably be more specific in an issue about what they'd like to be polished.

Also, in this conversation we've been discussing new ideas that would be worth to be implemented in the Incubation period. @mdedetrich has already open an issue to discuss ways to make Better files cross-compatible with Scalajs. Please, if you're interested in this, get in touch with him!

@larsrh or @tpolecat: Do you know of something that would be interested in creating a pure API based on better-files?

Regarding infrastructure, I'll contact library maintainers to make the transition in the next days. Thank you very much for getting involved in this collaborative effort to improve the experience of Scala developers all around the world.

[pathikrit]

Change name (@pathikrit has already agreed this is a good idea). Does someone have any idea for alternative names to Better files?

IMO, better-file belongs scala.nio._ with the understanding that these are wrappers around Java's NIO. A future pure library should go in scala.io._?

Clean up the API, especifically the extreme use of symbols. /cc Eugene Yokota and @Ichoran can probably be more specific in an issue about what they'd like to be polished.

Please comment here: https://github.com/pathikrit/better-files/issues/102
Basically plan to move all the symbolic DSL to an optional import.

Next steps:

Please file issues here: https://github.com/pathikrit/better-files/issues/
I created a new tag related to SPP here.
Please tag any issues related to SPP with above tag.

[tpolecat]

I don't think platform modules should be in the scala namespace, which implies that they are part of the language core.

[pathikrit]

You are right. Well I am open to better name for better-files (I am terrible at naming)

[larsrh]

IMO, better-file belongs scala.nio._ with the understanding that these are wrappers around Java's NIO. A future pure library should go in scala.io._?

That could be its name though: "scala-nio", plain and simple.

[pathikrit]

I am fine with scala-nio. Some questions:

1. Do I rename just the git repo or the package too?
2. What would be the new package and group-id be? scala.nio.File? I imagine that is an acceptable inconvenience for users?
3. Since we would publish under new group-id after renaming, do I restart semvar (i.e. start from v1.0.0) - currently better-files is at 2.17.x?
4. What are the next steps?

[pagoda_5b]

@pathikrit
A package id of scala.nio will get us back to the problem of using a scala core namespace.

Is is mandatory that the group-id for platform libs should be changed?

[pagoda_5b]

You are right. Well I am open to better name for better-files (I am terrible at naming)

Ideas that won't have scala in the name:
smart-files
x-files

[soronpo]

alasca.files