ENSIME: The Next Generation

[hedefalk]

@ckipp01

In this situation, what you’ll want is actually “Build Import”.

Cool, thanks. I’ve stayed away from that because of previous bloop behaviours. Will try, thx! Have follow-ups, but should move that not to hijack Sam’s thread…

[VlachJosef]

Hi Sam,

I’ve decided to test ensime-tng this morning and I must say I’m flabbergasted. It just works!

There was no problem to set it up following the README.md and two commands I need (the most) ensime-tng-jump-to-definition and ensime-tng-type-at-point are working without any issue.

I’m going to incorporate ensime-tng into my workflow, really great work there, thank you.

[fommil]

@VlachJosef

I’ve decided to test ensime-tng this morning and I must say I’m flabbergasted. It just works!

Awesome, glad you like it! I arbitrarily added a few people from hack-the-tower sessions who I knew would appreciate the invite

[vejeta]

@fommil

mmitting to contributing anything at this stage. Will open it up to users / testers later. I am still undecided if my repo will ever be public, but I’m sure public versions will pop up (i.e. be leaked) and I won’t discourage that. That’s how I imagine an LSP wrapper working.

If you are up to have an early new tester, count me up for it. I don´t know if I will have the skills to commit any PR, but surely I am up for it.

Thanks anyway for moving this forward.

[diesalbla]

@fommil

However, I’ve been seeing interest from private conversations. I’m starting to think that maybe the way to enjoy open source is via private invite-only groups, rather than in the public (which is quite toxic at the moment).

It may be a pity for anybody to feel shy of public light. However, open-source does not mean full transparency, even if they often concur. So, you may not need go for the full Github-style of published code, published history, published authors, forks, pull requests and issues, all open for anybody to read, fork, comment, and track, without restrain. Frankly, I am coming to doubt the sanity of this approach.

Privacy is good. Confidence (trust-based communication) is good.

So, for the particular case of Ensime V2, perhaps would any of the following middle stations suit?

1. To publish only the source code of specific releases, as tarballs, without version control metadata. No need to reveal the full list of contributors, or who edited what and when (who git the blame). If you worry that a particular association may embarrass the project, this can help.
2. To only publish a mirror of the main repository, with the full history of the few branches and releases, with all contributors. No need to reveal all branches or proposals that the main authors may have chosen to decline. No risk of slapping the proponents. This would let you keep internals communications private, openly speak your mind, without having to care for third parties.
3. To publish the main repository, with all its forks, and proposals, but keep all discussions of changes, and pull requests, private and confidential. Bit like the above.
4. To let Pull Requests, Issues, and its discussions, to be published for anyone to read, but restrict the writing on them. Alternatively, allow anyone to submit posts, but use some editorial review before anything is published. Thus you stop strangers from writing inflamatory comments, and allow others to withdraw messages that they may regret. In summary, Slow It Down.

Would you be willing to try any of these models for your project @fommil ?

[fommil]

@diesalbla

To publish only the source code of specific releases, as tarballs,

I’ve pretty much settled on this approach. But the exact timing of when tarballs will be made available is entirely up to my workload and/or whims.

[fommil]

thanks to the patient help of @smarter I now have Scala 3 working, although there are some inconsistencies in corner cases due to the way the api has evolved.

[fommil]

In the light of the article by @WojciechMazur / @julienrf Sustainable Scala | The Scala Programming Language I feel like I should point out that an explicit goal of mine was to reduce power consumption to extend laptop battery life, and it’s doing a pretty good job at this since there is no need for the semanticdb (30% cpu overhead) or a bloop server to calculate diagnostics (since diagnostics just come from the build tool when the user requests a build, not a secondary build just for diagnostics).

[tgodzik]

It’s awesome to hear that you are working on a more minimal and efficient solution, though I believe LSP/BSP is pretty lightweight and over pretty good battery life when working on projects.

I just wanted to add a bit of clarification:

• Bloop is not just a secondary build just for diagnostics. After the initial import you can compile run, test and debug your project within it and most likely in the future it will possible to publish packages with scala-cli. You can also use sbt via BSP, which means that only one tool will be used for compilation.

• semanticdb is used for such features as references, renames, implementations and much more. So yes it does cause about 30% more CPU usage (around 12% for Scala 3), but it’s used for a lot of features and we are doing a pretty good job of reusing it for anything new. So for users that need such features I don’t think it’s a huge price to pay. It’s also very important for beginners, since a lot of these features are expected by newcomers.

Again, glad to hear that you are working on your own solution, I think we can only benefit from that (more bugs reported in the compiler ).

Edit: I am talking about Metals here, forgot to mention, since I gather from what you wrote that you were aiming at it.

[fommil]

@tgodzik

more bugs reported in the compiler

I’m not reporting any more compiler bugs, although I have found quite a few in Scala 3. The team seems pretty swamped and I didn’t want to throw any more onto them.

I’d say we did a pretty good job of shaking out most of the Scala 2 bugs in ENSIME: The Original Series and had a very good handle on performance hacks (e.g. to avoid shapeless implicit search) and dealing with and testing compiler plugins that didn’t consider interactive compilers (that tooling was all latterly removed from the “community” build due to lack of interest).

@tgodzik

I gather from what you wrote that you were aiming at it.

Not aiming at anything. I’m intentionally not comparing ENSIME to Metals, as that is seen as toxic marketing by some. Most people assume that I’m using the semanticdb when they hear that there’s a compiler plugin, but my plugin is zero overhead. And others assume that I’m using bloop or scala-cli to get diagnostics, and I am not.

[tgodzik]

And others assume that I’m using bloop or scala-cli to get diagnostics, and I am not

I thought it’s pretty clear that you are trying to keep it as lightweight as possible so the fewer elements the better it is to control it.

Btw. I think people could use Bloop here if they wanted, since it will get info about any compiler plugins that a more traditional build tool gets. And then you could just use the CLI.

Not aiming at anything. I’m intentionally not comparing ENSIME to Metals, as that is seen as toxic marketing by some.

I think it’s fair to compare to Metals especially in terms of features or the actual experience you get, since it’s hard to gauge exactly what is it that the new ENSIME provides. As long as you don’t just bash it without reason Especially, if you point out any issue that you feel Metals suffers from that might help to improve some things.

FYI: I think it would be cool to at least see a gif demo if you don’t want to opensource it.

[fommil]

@tgodzik

. I think people could use Bloop here if they wanted, since it will get info about any compiler plugins that a more traditional build tool gets. And then you could just use the CLI.

In Emacs or Vim one typically just invoke the build tool to do the building, and the diagnostics are surfaced like in any other programming language via regexp matching from the output, including jump-to-error. Folk can use Bloop in addition to that if they really want but there’s no reason to do that; it’s an extra unnecessary component in the Emacs / Vim workflow. Especially if the build tool has no startup time as is the case with sbt client or warm maven.

@tgodzik

it’s hard to gauge exactly what is it that the new ENSIME provides.

@fommil

I’m limiting the features to:

• dot completion
• infer type
• import class
• jump to source

@tgodzik

if you don’t want to opensource it.

It is already “open source”, although I prefer to say “free software”. It is just not available for download from a public website, which has become an expectation of free software in recent years.

@tgodzik

I think it’s fair to compare to Metals

It’s good that you would feel that way, however I don’t think everybody feels that way so I’ll just steer away from making direct comparisons.

[fommil]

@tgodzik

it would be cool to at least see a gif demo

ok I just downloaded peek to do that and ended up recording a blank screen so I have no idea what I’m doing.

Here’s some input/output.

$ ensime complete src/main/scala.interactive/simple2.scala 106 src/main/scala.interactive/simplelib.scala
bar: String
foo: List[Boolean]
formatted(fmtstr: String)String
hashCode: ()Int
toString: ()String

[tgodzik]

@fommil

Folk can use Bloop in addition to that if they really want but there’s no reason to do that; it’s an extra unnecessary component in the Emacs / Vim workflow. Especially if the build tool has no startup time as is the case with sbt client or warm maven.

If you are working on a number of projects then keeping a warm sbt for each workspace can be problematic, which is where Bloop is most useful since you just keep one server. So in essence, it is more energy/CPU efficient than starting a build tool for each new workspace. You also don’t need to make sbt warm with Bloop, which can take some time.

It’s good that you would feel that way, however I don’t think everybody feels that way so I’ll just steer away from making direct comparisons.

You can send me that in private then I understand you have a different perspective on what you expect from tooling and it might be that Metals is a 180 degrees opposite of that.

It is already “open source”, although I prefer to say “free software”. It is just not available for download from a public website, which has become an expectation of free software in recent years.

It’s not the traditional approach and I didn’t mean bad anything by saying it’s not opensource, however there is a higher than usual barier of entry.

ok I just downloaded peek to do that and ended up recording a blank screen so I have no idea what I’m doing

Might some issue with a missing C library from what I remember (not 100% sure)

Btw. Please do report issues that you encountered, I try to fix them from time to time with a focus of the area:ide label. There is also a spree organized every 3 weeks to work on the compiler with the community, though we haven’t announced it recently since we are lacking resources for additional teams to get involved. If you are interested do let me know, though we have a break right now for the christmas season.

[AMatveev]

@tgodzik

expect from tooling and it might be that Metals is a 180 degrees opposite of that.

The ability to start and debug project from scratch quickly is an important use case at least for me.
Fortunately sbt can solve such task with sbt Reference Manual — Remote Caching
Unfortunately it is absolutely unclear when such ability is going to appear in bloop.

I have tried working with metal via sbt, it seems slow.

• it requires more than 8 seconds to finish outgoing compilation when sbt makes incremental compilation in less then a second.
• the initial start of metal requires some time for indexing.

It is not critical thing but ensime seems good alternative in that case. Unfortunately I don’t understand whether it will be ready for production someday. But it is toxic theme for discussion. :)))

[fommil]

@tgodzik

If you are working on a number of projects then keeping a warm sbt for each workspace can be problematic

If I’m working on multiple projects then the chances that they all share the same version of Java is minimal. I’d rather build everything according to the way the projects have been setup.

I understand that bloop and scala-cli work well for some people, but I just have no use-case for either of them. I work with projects that are setup by other people far more often than I start projects from scratch, and it’s just a lot safer for me to not stray from the beaten track, especially when sbt comes with so many plugins to make the devops painless. bloop also has lots of little corner cases that break in larger non-standard projects, such as the exact lifecycle of how resource files end up in the target directories, and whether the project is setup to use compile-to-jar or not. I can also customise sbt transient directories to use RAMdisk for speed boosts from a global plugin but have no control over that level of detail in bloop.

[fommil]

@AMatveev

I have tried working with metal via sbt, it seems slow.

• it requires more than 8 seconds to finish outgoing compilation when sbt makes incremental compilation in less then a second.
• the initial start of metal requires some time for indexing.

Oh boy, this is really turning into quite the Metals thread now

Without making a direct comparison to Metals, I can say that ensime

• doesn’t have a secondary compile, you just use your build tool
• starts up in 2 seconds and doesn’t do any indexing in that sense. There are trade-offs for this, the most significant one for me being that the “import type at point” feature does not support type aliases (that would require me to scalap index all class files) or def/val members (that would require me to ASM index all class files). Both of those indexes were created in ENSIME:The Original Series, and I don’t think they were worth it. I have the option to bring it back.

[tgodzik]

@AMatveev

it requires more than 8 seconds to finish outgoing compilation when sbt makes incremental compilation in less then a second.

Och, that is actually strange, it should be exactly the same compilation. When using sbt via Metals or via console it should not be any different. They share the compilation. There is probably a way to optimize it. Please do report it in Metals or sbt.

@fommil

doesn’t have a secondary compile, you just use your build tool

In the case @AMatveev is describing it’s actually the same build tool, so no secondary compile happens

@fommil

I understand that bloop and scala-cli work well for some people, but I just have no use-case for either of them…

I understand that, it’s really hard to map everything between sbt and Bloop, especially the plugins are usually not translatable. That’s why there is also an alternative of using sbt or Mill via BSP. In that case you will never need to wait for compilation after doing any changes, since the build tool will be notified by the editor beforehand.

@fommil

Oh boy, this is really turning into quite the Metals thread now

Sorry! I didn’t want to hijack the thread, just wanted to clarify some things. You are building your own tool that is suitable for your needs, so I am for sure not trying to convince you that any other approach is better (especially that for you for sure it isn’t). No more Metals talk!

[fommil]

I made a public release of ENSIME: The Next Generation at https://ensime.github.io/

I’ve been sitting on this since before xmas and it seems pretty stable to me.

[Jasper-M]

Who cares about stability when the stylesheet of your webpage is that cool.