Question about building scala and sbt proper

Hi again, after having done som digging, let me rephrase my question, after a bit of context:

I am trying to upgrade Debians Scala and sbt packages from 2.11.12. The debian process is new to me and strict and complex. I respect Debians political point of view, but it makes things a bit difficult to start with. A Deian package must be built from scatch on debian servers. And I only have scala-2.11 and sbt-ivy available.

Hence the question about correct version of jdk, sbt and scala to use when building different versions moving up to 2.12.8.

I know of the compatability matrix. But there are a couple of issues with it.
1- I tried using default scala on ubuntu/debian, which used 2.11.12 and jdk 10/11, to run a scala script and it crashed. when I downgraded to jdk 8 it ran without problems.
2- Github page for scala, states that 2.11 was targeted at 1.6 and 2.12 at java 8.

Since a jdk has library changes, I found it safest and more stable to run using jdk8. jdk 8 will still be around in the wild for a good many years. and its still a bit more work for scala 2.13 to be complete and scala to be completely jdk-11 compatible.

So my first question is: is this a reasonable basis?

My second question is about building scala. I am using scala-sources-2.12.8.tar.gz (first try was with git clone)

  • is it enough to just use “dist/mkPack” to build a complete scala release for a debian package or must I go through the bootstrapping procedure and run the stability tests and all, as well
    • also does it contain all that a release must contain, or are there other repos that needs to be compiled as well?
  • since I dont have sbt available, is there a way to compile sbt without sbt, using only scala 2.11?
  • is it correct that I actually only need sbt installed on the build system to compile scala, or must it contain a scala installation as well ro be able to run sbt (before sbt downloads and compiles scala 2.12.7)
  • can I install sbt 1.2.8 on the build system to compile sbt and scala or should I use 0.13 (again I have read documents that state that certain versions must be used to build sbt at different version)

Regards

Thomas

Yes, JDK 8 is the safe and stable choice here.

I think targeting 2.12 is reasonable. Whatever you do for 2.12 should work for 2.13 as well, once 2.13 is finalized. (You, or someone, could think later about whether your efforts could be backported to 2.11.)

I don’t see any need for you to bootstrap or run the stability test. Any merged scala/scala PR has already been through that, it’s part of testing/QA rather than part of the build proper.

does it contain all that a release must contain

no, you also need https://github.com/scala/scala-dist

since I dont have sbt available, is there a way to compile sbt without sbt, using only scala 2.11?

I don’t know, perhaps @eed3si9n could address this

can I install sbt 1.2.8 on the build system to compile sbt and scala or should I use 0.13

I don’t see any way sbt 0.13 would enter into this. Both Scala 2.12 and sbt 1 are built using sbt 1.

Hope this helps.

Some history of past efforts to add Scala to Debian are at https://github.com/scala/bug/issues/9299 and https://github.com/scala/bug/issues/10172 and https://github.com/scala/scala-lang/issues/295; of these, 9299 is the still-active ticket.

Not at the moment.
It would actually be an interesting contribution if someone wants to generate a shell script that requires only scalac to build sbt.

Scala used to be built with ant, so it was probably still possible for these distros to build Scala from source by only depending on the Ant dependency.

Wasn’t 2.12 the first version of Scala that was built with sbt and not ant? Seems like a distro would need to go back to 2.12.0 and try to build with Ant? But even 2.12.0 probably depends on a 2.12 milestone, and not 2.11.

Seems like it would be easier to just start from a jar for sbt-launcher or sbt-extras to build Scala 2.12 from scratch.

In all 2.12.0-Mx builds, both sbt and Ant buildfiles exists. The Ant buildfile was carried over from 2.11. I dont know how usable the later milestone Ant build files are, cause the 2.12.0 build does not seem to have an Ant buildfile. Anyone?

(One is not allowed to use externally downloaded tools/apps/libs/jars, to compile a package in Debian. All debian packages must be built with only what is already available in the current debian repo (main) and the source package alone. I.e it must be able to bootsrtap it self. 30 years ago a few people handwrote the first version of the tools and infrastructure, which has evolved into what is today. Those were the simpler days… :slight_smile:

Bootstrapping Scala is basically impossible. You will not be able to get around the fact that to compile Scala 2.12.0, you will need Scala 2.12.0-RC2, to compile Scala 2.12.0-RC2 you will need Scala 2.12.0-RC1-1e81a09, to compile Scala 2.12.0-RC1-1e81a09 you will need Scala 2.12.0-RC1-ceaf419, you can keep going back this way in theory until you get to Scala 1 from 15 years ago, which is written in PiCo, a language which is basically Java-with-pattern-matching and whose source code was never released, so at that point you’ll need to write your own compiler.

I don’t know how Scala 2.11.8 got in Debian, but it’s impossible that it got it in through a full bootstrapping process, they must have cheated, so you’ll have to cheat again to upgrade it.

Note that this isn’t unique to Scala at all, to compile the current version of Rust, you’ll need the previous version of Rust, until you get to the first versions of Rust which were written in OCaml, but to compile OCaml, you’ll need the previous version of Ocaml … Someone did break the cycle through heroic effort by writing an alternative Rust compiler in C++: https://github.com/thepowersgang/mrustc, but I’m not aware of any Scala compiler not written in Scala.

I thought the bootstrap process of scala was compiled with javac…

Well… since its already there, I can use it to build later version :slight_smile:

I was thinking of trying that, I found some sh compile command in version 0.4.9 of sbt, so I thought I could look at that and the current build file to see if its possible. The sh command used javac…

I see now that you used the word “generate”, did you mean have some sbt code “record” the build process and the actual commands performed and create a script from it?

You can maybe use it to build Scala 2.12.0-M1, which you can then use to build -M2, then -M3 then -M3-dc9effe, then -M4, … It’s going to take a while to get to 2.12.8.

Yes, that seems right, but once you have 2.12.0 you can skip to 2.12.8.

Since I use sbt, I wasn’t aware before this that Scala 2.11 was available in Debian. Having Scala 2.11 available seems like a toe in here. There won’t be a reason to go back to 2004.

Getting Scala on Debian is a gateway to a lot of distributions, so this has a lot of value for furthering Scala. Granted, the bootstrap-from-source requirement is onerous. Scala language enthusiasts won’t be interested in. That’s what the project is good at, though.

It’s too bad sbt was never packaged in Debian. That’s the bigger hurdle.

Should this discussion be moved to a GitHub issue(s) or the sbt Discourse?

Just an hypothetical question, would it be possible to write a simpler scala compiler / lib in e.g. java, that can be used to bootstrap the scala compiler process? that would be interesting to many people, also for learning about compilers and the as a scala language tutorial. (I would definately do that if it meant learning more about compilers and the inner workings of scala)

Another thought, would it be possible to use the disasembler on the class files, to get pure java source to be able to compile at least a basic scala compiler environment? or was that too naive? :slight_smile:

Yes, but it would take a huge amount of work (if the goal is to make a compiler that is only usable for bootstrapping you could cut some corners, but this is probably still one year of work).

You’d probably have to do some substantial changes to the decompiled code to get it to work, so again a lot of work.

Note that when rust is bootstrapped on a new architecture in Debian, they downloaded a binary of the reference compiler and use that to build it, so it’s clearly possible to bend the rules: https://salsa.debian.org/rust-team/rust/blob/debian/sid/debian/README.source#L69

Interresting, I suspect thats for one or two reasons. Either the maintainer team did not notice the infraction (which is what happened with the sbt package thats in Debian unstable) or the maintainer team consisted of other people which might have judged it slightly different than the java team. Another possibility is that for ports such as that, they decided to allow it, for some reason. Perhaps otherwise it would never happen or something… Pure speculation though…

Good to know which options are realistic and which are not :slight_smile:

I know getting Sbt into Debian is the better choice, but I am exploring alternatives so…

One thought that came to mind was to perhaps use the ant build files from v2.12.0-M5 to build the scala source up to 2.12.8? Would that require much change of the build.xml files and support files? Or put another way, did much change in the way scala is built from M5 and up to 2.12.8?

(I have been looking into alternatives for a couple of weeks, and I keep running into the chicken and egg problem of scala and sbt all the time. So just trying to figure out if there is some solution I can work with in the mean time…)

No clue, I don’t think anyone remembers how the ant-based build works at this point.

Also, even if you manage to do this, consider that to compile scala 2.12.8, you need all of its dependencies, including https://github.com/scala/scala-asm and https://github.com/scala/scala-xml (not 100% sure if this is one is actually required for the build ?) which both have an sbt-based build.

The structure of the sbt build hasn’t deviated much from the ant build. It shouldn’t be that hard to make the old scripts work again. But as long as you only need it to get sbt out of the way I wouldn’t bother. Building the compiler and standard library is really quite simple. You could do it with a few lines of code in a shell script. The sbt build is so big (and the old ant build was even bigger) because it does many things you don’t want or need for building a simple bootstrapping compiler (dependency management, incremental builds, packaging, source generation, testing, scaladoc/REPL/etc.). It’s probably the same for sbt.

scala-asm and scala-xml shouldn’t pose any problems. scala-asm is a fork of asm, written in Java and built with ant. If you can build asm, you should be able to build scala-asm. scala-xml is still needed to compile 2.12 but it’s just another source folder that you have to compile to a JAR, same as scala-library, scala-reflect and scala-compiler.

Of course, none of this matters if you don’t have the right Scala binaries to start the bootstrapping process.

1 Like

Excuse my total ignorance on this, but I’m confused what your requirements are. Certainly you need some kind of compiler to get any kind of binaries regardless of the language? So what exactly does it mean to bootstrap?

Would it be legitimate if could you decompile the compiler to Java to get compiler source code you could compile with a Java compiler? Or simply claim that Java byte code is a kind of source code?