I had not anticipated that. In order to address that, it seems we'll need to downgrade our dependency on scala-library.jar in the published scalajs-library (and our other artifacts). In theory we should downgrade all the way to 2.12.2 and 2.13.0, which are the earliest versions we still support. That's going to complicate our build process, because we want to use the most recent versions of the compiler to build scalajs-library, since they produce better code.

Unless we use this opportunity to cut ties with old (> 1 year old) Scala patch versions? I'm not sure what is best here.

@gzm0 Opinions?

to support backwards-only binary compatibility (SIP-51),
[error] the Scala 2.13 compiler cannot be older than scala-library on the
[error] dependency classpath.

I do not quite understand this part: If the library is backwards compatible, then why do we need a newer compiler? Shouldn't it be the other way around? A newer Scala compiler requires a newer library (I guess because it relies on symbols in it).

At least this is how I understood it initially, am I completely off?

That was the original plan. It was a late change to enforce compiler version = library version. The issue is that the compiler is compiled with full optimizations that allow it to inline from the library. If there is a newer library, the compiler can break. Moreover, the existence of macros and scala-reflect, also compiled with full inlining, means that the classpath of the compiler and of the compiled application must have the same reflect and library versions.

That was the original plan. It was a late change to enforce compiler version = library version

I see :-/ And I assume impact on the downstream ecosystem wasn't fully re-assesed :(

The issue is that the compiler is compiled with full optimizations that allow it to inline from the library. If there is a newer library, the compiler can break.

This I do not quite understand: If an older compiler inclined from an older library, how does providing a newer, backwards compatible library break anything?

Moreover, the existence of macros and scala-reflect, also compiled with full inlining, means that the classpath of the compiler and of the compiled application must have the same reflect and library versions.

Similar thing here: I do not understand how I lining changes the picture :-/

Also: What in these constraints makes it that they do not apply to Scala.js compilation? Because otherwise we'd have to ship all Scala library version for all SJS versions which we deemed unacceptable IIRC.

I'm doing some digging on

• Add support for SIP-51 (unfreezing the Scala library) sbt/sbt#7480

And found this (partial) on the commit that introduced this check

When expanding a macro compiled against a new Scala library, the
runtime classpath of the compiler should not contain an older library.
Otherwise a NoSuchMethodException can occur

sbt/sbt@4c74358

It feels like the comment / commit message is arguing for the other direction of the inequality sign: the scalalib must not be older 🤔

This I do not quite understand: If an older compiler inclined from an older library, how does providing a newer, backwards compatible library break anything?

Because the public ABI is backward compatible, but inlining across the artifact boundary means that we also rely on the private ABI. The latter is not checked for backward compatibility.

The same issue affects authors of compiler plugins. A compiler plugin for 2.13.N needs to be built with 2.13.N, as it was always the case. But if the plugin has a dependency on some library that depends on a newer 2.13.(N M) scala-library, dependency resolution will pull in that newer scala-library jar and sbt will fail the build.

So compiler plugin authors cannot just upgrade their dependencies, if they want to release a new version of a plugin for an older Scala version. They need to make sure none of their dependencies pulls in a newer scala-library. sbt/sbt#7480 (comment)

It feels like the comment / commit message is arguing for the other direction of the inequality sign: the scalalib must not be older 🤔

The scala lib of the project cannot be older than what any of the dependencies needs; that's the same as with other libraries. The difference is that we cannot upgrade the scala-library transparently, like the build tool does for other libraries. The reason is inlining: a 2.13.13 scala-compiler requires exactly the 2.13.13 scala-library on the runtime classpath.

We were considering upgrading the scala-library transparently only on the compile time classpath (assume the compiler is running as java -cp <runtime classpath> scala.tools.nsc.Main -cp <compile time classpath>), but that doesn't work due to macros.

A macro compiled with 2.13.14 can reference a new method added to scala-library.jar in that release. When using the macro on 2.13.13, it is loaded dynamically into the jvm running the 2.13.13 compiler, with the 2.13.13 standard library on the runtime classpath. So the method is not there when the macro is invoked.

We considered transparently upgrading the scalaVersion altogehter, but then decided that it's better if the user does so manually. It's not ideal if the user writes scalaVersion = 2.13.13 but then the project actually uses a newer compiler.

So I guess the question is this: how important is it for new library / compiler plugin / Scala.js releases to support older 2.13 versions? Can all the library / plugin authors assume that forcing users to the latest 2.13 is OK?

On Discord it was also suggested that scalaVersion := 2.13 or something like that should be allowed so that the build tool can automatically pick the latest compiler according to the dependency graph. https://discord.com/channels/632150470000902164/632628489719382036/1246367750054740029

TY for the explanation @lrytz. This is very helpful.

A macro compiled with 2.13.14 can reference a new method added to scala-library.jar in that release. When using the macro on 2.13.13, it is loaded dynamically into the jvm running the 2.13.13 compiler, with the 2.13.13 standard library on the runtime classpath. So the method is not there when the macro is invoked.

It feels to me that this is the core of the issue: Macro libraries should be on the runtime classpath, not the compile time classpath. Have you considered resolving them differently?

After all, macros are really just a form of compiler plugins :ducks:

So I guess the question is this: how important is it for new library / compiler plugin / Scala.js releases to support older 2.13 versions? Can all the library / plugin authors assume that forcing users to the latest 2.13 is OK?

@sjrd can answer this question better than me. But so far we have even back published newer Scala versions for older Scala.js versions (to some extent). Tying the Scala version to the Scala.js version would be a significant departure from our current versioning scheme.

My understanding of the discussion on SIP-51 was that it is not a change the Scala community is willing to do.

Indeed, we do backpublish newer Scala versions for older Scala.js versions. Usually several versions back. For the latest 2.13.14, I had only proactively backpublished the latest Scala.js version, and there was an explicit request to publish older ones.

The demand definitely exists. Now is it a demand that belongs to the "old system", and that we should nudge people away from? I don't know.

A macro compiled with 2.13.14 can reference a new method added to scala-library.jar in that release. When using the macro on 2.13.13, it is loaded dynamically into the jvm running the 2.13.13 compiler, with the 2.13.13 standard library on the runtime classpath. So the method is not there when the macro is invoked.

It feels to me that this is the core of the issue: Macro libraries should be on the runtime classpath, not the compile time classpath. Have you considered resolving them differently?

I don't really follow... A difference between macros and compiler plugins is that macros are binary compatible (unless they do funky stuff like casting to nsc.Global to access internal API). So a library with macros built on 2.13.10 can be used on a 2.13.13 compiler.

The compiled macro is bytecode executed at compile time, the macro code is loaded and executed within the compiler. It runs with the scala-library that the compiler has on its runtime classpath.

So a macro compiled against 2.13.14 referencing some new API will fail to expand in a 2.13.13 compiler.

The demand definitely exists. Now is it a demand that belongs to the "old system", and that we should nudge people away from?

At the end, Scala.js is not different from a library. After SIP-51, updating a dependency to new version built with 2.13.14 requires the project's scalaVersion to be updated to at least 2.13.14. While working on SIP-51 we realized that upgrading only scala-library but keeping the compiler doesn't work, and the we (the SIP commitee) agreed that it's fine requiring the Scala version upgrade.

The question is, what's the use case for people to update a dependency but remain on an older Scala compiler?

Maybe there's a stronger use case for Scala.js (or compiler plugins in general) than any other library, to be able to push a new plugin version for older compilers? For example, for Metals, it seems supporting older Scala versions in new releases is important. After all, the plugin is injected into the build when a user loads the project in the IDE, so the user doesn't make a choice to update some dependency.

At the end, Scala.js is not different from a library

I don't have time to write a full answer but I felt clarifying this is important: Scala.js does something no library does: It takes the Scala library sources (!) and compiles them for Scala.js. To do this, it needs to select a version. I think it is this selection that is at the core of how Scala.js is different.

The question is, what's the use case for people to update a dependency but remain on an older Scala compiler?

The main use case is to use a compiler plugin, which isn't released for a newer Scala version. Since most projects prefer to support newer Scala versions only by version bumps instead of re-publishing old tags/releases for newer Scala versions. Most tool chains aren't ready for selective republishing.

Combined with the fact, that most libraries frequently update their Scala library version (for no reason other than providing a newer transitive version), you quickly end up with hard challenges to pick the correct version.

As a rule of thumb: Libraries should always depend on the lowest possible version, applications should always pick the newest supported version.