The Scala compiler ships with its own JVM bytecode optimizer. While working on that feature I got curious what would be its effect when running on GraalVM.

I benchmarked the Scala compiler built without and with its own optimizer enabled, and compared the performance of these two compilers on HotSpot (1.8.0_181), Graal CE RC7 and Graal EE RC7.

Results when running on Linux (i7-6700, 64 GB RAM, Debian 8.8), unit is ms/op, in parentheses is the factor to the baseline:

(The Scala version numbers are not important; the first column is a compiler built without optimizer, the second is built with the optimizer enabled)

Graal is doing really well here, the unoptimized Scala compiler runs 22% faster on Graal EE than HotSpot. The Scala optimizer is helpful on HotSpot, less so on GraalVM.

I ran the same benchmarks on my MacBook Pro (i7-8850H, 16 GB RAM, macOS 10.14) and got very different results:

Graal CE is slower than HotSpot, GraalEE just a bit faster when running the non-optimized Scala compiler. The optimized Scala compiler runs faster on HotSpot than on Graal.

We were quite surprised by these results, so we ran the benchmarks on an iMac (i7-6700K) and on something-like-a-Mac-Mini (i7-7700, macOS 10.13.6). The results were the same as on my MacBook Pro.

By suggestion of @axel22, I ran the benchmarks on a Linux VM (Debian 8.11, 8 CPUs, 4 GB RAM) on my MacBook Pro. The results were the same as on the non-VM Linux machine, GraalVM is doing well.

So it looks like GraalVM has suboptimal performance on macOS.

To reproduce the benchmarks

• Clone https://github.com/scala/compiler-benchmark
• Install sbt (https://www.scala-sbt.org/download.html)
• Install HotSpot, Graal CE and Graal EE
• Run the script below (it uses https://github.com/shyiko/jabba to switch JVMs, so you need to adjust it if you use something else to manage your JVMs)

jabba use 1.8.191
for sv in 2.13.0-pre-84de40d-noopt 2.13.0-pre-8d17277-SNAPSHOT; do
  for vm in 1.8.191 1.8.172-graal-ce-rc7 1.8.172-graal-ee-rc7; do
    echo "----- $sv ----- $vm -----"
    sbt \
      'set resolvers in ThisBuild   = List("scala-integration" at "https://scala-ci.typesafe.com/artifactory/scala-integration/", "scala-release-temp" at "https://scala-ci.typesafe.com/artifactory/scala-release-temp/", "scala-pr" at "https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/")' \
      "set scalaVersion in ThisBuild := \"$sv\"" \
      clean \
      "hot -jvm $(jabba which $vm)/bin/java -psource=scalap -f 1 -wi 20 -w 10 -i 10 -r 10"
  done
done