Ubiquitous inlining

Language Design
1

**The ask:** An (optional) compiler flag reporting the emitted bytecode size of each method.

The compiler must have this somewhere when it writes the code out?

Possibly : A scalafix to check it

The Journey

More or less revolves around my hobby [vecxt]( GitHub - Quafadas/vecxt · GitHub ) which is me exploring and learning about arrays, matrices etc.

It starts with Array[A]. You’ll quickly notice catastrophic boxing… and you’ll also quickly notice, that you can inline round it… I’m posting (the obvious) concrete sum extension method for illustration.

    inline def sumSIMD: Double =
      var i: Int = 0
      var acc = DoubleVector.zero(spd)

      while i < spd.loopBound(vec.length) do
        acc = acc.add(DoubleVector.fromArray(spd, vec, i))
        i += spdl
      end while
      var temp = acc.reduceLanes(VectorOperators.ADD)
      // var temp = 0.0
      while i < vec.length do
        temp += vec(i)
        i += 1
      end while
      temp
    end sumSIMD

At first, I believed the inline looks like an asymmetric bet. Make all the things inline and don’t worry about generics / specialisation…

What happens next, was not obvious to me. Here’s the key JVM default info…

Bytecodes Threshold What happens above it
35 MaxInlineSize Not inlined at cold call sites; still inlined if the site is hot.
325 FreqInlineSize Not inlined into callers even when hot. Method is still JIT-compiled and fully optimised internally; what’s lost is optimisation across the boundary — escape analysis, constant propagation, loop fusion.
8000 DontCompileHugeMethods Never JIT-compiled at all. Runs interpreted for the life of the process, silently, with nothing in a profile pointing at the cause.
65535 JVMS §4.7.3 code_length Compile error — “Method too large”.

That method above? Something like 80 bytecodes, because you have two loops and the SIMD instructions… similar for many methods like +, * etc.

If you start inlining these together in a chain? I got this so badly wrong, that I blew out the 65535 limit :rofl: when trying to mess around with a neural network type thing. Which was my first hint that something… wasn’t quite right.

HugeMethodLimit is the danger and a nasty performance cliff - “Great microbenchmarks, horrible real world performance” wasn’t the tagline I had in mind for vecxt!

`-Xmax-inlines` (default 32) is the only guard rail I’m aware of. It fires on the right axis with the wrong framing. Its message names *successive* inlines and suggests a recursive inline method. Mine not recursion, but rather a pathological, aggressively layered inlining strategy in which I was confident there was no recursion. The message states no consequence and prints the flag to raise the limit, so I raised it. Wording like *“raising this limit may produce methods too large for the JIT to compile (>8000 bytecodes) and may materially impact JVM performance”* would, I hope, have triggered an alarm bell…

## Documentation

The macros best-practices page does say to avoid generating large methods for JIT reasons — one sentence and no numbers. The inline tutorial frames inlining as a metaprogramming entrypoint and does not discuss size. So a fair criticism is not “undocumented”. Rather qualitative, with no thresholds or a measurement path. It is very easy to ignore.

The numbers that matter — 35, 325, 8000, 65535 … were pulled from Claude’s memory (I’ve done my best to check them, I believe they represent the JVM defaults). I’m not aware of too many people with the confidence to start tampering with them.

Essentially, I was not aware, despite a reasonable best effort and genuine ecosystem curiosity, that I was playing with fire.

## Proposals

1. **`-Vprint-method-sizes`** — emitted bytecode size per method. No new analysis needed; makes the existing best-practices advice actionable, and lets authors diff a refactor or assert a CI bound.

2. **Reword the `-Xmax-inlines` message** to name the potential consequences.

  1. Add a hint which has a better chance of triggering follow up questions to the inline scala docs. inline is not an asymmetric, consequence free keyword - it actively eats the budget the JVM uses for it’s optimisations.
  2. A scalafix lint, for those two know they are playing with fire and want to manage it below a level

I suspect, that I hit a particularly sharp edge as I was mucking around with both the VectorAPI and the specialisation part. However, I do wonder how many methods out there silently go over that 8000 bytecode cliff. It’s a silent performance killer, and possibly a way to lose hearts and minds.

Here’s what I think I measured in terms of bytecode output…

ops universally inline after first de-inlining
chain01 — (a * b).sum 2 226 17
chain02 3 281 64
chain04 — (a * b + a - b + a).sum 5 391 158
chain08 9 770 273
chain16 16 1347 547
same arithmetic as chain04, hand-written loop — 47 47

Chain 16 is not unreasonable for numeric stuff. And did not appears unreasonable, to start chaining those methods together. Works great in the small, horrible as you scale it up. Hence the request for a discussion…

3 Likes
2

Discord wisdom has pointed out that this can read as though I think inlining itself as a language feature is a problem. That is not the intent and not a statement I agree with.

I abused this language feature and I’ll own that part. What I’m saying is that it was not obvious to me, that my use of it was, in fact abusive…

3

You can always disassemble the bytecode to get this; and with an LLM doing it, it’s barely harder than parsing the compiler output. Even if I’m coding by hand, I always have a LLM up these days to handle tedious tasks like that. The future will look even more like this.

But what would be really useful is a compiler warning flag that would yell at you if a method is large (especially beyond the Huge size). This tells you that there is even anything to care about. Once you already care, existing solutions are fine.

4

I don’t really agree with this, I would much rather we have deterministic, and checked, than have to trust the LLM

(There are things which cannot be done deterministically/procedurally, and there LLMs have their uses, but this falls squarely in the “we can compute this” camp)

4 Likes
5

The compiler already errors if the method/class is too large for JVM to read it. But it would be helpful to (even as an opt-in) have a warning that:

  • warns that method is beyond certain size - one could set it up to e.g. the same value is some popular JVM version uses as cutoff above which C2 does not run (HugeMethodLimit = 8000?)
  • warning about escape-analysis limits (MaxBCEAEstimateSize = 150?)
  • the documentation would have to explain that this number can have different default values on each JVM (and that non-HotSpot bases ones might not have it at all!), and that it can be customized, and what it means
  • there could also be warning about approaching class/method size limit, to make user aware that it still works, but anything more would make it stop working

I’m assuming it would either have very good warning explaining in detail what it means, or should be opt-in since most people should not run into these, and would probably not understand these anyway without some JVM expertise. (So they could just bump up the number without understanding what it means).

3 Likes
6

Thanks for this - I think your post is a better summary of the issue than mine.

I’m not sure the warning needs to be very comprehensive, it just needs to generate enough fear to ask the right follow up question… as has been pointed out, one you know you need to care, the answers can be found.

For me I’m pretty sure it’s the 8000 one that was the painful one. I think I carried it around for a rather long time knowing only that “something wasn’t working right”, but being all but clueless as to what or how to identify it.

7

This is a trivial operation for a frontier LLM, for a measurement that is almost never a critical single-use operation but rather part of a much more elaborate performance investigation.

I absolutely agree that determinism, especially in complex load-bearing cases, is a very nice feature to have. But this is an easy usually-not-load-bearing case. You can have a LLM write a short script or something, then have using that script be a skill. It’s fine if the compiler does it too, but because there are easy alternatives that work well, the high-value case is a flag that warns when something is too big. That changes a pull of information to a push, and that is very valuable.

Otherwise, if it’s something that you do almost never, have the LLM do it and pull the bytecode if you mistrust it; or if you do sizings often, have it write the external tooling.