Skip to content

Conversation

jchyb
Copy link
Contributor

@jchyb jchyb commented Sep 29, 2025

Closes #9013

Just a few small tweaks left:

  • extract MethodHandles.lookup into local variable in the static constructor
  • [x] maybe inline objCAS2 (should give us more freedom, allow us to backport into 3.3 in the worst case scenario (hopefully not))
    EDIT: done

This PR replaces the use of the deprecated Unsafe with VarHandles, available in java 9+. Important thing about VarHandles is that for private members (like our val dictating the value/resolution progress of the lazy val), the findVarHandle method must be called from the class containing the val, otherwise we end up with an error. This meant that some custom logic had to be added to the MoveStatics phase, which would usually move static constructor to the companion object (which we do not want for the VarHandles here).
Only the newer implementation was adjusted, the one available under -Ylegacy-lazy-vals was left untouched.

This PR also effectively drops support for java 8, with java 17 being the new required version. As part of that:

  • lowest supported version for -release/-java-target-version is now 17 (earlier versions will throw an error). -Xunchecked-java-output-version was left untouched.
  • managed community-build tests are now run with java 17. Options setting -release 8 were removed for projects that were using that. Projects that were using javax.xml.bind (dropped in java 11) were removed altogether (playJson and betterfiles)
  • Java version used for releases was set to 17 (Added separately in Require JDK 17+ #24146)

@jchyb jchyb force-pushed the lazyval-varhandle branch 5 times, most recently from a82a297 to 5aafc89 Compare October 2, 2025 14:32
@sjrd
Copy link
Member

sjrd commented Oct 3, 2025

extract MethodHandles.lookup into local variable in the static constructor

I actually wonder: is that worth it? It seems to bring additional complexity to the transformation. Sure, we need one call for every lazy val in the class, but does it matter? It's in the static initializer anyway, which is executed O(1) times.

maybe inline objCAS2

IMO that is really worth it, but for a different reason: it removes the last reference to the LazyVals$ module class in the generated code. That's good because we are then reasonably sure that we won't hit load-class-time warnings about the Unsafe val in that module class. AFAICT, generating the inlined code would not be any harder than generating the call to objCAS2. It's one method call.

(We can drop the if (debug) thing at this point anyway.)

@jchyb
Copy link
Contributor Author

jchyb commented Oct 3, 2025

extract MethodHandles.lookup into local variable in the static constructor

I actually wonder: is that worth it? It seems to bring additional complexity to the transformation. Sure, we need one call for every lazy val in the class, but does it matter? It's in the static initializer anyway, which is executed O(1) times.

It's an obvious improvement so I wanted to take it. Unfortunately, even without this we still had to introduce complexity in the MoveStatics phase (directly linking to the LazyVals phase), so we are not doing that much more (1st commit vs 2nd commit). I'll try running some benchmark tests to see if it's worth it.

maybe inline objCAS2

IMO that is really worth it, but for a different reason: it removes the last reference to the LazyVals$ module class in the generated code. That's good because we are then reasonably sure that we won't hit load-class-time warnings about the Unsafe val in that module class. AFAICT, generating the inlined code would not be any harder than generating the call to objCAS2. It's one method call.

(We can drop the if (debug) thing at this point anyway.)

Right, I hadn't even thought about that (we still reference the nested classes but now I realize that's not an issue). Weirdly, even with the objCAS2 call, the warnings disappeared for me.

@jchyb jchyb force-pushed the lazyval-varhandle branch 6 times, most recently from 9032701 to bb1b2ca Compare October 6, 2025 18:07
@jchyb jchyb marked this pull request as ready for review October 7, 2025 09:15
@jchyb jchyb requested a review from sjrd October 7, 2025 09:15
@jchyb
Copy link
Contributor Author

jchyb commented Oct 7, 2025

@sjrd Issues with CI took me a little more time than I anticipated, so I haven't done the promised benchmarks yet. Still I am willing to remove the initialization optimization if it complicates things too much.

I did test whether this works with native image though (it does).

Copy link
Member

@sjrd sjrd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work overall!

private lazy val minTargetVersion = classfileVersionMap.keysIterator.min
private lazy val maxTargetVersion = classfileVersionMap.keysIterator.max

private val minReleaseVersion = 17
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we change classfileVersionMap instead?

(minTargetVersion to maxTargetVersion).toList.map(_.toString)

def supportedReleaseVersions: List[String] =
if scala.util.Properties.isJavaAtLeast("9") then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true now.

val varHandleInfo = appendVarHandleDefs.getOrElseUpdate(claz, new VarHandleInfo(EmptyTree, Nil))
varHandleInfo.methodHandlesLookupDef match
case EmptyTree =>
val lookupSym: TermSymbol = newSymbol(claz, (s"${claz.name}${lazyNme.methodHandleLookupSuffix}").toTermName, Synthetic, defn.MethodHandlesLookupClass.typeRef).enteredAfter(this)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turning names into strings like that is not recommended. Consider something like

Suggested change
val lookupSym: TermSymbol = newSymbol(claz, (s"${claz.name}${lazyNme.methodHandleLookupSuffix}").toTermName, Synthetic, defn.MethodHandlesLookupClass.typeRef).enteredAfter(this)
val lookupSym: TermSymbol = newSymbol(claz, claz.name.toTermName.withSuffix(lazyNme.methodHandleLookupSuffix), Synthetic, defn.MethodHandlesLookupClass.typeRef).enteredAfter(this)

case _ =>

// create a VarHandle for this lazy val
val varHandleSymbol: TermSymbol = newSymbol(claz, s"$containerName${lazyNme.lzyHandleSuffix}".toTermName, Synthetic, defn.VarHandleClass.typeRef).enteredAfter(this)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here:

Suggested change
val varHandleSymbol: TermSymbol = newSymbol(claz, s"$containerName${lazyNme.lzyHandleSuffix}".toTermName, Synthetic, defn.VarHandleClass.typeRef).enteredAfter(this)
val varHandleSymbol: TermSymbol = newSymbol(claz, containerName.toTermName.withSuffix(lazyNme.lzyHandleSuffix), Synthetic, defn.VarHandleClass.typeRef).enteredAfter(this)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, can't we make those symbols Private?

}

extension (sym: Symbol) def isVarHandleForLazy(using Context) =
sym.name.endsWith(lazyNme.lzyHandleSuffix)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we should use a dedicated NameKind if we need to do that kind of test.


val staticAssigns = staticFields.map(x => Assign(ref(x.symbol), x.rhs.changeOwner(x.symbol, staticCostructor)))
tpd.DefDef(staticCostructor, Block(staticAssigns, tpd.unitLiteral)) :: newBody
val symbolRemap = localStaticDefs.map { x =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO this is too complex to introduce without a very good reason.

I suggest we stick to the simpler alternative where we repeat the calls to lookup().

localStaticDefs.addOne(valDef)
case memberDef: MemberDef if memberDef.symbol.isScalaStatic =>
if memberDef.symbol.isVarHandleForLazy then
staticTiedDefs.addOne(memberDef)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this could be

Suggested change
staticTiedDefs.addOne(memberDef)
remainingDefs.addOne(memberDef)

which would remove the need for staticTiedDefs altogether.

import dotty.tools.pc.utils.JRE

class CompletionRelease8Suite extends BaseCompletionSuite:
@Ignore class CompletionRelease8Suite extends BaseCompletionSuite:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this still needs to be addressed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't work anymore, because the -release 8 option throws an error now (on purpose). I'll add the comment explaining that. If it's clearer to just remove the test completely I can do that too

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Yes, in that case we should remove the test entirely.

Perhaps extract a PR that really gets rid of JDK 8 support. There seem to be several commits about that in this PR, and they should stand on their own.

lazy val i18231 = project
.in(file("i18231"))
.settings(scalacOptions += "-release:8")
// .settings(scalacOptions += "-release:8")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't that invalidate the purpose of the test? Should it be retargeted to 17?

@jchyb jchyb self-assigned this Oct 7, 2025
@jchyb jchyb mentioned this pull request Oct 7, 2025
hamzaremmal added a commit that referenced this pull request Oct 7, 2025
This is paving the way for #24109.
As part of this:
* the minimum version for `-release`/`-java-output-version` flag is set
to 17
* managed community build tests are done on JDK 17
  * some community-build projects are temporarily removed
  * some that were using -release 8 option now have that option removed
* compilation tests using `-release 8` were disabled
* presentation compiler tests using `-release 8` and `-release 11` were
removed
* releases are done on JDK 17
@jchyb jchyb force-pushed the lazyval-varhandle branch 2 times, most recently from c578c64 to 045cec4 Compare October 8, 2025 12:06
@jchyb jchyb force-pushed the lazyval-varhandle branch from 045cec4 to aa26211 Compare October 8, 2025 12:11
@jchyb jchyb requested a review from sjrd October 9, 2025 09:51
Copy link
Member

@sjrd sjrd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only an optional nit. LGTM. Nice improvement, even. :)

val varHandleSymbol: TermSymbol = newSymbol(claz, LazyVarHandleName(containerName), Private | Synthetic, defn.VarHandleClass.typeRef).enteredAfter(this)
varHandleSymbol.addAnnotation(Annotation(defn.ScalaStaticAnnot, varHandleSymbol.span))
val getVarHandle = Apply(
Select(Apply(Select(ref(defn.MethodHandlesClass), defn.MethodHandles_lookup.name), Nil), defn.MethodHandlesLookup_FindVarHandle.name),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Directly select the symbols?

Suggested change
Select(Apply(Select(ref(defn.MethodHandlesClass), defn.MethodHandles_lookup.name), Nil), defn.MethodHandlesLookup_FindVarHandle.name),
Select(Apply(Select(ref(defn.MethodHandlesClass), defn.MethodHandles_lookup), Nil), defn.MethodHandlesLookup_FindVarHandle),

or maybe it's

Suggested change
Select(Apply(Select(ref(defn.MethodHandlesClass), defn.MethodHandles_lookup.name), Nil), defn.MethodHandlesLookup_FindVarHandle.name),
Apply(ref(defn.MethodHandlesClass).select(defn.MethodHandles_lookup), Nil).select(defn.MethodHandlesLookup_FindVarHandle),

@jchyb jchyb merged commit f747007 into scala:main Oct 13, 2025
51 checks passed
@jchyb jchyb deleted the lazyval-varhandle branch October 13, 2025 13:40
@jchyb
Copy link
Contributor Author

jchyb commented Oct 13, 2025

Thank you for the reviews and support @sjrd!

@jchyb jchyb added the needs-minor-release This PR cannot be merged until the next minor release label Oct 13, 2025
@tgodzik
Copy link
Contributor

tgodzik commented Oct 13, 2025

Btw. this will not be backported to LTS, which means LTS will get the warnings about Unsafe deprecation and might eventually not work on a newer Java version. That said, it's encouraged if you want to work on newest JDK to just use Scala Next

Great work @jchyb !

@jivanic-demystdata
Copy link

@tgodzik Couldn't this be an issue for Libs authors, who will want to provide official support to JDK25 but can't use Scala Next to allow everyone to use the Scala version they want?

@tgodzik
Copy link
Contributor

tgodzik commented Oct 13, 2025

Might be, though if someone is using JDK 25 for the library, they are already forcing users pretty heavily with the JDK bump. Do you know any such library? I assumed that most libraries would release on max 21. Library authors were usually quite careful about that.

@mr-git
Copy link

mr-git commented Oct 13, 2025

From https://openjdk.org/jeps/471:

Phase 3, throwing exceptions by default, in JDK 26 or later; and
Phases 4 and 5, which will remove the methods, in releases following JDK 26.

and

.. this will not be backported to LTS..

Does this mean that library built with latest Scala 3 LTS might not be usable on JDK 26+?

@tgodzik
Copy link
Contributor

tgodzik commented Oct 13, 2025

That is possible, but we can't change the implementation of lazy vals within a minor. There will be a new LTS soon in the 3.9.x line.

As a side note we should probably make sure that this is detected during compilation.

@mr-git
Copy link

mr-git commented Oct 14, 2025

..we can't change the implementation of lazy vals within a minor.

I did the very quick and uneducated skim of changes, but I didn't notice public API changes (except those few ignored ones). Isn't implementation of lazy vals private?

@jivanic-demystdata
Copy link

@tgodzik

Do you know any such library? I assumed that most libraries would release on max 21. Library authors were usually quite careful about that.

When I say "providing official JDK 25 support", it's just running the CI with JDK 25 (additionally with 17 and 21, with a minimum required version of 17). Most libs will adopt this strategy.
If we can't run the CI with JDK 25 + Scala 3.3.x, we can really claim our libs work with it.

Hence, my question: will it be an issue not to backport this to Scala 3.3.x? Or will it work with just some warnings from JDK 25?

@tgodzik
Copy link
Contributor

tgodzik commented Oct 14, 2025

For JDK 25 it will only show warnings now, so we have some time to decide if the situation requires for us to try and backport this at some point anyway.

@Gedochao
Copy link
Contributor

Hence, my question: will it be an issue not to backport this to Scala 3.3.x? Or will it work with just some warnings from JDK 25?

I can't see how we could back port this to Scala 3.3.x, as the new API uses JDK API which is absent in Java 8.
We bump the required Java version to 17 in Scala 3.8, but I can't see how we could do that without changing the minor. That's a big compat no-no for 3.3 LTS. We just can't drop Java 8 there.

Library authors wanting to run on JDK 25 will have to bump to Scala 3.9 LTS, which is right behind the corner.
If your library needs it already, then all I can say is that you can be an early adopter of 3.8 until the new 3.9 LTS is out.

@jivanic-demystdata
Copy link

jivanic-demystdata commented Oct 14, 2025

Thanks for your clarifications 🙏

@MateuszKubuszok
Copy link

@Gedochao

I can't see how we could back port this to Scala 3.3.x, as the new API uses JDK API which is absent in Java 8.

Scala 2.13 does exactly that:

performs one-time feature discovery on JVM:

  • if there is java.lang.invoke.VarHandle.releaseFence it will be used (JDK9+)
  • if there is sun.misc.Unsafe.storeFence it will be used as a fallback (JDK8)

as a result all code compiled on 2.13.7+, should work on newer JDKs. With no recompilation. Since 7 years ago.

It means that currently libraries' authors would have less issues supporting 2.13 for both newer and older JDK than Scala 3 authors - on 2.13 code just works, on 3 I will have yet another issue to solve when it comes to supporting Scala 2.13 + Scala 3.

@sjrd
Copy link
Member

sjrd commented Oct 14, 2025

What Scala 2.13 does here has nothing to do with lazy vals. The use cases for releaseFence() are different, and also apply to what Scala 3 does.

For lazy vals with VarHandles, it is not possible to use the same trick. The reason is that you need to store a VarHandle at call site for the new system, but a Long for the old system. You can't even declare the field to store the VarHandle if you need to run on JDK 8.

Scala 2.13 does not have the lazy val problem for an entirely different reason: it never used Unsafe in the first place. It uses an older synchronized-based algorithm. The algorithm in Scala 3 used Unsafe to have a more performant replacement.

@MateuszKubuszok
Copy link

I'd argue that it's not possible to do it in exactly the same way, but still possible. OTOH ideas:

  • using Any for storing VarHandle | Long
  • a trait with 2 implementations: one using VarHandle, one using Long, the actual implementation being initialized via reflection

For me, the question is more about what utilities can and cannot be added to LTS (if they are not directly exposed to the users), and if none (because someone can still downgrade the patch version), how much extra code can be generated (possibly behind a flag, due to a probable performance hit), rather than whether it's possible or not.

@sjrd
Copy link
Member

sjrd commented Oct 14, 2025

Both of your solutions would be disastrous for performance. They're not viable in this context.

@MateuszKubuszok
Copy link

Another possibility is to allow using VarHandle if release is targetting something newer then "8".

I prefer any such option to dropping support in my libraries.

@mr-git
Copy link

mr-git commented Oct 14, 2025

.. if release is targetting something newer then "8".

And, during compilation, scalac should print warning about limitations of Unsafe usage on JDK 25+, when Scala 3 code gets targeted for soon-to-be-unsupported JDK 8

@lbialy
Copy link

lbialy commented Oct 14, 2025

Targeting something newer than "8" is mostly irrelevant - any of your dependencies can be compiled with a version of the compiler that generated code using Unsafe and you'll still crash on JDK 26.

I think the idea that 3.9 is the LTS that works with JDK 26 as a clean cut off barrier is actually a good choice, because:

  • our current compatibility scheme makes it a very good prevention mechanism - you want to target JDK 26, you need Scala 3.9, you set scalaVersion to 3.9 - deps that aren't ready crash the compilation with a tasty versioning error, way better than getting crashes in runtime
  • 3.9 is a LTS release after all.

It is an uncomfortable situation but the only way this could have been prevented was to drop JDK 8 before and that would make a lot of people very unhappy anyway.

@Gedochao
Copy link
Contributor

I prefer any such option to dropping support in my libraries.

Scala 3.3 LTS will be supported for at least a year after 3.9 LTS is out.
Technically speaking, 3.3 LTS will work with JDK 25 - that is, with warnings.
You don't have to upgrade immediately, if you care about older JDKs.
You might want to if you care about the new ones, though.
Scala 3.8+, including 3.9 LTS, will require JDK 17+, as previously communicated.

The way things are, it does not seem to me that JDK 26+ would be properly supported on versions older than the upcoming 3.9 LTS series.

And, during compilation, scalac should print warning about limitations of Unsafe usage on JDK 25+, when Scala 3 code gets targeted for soon-to-be-unsupported JDK 8

A warning already is being printed on JDK 25.

scala -e 'println("Hello")' -S 3.3.7--jvm 25
# Compiling project (Scala 3.3.7, JVM (25))
# Compiled project (Scala 3.3.7, JVM (25))
# WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
# WARNING: sun.misc.Unsafe::objectFieldOffset has been called by scala.runtime.LazyVals$ (file:~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.3.7/scala3-library_3-3.3.7.jar)
# WARNING: Please consider reporting this to the maintainers of class scala.runtime.LazyVals$
# WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
# Hello

Or do you mean that we should warn about JDK 25 limitations while targeting JDK 8? If so, I am lost here.

@lbialy
Copy link

lbialy commented Oct 14, 2025

The valid issue that @MateuszKubuszok is raising here is that it is going to be a problem for library maintainers that want to be able to release new releases for people on older JDKs. We already have a matrix because of 2.12, 2.13 and 3 and the promise was that 3.x would just work without any splits so once a lib drops 2.x, it's free of any versioning matrices altogether. @MateuszKubuszok's libs were hit with a backwards incompatible change when implicit resolution rules changed and it was a difficult problem to fix. @jchyb was involved with design and implementation of a new macro API that allowed to deal with that. Now this problem hits even more libs as it does introduce a compatibility matrix problem if you are unwilling to drop support for jdk <17 for your users by just bumping required Scala version to 3.9 for 3.x branch. So we will, indeed, have two 3.x variants for such maintainers - 3.3 with the jdk8-jdk25 range and 3.9 with the jdk17-jdk26+ range. I do understand the frustration but there's very little that can be done if we are to treat backwards compatibility in LTS seriously.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-minor-release This PR cannot be merged until the next minor release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runtime code implementing lazy val should not use sun.misc.Unsafe on Java 9+ (JEP-471)

8 participants