Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V2 #62

Open
wants to merge 35 commits into
base: master
Choose a base branch
from
Open

V2 #62

wants to merge 35 commits into from

Conversation

tadfisher
Copy link
Owner

  • Rewritten dependency resolution; now uses Gradle's artifact cache to map URLs to artifacts and calculate hashes. This is much simpler and should be much more reliable than the rather involved process in V1.
  • Add the "--task" flag, for locking only the artifacts that are resolved as part of running the given task(s).
  • Add flags for choosing the Gradle installation: a distribution URL, a local path, or a wrapper version.
  • The lockfile format has changed to be simpler to parse from the Nix side. It is now a mapping of (coordinates -> filename -> artifact), where artifact is a single URL and its SHA256 hash.
  • The default name of the lockfile has changed from gradle-env.json to gradle.lock.
  • gradle2nix no longer generates gradle-env.nix. The gradle2nix package includes a passthru.buildGradlePackage function, and the flake exposes builders.${system}.buildGradlePackage for consumers to import.
  • buildGradlePackage now applies the offline repository on-demand; that is, for each repository added by the project's build scripts, the URL and other parameters is replaced by the path to the offline repository. This is to provide compatibility for build scripts which use exclusiveContent repositories or call repositories.addFirst() themselves after script evaluation.
  • buildGradlePackage accepts a list of overlay functions for modifying the contents of the offline repository after parsing the lock file. This is useful for manually adding dependencies and working around pathological build scripts.

@rgoulter
Copy link

rgoulter commented May 30, 2024

FWIW, I ran into a snag where the flake doesn't have builders.aarch64-darwin.

A CI job using this gradle2nix v2 which succeeded with x86_64-linux and x86_64-darwin fails on aarch64-darwin:

copying path '/nix/store/dc477ngpdyz4wqn5yswppahrvr0vcghy-source' from 'https://cache.nixos.org/'...
error:
       … while evaluating the attribute 'packages.aarch64-darwin.c-worksheet-instrumentor'
         at /nix/store/mxkz6kricc021bpsn6n2i3gmhvyw3585-source/flake.nix:50:7:
           49|
           50|       c-worksheet-instrumentor = pkgs.callPackage ./c-worksheet-instrumentor.nix {inherit (gradle2nix) buildGradlePackage;};
             |       ^
           51|     });
       … while calling a functor (an attribute set with a '__functor' attribute)
         at /nix/store/dc477ngpdyz4wqn5yswppahrvr0vcghy-source/lib/customisation.nix:264:13:
          263|     in if missingArgs == {}
          264|        then makeOverridable f allArgs
             |             ^
          265|        # This needs to be an abort so it can't be caught with `builtins.tryEval`,
       (stack trace truncated; use '--show-trace' to show the full trace)
       error: attribute 'aarch64-darwin' missing
       at /nix/store/mxkz6kricc021bpsn6n2i3gmhvyw3585-source/flake.nix:46:20:
           45|       pkgs = nixpkgs.legacyPackages.${system};
           46|       gradle2nix = inputs.gradle2nix.builders.${system};
             |                    ^
           47|     in {

CI job run: https://github.com/rgoulter/c-worksheet-instrumentor/actions/runs/9299128759/job/25592407230?pr=27 from PR rgoulter/c-worksheet-instrumentor#27

Though, I do see aarch64-darwin in apps, packages from nix flake show.

EDIT: This seems to occur because my flake used grade2nix as the name of the flake input, as well as using it as the name for a let .. in variable.

i.e. gradle2nix = inputs.gradle2nix.packages.${system}.gradle2nix; did not work on aarch64-darwin, but gradle2nix = inputs.gradle2nixF.packages.${system}.gradle2nix; worked. -- This seems like

@collinarnett
Copy link

I'm somewhat new to gradle but I ran into the following error when trying to use openjdk17 with the v2 branch:

datahub on  master [?] via 🅶 v8.0.2 via ☕ v17.0.7 on ☁️  (us-east-1) on ☁️   
$ nix run github:tadfisher/gradle2nix/v2 -- -p . -j $(nix eval --raw nixpkgs#openjdk17.home)
Error: LinkageError occurred while loading main class org.nixos.gradle2nix.MainKt
	java.lang.UnsupportedClassVersionError: org/nixos/gradle2nix/MainKt has been compiled by a more recent version of the Java Runtime (class file version 65.0), this version of the Java Runtime only recognizes class file versions up to 61.0

@tadfisher
Copy link
Owner Author

I'm somewhat new to gradle but I ran into the following error when trying to use openjdk17 with the v2 branch:

datahub on  master [?] via 🅶 v8.0.2 via ☕ v17.0.7 on ☁️  (us-east-1) on ☁️   
$ nix run github:tadfisher/gradle2nix/v2 -- -p . -j $(nix eval --raw nixpkgs#openjdk17.home)
Error: LinkageError occurred while loading main class org.nixos.gradle2nix.MainKt
	java.lang.UnsupportedClassVersionError: org/nixos/gradle2nix/MainKt has been compiled by a more recent version of the Java Runtime (class file version 65.0), this version of the Java Runtime only recognizes class file versions up to 61.0

Thanks, we need to set target compatibility to something like JDK 8.

@tadfisher
Copy link
Owner Author

@collinarnett Should be fixed with the latest commit.

@collinarnett
Copy link

Heres the latest try:

datahub on  master [?] via 🅶 v8.0.2 via ☕ v17.0.7 on ☁️  (us-east-1) on ☁️   took 45s 
$ nix run github:tadfisher/gradle2nix/v2 -- -p .

FAILURE: Build failed with an exception.

* Where:
Initialization script '/nix/store/d3ma4j1y9cpb05yfwybvlbz22wjncvyg-gradle2nix-2.0.0/lib/gradle2nix/share/init.gradle' line: 7

* What went wrong:
A problem occurred evaluating initialization script.
> 'org.gradle.api.provider.Provider org.gradle.api.services.BuildServiceRegistry.registerIfAbsent(java.lang.String, java.lang.Class)'

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s
Exception in thread "main" org.gradle.tooling.BuildException: Could not run build action using connection to Gradle distribution 'https://services.gradle.org/distributions/gradle-8.0.2-bin.zip'.
	at org.gradle.tooling.internal.consumer.ExceptionTransformer.transform(ExceptionTransformer.java:51)
	at org.gradle.tooling.internal.consumer.ExceptionTransformer.transform(ExceptionTransformer.java:29)
	at org.gradle.tooling.internal.consumer.ResultHandlerAdapter.onFailure(ResultHandlerAdapter.java:43)
	at org.gradle.tooling.internal.consumer.async.DefaultAsyncConsumerActionExecutor$1$1.run(DefaultAsyncConsumerActionExecutor.java:69)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.gradle.internal.exceptions.LocationAwareException: Initialization script '/nix/store/d3ma4j1y9cpb05yfwybvlbz22wjncvyg-gradle2nix-2.0.0/lib/gradle2nix/share/init.gradle' line: 7
A problem occurred evaluating initialization script.
	at org.gradle.initialization.exception.DefaultExceptionAnalyser.transform(DefaultExceptionAnalyser.java:128)
	at org.gradle.initialization.exception.DefaultExceptionAnalyser.collectFailures(DefaultExceptionAnalyser.java:92)
	at org.gradle.initialization.exception.MultipleBuildFailuresExceptionAnalyser.transform(MultipleBuildFailuresExceptionAnalyser.java:55)
	at org.gradle.initialization.exception.StackTraceSanitizingExceptionAnalyser.transform(StackTraceSanitizingExceptionAnalyser.java:39)
	at org.gradle.internal.buildtree.DefaultBuildTreeFinishExecutor.finishBuildTree(DefaultBuildTreeFinishExecutor.java:54)
	at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.lambda$runBuild$4(DefaultBuildTreeLifecycleController.java:103)
	at org.gradle.internal.model.StateTransitionController.lambda$transition$6(StateTransitionController.java:177)
	at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:258)
	at org.gradle.internal.model.StateTransitionController.lambda$transition$7(StateTransitionController.java:177)
	at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:44)
	at org.gradle.internal.model.StateTransitionController.transition(StateTransitionController.java:177)
	at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.runBuild(DefaultBuildTreeLifecycleController.java:95)
	at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.fromBuildModel(DefaultBuildTreeLifecycleController.java:73)
	at org.gradle.tooling.internal.provider.runner.AbstractClientProvidedBuildActionRunner.runClientAction(AbstractClientProvidedBuildActionRunner.java:43)
	at org.gradle.tooling.internal.provider.runner.ClientProvidedBuildActionRunner.run(ClientProvidedBuildActionRunner.java:48)
	at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
	at org.gradle.internal.buildtree.ProblemReportingBuildActionRunner.run(ProblemReportingBuildActionRunner.java:49)
	at org.gradle.launcher.exec.BuildOutcomeReportingBuildActionRunner.run(BuildOutcomeReportingBuildActionRunner.java:65)
	at org.gradle.tooling.internal.provider.FileSystemWatchingBuildActionRunner.run(FileSystemWatchingBuildActionRunner.java:140)
	at org.gradle.launcher.exec.BuildCompletionNotifyingBuildActionRunner.run(BuildCompletionNotifyingBuildActionRunner.java:41)
	at org.gradle.launcher.exec.RootBuildLifecycleBuildActionExecutor.lambda$execute$0(RootBuildLifecycleBuildActionExecutor.java:40)
	at org.gradle.composite.internal.DefaultRootBuildState.run(DefaultRootBuildState.java:122)
	at org.gradle.launcher.exec.RootBuildLifecycleBuildActionExecutor.execute(RootBuildLifecycleBuildActionExecutor.java:40)
	at org.gradle.internal.buildtree.DefaultBuildTreeContext.execute(DefaultBuildTreeContext.java:40)
	at org.gradle.launcher.exec.BuildTreeLifecycleBuildActionExecutor.lambda$execute$0(BuildTreeLifecycleBuildActionExecutor.java:65)
	at org.gradle.internal.buildtree.BuildTreeState.run(BuildTreeState.java:53)
	at org.gradle.launcher.exec.BuildTreeLifecycleBuildActionExecutor.execute(BuildTreeLifecycleBuildActionExecutor.java:65)
	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor$3.call(RunAsBuildOperationBuildActionExecutor.java:61)
	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor$3.call(RunAsBuildOperationBuildActionExecutor.java:57)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor.execute(RunAsBuildOperationBuildActionExecutor.java:57)
	at org.gradle.launcher.exec.RunAsWorkerThreadBuildActionExecutor.lambda$execute$0(RunAsWorkerThreadBuildActionExecutor.java:36)
	at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:249)
	at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:109)
	at org.gradle.launcher.exec.RunAsWorkerThreadBuildActionExecutor.execute(RunAsWorkerThreadBuildActionExecutor.java:36)
	at org.gradle.tooling.internal.provider.continuous.ContinuousBuildActionExecutor.execute(ContinuousBuildActionExecutor.java:110)
	at org.gradle.tooling.internal.provider.SubscribableBuildActionExecutor.execute(SubscribableBuildActionExecutor.java:64)
	at org.gradle.internal.session.DefaultBuildSessionContext.execute(DefaultBuildSessionContext.java:46)
	at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter$ActionImpl.apply(BuildSessionLifecycleBuildActionExecuter.java:100)
	at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter$ActionImpl.apply(BuildSessionLifecycleBuildActionExecuter.java:88)
	at org.gradle.internal.session.BuildSessionState.run(BuildSessionState.java:69)
	at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter.execute(BuildSessionLifecycleBuildActionExecuter.java:62)
	at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter.execute(BuildSessionLifecycleBuildActionExecuter.java:41)
	at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:63)
	at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:31)
	at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:50)
	at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:38)
	at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:47)
	at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:31)
	at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:65)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:39)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:29)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:35)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:78)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:75)
	at org.gradle.util.internal.Swapper.swap(Swapper.java:38)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:75)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:64)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:63)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:84)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:52)
	at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:297)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
Caused by: org.gradle.api.GradleScriptException: A problem occurred evaluating initialization script.
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:93)
	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.lambda$apply$0(DefaultScriptPluginFactory.java:135)
	at org.gradle.configuration.DefaultScriptTarget.addConfiguration(DefaultScriptTarget.java:74)
	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:138)
	at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
	at org.gradle.configuration.BuildOperationScriptPlugin.lambda$apply$0(BuildOperationScriptPlugin.java:62)
	at org.gradle.configuration.internal.DefaultUserCodeApplicationContext.apply(DefaultUserCodeApplicationContext.java:44)
	at org.gradle.configuration.BuildOperationScriptPlugin.apply(BuildOperationScriptPlugin.java:62)
	at org.gradle.configuration.DefaultInitScriptProcessor.process(DefaultInitScriptProcessor.java:50)
	at org.gradle.initialization.InitScriptHandler$1.run(InitScriptHandler.java:56)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
	at org.gradle.initialization.InitScriptHandler.executeScripts(InitScriptHandler.java:51)
	at org.gradle.initialization.InitScriptHandlingSettingsLoader.findAndLoadSettings(InitScriptHandlingSettingsLoader.java:32)
	at org.gradle.api.internal.initialization.CacheConfigurationsHandlingSettingsLoader.findAndLoadSettings(CacheConfigurationsHandlingSettingsLoader.java:36)
	at org.gradle.initialization.GradlePropertiesHandlingSettingsLoader.findAndLoadSettings(GradlePropertiesHandlingSettingsLoader.java:38)
	at org.gradle.initialization.DefaultSettingsPreparer.prepareSettings(DefaultSettingsPreparer.java:31)
	at org.gradle.initialization.BuildOperationFiringSettingsPreparer$LoadBuild.doLoadBuild(BuildOperationFiringSettingsPreparer.java:71)
	at org.gradle.initialization.BuildOperationFiringSettingsPreparer$LoadBuild.run(BuildOperationFiringSettingsPreparer.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
	at org.gradle.initialization.BuildOperationFiringSettingsPreparer.prepareSettings(BuildOperationFiringSettingsPreparer.java:54)
	at org.gradle.initialization.VintageBuildModelController.lambda$prepareSettings$1(VintageBuildModelController.java:80)
	at org.gradle.internal.model.StateTransitionController.lambda$doTransition$13(StateTransitionController.java:247)
	at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:258)
	at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:246)
	at org.gradle.internal.model.StateTransitionController.lambda$transitionIfNotPreviously$11(StateTransitionController.java:221)
	at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:34)
	at org.gradle.internal.model.StateTransitionController.transitionIfNotPreviously(StateTransitionController.java:217)
	at org.gradle.initialization.VintageBuildModelController.prepareSettings(VintageBuildModelController.java:80)
	at org.gradle.initialization.VintageBuildModelController.prepareToScheduleTasks(VintageBuildModelController.java:70)
	at org.gradle.internal.build.DefaultBuildLifecycleController.lambda$prepareToScheduleTasks$3(DefaultBuildLifecycleController.java:134)
	at org.gradle.internal.model.StateTransitionController.lambda$doTransition$13(StateTransitionController.java:247)
	at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:258)
	at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:246)
	at org.gradle.internal.model.StateTransitionController.lambda$maybeTransition$9(StateTransitionController.java:198)
	at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:34)
	at org.gradle.internal.model.StateTransitionController.maybeTransition(StateTransitionController.java:194)
	at org.gradle.internal.build.DefaultBuildLifecycleController.prepareToScheduleTasks(DefaultBuildLifecycleController.java:132)
	at org.gradle.internal.buildtree.DefaultBuildTreeWorkPreparer.scheduleRequestedTasks(DefaultBuildTreeWorkPreparer.java:36)
	at org.gradle.configurationcache.VintageBuildTreeWorkController$scheduleAndRunRequestedTasks$1.apply(VintageBuildTreeWorkController.kt:36)
	at org.gradle.configurationcache.VintageBuildTreeWorkController$scheduleAndRunRequestedTasks$1.apply(VintageBuildTreeWorkController.kt:35)
	at org.gradle.composite.internal.DefaultIncludedBuildTaskGraph.withNewWorkGraph(DefaultIncludedBuildTaskGraph.java:109)
	at org.gradle.configurationcache.VintageBuildTreeWorkController.scheduleAndRunRequestedTasks(VintageBuildTreeWorkController.kt:35)
	at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.lambda$fromBuildModel$2(DefaultBuildTreeLifecycleController.java:76)
	at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.lambda$runBuild$4(DefaultBuildTreeLifecycleController.java:98)
	... 76 more
Caused by: java.lang.NoSuchMethodError: 'org.gradle.api.provider.Provider org.gradle.api.services.BuildServiceRegistry.registerIfAbsent(java.lang.String, java.lang.Class)'
	at org.nixos.gradle2nix.Gradle2NixPlugin.apply(Gradle2NixPlugin.kt:45)
	at org.nixos.gradle2nix.Gradle2NixPlugin.apply(Gradle2NixPlugin.kt:18)
	at org.gradle.api.internal.plugins.ImperativeOnlyPluginTarget.applyImperative(ImperativeOnlyPluginTarget.java:43)
	at org.gradle.api.internal.plugins.DefaultPluginManager.addPlugin(DefaultPluginManager.java:187)
	at org.gradle.api.internal.plugins.DefaultPluginManager.access$100(DefaultPluginManager.java:52)
	at org.gradle.api.internal.plugins.DefaultPluginManager$AddPluginBuildOperation.run(DefaultPluginManager.java:282)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
	at org.gradle.api.internal.plugins.DefaultPluginManager.lambda$doApply$0(DefaultPluginManager.java:167)
	at org.gradle.configuration.internal.DefaultUserCodeApplicationContext.apply(DefaultUserCodeApplicationContext.java:44)
	at org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:166)
	at org.gradle.api.internal.plugins.DefaultPluginManager.apply(DefaultPluginManager.java:151)
	at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.applyType(DefaultObjectConfigurationAction.java:177)
	at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.applyPlugin(DefaultObjectConfigurationAction.java:161)
	at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.access$100(DefaultObjectConfigurationAction.java:43)
	at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction$2.run(DefaultObjectConfigurationAction.java:87)
	at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.execute(DefaultObjectConfigurationAction.java:190)
	at org.gradle.groovy.scripts.DefaultScript.apply(DefaultScript.java:133)
	at org.gradle.api.Script$apply.callCurrent(Unknown Source)
	at init_9igmt6jvgo5sdpetnkjy70cd9.run(/nix/store/d3ma4j1y9cpb05yfwybvlbz22wjncvyg-gradle2nix-2.0.0/lib/gradle2nix/share/init.gradle:7)
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:91)
	... 141 more

@tadfisher
Copy link
Owner Author

@collinarnett Thanks, we were calling a version of BuildServiceRegistry.registerIfAbsent that was only added in Gradle 8.7. Should be fixed now.

@collinarnett
Copy link

Ok this is great so far! It's listing tasks and I can see build targets. At this point I'm not sure if the error I'm getting is associated with the fact that datahub is extremely imperative with it's use of gradle or a gradle2nix error. Here's the relevant part of the stack trace:

datahub on  master [?] via 🅶 v8.0.2 via ☕ v17.0.7 on ☁️  (us-east-1) on ☁️   took 48s 
$ nix run github:tadfisher/gradle2nix/v2 -- -p . -t :metadata-events:mxe-schemas:build
...
FAILURE: Build failed with an exception.

* What went wrong:
'org.gradle.api.internal.artifacts.ivyservice.ArtifactCacheLockingAccessCoordinator org.gradle.api.internal.artifacts.ivyservice.ArtifactCachesProvider.getWritableCacheAccessCoordinator()'
...
Caused by: org.gradle.internal.exceptions.LocationAwareException: 'org.gradle.api.internal.artifacts.ivyservice.ArtifactCacheLockingAccessCoordinator org.gradle.api.internal.artifacts.ivyservice.ArtifactCachesProvider.getWritableCacheAccessCoordinator()'
...
Caused by: java.lang.NoSuchMethodError: 'org.gradle.api.internal.artifacts.ivyservice.ArtifactCacheLockingAccessCoordinator org.gradle.api.internal.artifacts.ivyservice.ArtifactCachesProvider.getWritableCacheAccessCoordinator()'

@tadfisher
Copy link
Owner Author

@collinarnett This is definitely a gradle2nix problem, Gradle renamed that type for 8.1.0. I'll need to use reflection or create a separate build to target these versions.

@tadfisher
Copy link
Owner Author

@collinarnett I made that change, and tested it on the datahub repo using that task, which produced a lock file.

@collinarnett
Copy link

I tried it on one of the code generation tasks and it works! This is amazing thank you so much for all your hard work. 😄

@lunaticare
Copy link

Hi @tadfisher, I suggested some changes to allow Android app builds #67. Could you please have a look?

@njaremko
Copy link

njaremko commented Jun 21, 2024

EDIT: #69

I'm trying to test this out, and I need to use the overrides feature, but it seems nix is doing something wonky with overrides

I have something like this:

{
  packages.queryrunner = gradle2nix.packages.gradle2nix.buildGradlePackage {
    pname = "someProject";
    version = "1.0";
    lockFile = ../someProject/gradle.lock;
    gradleInstallFlags = [ "installDist" ];
    src = nix-filter {
      root = ../someProject;
      include = [
        "src"
        "local-repo"
        "build.gradle.kts"
        "gradle.properties"
        "settings.gradle.kts"
      ];
      exclude = [
        "build"
        ".gradle"
      ];
    };
    overrides = {
      "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.7.21" = {
        "kotlin-stdlib-jdk8-1.7.21.jar" = _: fetchurl {
          url = "https://repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-jdk8/1.7.21/kotlin-stdlib-jdk8-1.7.21.jar";
          hash = "13nkj6a3x4ygzqbd91f5nvlawpsg0fzqb4s3zqrjfmxhxgkllbmk";
          downloadToTemp = true;
          postFetch = "install -Dt $out/org/jetbrains/kotlin/kotlin-stdlib-jdk8/1.7.21/ $downloadedFile";
        };
      };
    };
  };
}

But I'm getting:

error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:9:12:
            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'someProject-1.0'
         whose name attribute is located at /nix/store/dydg48djlykksz8cxq0xjplyxpa9pvf4-source/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'overrides' of derivation 'someProject-1.0'
         at /nix/store/nj21vkq6wrskcw88c8slf2nr1mh06p6w-source/nix/someProject.nix:26:9:
           25|         };
           26|         overrides = {
             |         ^
           27|           "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.7.21" = {

       error: cannot coerce a set to a string: { "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.7.21" = «thunk»; }

sourceRoot = ".";

preferLocalBuild = true;
allowSubstitutes = false;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we do this? These seem to just be hashed binaries, and it forces people to manually build every dependency instead of download from binary cache

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the equivalent of what runCommandLocal or fetchurl does; in general, it doesn't make sense to have the substituter perform a fetch, then fetch the mostly-unchanged result from the substituter. You also want to know sooner rather than later that your ultimate source of truth (usually one or more Maven repositories) is providing stable downloads, otherwise you get problems reproducing builds (as you intentionally bypass substituters to do so).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For posterity: https://nix.dev/manual/nix/2.22/language/advanced-attributes.html?highlight=allowsubstitutes#adv-attr-allowSubstitutes

I forked this branch and tried it with allowSubstitues = true;, but it didn't cause me to download everything from my binary cache anyway, so I no longer feel strongly either way on this :P

@milahu
Copy link

milahu commented Jul 9, 2024

+1 for #69

fixme: attributes like gradleBuildFlags or gradleInstallFlags
should have no effect on the mavenRepo derivation

todo: split building and linking into 2 derivations
so once i have the main project's jar files
i can modify propagatedOverrides to fix runtime dependencies
without having to rebuild the main project

todo: add a generic mkOverride function

part of haveno.nix

      mkOverride = src: args: stdenvNoCC.mkDerivation ({
        name = src.name;
        nativeBuildInputs = [
          autoPatchelfHook
          #zip
          jdk
        ];
        dontAutoPatchelf = true;
        unpackPhase = ''
          ext=''${name##*.}
          runHook preUnpack
          sourceRoot=source
          mkdir $sourceRoot
          cd $sourceRoot
          libs=(${if (args.libs or []) == false then "" else lib.escapeShellArgs (args.libs or [])})
          doLibs=${if (args.libs or []) == false then "false" else "true"}
          bins=(${if (args.bins or []) == false then "" else lib.escapeShellArgs (args.bins or [])})
          archives=(${if (args.archives or []) == false then "" else lib.escapeShellArgs (args.archives or [])})
          doArchives=${if (args.archives or []) == false then "false" else "true"}
          archive_libs=()
          archive_bins=()
          if [ "$ext" = jar ]; then
            #echo name $name; echo src ${src}; echo src_nix ${src}; exit 1
            #unzip ${src}
            echo "unpacking ${src}"
            if $doLibs && [ ''${#libs[@]} == 0 ]; then
              while IFS= read -r lib; do
                libs+=("$lib")
              done < <(jar tf ${src} | grep '\.so$')
            fi
            if $doArchives && [ ''${#archives[@]} == 0 ]; then
              while IFS= read -r archive; do
                archives+=("$archive")
              # TODO more archive types
              done < <(jar tf ${src} | grep -E '\.(tar(\.(xz|gz|bz|bz2))?|zip)$')
            fi
            jar xf ${src} ''${libs[@]} ''${bins[@]} ''${archives[@]}
            idx=0
            for archive in ''${archives[@]}; do
              echo "unpacking $archive"
              archive_file="''${archive##*/}"
              archive_file="$TMP/temp_''${archive_file: -100}"
              mv "$archive" "$archive_file"
              archive_dir="$archive"
              mkdir "$archive_dir"
              # TODO more archive types
              # TODO generic unpack: zip, 7z, ...
              tar xf "$archive_file" -C "$archive_dir"
              rm "$archive_file"
              while IFS= read -r lib; do
                archive_libs+=("$lib")
              done < <(find "$archive_dir" -name '*.so*' -not -type d)
              while IFS= read -r bin; do
                archive_bins+=("$bin")
              done < <(find "$archive_dir" -executable -not -type d -not -name '*.so*')
            done
          elif [ "$ext" = exe ]; then
            cp ${src} $name
            chmod +w $name
          fi
          cd $NIX_BUILD_TOP
          runHook postUnpack
        '';
        buildPhase = ''
          runHook preBuild
          unpinLibs=$(
            for lib_out in ${lib.escapeShellArgs (args.unpinLibs or [])}; do
              #echo "lib_out $lib_out" >&2 # debug
              for lib in $lib_out/lib/*.so*; do
                # echo "lib_out lib $lib" >&2 # debug
                lib=''${lib##*/}
                lib=''${lib%.so*}.so
                echo "$lib"
              done
            done | sort -u
          )
          # echo unpinLibs: $unpinLibs # debug
          if [ "$ext" = jar ]; then
            # unpin libs
            echo "unpinning libs"
            for lib in "''${libs[@]}" "''${archive_libs[@]}"; do
              echo "$lib: unpinning"
              args=(patchelf)
              for dep in $(patchelf --print-needed $lib); do
                unp=''${dep%.so*}.so
                if echo "$unpinLibs" | grep -q -m1 -x -F "$unp"; then
                  args+=(--replace-needed "$dep" "$unp")
                  echo "$lib: unpinning $dep to $unp"
                fi
              done
              "''${args[@]}" "$lib"
            done
            autoPatchelf .
          elif [ "$ext" = exe ]; then
            chmod +x $name
            autoPatchelf $name
          fi
          runHook postBuild
        '';
        installPhase = ''
          runHook preInstall
          if [ "$ext" = jar ]; then
            cp ${src} $out
            chmod +w $out
            for archive in ''${archives[@]}; do
              echo "packing $archive"
              archive_dir="$archive"
              archive_file="$TMP/temp.tar"
              pushd "$archive_dir" >/dev/null
              case "$archive" in
                *.tar.gz|*.tar.xz|*.tar.bz|*.tar.bz2)
                  tar cf "$archive_file" .
                ;;
                *)
                  echo "FIXME internal error: unhandled archive type $archive"
                  exit 1
                ;;
              esac
              popd >/dev/null
              # dont use "tar cJf" as that is slooow
              # use minimal compression to make this fast
              case "$archive" in
                *.tar.gz) gzip -1 "$archive_file"; archive_file+=".gz";;
                *.tar.xz) xz -1 "$archive_file"; archive_file+=".xz";;
                *.tar.bz2) bzip2 -1 "$archive_file"; archive_file+=".bz2";;
                # TODO more archive types
                *)
                  echo "FIXME internal error: unhandled archive type $archive"
                  exit 1
                ;;
              esac
              stat "$archive_file" || true
              stat "$archive_dir" || true
              rm -rf "$archive_dir"
              # FIXME mv: cannot move '/build/temp.tar.xz' to 'native/linux/x64/tor.tar.xz': No such file or directory
              mv "$archive_file" "$archive"
            done
            jar uf $out ''${libs[@]} ''${bins[@]} ''${archives[@]}
          elif [ "$ext" = exe ]; then
            cp $name $out
          fi
          runHook postInstall
        '';
      } // (builtins.removeAttrs args [ "libs" "bins" "unpinLibs" "archives" ]));

overrides should allow to replace the fetchurl call
to avoid extra derivations
to avoid extra network requests

nix/build-maven-repo.nix

{
  fetch =
    coords: overrides: name:
    { url, hash }:
    let
      scheme = head (builtins.match "([a-z0-9+.-]+)://.*" url);
      fetch' = getAttr scheme fetchers';
      artifact = fetch' { inherit url hash; };
      override = overrides.${name} or lib.id;
    in
    override artifact;

should be something like

{
  fetch =
    coords: overrides: name:
    { url, hash }:
    let
      scheme = head (builtins.match "([a-z0-9+.-]+)://.*" url);
      fetch' = getAttr scheme fetchers';
      fetchArgs = { inherit url hash; };
      artifact = fetch' fetchArgs;
    in
    if overrides ? name then
      if overrides.${name} == null then null else
      overrides.${name} fetchArgs
    else
    artifact;

setting an override to null ignores the dependency
this case would also have to be handled by the caller of fetch

now trivial transformations like chmod +x can be implemented with postFetch

{
  overrides = {
    "io.grpc:protoc-gen-grpc-java:1.42.1" = {
      "protoc-gen-grpc-java-1.42.1-linux-x86_64.exe" = args: fetchurl (args // {
        postFetch = "chmod +x $out";
      });
    };
  };

maybe overrides should allow
to replace one dependency with multiple dependencies

for example an override function could return a list of sets

[
  {
    pkg = "io.grpc:protoc-gen-grpc-java:1.42.1";
    # the filename is taken from out.name
    out = fetchurl { /* ... */ };
  }
  # ...
]

maybe there should be an option to transform the parsed lockfile set
to allow arbitrary dynamic modifications of the lockfile
to avoid double-parsing the lockfile

gradle2nix.buildGradlePackage rec {
  transformLockfile = lockfile: lockfile // {
    "somepkg" = {
      "somefile" = { /* somespec */ };
    };
  };

mavenRepo = symlinkJoin { ... } in nix/build-maven-repo.nix
should accept a postBuild option
so we can modify the gradle-maven-repo derivation

example override: fix executable file protoc-gen-grpc-java
problem: verification fails because the checksum changes
quickfix: gradle --dependency-verification=off

part of haveno.nix

gradle2nix.buildGradlePackage rec {
  overrides = {
    # fix: Execution failed for task ':proto:generateProto'.
    # Cannot set /nix/store/.../protoc-gen-grpc-java-1.42.1-linux-x86_64.exe as executable
    "io.grpc:protoc-gen-grpc-java:1.42.1" = {
      "protoc-gen-grpc-java-1.42.1-linux-x86_64.exe" = src: stdenvNoCC.mkDerivation {
        name = src.name;
        dontUnpack = true;
        nativeBuildInputs = [
          autoPatchelfHook
        ];
        buildPhase = ''
          cp ${src} $out
          chmod +x $out
        '';
      };
    };
  };

error: verification fails

Execution failed for task ':proto:generateProto'.
> Dependency verification failed for configuration ':proto:protobufToolsLocator_grpc'
  One artifact failed verification: protoc-gen-grpc-java-1.42.1-linux-x86_64.exe (io.grpc:protoc-gen-grpc-java:1.42.1) from repository maven2
  This can indicate that a dependency has been compromised. Please carefully verify the checksums.

fix: patch checksum in gradle/verification-metadata.xml

ideally gradle2nix should set the mavenRepo variable

  postPatch = ''
    # TODO gradle2nix should set this variable
    mavenRepo=$(grep -m1 -F "repo.url 'file:/" $gradleInitScript | sed -E "s/^.*'file:(.*)'.*/\1/")
    # fix: Dependency verification failed
    xmlstarlet edit --inplace \
      --update '//_:artifact[@name="protoc-gen-grpc-java-1.42.1-linux-x86_64.exe"]/_:sha256/@value' \
      --value $(sha256sum $mavenRepo/io/grpc/protoc-gen-grpc-java/1.42.1/protoc-gen-grpc-java-1.42.1-linux-x86_64.exe | head -c64) \
      gradle/verification-metadata.xml
  '';

unnecessary nondeterminism in nix/setup-hook.sh
export GRADLE_USER_HOME="$(mktemp -d)"
should be
export GRADLE_USER_HOME="$TMP/gradle"

example: patch paths to protobuf executables in build.gradle

let
  protobuf3_19 = callPackage (<nixpkgs> + "/pkgs/development/libraries/protobuf/generic-v3.nix") {
    version = "3.19.1";
    sha256 = "sha256-IQAlnpsO3AYfzXVnIHxLOKo1XzdWDmwwv+W/OanAl+s=";
  };
in

# fix: error: cannot find symbol: method parseUnknownField
# https://github.com/protocolbuffers/protobuf/issues/10695
let protobuf = protobuf3_19; in

gradle2nix.buildGradlePackage rec {
  postPatch = ''
    substituteInPlace build.gradle \
      --replace-fail \
        'artifact = "com.google.protobuf:protoc:' \
        'path = "${protobuf}/bin/protoc" //' \
      --replace-fail \
        'artifact = "io.grpc:protoc-gen-grpc-java:' \
        'path = "${grpc-java}/bin/protoc-gen-grpc-java" //' \
  '';

grpc-java-bin.nix

{ lib
, stdenvNoCC
, fetchurl
#, protobuf
, autoPatchelfHook
}:

stdenvNoCC.mkDerivation rec {
  pname = "grpc-java";
  #version = "1.65.0";
  version = "1.42.1";

  # io.grpc:protoc-gen-grpc-java
  src = fetchurl {
    url = "https://repo.maven.apache.org/maven2/io/grpc/protoc-gen-grpc-java/${version}/protoc-gen-grpc-java-${version}-linux-x86_64.exe";
    hash = "sha256-VZa2nrae1Qi8aTB/FDQL5wK97i/w7TPByv3gKu/J+q8=";
  };

  dontUnpack = true;

  #dontStrip = true;

  nativeBuildInputs = [
    autoPatchelfHook
  ];

  #dontPatchELF = true;

  buildPhase = ''
    mkdir -p $out/bin
    cp $src $out/bin/protoc-gen-grpc-java
    chmod +x $out/bin/protoc-gen-grpc-java
  '';

  meta = with lib; {
    description = "The Java gRPC implementation. HTTP/2 based RPC [binary build]";
    homepage = "https://github.com/grpc/grpc-java";
    license = licenses.asl20;
    maintainers = with maintainers; [ ];
    mainProgram = "protoc-gen-grpc-java";
    platforms = platforms.all;
  };
}

fixme: copy fails when the destination file exists and is read-only gradle/gradle#1544
this happens when gradle copies files from the nix store
for example .jar files from mavenRepo

Execution failed for task ':relay:installDist'.
> Could not copy file '/build/source/build/lib/some-file-1.2.3.jar' to '/build/source/lib/some-file-1.2.3.jar'.
   > /build/source/lib/some-file-1.2.3.jar (Permission denied)

... maybe some of these snippets belong to https://nixos.wiki/wiki/Gradle

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants