Skip to content

feat: new launcher using Apache Mina sshd library #570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

kuisathaverat
Copy link
Contributor

@kuisathaverat kuisathaverat commented Apr 6, 2025

This pull request introduces several changes to enhance SSH connection management, update dependencies, and improve code formatting. The most notable updates include the addition of new Java interfaces and classes for SSH functionality, modifications to the pom.xml file to refine dependencies, and updates to .editorconfig and Jenkinsfile for better code consistency and build configuration.

SSH Connection Management Enhancements:

  • src/main/java/io/jenkins/plugins/sshbuildagents/ssh/Connection.java: Added a new interface to manage SSH connections, including methods for executing commands, transferring files, and configuring connection parameters.
  • src/main/java/io/jenkins/plugins/sshbuildagents/ssh/KeyAlgorithm.java, KeyAlgorithmManager.java, KnownHosts.java, ServerHostKeyVerifier.java, ShellChannel.java: Introduced new classes and interfaces for managing key algorithms, host verification, and non-interactive SSH sessions. [1] [2] [3] [4] [5]
  • src/main/java/io/jenkins/plugins/sshbuildagents/ssh/mina/FakeURI.java, KeyAlgorithmManagerImpl.java: Added classes for testing and implementing key algorithm management. [1] [2]

Dependency Updates:

  • pom.xml: Refined dependencies by adding new plugins (eddsa-api, mina-sshd-api) and excluding redundant ones (trilead-api). Introduced placeholders for future dependency removals. [1] [2]

Code Formatting and Build Configuration:

  • .editorconfig: Updated rules for *.java files, specifying 4-space indentation, LF line endings, and UTF-8 charset.
  • Jenkinsfile: Changed the Windows build configuration to use JDK 21 instead of JDK 17.

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests - that demonstrates feature works or fixes the issue

JENKINS-64106

@kuisathaverat kuisathaverat force-pushed the apache_mina branch 2 times, most recently from db78481 to beb4353 Compare April 6, 2025 17:35
@kuisathaverat kuisathaverat self-assigned this Apr 6, 2025
@kuisathaverat
Copy link
Contributor Author

kuisathaverat commented Apr 7, 2025

Not sure about the CI errors, 106 test run, and there is no failures. It fails when it try to make the Surefire report

[2025-04-07T07:35:51.011Z] [INFO] 
[2025-04-07T07:35:51.011Z] [INFO] Results:
[2025-04-07T07:35:51.012Z] [INFO] 
[2025-04-07T07:35:51.012Z] [WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 11
[2025-04-07T07:35:51.012Z] [INFO] 
[2025-04-07T07:35:51.012Z] [INFO] ------------------------------------------------------------------------
[2025-04-07T07:35:51.012Z] [INFO] BUILD FAILURE
[2025-04-07T07:35:51.012Z] [INFO] ------------------------------------------------------------------------
[2025-04-07T07:35:51.012Z] [INFO] Total time:  40:50 min
[2025-04-07T07:35:51.012Z] [INFO] Finished at: 2025-04-07T07:35:50Z
[2025-04-07T07:35:51.013Z] [INFO] ------------------------------------------------------------------------
[2025-04-07T07:35:51.013Z] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.5.2:test (default-test) on project ssh-slaves: 
[2025-04-07T07:35:51.013Z] [ERROR] 
[2025-04-07T07:35:51.013Z] [ERROR] See /home/jenkins/agent/workspace/Plugins_ssh-agents-plugin_PR-570/target/surefire-reports for the individual test results.
[2025-04-07T07:35:51.013Z] [ERROR] See dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[2025-04-07T07:35:51.013Z] [ERROR] ExecutionException There was an error in the forked process
[2025-04-07T07:35:51.013Z] [ERROR] [SUREFIRE] std/in stream corrupted
[2025-04-07T07:35:51.014Z] [ERROR] java.io.IOException: Stream closed
[2025-04-07T07:35:51.014Z] [ERROR] 	at java.base/java.io.BufferedInputStream.ensureOpen(BufferedInputStream.java:206)
[2025-04-07T07:35:51.014Z] [ERROR] 	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:411)
[2025-04-07T07:35:51.014Z] [ERROR] 	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)
[2025-04-07T07:35:51.014Z] [ERROR] 	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)
[2025-04-07T07:35:51.014Z] [ERROR] 	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
[2025-04-07T07:35:51.015Z] [ERROR] 	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
[2025-04-07T07:35:51.015Z] [ERROR] 	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)
[2025-04-07T07:35:51.015Z] [ERROR] 	at org.apache.maven.surefire.api.util.internal.Channels$3.readImpl(Channels.java:169)
[2025-04-07T07:35:51.015Z] [ERROR] 	at org.apache.maven.surefire.api.util.internal.AbstractNoninterruptibleReadableChannel.read(AbstractNoninterruptibleReadableChannel.java:50)
[2025-04-07T07:35:51.015Z] [ERROR] 	at org.apache.maven.surefire.api.stream.AbstractStreamDecoder.read(AbstractStreamDecoder.java:430)
[2025-04-07T07:35:51.015Z] [ERROR] 	at org.apache.maven.surefire.api.stream.AbstractStreamDecoder.read(AbstractStreamDecoder.java:419)
[2025-04-07T07:35:51.015Z] [ERROR] 	at org.apache.maven.surefire.api.stream.AbstractStreamDecoder.readMessageType(AbstractStreamDecoder.java:116)
[2025-04-07T07:35:51.016Z] [ERROR] 	at org.apache.maven.surefire.booter.stream.CommandDecoder.decode(CommandDecoder.java:77)
[2025-04-07T07:35:51.016Z] [ERROR] 	at org.apache.maven.surefire.booter.spi.CommandChannelDecoder.decode(CommandChannelDecoder.java:60)
[2025-04-07T07:35:51.016Z] [ERROR] 	at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:290)
[2025-04-07T07:35:51.016Z] [ERROR] 	at java.base/java.lang.Thread.run(Thread.java:1583)

@jglick
Copy link
Member

jglick commented Apr 9, 2025

Are any tests attempting to write to stdout/stderr (not using System.out / System.err)? I recall that for example https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/ProcessBuilder.html#inheritIO() can crash Surefire.

@kuisathaverat
Copy link
Contributor Author

I do not remember any, but I will review it.

@kuisathaverat
Copy link
Contributor Author

kuisathaverat commented Apr 14, 2025

Jesse was right; the cause was a bunch of outputs to System.err and System.out. Now, I have to resolve a few SpotBugs warnings.

@kuisathaverat kuisathaverat force-pushed the apache_mina branch 11 times, most recently from 0b2fcc4 to df049bb Compare June 9, 2025 17:48
@kuisathaverat kuisathaverat marked this pull request as ready for review June 10, 2025 05:42
@kuisathaverat kuisathaverat requested a review from a team as a code owner June 10, 2025 05:42
@kuisathaverat
Copy link
Contributor Author

kuisathaverat commented Jun 10, 2025

This is the initial implementation; it lacks host key verifications and other features the current launcher has. These features will be added in follow-up PRs.

@jglick
Copy link
Member

jglick commented Jun 20, 2025

Did you mean to update the PR title?

@kuisathaverat kuisathaverat changed the title test: implement launcher using Apache Mina sshd library feat: new launcher using Apache Mina sshd library Jun 22, 2025
*/
public boolean getTrackCredentials() {
String trackCredentials = System.getProperty(SSHApacheMinaLauncher.class.getName() + ".trackCredentials");
return !"false".equalsIgnoreCase(trackCredentials);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe trim trackCredentials just in case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, if the user is not able to put "true" or "false" in a system property, it is not setting the property correctly, so the property is invalid. You cannot manage all the wrong property values that a user can give you, and only make sense to fix their errors when they are common (which denotes there is another issue).
To provide a io.jenkins.plugins.sshbuildagents.ssh.mina.SSHApacheMinaLauncher.trackCredentials with a value that you must trim. You have to provide something like -Dio. Jenkins.plugins.sshbuildagents.ssh.mina.SSHApacheMinaLauncher.trackCredentials=" true " that it is enough bizarre to do not thing anyone is so dumb.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use SystemProperties instead please.

/*
* The MIT License
*
* Copyright (c) 2016, Michael Clarke
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why you use this copyright here?
It looks to be a new class written by you. Either you put yourself or nobody but not someone who was not part of writing this code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C&P from the old class, so I keep the same header

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will review it, as it is likely new content, even though I started by copying it from the old file.

</dependency>
</dependencies>
</dependencyManagement>

<dependencies>
<dependency>
<groupId>io.jenkins.plugins</groupId>
<artifactId>eddsa-api</artifactId>
Copy link
Member

@olamy olamy Jun 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we remove this dependency and use this new feature from mina-sshd see apache/mina-sshd#639
The project https://github.com/str4d/ed25519-java is not maintained anymore and has some known security issues such str4d/ed25519-java#95

import org.apache.sshd.common.config.keys.loader.pem.PKCS8PEMResourceKeyPairParser;
import org.apache.sshd.common.config.keys.loader.pem.RSAPEMResourceKeyPairParser;
import org.apache.sshd.common.util.security.bouncycastle.BouncyCastleKeyPairResourceParser;
import org.apache.sshd.common.util.security.eddsa.Ed25519PEMResourceKeyParser;
Copy link
Member

@olamy olamy Jun 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be good to get rid of net.i2p.crypto.eddsa with using BC

final Node node = computer.getNode();
final String host = this.host;
final int port = this.port;
if (computer == null || listener == null) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure it's needed as both have @NonNull annotation :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I struggle to remove a warning from SpotBugs; this is a leftover. I will remove it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no worries. not a big blocking problem

@olamy
Copy link
Member

olamy commented Jun 26, 2025

Nice PR.
Some questions:

  • should we maintain both implementations?
    Maybe it could be more simple (and less maintenance) to have a single implementation based on Apache Mina?
  • Maybe we could get rid of trilead as well?

this.j.timeout = 0;
}

@BeforeAll
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sonds like a bit redundant with @Testcontainers(disabledWithoutDocker = true)?

@kuisathaverat
Copy link
Contributor Author

Nice PR. Some questions:

  • should we maintain both implementations?

At least until we are confident with this new launcher, it lacks some features the other has, like host verification.

Maybe it could be more simple (and less maintenance) to have a single implementation based on Apache Mina?

  • Maybe we could get rid of trilead as well?

That's the goal; I will stop accepting and processing changes in Trilead because I will focus my time on this new launcher and a way to migrate the old configuration to the latest in a seamless manner. It will take a while for sure.

olamy
olamy previously requested changes Jun 27, 2025
List<PosixFilePermission> permissions = new ArrayList<>();
permissions.add(PosixFilePermission.GROUP_READ);
permissions.add(PosixFilePermission.OWNER_WRITE);
scp.upload(bytes, remotePath, permissions, null);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to ensure remote path exists first

try (ClientSession session = connect()) {
DefaultScpClient scp = new DefaultScpClient(session);
List<PosixFilePermission> permissions = new ArrayList<>();
permissions.add(PosixFilePermission.GROUP_READ);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you sure we want to force such perms. Remote server may have mask in place with different permissions. (sftp could simplify as you will not need to set permissions)


@BeforeAll
static void beforeAll() {
assumeFalse(SystemUtils.IS_OS_WINDOWS);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about annotations at class level such @DisabledOnOs(OS.WINDOWS)

@@ -0,0 +1,42 @@
FROM ubuntu:14.04
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file doesn't seem to be used in any test?

@@ -0,0 +1,42 @@
FROM ubuntu:16.04
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto
this file doesn't seem to be used in any test?

@@ -0,0 +1,42 @@
FROM ubuntu:18.04
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

ConnectFuture connectionFuture = client.connect(this.credentials.getUsername(), this.host, this.port);
connectionFuture.verify(this.timeoutMillis);
session = connectionFuture.getSession();
addAuthentication();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should be able to use something

SSHAuthenticator.newInstance(session, credentials).authenticate(TaskListener.NULL)

/*
* The MIT License
*
* Copyright (c) 2016, Michael Clarke
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

/*
* The MIT License
*
* Copyright (c) 2016, Michael Clarke
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

/*
* The MIT License
*
* Copyright (c) 2016, Michael Clarke
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

/*
* The MIT License
*
* Copyright (c) 2016, Michael Clarke
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@@ -0,0 +1,26 @@
package io.jenkins.plugins.sshbuildagents.ssh.mina;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing license but we probably will not need this file anymore

/*
* The MIT License
*
* Copyright (c) 2016, Michael Clarke
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

/*
* The MIT License
*
* Copyright (c) 2016, Michael Clarke
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto.

but not sure we really need this file

/*
* The MIT License
*
* Copyright (c) 2004-, all the contributors
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah :)


@Before
public void setup() throws IOException {
Logger.getLogger("org.apache.sshd").setLevel(Level.FINE);
Copy link
Member

@olamy olamy Jun 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test is really slow as the same server is restarted for every single test.
why not using test container here to avoid some collision between server and client code (normally shouldn't be but...)
well I guess the main issue is more the 5 minutes loop :)

connection.setCredentials(credentials);
ShellChannel shellChannel = connection.shellChannel();
shellChannel.execCommand("sleep 500s");
for (int i = 0; i < 300; i++) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks that's plenty of time to have a coffee :)
do we really need this? Can't we skip this on normal dev life and run it on CI server only?

@olamy
Copy link
Member

olamy commented Jun 27, 2025

Right because it's too easy to talk and do nothing, I have started to put my comments into code with this kuisathaverat#102
Obviously some tests are failing, as now ConnectionImpl is using the existing authz mod,e which doesn't work without Jenkins

@kuisathaverat
Copy link
Contributor Author

I will address some issues and then merge them as is. This PR is a WIP new launcher, and this is the first iteration. It is not perfect, and it should not be. I do not want to make this PR an open issue that remains unmerged. Also, it helps to work on different improvements in parallel and track the changes better

@kuisathaverat kuisathaverat dismissed olamy’s stale review June 27, 2025 13:35

The proposed changes are opposed to the structure I have planned for the launcher, or are not in the scope of this PR.

@olamy
Copy link
Member

olamy commented Jun 28, 2025

I will address some issues and then merge them as is. This PR is a WIP new launcher, and this is the first iteration. It is not perfect, and it should not be. I do not want to make this PR an open issue that remains unmerged. Also, it helps to work on different improvements in parallel and track the changes better

If you do so, you must at least mark and inform users about the experimental aspect of your PR when they use the new launcher because many features are not working now.

I completely understand the rationale behind adding an abstraction layer on top of the SSH connection. In theory, it makes sense and can offer flexibility.
That said, I'm wondering if it's really necessary in the specific context of the Jenkins SSH Agents plugin. Do we foresee switching from Apache Mina to another implementation anytime soon (while it took already so long having some changes out of trilead :))? From what I can see, both the SSH protocol and the Apache Mina library are quite stable and well-maintained, especially when it comes to bugs and security.

Introducing a new abstraction would mean committing to a public API that we'll need to support in the long term. Given that this is currently scoped just to the Jenkins SSH Agents plugin. I'm not sure the added complexity is justified.

Also, I'm a bit hesitant about re-implementing the authentication layer from scratch. The existing solution is widely used, battle-tested, and already aligns with security requirements.
https://github.com/jenkinsci/mina-sshd-api-plugin/tree/main/mina-sshd-api-core/src/main/java/io/jenkins/plugins/mina_sshd_api/core/authenticators

image

I wish to have you open to comments as definitely such changes need some discussion and acceptance by the broader community, as this has some large effect on the long term maintenance of the plugin

*
* @author Ivan Fernandez Calvo
*/
public interface Connection extends AutoCloseable {
Copy link
Member

@olamy olamy Jun 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need such abstraction here? maybe for simplicity we could use directly a class using Apache Mina.
I can't imagine to what other libraries we could switch?
At least this should be marked as internal (in the package name?) or clearly documented in the Javadoc as internal. Otherwise we have to commit to support this as a long term API (is it the goal of this Jenkins plugin?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants