File is getting corrupted when Chunk Mode file transfer interrupted #315

anamikagsingh · 2023-04-26T11:12:23Z

Hi,

This change will fix the Chunk mode issue raised in #308

Regards,
Anamika

norrisjeremy · 2023-04-27T11:39:40Z

Hi @anamikagsingh,

Can you execute mvn formatter:format so that the code changes are formatted correctly?
Can you provide more details on the issue you are seeing and how your proposed changes help to resolve it?
-- Specifically, can you describe the overall interaction that a JSch consumer would utilize in order to make use of the new checkChunkFailed() & setChunkStart() methods?
-- It may help if you can provide some short code examples that depict how the file corruption can occur during an interrupted transfer, and then how a user would mitigate the file corruption using the proposed changes.

Thanks,
Jeremy

anamikagsingh · 2023-05-02T07:56:47Z

Hi Jeremy,

Could you please let me know if I need to format the file and commit once again or it can be done on committed file too?

In our application, we have an option to transfer large files in smaller chunks and append them in the target in the sequence they have been transferred. But in case the transfer of chunk to the target directory is interrupted due to any issues (i.e., network failure, engine restart etc.), it will transfer the complete chunk again and as a result, the target file gets corrupted.

To solve this, chunk transfer should be resumed instead of transferring the complete chunk again. And, to achieve this, we added a method checkChunkFailed() in ChannelSftp class which checks the bytes which is already transferred and accordingly set the information whether the chunk transfer is interrupted or not. Also, in SFTP application, we have used the below method which will try to resume the chunk transfer if transfer of chunk to the target directory is interrupted.

public void checkChunkTransferInterrupted(byte[] file, String absoluteFilePath, String chunk_start){
String SIGNATURE = "checkChunkTransferInterrupted(byte[] file, String absoluteFilePath, String chunk_start)";
TRACE.entering(SIGNATURE);
InputStream is = new ByteArrayInputStream(file);
try {
	if(sftpChannel.checkChunkFailed(is, absoluteFilePath, chunk_start)){
	setChunkFailed(true, chunk_start);
	TRACE.warningT(SIGNATURE, "Chunk Mode: Chunk transfer was interrupted");
	TRACE.debugT(SIGNATURE, "Chunk Start: " + chunk_start);
	}
} catch (SftpException e) {
	TRACE.warningT(SIGNATURE, "Chunk Mode: Error while checking if chunk transfer interrupted -"+ e.toString()+ ".Continue without check for chunk transfer interruption.");
}
TRACE.exiting(SIGNATURE);
}

public void setChunkFailed(boolean chunk_failed, String chunk_start){
this.chunk_failed = chunk_failed;
sftpChannel.setChunkStart(chunk_start);
}

anamikagsingh · 2023-05-02T09:15:49Z

Hi Jeremy,

I also have added changed files 'Session.java' and 'UserAuthKeyboardInteractive.java' to this pull request. This change is to fix the issue raised in #302 and also to support multiple server fingerprint.

Regards,
Anamika

norrisjeremy · 2023-05-02T10:40:55Z

Hi @anamikagsingh,

You need to execute mvn formatter:format and commit, then push the resulting changes to this PR.
Can you please split the changes to Session.java and UserAuthKeyboardInteractive.java into individual commits, instead of a single combined commit?
Can you modify your commit messages to be more descriptive than simply ChannelSftp and SessionAndUserAuthKeyboardInteractive?
I'm not sure I follow what you mean by we have an option to transfer large files in smaller chunks and append them in the target in the sequence they have been transferred: how are you transferring these large files via smaller chunks using JSch's existing ChannelSftp API methods?
-- I also do not understand how an application would determine the chunkStart value to use for the checkChunkFailed() method: can you better explain how you are determining this value?
I'm not sure I understand the purpose of the changes to Session.java: can't you implement this custom logic of authenticating hosts by their host key fingerprint by simply implementing your own custom HostKeyRepository and then instructing JSch use it (via either jsch.setHostKeyRepository() or session.setHostKeyRepository())?
I'm not sure I understand the proposed changes to UserAuthKeyboardInteractive.java: why not simply implement a custom UserInfo as suggested in UnsupportedOperationException occurs when password prompt is in a different format #302?
-- Also, you mentioned in UnsupportedOperationException occurs when password prompt is in a different format #302 that you had a server that presented a prompt of Enter password:, but doesn't that prompt already work without any changes since that should already match the existing prompt[0].toLowerCase().indexOf("password:") >= 0?

Thanks,
Jeremy

src/main/java/com/jcraft/jsch/Session.java

norrisjeremy · 2023-05-02T12:09:49Z

src/main/java/com/jcraft/jsch/UserAuthKeyboardInteractive.java

@@ -129,7 +129,8 @@ public boolean start(Session session) throws Exception {
          byte[][] response = null;

          if (password != null && prompt.length == 1 && !echo[0]
-              && prompt[0].toLowerCase().indexOf("password:") >= 0) {
+              && (prompt[0].toLowerCase().indexOf("password:") >= 0
+            		 || prompt[0].toLowerCase().indexOf("password") >= 0)){


In #302 you mention that this change is needed in order to support a server that has a prompt of Enter password:.
However, the string Enter password: already should match prompt[0].toLowerCase().indexOf("password:") >= 0, so I'm not sure I understand the purpose of this change (see below)?

jshell> String foo = "Enter password:"; foo ==> "Enter password:" jshell> if (foo.toLowerCase().indexOf("password:") >= 0) System.out.println("matches"); else throw new Exception("fail"); matches jshell>

I am not able to reproduce this issue in my test system so cannot test it. We can leave the changes done in UserAuthKeyboardInteractive.java for now. Please let me know if I need to revert the changes and commit again. Thanks.

If you are not able to reproduce the issue that you believed was fixed by this change, then it should be removed.

The or also overlaps, you could just remove the colon..

norrisjeremy · 2023-05-02T12:11:55Z

src/main/java/com/jcraft/jsch/ChannelSftp.java

@@ -643,6 +653,32 @@ public void _put(InputStream src, String dst, SftpProgressMonitor monitor, int m
        throw (SftpException) e;
      throw new SftpException(SSH_FX_FAILURE, e.toString(), e);
    }
+
+  public boolean checkChunkFailed(InputStream src, String dst, String ChunkStart)throws SftpException{


The argument InputStream src appears to be unused in this method, so why is it needed?

The argument String ChunkStart should be changed to not start with a capital letter.
-- Perhaps something like String chunkStart instead?

Removed the argument InputStream src from the method.

Changed the parameter name to startChunk

norrisjeremy · 2023-05-02T12:16:55Z

src/main/java/com/jcraft/jsch/ChannelSftp.java

@@ -32,6 +32,9 @@
 import java.util.Hashtable;
 import java.util.Vector;

+import com.jcraft.jsch.Channel.MyPipedInputStream;


This is unnecessary.
Since ChannelSftp extends Channel, it can directly reference the MyPipedInputStream class without an import statement.

Removed the import as it was not needed. Thanks for the suggestions.

norrisjeremy · 2023-05-02T12:18:37Z

src/main/java/com/jcraft/jsch/ChannelSftp.java

+  	    	throw new SftpException(SSH_FX_FAILURE, "failed to resume for "+dst);
+  	    }
+  	 }
+  	else if(mode==RESUME && skip>0){


I think this could be better written as simply something like this to avoid code duplication:

if (mode == RESUME && skip > 0) { long srcSkip = chunkStart != null ? skip - Long.parseLong(chunkStart) : skip; long skipped = src.skip(srcSkip); if (skipped < srcSkip) { throw new SftpException(SSH_FX_FAILURE, "failed to resume for " + dst); } }

I have made the changes as suggested.

norrisjeremy · 2023-05-02T12:19:25Z

src/main/java/com/jcraft/jsch/ChannelSftp.java

@@ -158,6 +161,7 @@ public class ChannelSftp extends ChannelSession {
  private boolean fEncoding_is_utf8 = true;

  private RequestQueue rq = new RequestQueue(16);
+  private String chunkStart = null;


Why does chunkStart need to be a String instead of having it simply be a long?
This would avoid a lot of unwieldy calls to Long.parseLong(chunkStart)?

Changed it to Long now and made the other changes accordingly.

norrisjeremy · 2023-05-02T12:22:02Z

src/main/java/com/jcraft/jsch/Session.java

+    			  break;
+    		  }
+    	  }	
+		}


Also, can you point at any other applications or libraries that perform SSH host key verification using fingerprints?
This seems like a pretty unusual (and potentially insecure) practice that I am unaware of, and I'm not sure we want to directly enable this sort of thing natively within JSch.

I have no idea if any other application which is using Jsch library is also using fingerprints to perform SSH host key verification. It is being used in SAP PI SFTP Adapter.

norrisjeremy · 2023-05-02T12:23:43Z

src/main/java/com/jcraft/jsch/Session.java

@@ -129,6 +129,8 @@ public class Session {

  Buffer buf;
  Packet packet;
+
+  List<String> fingerprintList = new ArrayList<String>();


This can be shortened to simply new ArrayList<>() instead of new ArrayList<String>().

Changed it as suggested.

…g network failure

norrisjeremy · 2023-05-22T12:08:33Z

HI @anamikagsingh,

Can you please answer the following questions?

I'm not sure I follow what you mean by we have an option to transfer large files in smaller chunks and append them in the target in the sequence they have been transferred: how are you transferring these large files via smaller chunks using JSch's existing ChannelSftp API methods? Can you provide some example code of how you are using ChannelSftp and how your proposed API changes are utilized to remedy the file corruption issue you are having?
I'm not sure I understand the purpose of the changes to Session.java: is there a reason you cannot simply achieve this custom logic of authenticating hosts by their host key fingerprint by simply implementing your own custom HostKeyRepository and then instructing JSch use it (via either jsch.setHostKeyRepository() or session.setHostKeyRepository())?

Thanks,
Jeremy

anamikagsingh · 2023-07-11T14:42:55Z

HI @anamikagsingh,

Can you please answer the following questions?

I'm not sure I follow what you mean by we have an option to transfer large files in smaller chunks and append them in the target in the sequence they have been transferred: how are you transferring these large files via smaller chunks using JSch's existing ChannelSftp API methods? Can you provide some example code of how you are using ChannelSftp and how your proposed API changes are utilized to remedy the file corruption issue you are having?

I'm not sure I understand the purpose of the changes to Session.java: is there a reason you cannot simply achieve this custom logic of authenticating hosts by their host key fingerprint by simply implementing your own custom HostKeyRepository and then instructing JSch use it (via either jsch.setHostKeyRepository() or session.setHostKeyRepository())?

Thanks, Jeremy

About point 1, when chunkmode is enabled, we set the chunk size (i.e. 10 MB). Based on chunk size, the bytes will be read from the input file and processed further. We are using the below snippet for chunk mode transfer in Adapter:


  if(chunkMode) {					
	long readBytes = 0;
	String user = propHandler.getPropertyValueAsString(ChannelConstants.AUTH_USERNAME);
	String chunkID = getChunkID(con.getHost(), user, fileAtts);
	//for recovery in chunk mode
	boolean retry = false;
	String chunkEntry = checkChunkStatus(chunkID);
	if(chunkEntry != null) {
		readBytes = Long.parseLong(chunkEntry);
		if(readBytes > 0) {
		retry = true;
		 }
		}				
		//read from inputstream
		byte[] chunk = new byte[chunkSize];
		int curChunkSize = -1;
		curChunkSize = readChunk(chunk, is, chunkSize);
		TRACE.debugT(SIGNATURE, "Bytes read: " + curChunkSize);
		if(curChunkSize >0 || processEmptyFile ==true) {
		audit.addAuditLogEntry(msg.getMessageKey(), AuditLogStatus.SUCCESS, "Starting to read file from byte: " + readBytes);
		readBytes += curChunkSize;
		TRACE.debugT(SIGNATURE, "Total read Bytes: " + readBytes);

		TxTicket ticket = null;
		//sending message to XI
		try {
			//create transaction
			ticket = createTransaction(channelDeliverySemantics);
			updateChunkStatus(chunkID, readBytes + "");
			if(curChunkSize == chunkSize) {
				sendMessageFromSftp(chunk, fileAtts, attachments, channel, con, home, msg,isSpecialFileHandling, specialFileStorageLocation);
			} else {
				byte[] trimChunk = new byte[curChunkSize];
				System.arraycopy(chunk, 0, trimChunk, 0, curChunkSize);
				sendMessageFromSftp(trimChunk, fileAtts, attachments, channel, con, home, msg,isSpecialFileHandling, specialFileStorageLocation);
				
				//final chunk; hence add it to message ID mapper for duplicate check
				addToIDMapper(con.getHost(), fileAtts, msg.getMessageKey().getMessageId());
			}
			if(ticket != null) {
				commitTransaction(ticket);
			}
			xiMsgSent = true;
		} catch(MessagingException e) {
		 }
	}

About point 2, I will try to figure it out.

Thanks.

norrisjeremy · 2023-08-09T12:31:33Z

Hi @anamikagsingh,

From what I can understand, this chunkmode handling seems to be specific to your application.
As such, I'm not sure if I believe it is appropriate to merge it into JSch, lest it introduce regressions for other JSch users.

Thanks,
Jeremy

sonarcloud · 2024-03-15T16:22:26Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

ChannelSftp

7c13d8e

SessionAndUserAuthKeyboardInteractive

f55eaae

norrisjeremy suggested changes May 2, 2023

View reviewed changes

anamikagsingh added 2 commits May 15, 2023 12:01

Support for Multiple Fingerprints And Chunk transfer resumption durin…

9aee720

…g network failure

Chunk transfer resumption during network failure

8d05634

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

File is getting corrupted when Chunk Mode file transfer interrupted #315

File is getting corrupted when Chunk Mode file transfer interrupted #315

anamikagsingh commented Apr 26, 2023

norrisjeremy commented Apr 27, 2023

anamikagsingh commented May 2, 2023

anamikagsingh commented May 2, 2023

norrisjeremy commented May 2, 2023

norrisjeremy May 2, 2023

anamikagsingh May 15, 2023

norrisjeremy May 22, 2023

ecki Oct 24, 2023

norrisjeremy May 2, 2023

anamikagsingh May 15, 2023

norrisjeremy May 2, 2023

anamikagsingh May 15, 2023

norrisjeremy May 2, 2023

anamikagsingh May 15, 2023

norrisjeremy May 2, 2023

anamikagsingh May 15, 2023

norrisjeremy May 2, 2023

anamikagsingh May 15, 2023

norrisjeremy May 2, 2023

anamikagsingh May 15, 2023

norrisjeremy commented May 22, 2023

anamikagsingh commented Jul 11, 2023

norrisjeremy commented Aug 9, 2023

sonarcloud bot commented Mar 15, 2024

File is getting corrupted when Chunk Mode file transfer interrupted #315

Are you sure you want to change the base?

File is getting corrupted when Chunk Mode file transfer interrupted #315

Conversation

anamikagsingh commented Apr 26, 2023

norrisjeremy commented Apr 27, 2023

anamikagsingh commented May 2, 2023

anamikagsingh commented May 2, 2023

norrisjeremy commented May 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

norrisjeremy commented May 22, 2023

anamikagsingh commented Jul 11, 2023

norrisjeremy commented Aug 9, 2023

sonarcloud bot commented Mar 15, 2024

Quality Gate passed