Skip to content

Sftp private key #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

Sftp private key #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5bcdca1
first commit
Jun 26, 2020
77381d9
adding privateKey functions to SFTP-Action, and new SCP action plugin
Jul 16, 2020
fc90da8
Merge pull request #1 from bajram-adapt/adapt
bajram-adapt Jul 16, 2020
e5469a5
Changed formatting
Jul 17, 2020
ef680a3
Merge pull request #2 from bajram-adapt/adapt
bajram-adapt Jul 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions docs/SCPRemote-action.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# SCP Remote to Remote

Description
-----------
This Action will connect to a Bastion Host and execute a SCP command over SSH to copy a file path from Host A to Host B

Use Case
--------
The Use Cases for this Plugin is when we want to connect to a Bastion Host in order to execute an SCP command that will copy files between to other Remote hosts.

Properties
----------
| Configuration | Required | Default | Description |
| :------------ | :------: | :------ | :---------- |
| **Bastion Host** | **Y** | None | This is the Hostname for the Bastion host that will be executing the SCP command. Can be a hostname or IP Address |
| **Port** | **N** | 22 | Specifies the Port that will be used. Defaults to 22 |
| **Bastion User Name** | **Y** | None | Specifies the Bastion Hosts User Name. |
| **Private Key** | **Y** | None | The private RSA key to be used to connect over SHH to the Bastion host. This should be an RSA key. |
| **Passphrase** | **N** | None | Passphrase to be used with the Private RSA key if a Passphrase was setup when the key was created. |
| **Compression Flag** | **N** | None | Flag for SCP command to enable compression of files. |
| **Verbose Flag** | **N** | None | Flag for SCP command to enable verbose mode which extends Logs. |
| **Directory Flag** | **N** | None | Flag for SCP command to enable movement of directories. |
| **Host A User Name** | **Y** | None | The User Name of Host A. This is the host that the source files are on that need to be moved |
| **Host A Host** | **Y** | None |The Hostname of Host A. Can be a hostname or IP Address|
| **Source Path** | **Y** | None | The Absolute Path of the File that needs to be copied. |
| **Host B User Name** | **Y** | None | The User Name of Host B. This is the host that the source files are being copied to |
| **Host B Host** | **Y** | None |The Hostname of Host B. Can be a hostname or IP Address|
| **Destination Path** | **Y** | None | The Absolute Path of the location the files need to be copied to. |



Usage Notes
--------

In order to perform SCP between to remote hosts, we require a Bastion Host. An SCP command based on the configuration supplied will be created to perform a file copy. Authentication setup between all hosts will need to be setup before hand. This includes being able to SSH on the bastion host with the private key being supplied in the configuration and have the 2 remote hosts that files are being moved on having known host/authenticated_keys setup for SSH communication.
7 changes: 5 additions & 2 deletions docs/SFTPCopy-action.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,11 @@ Plugin Configuration
| :------------ | :------: | :----- | :---------- |
| **Host** | **Y** | N/A | Specifies the host name of the SFTP server.|
| **Port** | **N** | 22 | Specifies the port on which SFTP server is running.|
| **User** | **Y** | N/A | Specifies the name of the user which will be used to connect to the SFTP server.|
| **Password** | **Y** | N/A | Specifies the password of the user.|
| **User Name** | **Y** | N/A | Specifies the name of the user which will be used to connect to the SFTP server.|
|**Authentication**|**Y**|**PrivateKey**| Specifies the type of Authentication that will be used to connect to the SFTP Server|
|**Private Key**|**N**|N/A| Private RSA Key to be used to connect to the SFTP Server. This key can be stored in the Secure Keys Store and macro called into the Configuration. Must be a RSA key example: -----BEGIN RSA PRIVATE KEY-----|
|**Private Key Passphrase**|**N**|N/A| Passphrase to be used with RSA Private Key if a Passphrase was specified when key was generated|
| **Password** | **N** | N/A | Specifies the password of the user. Only Required if Private Key is not being used|
| **Source Directory** | **Y** | N/A | Absolute path of the directory on the SFTP server which is to be copied. If the directory is empty, the execution of the plugin will be no-op.|
| **Destination Directory** | **Y** | N/A | Destination directory on the file system, where files need to be copied. If directory does not exist, it will lbe created.|
| **Uncompress** | **N** | true | Boolean flag to determine whether to uncompress the `.zip` files while copying.|
Expand Down
7 changes: 5 additions & 2 deletions docs/SFTPDelete-action.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,11 @@ Plugin Configuration
| :------------ | :------: | :----- | :---------- |
| **Host** | **Y** | N/A | Specifies the host name of the SFTP server.|
| **Port** | **N** | 22 | Specifies the port on which SFTP server is running.|
| **User** | **Y** | N/A | Specifies the name of the user which will be used to connect to the SFTP server.|
| **Password** | **Y** | N/A | Specifies the password of the user.|
| **User Name** | **Y** | N/A | Specifies the name of the user which will be used to connect to the SFTP server.|
|**Authentication**|**Y**|**PrivateKey**| Specifies the type of Authentication that will be used to connect to the SFTP Server|
|**Private Key**|**N**|N/A| Private RSA Key to be used to connect to the SFTP Server. This key can be stored in the Secure Keys Store and macro called into the Configuration. Must be a RSA key example: -----BEGIN RSA PRIVATE KEY-----|
|**Private Key Passphrase**|**N**|N/A| Passphrase to be used with RSA Private Key if a Passphrase was specified when key was generated|
| **Password** | **N** | N/A | Specifies the password of the user. Only Required if Private Key is not being used|
| **Files to be deleted** | **Y** | ${sftp.copied.file.names} | Comma separated list of files on the SFTP server to be deleted. Default value for this field is a Macro which will be substituted by SFTP copy plugin when this plugin is used with it. |
| **Destination Directory** | **Y** | N/A | Destination directory on the file system, where files need to be copied. If directory does not exist, it will lbe created.|
| **Continue execution on error** | **N** | false | Boolean flag to determine whether to proceed with next files in case there is a failure in deletion of any particular file. |
Expand Down
7 changes: 5 additions & 2 deletions docs/SFTPPut-action.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,11 @@ Plugin Configuration
| :------------ | :------: | :----- | :---------- |
| **Host** | **Y** | N/A | Specifies the host name of the SFTP server.|
| **Port** | **N** | 22 | Specifies the port on which SFTP server is running.|
| **User** | **Y** | N/A | Specifies the name of the user which will be used to connect to the SFTP server.|
| **Password** | **Y** | N/A | Specifies the password of the user.|
| **User Name** | **Y** | N/A | Specifies the name of the user which will be used to connect to the SFTP server.|
|**Authentication**|**Y**|**PrivateKey**| Specifies the type of Authentication that will be used to connect to the SFTP Server|
|**Private Key**|**N**|N/A| Private RSA Key to be used to connect to the SFTP Server. This key can be stored in the Secure Keys Store and macro called into the Configuration. Must be a RSA key example: -----BEGIN RSA PRIVATE KEY-----|
|**Private Key Passphrase**|**N**|N/A| Passphrase to be used with RSA Private Key if a Passphrase was specified when key was generated|
| **Password** | **N** | N/A | Specifies the password of the user. Only Required if Private Key is not being used|
| **Source Path** | **Y** | N/A | Path of file or directory on the file system which is to be copied.|
| **Destination Directory** | **Y** | N/A | Destination directory on the FTP Server, where files need to be copied. If directory does not exist, it will lbe created.|
| **File Name Regex** | **N** | .* | Regex to choose only the files that are of interest. All files will be copied by default.|
Expand Down
46 changes: 36 additions & 10 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>io.cdap.plugin</groupId>
<artifactId>sftp-actions</artifactId>
<artifactId>sftp-actions-ssh</artifactId>
<packaging>jar</packaging>
<version>1.5.0-SNAPSHOT</version>
<name>SFTP Actions</name>
Expand All @@ -29,27 +29,22 @@
<jsch.version>0.1.53</jsch.version>
<hadoop.version>2.3.0</hadoop.version>
<guava.version>19.0</guava.version>

<!-- properties for script build step that creates the config files for the artifacts -->
<widgets.dir>widgets</widgets.dir>
<docs.dir>docs</docs.dir>

<!-- This is the version range for the app.parents that this plugin is valid for. Usually this will correspond with
the CDAP version, but not always. -->
<etl.versionRange>[6.0.0-SNAPSHOT,7.0.0-SNAPSHOT)</etl.versionRange>

<!-- These are the application template artifacts that this plugin will be available for. -->
<app.parents>
system:cdap-data-pipeline,
system:cdap-data-streams
</app.parents>

<!-- this is here because project.basedir evaluates to null in the script build step -->
<main.basedir>${project.basedir}</main.basedir>
</properties>

<url>http://maven.apache.org</url>

<dependencies>
<dependency>
<groupId>io.cdap.cdap</groupId>
Expand All @@ -62,6 +57,24 @@
<artifactId>hydrator-common</artifactId>
<version>${hydrator.version}</version>
</dependency>
<dependency>
<groupId>io.cdap.cdap</groupId>
<artifactId>cdap-unit-test</artifactId>
<version>${cdap.version}</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.cdap.cdap</groupId>
<artifactId>hydrator-test</artifactId>
<version>${cdap.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
Expand All @@ -81,11 +94,24 @@
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<version>4.12</version>
<scope>test</scope>
</dependency>
<!-- https://github.com/shamsoftware/sham-ssh :Mock SSH Server used for testing-->
<dependency>
<groupId>software.sham</groupId>
<artifactId>sham-ssh</artifactId>
<version>0.1.0</version>
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a <scope>test</scope> for test only dependency

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.mockito/mockito-all -->
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.10.19</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
Expand Down Expand Up @@ -173,8 +199,8 @@
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
<plugin>
Expand Down
Loading