initial public release

.gitignore

@@ -0,0 +1,9 @@
+# the .jar ist to be installed on the target machine
+/dataloader/dataloader-58.0.1.jar
+
+# ignore local testing artifacts
+/.SfDataloaderCli
+/*.csv
+/*.txt
+/*.sdl
+/*.ps1

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2023 U. Braig
+Copyright (c) 2023 Uwe Braig
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,2 +1,71 @@
-# sf-dataloader-cli
-Command Line Interface for Salesforce Data Loader: A PowerShell wrapper to easily use the Salesforce Data Loader via command line. 
+# Command Line Interface for Salesforce Data Loader
+
+## What is it?
+A PowerShell module intended to simplify the usage of Scripted [Data Loader](https://developer.salesforce.com/tools/data-loader).
+
+## Why would you need it?
+The Salesforce Data Loader "as is" offers two ways of using it:
+* Data Loader __GUI mode__ offers only limited options to save configuration settings for repetitive tasks.
+* Data Loader __Scripting__ (a.k.a. "__Scripted Data Loader__") is quite complex to configure and rather unflexible for ad-hoc changes: It requires to write .xml  configuration files with the details of the operation to be executed.
+
+This module is intended to fill the gap: You can run Salesforce Data Loader from of the PowerShell command line ([What is PowerShell?](https://learn.microsoft.com/en-us/powershell/scripting/overview)) for ad-hoc as well as for repetitive tasks:
+* Provides a set of straightforward commands and easy-to-remember command aliases for all operations like __EXTRACT__, __INSERT__, __UPDATE__, __UPSERT__ and (HARD) __DELETE__.
+* Easy to choose between __SOAP API__ or __Bulk API__ in either __serial__ or __parallel__ mode. Allows to set the __batch size__ via command line option.
+* Auto-creates __mapping files__ in many scenarios.
+* Encapsulates the handling of Salesforce authentication, e.g. 
+    * by generating key files and encrypting passwords, i.e. the "username + password + security token" style,
+    * optionally by using the authorization information as handled by the [Salesforce SFDX CLI](https://developer.salesforce.com/tools/sfdxcli).
+* Supports all PowerShell standard features like help pages, tab completion for parameters, approved verb best practices etc.
+
+
+## How does it look like?
+
+A very basic example to copy Leads from one Org to another in 4 simple steps. 
+For more examples and command reference see the [Wiki Pages](../../wiki)
+
+### Step 1: Authorize the Source Org
+In this  example we use the default org of your SFDX project:
+
+`$MySourceOrg = sfauth`
+
+### Step 2: Authorize the Target Org
+In this example for a Sandbox org we will give the username and have the password and security token prompted via console input: 
+
+`$MyTargetOrg = sfauth [email protected] -ConsoleInput -InstanceUrl https://test.salesforce.com`
+
+### Step 3: Extract all Lead Records from Source Org
+Simple example to extract all Leads to a default target file 'Lead.csv':
+
+`sfextract $MySourceOrg Lead "SELECT Id, FirstName, LastName, Company FROM Lead"`
+
+### Step 4: Insert all Leads to Target Org
+Simple example to import those Leads from the default source file 'Lead.csv' to another org. A default Data Loader mapping file (.sdl) ist automatically created on the fly from the column headers in the .csv file:
+
+`sfinsert $MyTargetOrg Lead`
+
+## How to get it?
+
+### Prerequisites
+Mandatory
+- Windows 10 or newer with __PowerShell v5.1__ or newer. NOTE: v5.1 is already installed by default on Windows 10 and newer. For further details see [Installing Windows PowerShell](https://learn.microsoft.com/en-us/powershell/scripting/windows-powershell/install/installing-windows-powershell?view=powershell-7.3)
+- A __Java Runtime Environment (JRE)__ version 11 or later as described in [Considerations for Installing Data Loader](https://developer.salesforce.com/docs/atlas.en-us.dataLoader.meta/dataLoader/installing_the_data_loader.htm), e.g. [Zulu OpenJDK](https://www.azul.com/downloads/zulu-community/?package=jdk). Make sure that the __JAVA_HOME__ environment variable is set properly.
+- A download of the latest __Salesforce Data Loader .zip file__ from https://developer.salesforce.com/tools/data-loader or from https://github.com/forcedotcom/dataloader/releases. __NOTE:__ There is no need to run the installation batch file from inside the .zip. This would only be needed if you also want to use Data Loader independently of the SfDataloaderCli module.
+- Make sure, you have got the proper permissions in your Salesforce Org as described in [Salesforce Data Loader Installation Considerations](https://developer.salesforce.com/docs/atlas.en-us.dataLoader.meta/dataLoader/installing_the_data_loader.htm).
+
+Optionally
+- [Salesforce CLI](https://developer.salesforce.com/tools/sfdxcli) if you want to use the authentication methods provided by SFDX.
+
+### Download and Install SfDataloaderCli PowerShell Module
+- Download the .zip file of the latest stable version from [Releases](./releases).
+- Extract the files to a target directory of your choice, e.g. `D:\sf-dataloader-cli-0.0.1-beta`
+- From the Data Loader .zip file (see prerequisites above), locate the Data Loader .jar file: e.g. in the `dataloader_v58.0.2.zip` this would be the file `dataloader_v58.0.2.jar`.
+- Copy this .jar file to the corresponding directory in the PowerShell module directory, e.g. to `D:\sf-dataloader-cli-0.0.1-beta\dataloader`
+
+### Import the SfDataloaderCli module into your PowerShell session
+- Open a PowerShell console window.
+- Run `Import-Module D:\sf-dataloader-cli-0.0.1-beta\dataloader\SfDataloaderCli.psd1`.
+
+## Get Started!
+
+See the [Wiki Pages](../../wiki) on how to get started.
+

SfDataloaderCli.psm1

@@ -0,0 +1,9 @@
+foreach ($directory in @('Private', 'Public')) {
+    Get-ChildItem -Path "$PSScriptRoot\$directory\*.ps1" | ForEach-Object {. $_.FullName}
+}
+
+New-Alias sfextract Export-SfRecords -Force
+New-Alias sfinsert Add-SfRecords -Force
+New-Alias sfupdate Update-SfRecords -Force
+New-Alias sfupsert Import-SfRecords -Force
+New-Alias sfauth Get-SfAuthToken -Force

configs/config.properties

@@ -0,0 +1,51 @@
+# This is the base configuration that will be copied on the fly on each call.
+# The actual settings are determined by this file + process-conf.xml (created on the fly) + command line parameters
+dataAccess.readBatchSize=200
+dataAccess.readUTF8=true
+dataAccess.writeBatchSize=500
+dataAccess.writeUTF8=true
+loader.csvComma=true
+loader.csvOther=false
+loader.csvOtherValue=-
+loader.csvTab=true
+loader.hideWelcome=true
+process.enableExtractStatusOutput=false
+process.enableLastRunOutput=false
+process.loadRowToStartAt=0
+process.useEuropeanDates=false
+sfdc.bulkApiCheckStatusInterval=5000
+sfdc.bulkApiSerialMode=false
+sfdc.bulkApiZipContent=false
+sfdc.connectionTimeoutSecs=60
+sfdc.enableRetries=true
+sfdc.extractionRequestSize=500
+sfdc.insertNulls=true
+sfdc.loadBatchSize=200
+sfdc.maxRetries=3
+sfdc.minRetrySleepSecs=2
+sfdc.noCompression=false
+# --- new oauth
+#sfdc.oauth.loginfrombrowser=false
+#
+#sfdc.oauth.environment=Sandbox
+#sfdc.oauth.environments=Production,Sandbox
+#sfdc.oauth.Production.bulk.clientid=DataLoaderBulkUI/
+#sfdc.oauth.Production.partner.clientid=DataLoaderPartnerUI/
+#sfdc.oauth.Production.redirecturi=https\://login.salesforce.com/services/oauth2/success
+#sfdc.oauth.Production.server=https\://login.salesforce.com/
+#sfdc.oauth.redirecturi=https\://login.salesforce.com
+#sfdc.oauth.Sandbox.bulk.clientid=DataLoaderBulkUI/
+#sfdc.oauth.Sandbox.partner.clientid=DataLoaderPartnerUI/
+#sfdc.oauth.Sandbox.redirecturi=https\://test.salesforce.com/services/oauth2/success
+#sfdc.oauth.Sandbox.server=https\://test.salesforce.com/
+#sfdc.oauth.server=https\://test.salesforce.com
+#sfdc.reuseClientConnection=true
+# --- END new oauth
+#sfdc.resetUrlOnLogin=true
+sfdc.timeoutSecs=540
+sfdc.timezone=Europe/Berlin
+sfdc.truncateFields=true
+sfdc.useBulkApi=false
+sfdc.wireOutput=false
+sfdcInternal=false
+sfdcInternal.isSessionIdLogin=false

configs/log-conf.xml

@@ -0,0 +1,35 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+  <appenders>
+    <RollingFile name="fileAppender" 
+                 fileName="{!LogFilePath}.log"
+                 filePattern="{!LogFilePath}-%d{yyyy-MM-dd}.log"
+                 ignoreExceptions="false">
+      <PatternLayout>
+          <Pattern>"%d %-5p [%t] %C{2} %M (%F:%L) - %m%n"</Pattern>
+      </PatternLayout>
+      <Policies>
+          <SizeBasedTriggeringPolicy size="100KB" />
+      </Policies>
+      <DefaultRolloverStrategy max="1" />
+    </RollingFile>
+    <CONSOLE name="STDOUT" target="SYSTEM_OUT">
+      <PatternLayout pattern="%d %-5p [%t] %C{2} %M (%F:%L) - %m%n"/>
+    </CONSOLE>
+  </appenders>
+  <loggers>
+    <logger name="org.apache.log4j.xml" level="warn">
+      <AppenderRef ref="fileAppender"/>
+      <AppenderRef ref="STDOUT" />
+    </logger>
+    <logger name="org.apache" level="warn">
+      <AppenderRef ref="fileAppender"/>
+      <AppenderRef ref="STDOUT" />
+    </logger>
+    <root level="debug">
+      <AppenderRef ref="fileAppender"/>
+      <!-- <AppenderRef ref="STDOUT" /> -->
+    </root>
+  </loggers>
+</configuration>
+

configs/readme.txt

@@ -0,0 +1,3 @@
+In this folder: Config files templates.
+Will be copied from here by the scripts to a temp working directory.
+

dataloader/README.md

@@ -0,0 +1,5 @@
+Dont't forget to complete the installation here:
+
+- Download the latest version of dataloader from https://developer.salesforce.com/tools/data-loader
+- Find the dataloader-nn.n.n.jar file and copy it here
+- The scripts expect EXACTLY ONE .jar file here and will use that.

private/BuildBulkModeConfigMap.ps1

@@ -0,0 +1,15 @@
+Set-StrictMode -Version 3
+
+function BuildBulkModeConfigMap {
+    param (
+        [string]$BulkMode
+    )
+
+    $ConfigOverrideMap = switch ($BulkMode) {
+        'Parallel' { @{ 'sfdc.useBulkApi' = 'true'; 'sfdc.bulkApiSerialMode' = 'false' } }
+        'Serial' { @{ 'sfdc.useBulkApi' = 'true'; 'sfdc.bulkApiSerialMode' = 'true' } }
+        Default { @{ 'sfdc.useBulkApi' = 'false' } }
+    }
+
+    return $ConfigOverrideMap
+}

private/BuildDataloaderLogConfXml.ps1

@@ -0,0 +1,27 @@
+Set-StrictMode -Version 3
+
+function BuildDataloaderLogConfXml{
+    [CmdletBinding()]
+    param (
+        [Parameter(Mandatory)][string]$ConfigFilesPath,
+        [Parameter(Mandatory)][string]$LogFilePath
+    )
+
+    # ------------------------------------ load the template
+    [string[]]$LogConfTemplateLines = Get-Content -Path "$PSScriptRoot\..\configs\log-conf.xml"
+
+    # ------------------------------------ patch the template
+    $LogFilePath = $LogFilePath -replace '\\', '/'
+    [string[]]$LogConfXmlLines = @()   # empty array to collect all XML strings
+    foreach ($LogConfTemplateLine in $LogConfTemplateLines) {
+        $NewLogConfXmlLine = $LogConfTemplateLine -replace '{!LogFilePath}', $LogFilePath
+        $LogConfXmlLines += $NewLogConfXmlLine
+    }
+
+    # ------------------------------------------------------------------------------------------- generate the log-conf.xml
+    $LogConfXmlFilename = Join-Path $ConfigFilesPath 'log-conf.xml'
+    Write-Debug "Generating log-conf: <$LogConfXmlFilename>"
+    Set-Content -Path $LogConfXmlFilename -Value $LogConfXmlLines
+
+    return $LogConfXmlFilename
+}

private/BuildDataloaderProcessConfXml.ps1

@@ -0,0 +1,35 @@
+Set-StrictMode -Version 3
+
+function BuildDataloaderProcessConfXml{
+    [CmdletBinding(DefaultParameterSetName = 'authentication_sfdx_oauth')]
+    param (
+        [Parameter(Mandatory, Position=0)]
+        [string]$ProcessName,
+
+        [Parameter(Mandatory, Position=1)]
+        $ConfigOverrideMap
+    )
+
+    # --------------------------------------- Build the process-conf.xml
+	[string[]]$ProcessConfXml = @()   # empty array to collect all XML strings
+	$ProcessConfXml += '<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">'
+	$ProcessConfXml += '<beans>'
+	$ProcessConfXml += "<bean id=""$ProcessName"" class=""com.salesforce.dataloader.process.ProcessRunner"">"
+	$ProcessConfXml += '<description>Auto-generated by SfDataloader module</description>'    
+    $ProcessConfXml += "<property name=""name"" value=""$ProcessName"" />"
+    $ProcessConfXml += '<property name="configOverrideMap">'
+    $ProcessConfXml += '<map>'
+
+#	foreach($key in $ConfigOverrideMap.keys) {
+    foreach($item in $ConfigOverrideMap.getEnumerator() | Sort-Object Name) {
+            $key = $item.Name
+            $ProcessConfXml += "<entry key=""$key"" value=""$($ConfigOverrideMap[$key])"" />" 
+	}
+
+    $ProcessConfXml += '</map>'
+    $ProcessConfXml += '</property>'
+    $ProcessConfXml += '</bean>'
+    $ProcessConfXml += '</beans>'
+
+    return $ProcessConfXml
+}

private/BuildErrorSuccessFileNames.ps1

@@ -0,0 +1,21 @@
+Set-StrictMode -Version 3
+
+function BuildErrorSuccessFileNames {
+    param (
+        [string]$Path
+    )
+
+    # --------------------------------------- Build error/success file names
+    $CsvFilePath = Split-Path -Path $Path
+    $CsvFileName = Split-Path -Path $Path -Leaf
+    $PathSuccessFile = Join-Path $CsvFilePath $CsvFileName.Replace('.csv', '-SUCCESS.csv')
+    $PathErrorFile = Join-Path $CsvFilePath $CsvFileName.Replace('.csv', '-ERROR.csv')
+
+    # --------------------------------------- Build Config Override Map
+    $ConfigOverrideMap = @{
+        'process.outputSuccess' = $PathSuccessFile
+        'process.outputError'   = $PathErrorFile
+    }
+
+    return $ConfigOverrideMap
+}

private/InitializeConfigTempDir.ps1

@@ -0,0 +1,14 @@
+Set-StrictMode -Version 3
+
+function InitializeConfigTempDir {
+#    $ConfigTempDir = (Get-Location).Path
+#    $ConfigTempDir =  Join-Path $Env:temp 'SfDataloader'
+    $ConfigTempDir =  Join-Path (Get-Location).Path '.SfDataloaderCli'
+
+    if (!(Test-Path $ConfigTempDir)) {
+        $dummy = New-Item -ItemType "directory" -Path $ConfigTempDir
+        Write-Debug "Dummy: <$dummy>"
+    }
+    Write-Verbose "ConfigTempDir: <$ConfigTempDir>"
+    return $ConfigTempDir
+}

private/InvokeSfDataloaderJavaClass.ps1

@@ -0,0 +1,39 @@
+Set-StrictMode -Version 3
+
+function InvokeSfDataloaderJavaClass{
+	[CmdletBinding()]
+	param (
+		[Parameter()][string[]]$SystemPropertiesList,
+		[Parameter(Mandatory)][string]$ClassName,
+		[Parameter()][string[]]$ArgumentList
+	)
+
+    $JavaHome = $env:JAVA_HOME
+    $ClassPath = "$PSScriptRoot\..\dataloader\*.jar"
+
+    # --- find a jar. NOTE: We expect exactly one .jar file here!
+    $ClassPathFiles = @(Get-ChildItem -Path $ClassPath)
+    if (!($ClassPathFiles)) {
+        throw "NOT FOUND <$ClassPath> - Check Installation instructions!"
+    } elseif ($($ClassPathFiles.Count) -ne 1) {
+        throw "MORE THAN ONE FILE FOUND <$ClassPath> - Check Installation instructions!"
+    } else {
+        $ClassPath = $ClassPathFiles[0].FullName
+    }
+
+	Write-Verbose "JavaHome: $JavaHome"
+	Write-Debug   "PSScriptRoot: $PSScriptRoot"
+	Write-Verbose "ClassPath: $ClassPath"
+	Write-Debug   "ClassName: $ClassName"
+    Write-Debug   "ArgumentList: $ArgumentList"
+    if ($debug) { 
+		Write-Debug '--- BEGIN EchoArgs'
+		& "$PSScriptRoot\..\private\EchoArgs.exe" '-cp' $ClassPath $SystemPropertiesList $ClassName $ArgumentList | Write-Debug
+		Write-Debug '--- END EchoArgs'
+		pause 
+	}
+    $Result = & "$JavaHome\bin\java" '-cp' $ClassPath $SystemPropertiesList $ClassName $ArgumentList
+    [string]$s = $Result
+    return $s
+}
+