Skip to content

Commit

Permalink
Option to validate system datasets (IBM#471)
Browse files Browse the repository at this point in the history
* Validate System Datasets

Signed-off-by: Dennis Behm <[email protected]>
  • Loading branch information
dennis-behm authored Mar 7, 2024
1 parent 27d902a commit 452864a
Show file tree
Hide file tree
Showing 5 changed files with 215 additions and 10 deletions.
8 changes: 7 additions & 1 deletion build-conf/build.properties
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,17 @@
# These properties files expect to contain centrally managed defaults
# such as system datasets, language script specific settings
#
buildPropFiles=datasets.properties,dependencyReport.properties,Assembler.properties,BMS.properties,\
buildPropFiles=${systemDatasets},dependencyReport.properties,Assembler.properties,BMS.properties,\
MFS.properties,PSBgen.properties,DBDgen.properties,ACBgen.properties,Cobol.properties,\
LinkEdit.properties,PLI.properties,REXX.properties,ZunitConfig.properties,Transfer.properties,\
CRB.properties,zCEE3.properties,zCEE2.properties

#
# Comma separated list of property files defining system datasets
# This property is validated during the initialization process of zAppBuild
#
systemDatasets=datasets.properties

#
# Comma separated list of default application configuration property files to load
# Supports both relative path (to zAppBuild/build-conf/) and absolute path
Expand Down
8 changes: 7 additions & 1 deletion build.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import groovy.cli.commons.*
@Field def reportingUtils= loadScript(new File("utilities/ReportingUtilities.groovy"))
@Field def filePropUtils= loadScript(new File("utilities/FilePropUtilities.groovy"))
@Field def dependencyScannerUtils= loadScript(new File("utilities/DependencyScannerUtilities.groovy"))
@Field def validationUtils= loadScript(new File("utilities/DatasetValidationUtilities.groovy"))
@Field String hashPrefix = ':githash:'
@Field String giturlPrefix = ':giturl:'
@Field String gitchangedfilesPrefix = ':gitchangedfiles:'
Expand Down Expand Up @@ -268,14 +269,15 @@ options:
cli.r(longOpt:'reset', 'Deletes the dependency collections and build result group from the MetadataStore')
cli.v(longOpt:'verbose', 'Flag to turn on script trace')
cli.pv(longOpt:'preview', 'Supplemental flag indicating to run build in preview mode without processing the execute commands')
cli.cd(longOpt:'checkDatasets', 'Optional flag to validate the presense of the defined system datasets. ')

// scan options
cli.s(longOpt:'scanOnly', 'Flag indicating to only scan source files for application without building anything (deprecated use --scanSource)')
cli.ss(longOpt:'scanSource', 'Flag indicating to only scan source files for application without building anything')
cli.sl(longOpt:'scanLoad', 'Flag indicating to only scan load modules for application without building anything')
cli.sa(longOpt:'scanAll', 'Flag indicating to scan both source files and load modules for application without building anything')

// web application credentials (overrides properties in build.properties)
// DBB metadatastore credentials (overrides properties in build.properties)
cli.url(longOpt:'url', args:1, 'Db2 JDBC URL for the MetadataStore. Example: jdbc:db2:<Db2 server location>')
cli.id(longOpt:'id', args:1, 'Db2 user id for the MetadataStore')
cli.pw(longOpt:'pw', args:1, 'Db2 password (encrypted with DBB Password Utility) for the MetadataStore')
Expand Down Expand Up @@ -469,6 +471,7 @@ def populateBuildProperties(def opts) {
if (opts.b) props.baselineRef = opts.b
if (opts.m) props.mergeBuild = 'true'
if (opts.pv) props.preview = 'true'
if (opts.cd) props.checkDatasets = 'true'

// scan options
if (opts.s) props.scanOnly = 'true'
Expand Down Expand Up @@ -541,6 +544,9 @@ def populateBuildProperties(def opts) {
if(props.reportExternalImpactsAnalysisDepths) assert (props.reportExternalImpactsAnalysisDepths == 'simple' || props.reportExternalImpactsAnalysisDepths == 'deep' ) : "*! Build Property props.reportExternalImpactsAnalysisDepths has an invalid value"
if(props.baselineRef) assert (props.impactBuild) : "*! Build Property props.baselineRef is exclusive to an impactBuild scenario"

// Validate system datasets
if (props.checkDatasets && props.systemDatasets) validationUtils.validateSystemDatasets(props.systemDatasets, props.verbose)

// Print all build properties + some envionment variables
if (props.verbose) {
println("java.version="+System.getProperty("java.runtime.version"))
Expand Down
75 changes: 74 additions & 1 deletion docs/BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ build options:
-cco,--cccOptions Headless Code Coverage Collector Options
-re,--reportExternalImpacts Flag to activate analysis and report of external impacted files within DBB collections
-cd,--checkDatasets Flag to enable validation of the defined system dataset definitions.
Db2 MetadataStore configuration options
-url,--url <arg> Db2 JDBC URL for the MetadataStore.
Expand Down Expand Up @@ -256,6 +256,7 @@ utility options
- [Perform a Scan Source build](#perform-a-scan-source-build)
- [Perform a Scan Source + Outputs build](#perform-a-scan-source--outputs-build)
- [Dynamically Overwrite build properties](#dynamically-overwrite-build-properties)
- [Validate System Datasets](#validate-system-datasets)

<!-- /TOC -->
### Build a Single Program
Expand Down Expand Up @@ -2094,3 +2095,75 @@ required props = linkedit_srcPDS,linkedit_objPDS,linkedit_loadPDS,linkedit_linkE
</details>


### Validate System Datasets

During the initialization phase of the build, a validation of the defined system datasets can be performed. The system datasets are configured with the build property `systemDatasets` in [build-conf/build.properties](../build-conf/build.properties), which contains one or multiple references to build property files defining key-value pairs listing the system datasets. In zAppBuild the default file is called [datasets.properties](../build-conf/datasets.properties) managed in the `build-conf` folder.

To enable validation of system datasets specify the option `--checkDatasets`. It is available in any build scenario. Be aware that this functionality is also available as a stand-alone script and find the instructions [here](../utilities/README.md#dataset-validation-utilities)

```
groovyz dbb-zappbuild/build.groovy \
--workspace /var/dbb/dbb-zappbuild/samples \
--hlq DBB.ZAPP.CLEAN.MASTER \
--workDir /var/dbb/out/MortgageApplication \
--application MortgageApplication \
--logEncoding UTF-8 \
--impactBuild \
--verbose \
--checkDatasets
```
<details>
<summary>Build log</summary>

```
** Build start at 20210622.082942.029
** Input args = /var/dbb/dbb-zappbuild/samples --hlq DBB.ZAPP.CLEAN.MASTER --workDir /var/dbb/out/MortgageApplication --application MortgageApplication --logEncoding UTF-8 --impactBuild --verbose --propOverwrite mainBuildBranch=develop
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/datasets.properties
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/Assembler.properties
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/BMS.properties
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/MFS.properties
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/PSBgen.properties
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/DBDgen.properties
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/ACBgen.properties
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/Cobol.properties
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/LinkEdit.properties
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/PLI.properties
** Loading property file /ZT01/var/dbb/dbb-zappbuild/build-conf/ZunitConfig.properties
** appConf = /var/dbb/dbb-zappbuild/samples/MortgageApplication/application-conf
** Loading property file /var/dbb/dbb-zappbuild/samples/MortgageApplication/application-conf/file.properties
** Loading property file /var/dbb/dbb-zappbuild/samples/MortgageApplication/application-conf/BMS.properties
** Loading property file /var/dbb/dbb-zappbuild/samples/MortgageApplication/application-conf/Cobol.properties
** Loading property file /var/dbb/dbb-zappbuild/samples/MortgageApplication/application-conf/LinkEdit.properties
** Overwriting build property mainBuildBranch from cli argument --propOverwrite with value develop
** The dataset PLI.V5R2.SIBMZCMP referenced for property IBMZPLI_V52 was found.
*! No dataset defined for property IBMZPLI_V51 specified in /ZT01/var/dbb/dbb-zappbuild/build-conf/datasets.properties.
** The dataset WMQ.V9R2M4.SCSQPLIC referenced for property SCSQPLIC was found.
** The dataset COBOL.V6R1.SIGYCOMP referenced for property SIGYCOMP_V6 was found.
** The dataset CICSTS.V5R4.CICS.SDFHCOB referenced for property SDFHCOB was found.
*! No dataset defined for property SIGYCOMP_V4 specified in /ZT01/var/dbb/dbb-zappbuild/build-conf/datasets.properties.
** The dataset HLASM.SASMMOD1 referenced for property SASMMOD1 was found.
** The dataset SYS1.MACLIB referenced for property MACLIB was found.
** The dataset PDTCC.V1R8.SIPVMODA referenced for property PDTCCMOD was found.
** The dataset CICSTS.V5R4.CICS.SDFHLOAD referenced for property SDFHLOAD was found.
** The dataset CICSTS.V5R4.CICS.SDFHMAC referenced for property SDFHMAC was found.
** The dataset CEE.SCEEMAC referenced for property SCEEMAC was found.
** The dataset WMQ.V9R2M4.SCSQCOBC referenced for property SCSQCOBC was found.
** The dataset IMS.V15R1.SDFSMAC referenced for property SDFSMAC was found.
** The dataset RDZ.V14R1.SFELLOAD referenced for property SFELLOAD was found.
** The dataset DBC0CFG.DB2.V12.SDSNLOAD referenced for property SDSNLOAD was found.
** The dataset CICSTS.V5R4.CICS.SDFHPL1 referenced for property SDFHPL1 was found.
** The dataset WMQ.V9R2M4.SCSQLOAD referenced for property SCSQLOAD was found.
** The dataset IMSCFG.IMSC.REFERAL referenced for property REFERAL was found.
** The dataset DEBUG.V14R1.SEQAMOD referenced for property SEQAMOD was found.
** The dataset DBC0CFG.SDSNEXIT referenced for property SDSNEXIT was found.
** The dataset IMS.V15R1.SDFSRESL referenced for property SDFSRESL was found.
** The dataset RATCFG.ZUNIT.SBZUSAMP referenced for property SBZUSAMP was found.
** The dataset CEE.SCEELKED referenced for property SCEELKED was found.
..... // lists of all build properties
...
...
```

</details>


66 changes: 66 additions & 0 deletions utilities/DatasetValidationUtilities.groovy
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
@groovy.transform.BaseScript com.ibm.dbb.groovy.ScriptLoader baseScript
import com.ibm.dbb.build.*
import groovy.transform.*
import groovy.cli.commons.*
import java.util.Properties
import com.ibm.jzos.ZFile

// define script properties

@Field BuildProperties props = BuildProperties.getInstance()

validateDatasets(args)

def validateSystemDatasets(String propertyFiles, String verbose) {

propertyFiles.split(",").each { propertyFile ->

// load property file using java.util.Properties
def properties = new Properties()

// convert to absolute path based on zAppBuild conventions and structure
if (props && !propertyFile.startsWith('/')) propertyFile = "${props.zAppBuildDir}/build-conf/$propertyFile"

File propFile = new File(propertyFile)
if (propFile.exists()) {

propFile.withInputStream { properties.load(it) }
properties.each { key,dataset ->
if (dataset) {
if (ZFile.dsExists("'$dataset'")) {
if (verbose.toBoolean()) println "** The dataset $dataset referenced for property $key was found."
} else {
println "*! The dataset $dataset referenced for property $key was not found. Process exits."
System.exit(1)
}
} else {
if (verbose.toBoolean()) println "*! No dataset defined for property $key specified in $propertyFile."
}
}
} else {
println "*!* The specified $propertyFile (in the list [$propertyFiles]) does not exist."
}
}
}

def validateDatasets(String[] cliArgs)
{
def cli = new CliBuilder(usage: "DatasetValidationUtilites.groovy [options]", header: '', stopAtNonOption: false)
cli.d(longOpt:'systemDatasetDefinition', args:1, required:true, 'List of property files containing system dataset definitions.')
cli.h(longOpt:'help', 'Flag to print the Help message.')

def opts = cli.parse(cliArgs)

// if opt parse fail exit.
if (! opts) {
System.exit(1)
}

if (opts.help)
{
cli.usage()
System.exit(0)
}

validateSystemDatasets(opts.d, "true")
}
68 changes: 61 additions & 7 deletions utilities/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,64 @@ This folder contains common utility and helper files used by the zAppBuild `buil

File | Description
--- | ---
BindUtilities.groovy | Use for the DB2 binding.
BuildReportUtilities.groovy | Helper to populate additional build report entries.
BuildUtilities.groovy | Common build utility methods.
DependencyScannerUtilities.groovy | Populates the DependencyScannerRegistry and returns the mapped dependency scanner for a build file.
FilePropUtilities.groovy | Helper util to load and validate file level properties overrides.
GitUtilities.groovy | Git command methods.
ImpactUtilities.groovy | Methods used for ImpactBuilds.
[BindUtilities.groovy](BindUtilities.groovy) | Use for the DB2 binding.
[BuildReportUtilities.groovy](BuildReportUtilities.groovy) | Helper to populate additional build report entries.
[BuildUtilities.groovy](BuildUtilities.groovy) | Common build utility methods.
[DatasetValidationUtilities.groovy](DatasetValidationUtilities.groovy) | Helper to validate that system dataset definitions are correctly defined.
[DependencyScannerUtilities.groovy](DependencyScannerUtilities.groovy) | Populates the DependencyScannerRegistry and returns the mapped dependency scanner for a build file.
[FilePropUtilities.groovy](FilePropUtilities.groovy) | Helper util to load and validate file level properties overrides.
[GitUtilities.groovy](GitUtilities.groovy) | Git command methods.
[ImpactUtilities.groovy](ImpactUtilities.groovy) | Methods used for ImpactBuilds.

## Stand-alone utils

A few utilities are used both in the zAppBuild build process, but can also be invoked stand-alone:
* Dataset Validation Utilities
* Bind Utilities

### Dataset Validation Utilities

The [DatasetValidationUtilities.groovy](DatasetValidationUtilities.groovy) can be used as a standalone script to validate that the configured system datasets such as the [dataset.properties](../build-conf/datasets.properties) are available on the system where this script is executed. Using the script as a standalone utility targets situations where builds runs into allocation exceptions such as `BGZTK0016E An error occurred running BPXWDYN command 'alloc dd(TASKLIB) dsn(COBOL.V6R1.SIGYCOM) shr'` and helps the build script administrator to customize zAppBuild.

#### Usage

```
groovyz /u/ibmuser/zAppBuild/utilities/DatasetValidationUtilities.groovy --help
usage: DatasetValidationUtilites.groovy [options]
-d,--systemDatasetDefinition <arg> List of property files containing
system dataset definitions.
-h,--help Flag to print the Help message.
```

#### Sample invocation

```
groovyz /u/ibmuser/zAppBuild/utilities/DatasetValidationUtilities.groovy -d /var/dbb/dbb-zappbuild-config/datasets.properties
** The dataset PLI.V5R2.SIBMZCMP referenced for property IBMZPLI_V52 was found.
*! No dataset defined for property IBMZPLI_V51 specified in /var/dbb/dbb-zappbuild-config/datasets.properties.
** The dataset WMQ.V9R2M4.SCSQPLIC referenced for property SCSQPLIC was found.
** The dataset COBOL.V6R1.SIGYCOMP referenced for property SIGYCOMP_V6 was found.
** The dataset CICSTS.V5R4.CICS.SDFHCOB referenced for property SDFHCOB was found.
*! No dataset defined for property SIGYCOMP_V4 specified in /var/dbb/dbb-zappbuild-config/datasets.properties.
** The dataset HLASM.SASMMOD1 referenced for property SASMMOD1 was found.
** The dataset SYS1.MACLIB referenced for property MACLIB was found.
** The dataset PDTCC.V1R8.SIPVMODA referenced for property PDTCCMOD was found.
** The dataset CICSTS.V5R4.CICS.SDFHLOAD referenced for property SDFHLOAD was found.
** The dataset CICSTS.V5R4.CICS.SDFHMAC referenced for property SDFHMAC was found.
** The dataset CEE.SCEEMAC referenced for property SCEEMAC was found.
** The dataset WMQ.V9R2M4.SCSQCOBC referenced for property SCSQCOBC was found.
** The dataset IMS.V15R1.SDFSMAC referenced for property SDFSMAC was found.
** The dataset RDZ.V14R1.SFELLOAD referenced for property SFELLOAD was found.
** The dataset DBC0CFG.DB2.V12.SDSNLOAD referenced for property SDSNLOAD was found.
** The dataset CICSTS.V5R4.CICS.SDFHPL1 referenced for property SDFHPL1 was found.
** The dataset WMQ.V9R2M4.SCSQLOAD referenced for property SCSQLOAD was found.
** The dataset IMSCFG.IMSC.REFERAL referenced for property REFERAL was found.
** The dataset DEBUG.V14R1.SEQAMOD referenced for property SEQAMOD was found.
** The dataset DBC0CFG.SDSNEXIT referenced for property SDSNEXIT was found.
** The dataset IMS.V15R1.SDFSRESL referenced for property SDFSRESL was found.
** The dataset RATCFG.ZUNIT.SBZUSAMP referenced for property SBZUSAMP was found.
** The dataset CEE.SCEELKED referenced for property SCEELKED was found.
```

0 comments on commit 452864a

Please sign in to comment.