Fix script dir link (#39)

* Fix SCRIPT_DIR link

Fix style issues in build_aws script:
* remove unnecessary braces
* replace tabs with spaces

* update client build script name

* was build_client.sh
* now build_dss_client.sh

* Fix typo

* Update README.md

Update name of aws-sdk-cpp RPM

* fix spacing

* Update README

- prioritize docker build
- update build deps

* Refactor build_aws-sdk.sh

- add check if RPM is already built and skip
- add check if RPM is already installed and fail
- use RPMBUILD_DIR var
- clone single branch - avoid needless double clone
- use ARTIFACT var instead of static string

* Update README

- deemphasize native build
- promote Docker build
- Reword intro for clarity
- Improve native build flow
- Streamline deps install

* replace tab with space

* Remove UFM

- remove deps from README
- remove artifacts from dss-sdk build script
- remove deps from dockerfile

* Update commit IDs to remove UFM

* Add missing yamllint.

- fix double pip install

* add .vscode to gitignore

* Update commit IDs

.gitignore

@@ -1,2 +1,3 @@
 go_*
 *.tgz
+.vscode

README.md

@@ -4,58 +4,80 @@ Disaggregated Storage Solution
 
 ## What is DSS
 
-Samsung has developed DSS, a rack-scalable, very high read-bandwidth-optimized, Amazon S3-compatible object storage solution. It utilizes a disaggregated architecture, enabling independent scaling of storage and compute. It features an end-to-end KV semantic communication stack, entirely eliminating the legacy software storage stack. All storage communication uses the NVMeOf-KV-RDMA protocol introduced and open sourced by Samsung. With zero-copy transfer, it achieves high end-to-end performance. The DSS client-side stack includes a high performance wrapper library for simple application integration. Applications utilizing the DSS client library eliminate the need for bucket semantics, key distribution and load balancing between server-side S3 endpoints.
+DSS is a rack-scalable, very high read-bandwidth-optimized, Amazon S3-compatible object storage solution developed by Samsung. It utilizes a disaggregated architecture, enabling independent scaling of storage and compute. It features an end-to-end KV semantic communication stack, entirely eliminating the legacy software storage stack. All storage communication uses the NVMeOf-KV-RDMA protocol introduced and open sourced by Samsung. With zero-copy transfer, it achieves high end-to-end performance. The DSS client-side stack includes a high performance wrapper library for simple application integration. Applications utilizing the DSS client library eliminate the need for bucket semantics, key distribution and load balancing between server-side S3 endpoints.
 
 [![How to build, deploy, and use DSS software](https://img.youtube.com/vi/fpAFvLhTpqw/0.jpg)](https://youtu.be/fpAFvLhTpqw "How to build, deploy, and use DSS software")
 
 [How to build, deploy, and use DSS software](https://youtu.be/fpAFvLhTpqw)
 
-## Prerequisites
+## Build DSS - Docker
 
-### Operating system requirements
+DSS is optimally built via Docker using the scripts documented below.
 
-DSS build and runtime is presently supported on CentOS 7.8.
+### Build All - Docker
 
-### Build package dependencies
+Build all of the DSS artifacts and its dependency artifacts using one script:
 
-Install the following packages / modules to build DSS and its external dependencies:
+```bash
+./scripts/docker/build_all.sh
+```
+
+### Build Dependencies - Docker
+
+Optionally, build only the dependencies artifacts:
 
 ```bash
-sudo yum install epel-release -y
-sudo yum group install "Development Tools" -y
-sudo yum install bc boost-devel check cmake cmake3 CUnit-devel dejagnu dpkg elfutils-libelf-devel expect \
-  glibc-devel jemalloc-devel Judy-devel libaio-devel libcurl-devel libuuid-devel meson ncurses-devel numactl-devel \
-  openssl-devel pulseaudio-libs-devel python3 python3-devel python3-pip rdma-core-devel redhat-lsb ruby-devel \
-  snappy-devel tbb-devel wget zlib-devel -y
-sudo python3 -m pip install pybind11
-sudo gem install ffi -v 1.12.2
-sudo gem install git -v 1.6.0
-sudo gem install rb-inotify -v 0.9.10
-sudo gem install rexml -v 3.2.3
-sudo gem install backports -v 3.21.0
-sudo gem install fpm
+./scripts/docker/build_gcc.sh
+./scripts/docker/build_aws-sdk.sh
+./scripts/docker/build_kernel.sh
+./scripts/docker/build_mlnx-tools.sh
+```
+
+### Build DSS Artifacts - Docker
+
+Optionally, build only the DSS artifacts:
+
+```bash
+./scripts/docker/build_dss-sdk.sh
+./scripts/docker/build_minio.sh
+./scripts/docker/build_dss-client.sh
+./scripts/docker/build_datamover.sh
 ```
 
 ## Build DSS
 
-**NOTE: User-built GCC and AWS-SDK-CPP RPMs must be installed on the build machine.**
+Alternatively, DSS can be built natively, but all dependencies must be installed first.
 
-On initial build:
+### Prerequisites
 
-1. Build GCC
-2. Install the resulting GCC RPM
-3. Build AWS-SDK-CPP
-4. Install the resulting AWS-SDK-CPP RPM.
-5. Run the `build_all.sh` script
+#### Operating system requirements
+
+DSS build and runtime is presently supported on CentOS 7.8.
+
+#### Build package dependencies
+
+Install the following packages / modules to build DSS and its external dependencies:
 
 ```bash
-./scripts/build_gcc.sh
-sudo yum install ./dss-ansible/artifacts/dss-gcc510*.rpm -y
-./scripts/build_aws-sdk.sh
-sudo yum install ./dss-ansible/artifacts/aws-sdk-cpp-1.8.99-0.x86_64.rpm -y
-./scripts/build_all.sh
+sudo yum install epel-release centos-release-scl-rh -y
+sudo yum install bc bison boost-devel cmake cmake3 CUnit-devel devtoolset-11 dpkg elfutils-libelf-devel \
+  flex gcc gcc-c++ git glibc-devel gmp-devel jemalloc-devel Judy-devel libaio-devel libcurl-devel libmpc-devel \
+  libuuid-devel make man-db meson mpfr-devel ncurses-devel numactl-devel openssl openssl-devel patch \
+  pulseaudio-libs-devel python3 python3-devel python3-pip rdma-core-devel redhat-lsb-core rpm-build \
+  snappy-devel tbb-devel wget zlib-devel -y
+sudo python3 -m pip install pybind11 gcovr==5.0
 ```
 
+**NOTE: User-built GCC and AWS-SDK-CPP RPMs must be installed on the build machine.**
+
+On initial build:
+
+1. Build GCC: `./scripts/build_gcc.sh`
+2. Install the resulting GCC RPM: `sudo yum install ./dss-ansible/artifacts/dss-gcc510*.rpm -y`
+3. Build AWS-SDK-CPP: `./scripts/build_aws-sdk.sh`
+4. Install the resulting AWS-SDK-CPP RPM: `sudo yum install ./dss-ansible/artifacts/aws-sdk-cpp-*.rpm -y`
+5. Run the `build_all.sh` script: `./scripts/build_all.sh`
+
 Once the GCC and AWS RPMs are installed, only the `build_all.sh` script needs to be run on subsequent builds.
 
 Dependency artifacts for GCC, kernel, aws-sdk-cpp, and mlnx-tools are staged under `rpmbuilder` and `workspace` directories of your home directory by default. By leaving them in-place, re-build of these upstream components will be skipped on subsequent builds.
@@ -76,32 +98,6 @@ DSS individual components:
 * Build dss-client: `./scripts/build_dss-client.sh`
 * Build dss-datamover: `./scripts/build_datamover.sh`
 
-## Build Docker
-
-DSS can alternatively be built via Docker.
-
-```bash
-./scripts/docker/build_all.sh
-```
-
-To build individual DSS dependencies via Docker:
-
-```bash
-./scripts/docker/build_gcc.sh
-./scripts/docker/build_aws-sdk.sh
-./scripts/docker/build_kernel.sh
-./scripts/docker/build_mlnx-tools.sh
-```
-
-To build individual DSS components from Docker:
-
-```bash
-./scripts/docker/build_dss-sdk.sh
-./scripts/docker/build_minio.sh
-./scripts/docker/build_dss-client.sh
-./scripts/docker/build_datamover.sh
-```
-
 ## Deploy DSS
 
 See [dss-ansible README](https://github.com/OpenMPDK/dss-ansible/blob/master/README.md)

scripts/build_all.sh

@@ -34,7 +34,7 @@
 set -e
 
 # Load utility functions
-SCRIPT_DIR=$(readlink -f "$(dirname "$0")")
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 . "$SCRIPT_DIR/utils.sh"
 
 # Build dependency artificats
@@ -43,8 +43,8 @@ SCRIPT_DIR=$(readlink -f "$(dirname "$0")")
 "$SCRIPT_DIR/build_kernel.sh"
 "$SCRIPT_DIR/build_mlnx-tools.sh"
 
-# Build DSS
+# # Build DSS
 "$SCRIPT_DIR/build_dss-sdk.sh"
 "$SCRIPT_DIR/build_minio.sh"
-"$SCRIPT_DIR/build_client.sh"
+"$SCRIPT_DIR/build_dss-client.sh"
 "$SCRIPT_DIR/build_datamover.sh"

scripts/build_aws-sdk.sh

@@ -1,28 +1,51 @@
-#!/usr/bin/bash
+#!/usr/bin/env bash
 # shellcheck disable=SC1090,SC1091
 set -e
 
-DIR="$PWD/$(dirname "$0")"
-source "${DIR}/utils.sh"
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+source "$SCRIPT_DIR/utils.sh"
 
 GIT_AWS_RELEASE=1.9
 GIT_CHECKOUT_TAG="1.9.343-elbencho-tag"
+ARTIFACT='aws-sdk-cpp-*.rpm'
+
+# Check if aws-sdk-cpp RPM is already built and present in dss-ansible/artifacts dir
+CHECK_ARTIFACT=$(find "$ARTIFACTS_DIR/" -name "$ARTIFACT" | wc -l)
+
+# Skip build if RPM is present
+if [ "$CHECK_ARTIFACT" -gt 0 ]
+then
+    echo "aws-sdk-cpp RPM is present in $ARTIFACTS_DIR" - Skipping...
+    exit 0
+fi
+
+# Check if aws-sdk-cpp RPM is already installed
+set +e
+CHECK_RPM_INSTALLED=$(rpm -q aws-sdk-cpp)
+set -e
+
+# Abort if aws-sdk-cpp RPM is installed - must be uninstalled before build
+if [ "$CHECK_RPM_INSTALLED" != 'package aws-sdk-cpp is not installed' ]
+then
+    echo "aws-sdk-cpp RPM is installed."
+    echo "aws-sdk-cpp RPM must be uninstalled before rebuilding: \"sudo yum remove aws-sdk-cpp -y\""
+    exit 1
+fi
 
 echo "Preparing the environment and the spec file"
-mkdir -p "${HOME}"/rpmbuild/{SOURCES,BUILD,RPMS,SPECS}
-cp "$DIR/aws-git-${GIT_AWS_RELEASE}.patch" "$HOME"/rpmbuild/SOURCES/
-AWS_SPEC_FILE="$HOME/rpmbuild/SPECS/aws-${GIT_AWS_RELEASE}.spec"
-rpm -q aws-sdk-cpp &>/dev/null &&  rpm -e aws-sdk-cpp
+mkdir -p "$RPMBUILD_DIR"/{SOURCES,BUILD,RPMS,SPECS}
+cp "$SCRIPT_DIR/aws-git-$GIT_AWS_RELEASE.patch" "$RPMBUILD_DIR/SOURCES/"
+AWS_SPEC_FILE="$RPMBUILD_DIR/SPECS/aws-$GIT_AWS_RELEASE.spec"
 
 # Create spec file for RPM build
-cat > "${AWS_SPEC_FILE}" << EOF
+cat > "$AWS_SPEC_FILE" << EOF
 Name:           aws-sdk-cpp
-Version:        ${GIT_AWS_RELEASE}
+Version:        $GIT_AWS_RELEASE
 Release:        0
 Summary:        Amazon Web Services SDK for C++
 License:        ASL 2.0
 URL:            https://github.com/aws/%{name}
-Patch0:		aws-git-${GIT_AWS_RELEASE}.patch
+Patch0:         aws-git-$GIT_AWS_RELEASE.patch
 
 
 %define _unpackaged_files_terminate_build 0
@@ -35,9 +58,8 @@ portability (Windows, OSX, Linux, and mobile).
 
 %prep
 rm -rf aws-sdk-cpp
-git clone --recursive -q https://github.com/breuner/aws-sdk-cpp.git 
+git clone --recursive -q https://github.com/breuner/aws-sdk-cpp.git --branch $GIT_CHECKOUT_TAG --single-branch
 cd aws-sdk-cpp
-git checkout ${GIT_CHECKOUT_TAG}
 
 %patch0 -p1
 
@@ -63,9 +85,9 @@ EOF
 echo -n "Building AWS RPM ...."
 if ! rpmbuild -bb "$AWS_SPEC_FILE";
 then
-	echo "[Failed]" 
-	exit 1
+    echo "[Failed]" 
+    exit 1
 fi
 echo "[Success]"
 
-find "$RPM_DIR" -name 'aws-sdk-cpp*.rpm' -exec cp {} "$ARTIFACTS_DIR/" \;
+find "$RPM_DIR" -name "$ARTIFACT" -exec cp {} "$ARTIFACTS_DIR/" \;

scripts/build_datamover.sh

@@ -33,7 +33,7 @@
 set -e
 
 # Load utility functions
-SCRIPT_DIR=$(readlink -f "$(dirname "$0")")
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 . "$SCRIPT_DIR/utils.sh"
 
 # Check for submodules in update init recursive if missing

scripts/build_dss-client.sh

@@ -33,7 +33,7 @@
 set -e
 
 # Load utility functions
-SCRIPT_DIR=$(readlink -f "$(dirname "$0")")
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 . "$SCRIPT_DIR/utils.sh"
 
 # Check for submodules in update init recursive if missing

scripts/build_dss-sdk.sh

@@ -35,7 +35,7 @@
 set -e
 
 # Load utility functions
-SCRIPT_DIR=$(readlink -f "$(dirname "$0")")
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 . "$SCRIPT_DIR/utils.sh"
 
 # Set build variables
@@ -50,20 +50,11 @@ checksubmodules
 # Set artifacts build directory paths
 target_build_dir="${DSS_DIR}/dss-sdk/df_out"
 host_build_dir="${DSS_DIR}/dss-sdk/host_out"
-nkv_agent_build_dir="${DSS_DIR}/dss-sdk/ufm/agents/nkv_agent"
-ufm_build_dir="${DSS_DIR}/dss-sdk/ufm/fabricmanager"
-ufm_broker_build_dir="${DSS_DIR}/dss-sdk/ufm/ufm_msg_broker"
 
 echo "Removing existing artifacts from artifacts directory..."
 rm -f "${ARTIFACTS_DIR}"/nkv-target-*.tgz
 rm -f "${ARTIFACTS_DIR}"/nkv-sdk-bin-*.tgz
-rm -f "${ARTIFACTS_DIR}"/nkvagent-*.rpm
-rm -f "${ARTIFACTS_DIR}"/ufm-*.rpm
-rm -f "${ARTIFACTS_DIR}"/ufmbroker-*.rpm
 
 echo "Copying artifacts to artifacts directory..."
 cp "${target_build_dir}"/nkv-target-*.tgz "${ARTIFACTS_DIR}"
 cp "${host_build_dir}"/nkv-sdk-bin-*.tgz "${ARTIFACTS_DIR}"
-cp "${nkv_agent_build_dir}"/nkvagent-*.rpm "${ARTIFACTS_DIR}"
-cp "${ufm_build_dir}"/ufm-*.rpm "${ARTIFACTS_DIR}"
-cp "${ufm_broker_build_dir}"/ufmbroker-*.rpm "${ARTIFACTS_DIR}"

scripts/build_gcc.sh

@@ -34,7 +34,7 @@
 set -e
 
 # Load utility functions
-SCRIPT_DIR=$(readlink -f "$(dirname "$0")")
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 . "$SCRIPT_DIR/utils.sh"
 
 # Set build variables

scripts/build_kernel.sh

@@ -34,7 +34,7 @@
 set -e
 
 # Load utility functions
-SCRIPT_DIR=$(readlink -f "$(dirname "$0")")
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 . "$SCRIPT_DIR/utils.sh"
 
 # Set build variables

scripts/build_minio.sh

@@ -33,7 +33,7 @@
 set -e
 
 # Load utility functions
-SCRIPT_DIR=$(readlink -f "$(dirname "$0")")
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 . "$SCRIPT_DIR/utils.sh"
 
 # Check for submodules in update init recursive if missing

scripts/build_mlnx-tools.sh

@@ -34,7 +34,7 @@
 set -e
 
 # Load utility functions
-SCRIPT_DIR=$(readlink -f "$(dirname "$0")")
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 . "$SCRIPT_DIR/utils.sh"
 
 # Set build variables