Skip to content

Commit

Permalink
Merge pull request #18 from TIK-NFL/update_v5.5
Browse files Browse the repository at this point in the history
Merged update_v5.5 branch into staging
  • Loading branch information
aseeland authored Jul 28, 2021
2 parents f2a0f66 + c646792 commit 2feb4db
Show file tree
Hide file tree
Showing 650 changed files with 33,126 additions and 22,789 deletions.
2 changes: 1 addition & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Closes #

**Suggestions on how to test this**:

**Does this PR introduce a user interface change?**:
**Does this PR introduce a user interface change? If mockups are available, please link/include them here**:

**Is there a release notes update needed for this change?**:

Expand Down
14 changes: 14 additions & 0 deletions .github/workflows/guides_build_sphinx.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: "Guides Build Status"
on:
pull_request:
paths:
- 'doc/sphinx-guides/**/*.rst'

jobs:
docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: ammaraskar/sphinx-action@master
with:
docs-folder: "doc/sphinx-guides/"
11 changes: 10 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ oauth-credentials.md

/src/main/webapp/oauth2/newAccount.html
scripts/api/setup-all.sh*
scripts/api/setup-all.*.log

# ctags generated tag file
tags
Expand All @@ -45,6 +46,10 @@ conf/docker-aio/dv/deps/
conf/docker-aio/dv/install/dvinstall.zip
# or copy of test data
conf/docker-aio/testdata/

# docker-aio creates maven/ which reports 86 new files. ignore this wd.
maven/

scripts/installer/default.config
*.pem

Expand All @@ -60,4 +65,8 @@ scripts/installer/default.config
tests/node_modules
tests/package-lock.json
venv
/build/

# from thumbnail tests in SearchIT
scripts/search/data/binary/trees.png.thumb140
src/main/webapp/resources/images/cc0.png.thumb140
src/main/webapp/resources/images/dataverseproject.png.thumb140
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
language: java
jdk:
- openjdk8
- openjdk11
script: mvn -DcompilerArgument=-Xlint:unchecked test -P all-unit-tests
after_success: mvn jacoco:report coveralls:report
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1 +1 @@
# See `conf/docker` for Docker images
# See http://guides.dataverse.org/en/latest/developers/containers.html
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,10 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
[![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg?raw=true "Dataverse Project")](http://dataverse.org)

[![API Test Status](https://jenkins.dataverse.org/buildStatus/icon?job=IQSS-dataverse-develop&subject=API%20Test%20Status)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/)
[![API Test Coverage](https://img.shields.io/jenkins/coverage/jacoco?jobUrl=https%3A%2F%2Fjenkins.dataverse.org%2Fjob%2FIQSS-dataverse-develop&label=API%20Test%20Coverage)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/)
[![API Test Coverage](https://img.shields.io/jenkins/coverage/jacoco?jobUrl=https%3A%2F%2Fjenkins.dataverse.org%2Fjob%2FIQSS-dataverse-develop&label=API%20Test%20Coverage)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/ws/target/coverage-it/index.html)
[![Unit Test Status](https://img.shields.io/travis/IQSS/dataverse?label=Unit%20Test%20Status)](https://travis-ci.org/IQSS/dataverse)
[![Unit Test Coverage](https://img.shields.io/coveralls/github/IQSS/dataverse?label=Unit%20Test%20Coverage)](https://coveralls.io/github/IQSS/dataverse?branch=develop)
[![Guides Build Status](https://github.com/IQSS/dataverse/actions/workflows/guides_build_sphinx.yml/badge.svg)](https://github.com/IQSS/dataverse/actions/workflows/guides_build_sphinx.yml)

[dataverse.org]: https://dataverse.org
[demo.dataverse.org]: https://demo.dataverse.org
Expand Down
80 changes: 15 additions & 65 deletions Vagrantfile
Original file line number Diff line number Diff line change
@@ -1,77 +1,27 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "bento/centos-8.2"

config.vm.define "standalone", primary: true do |standalone|
config.vm.hostname = "standalone"
# Uncomment this temporarily to get `vagrant destroy` to work
#standalone.vm.box = "puppetlabs/centos-7.2-64-puppet"

operating_system = "centos"
if ENV['OPERATING_SYSTEM'].nil?
config.vm.box = "puppetlabs/centos-7.2-64-puppet"
config.vm.box_version = '1.0.1'
elsif ENV['OPERATING_SYSTEM'] == 'debian'
puts "WARNING: Debian specified. Here be dragons! https://github.com/IQSS/dataverse/issues/1059"
config.vm.box_url = "http://puppet-vagrant-boxes.puppetlabs.com/debian-73-x64-virtualbox-puppet.box"
config.vm.box = "puppet-vagrant-boxes.puppetlabs.com-debian-73-x64-virtualbox-puppet.box"
else
operating_system = ENV['OPERATING_SYSTEM']
puts "Not sure what do to with operating system: #{operating_system}"
exit 1
end

mailserver = "localhost"
if ENV['MAIL_SERVER'].nil?
puts "MAIL_SERVER environment variable not specified. Using #{mailserver} by default.\nTo specify it in bash: export MAIL_SERVER=localhost"
else
mailserver = ENV['MAIL_SERVER']
puts "MAIL_SERVER environment variable found, using #{mailserver}"
end

config.vm.provider "virtualbox" do |v|
v.memory = 2048
v.cpus = 1
end
config.vm.provision "shell", path: "scripts/vagrant/setup.sh"
config.vm.provision "shell", path: "scripts/vagrant/setup-solr.sh"
config.vm.provision "shell", path: "scripts/vagrant/install-dataverse.sh", args: mailserver
# FIXME: get tests working and re-enable them!
#config.vm.provision "shell", path: "scripts/vagrant/test.sh"

config.vm.network "private_network", type: "dhcp"
config.vm.network "forwarded_port", guest: 80, host: 8888
config.vm.network "forwarded_port", guest: 443, host: 9999
config.vm.network "forwarded_port", guest: 8983, host: 8993
config.vm.network "forwarded_port", guest: 8080, host: 8088
config.vm.network "forwarded_port", guest: 8181, host: 8188

# FIXME: use /dataverse/downloads instead
config.vm.synced_folder "downloads", "/downloads"
# FIXME: use /dataverse/conf instead
config.vm.synced_folder "conf", "/conf"
# FIXME: use /dataverse/scripts instead
config.vm.synced_folder "scripts", "/scripts"
config.vm.synced_folder ".", "/dataverse"
config.vm.provider "virtualbox" do |vbox|
vbox.cpus = 4
vbox.memory = 4096
end

config.vm.define "solr", autostart: false do |solr|
config.vm.hostname = "solr"
solr.vm.box = "puppet-vagrant-boxes.puppetlabs.com-centos-65-x64-virtualbox-puppet.box"
config.vm.synced_folder ".", "/dataverse"
config.vm.network "private_network", type: "dhcp"
config.vm.network "forwarded_port", guest: 8983, host: 9001
end
config.vm.provision "shell", path: "scripts/vagrant/setup.sh"
config.vm.provision "shell", path: "scripts/vagrant/setup-solr.sh"
config.vm.provision "shell", path: "scripts/vagrant/install-dataverse.sh"

config.vm.define "test", autostart: false do |test|
config.vm.hostname = "test"
test.vm.box = "puppet-vagrant-boxes.puppetlabs.com-centos-65-x64-virtualbox-puppet.box"
config.vm.synced_folder ".", "/dataverse"
config.vm.network "private_network", type: "dhcp"
end
config.vm.network "private_network", type: "dhcp"
config.vm.network "forwarded_port", guest: 80, host: 8888
config.vm.network "forwarded_port", guest: 443, host: 9999
config.vm.network "forwarded_port", guest: 8983, host: 8993
config.vm.network "forwarded_port", guest: 8080, host: 8088
config.vm.network "forwarded_port", guest: 8181, host: 8188

config.vm.synced_folder ".", "/dataverse"

end
24 changes: 8 additions & 16 deletions conf/docker-aio/0prep_deps.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,18 @@ if [ ! -d dv/deps ]; then
mkdir -p dv/deps
fi
wdir=`pwd`
if [ ! -e dv/deps/glassfish4dv.tgz ]; then
echo "glassfish dependency prep"
mkdir -p /tmp/dv-prep/gf
cd /tmp/dv-prep/gf
wget http://download.java.net/glassfish/4.1/release/glassfish-4.1.zip
wget https://search.maven.org/remotecontent?filepath=org/jboss/weld/weld-osgi-bundle/2.2.10.Final/weld-osgi-bundle-2.2.10.Final-glassfish4.jar -O weld-osgi-bundle-2.2.10.Final-glassfish4.jar
unzip glassfish-4.1.zip
rm glassfish4/glassfish/modules/weld-osgi-bundle.jar
mv weld-osgi-bundle-2.2.10.Final-glassfish4.jar glassfish4/glassfish/modules
tar zcf $wdir/dv/deps/glassfish4dv.tgz glassfish4
cd $wdir
# assuming that folks usually have /tmp auto-clean as needed

if [ ! -e dv/deps/payara-5.2020.6.zip ]; then
echo "payara dependency prep"
# no more fiddly patching :)
wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2020.6/payara-5.2020.6.zip -O dv/deps/payara-5.2020.6.zip
fi

if [ ! -e dv/deps/solr-7.7.2dv.tgz ]; then
if [ ! -e dv/deps/solr-8.8.1dv.tgz ]; then
echo "solr dependency prep"
# schema changes *should* be the only ones...
cd dv/deps/
#wget https://archive.apache.org/dist/lucene/solr/7.3.0/solr-7.3.0.tgz -O solr-7.3.0dv.tgz
wget https://archive.apache.org/dist/lucene/solr/7.7.2/solr-7.7.2.tgz -O solr-7.7.2dv.tgz
cd dv/deps/
wget https://archive.apache.org/dist/lucene/solr/8.8.1/solr-8.8.1.tgz -O solr-8.8.1dv.tgz
cd ../../
fi

22 changes: 16 additions & 6 deletions conf/docker-aio/1prep.sh
Original file line number Diff line number Diff line change
@@ -1,22 +1,32 @@
#!/bin/sh
#!/bin/bash

# move things necessary for integration tests into build context.
# this was based off the phoenix deployment; and is likely uglier and bulkier than necessary in a perfect world

mkdir -p testdata/doc/sphinx-guides/source/_static/util/
cp ../solr/7.7.2/schema*.xml testdata/
cp ../solr/7.7.2/solrconfig.xml testdata/
cp ../solr/7.7.2/updateSchemaMDB.sh testdata/
cp ../solr/8.8.1/schema*.xml testdata/
cp ../solr/8.8.1/solrconfig.xml testdata/
cp ../solr/8.8.1/updateSchemaMDB.sh testdata/
cp ../jhove/jhove.conf testdata/
cp ../jhove/jhoveConfig.xsd testdata/
cd ../../
cp -r scripts conf/docker-aio/testdata/
cp doc/sphinx-guides/source/_static/util/createsequence.sql conf/docker-aio/testdata/doc/sphinx-guides/source/_static/util/

wget -q https://downloads.apache.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
tar xfz apache-maven-3.6.3-bin.tar.gz
mkdir maven
mv apache-maven-3.6.3/* maven/
echo "export JAVA_HOME=/usr/lib/jvm/jre-openjdk" > maven/maven.sh
echo "export M2_HOME=../maven" >> maven/maven.sh
echo "export MAVEN_HOME=../maven" >> maven/maven.sh
echo "export PATH=../maven/bin:${PATH}" >> maven/maven.sh
chmod 0755 maven/maven.sh

# not using dvinstall.zip for setupIT.bash; but still used in install.bash for normal ops
mvn clean
source maven/maven.sh && mvn clean
./scripts/installer/custom-build-number
mvn package
source maven/maven.sh && mvn package
cd scripts/installer
make clean
make
Expand Down
41 changes: 23 additions & 18 deletions conf/docker-aio/c7.dockerfile → conf/docker-aio/c8.dockerfile
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
FROM centos:7
FROM centos:8
# OS dependencies
RUN yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm
#RUN yum install -y java-1.8.0-openjdk-headless postgresql-server sudo epel-release unzip perl curl httpd
RUN yum install -y java-1.8.0-openjdk-devel postgresql96-server sudo epel-release unzip perl curl httpd
# PG 10 is the default in centos8; keep the repo comment for when we bump to 11+
#RUN yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm

RUN echo "fastestmirror=true" >> /etc/dnf/dnf.conf
RUN yum install -y java-11-openjdk-devel postgresql-server sudo epel-release unzip curl httpd python2 diffutils
RUN yum install -y jq lsof awscli

# copy and unpack dependencies (solr, glassfish)
# for older search scripts
RUN ln -s /usr/bin/python2 /usr/bin/python

# copy and unpack dependencies (solr, payara)
COPY dv /tmp/dv
COPY testdata/schema*.xml /tmp/dv/
COPY testdata/solrconfig.xml /tmp/dv
Expand All @@ -17,30 +22,30 @@ COPY testdata/sushi_sample_logs.json /tmp/
COPY disableipv6.conf /etc/sysctl.d/
RUN rm /etc/httpd/conf/*
COPY httpd.conf /etc/httpd/conf
RUN cd /opt ; tar zxf /tmp/dv/deps/solr-7.7.2dv.tgz
RUN cd /opt ; tar zxf /tmp/dv/deps/glassfish4dv.tgz
RUN cd /opt ; tar zxf /tmp/dv/deps/solr-8.8.1dv.tgz
RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2020.6.zip ; ln -s /opt/payara5 /opt/glassfish4

# this copy of domain.xml is the result of running `asadmin set server.monitoring-service.module-monitoring-levels.jvm=LOW` on a default glassfish installation (aka - enable the glassfish REST monitir endpoint for the jvm`
COPY domain-restmonitor.xml /opt/glassfish4/glassfish/domains/domain1/config/domain.xml
# this dies under Java 11, do we keep it?
#COPY domain-restmonitor.xml /opt/payara5/glassfish/domains/domain1/config/domain.xml

#RUN sudo -u postgres /usr/bin/initdb -D /var/lib/pgsql/data
RUN sudo -u postgres /usr/pgsql-9.6/bin/initdb -D /var/lib/pgsql/data
RUN sudo -u postgres /usr/bin/initdb /var/lib/pgsql/data

# copy configuration related files
RUN cp /tmp/dv/pg_hba.conf /var/lib/pgsql/data/
RUN cp -r /opt/solr-7.7.2/server/solr/configsets/_default /opt/solr-7.7.2/server/solr/collection1
RUN cp /tmp/dv/schema*.xml /opt/solr-7.7.2/server/solr/collection1/conf/
RUN cp /tmp/dv/solrconfig.xml /opt/solr-7.7.2/server/solr/collection1/conf/solrconfig.xml
RUN cp -r /opt/solr-8.8.1/server/solr/configsets/_default /opt/solr-8.8.1/server/solr/collection1
RUN cp /tmp/dv/schema*.xml /opt/solr-8.8.1/server/solr/collection1/conf/
RUN cp /tmp/dv/solrconfig.xml /opt/solr-8.8.1/server/solr/collection1/conf/solrconfig.xml

# skipping glassfish user and solr user (run both as root)
# skipping payara user and solr user (run both as root)

#solr port
EXPOSE 8983

# postgres port
EXPOSE 5432

# glassfish port
# payara port
EXPOSE 8080

# apache port, http
Expand All @@ -52,8 +57,8 @@ EXPOSE 9009

RUN mkdir /opt/dv

# yeah - still not happy if glassfish isn't in /usr/local :<
RUN ln -s /opt/glassfish4 /usr/local/glassfish4
# keeping the symlink on the off chance that something else is still assuming /usr/local/glassfish4
RUN ln -s /opt/payara5 /usr/local/glassfish4
COPY dv/install/ /opt/dv/
COPY install.bash /opt/dv/
COPY entrypoint.bash /opt/dv/
Expand All @@ -76,7 +81,7 @@ ENV doi_username=${doi_username}
ENV doi_password=${doi_password}
COPY configure_doi.bash /opt/dv

# healthcheck for glassfish only (assumes modified domain.xml);
# healthcheck for payara only (assumes modified domain.xml);
# does not check dataverse application status.
HEALTHCHECK CMD curl --fail http://localhost:4848/monitoring/domain/server.json || exit 1
CMD ["/opt/dv/entrypoint.bash"]
16 changes: 8 additions & 8 deletions conf/docker-aio/configure_doi.bash
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
#!/usr/bin/env bash

cd /usr/local/glassfish4
cd /opt/payara5

# if appropriate; reconfigure PID provider on the basis of environmental variables.
if [ ! -z "${DoiProvider}" ]; then
curl -X PUT -d ${DoiProvider} http://localhost:8080/api/admin/settings/:DoiProvider
fi
if [ ! -z "${doi_username}" ]; then
bin/asadmin create-jvm-options "-Ddoi.username=${doi_password}"
bin/asadmin create-jvm-options "-Ddoi.username=${doi_username}"
fi
if [ ! -z "${doi_password}" ]; then
bin/asadmin create-jvm-options "-Ddoi.password=${doi_password}"
fi
if [ ! -z "${doi_baseurl}" ]; then
bin/asadmin delete-jvm-options "-Ddoi.baseurlstring=https\://mds.test.datacite.org"
doi_baseurl_esc=`echo ${doi_baseurl} | sed -e 's/:/\\:/'`
bin/asadmin create-jvm-options "\"-Ddoi.baseurlstring=${doi_baseurl_esc}\""
doi_baseurl_esc=`echo ${doi_baseurl} | sed -e 's/:/\\\:/'`
bin/asadmin create-jvm-options "-Ddoi.baseurlstring=${doi_baseurl_esc}"
fi
if [ ! -z "${doi_mdcbaseurl}" ]; then
bin/asadmin delete-jvm-options "-Ddoi.mdcbaseurlstring=https\://api.test.datacite.org"
doi_mdcbaseurl_esc=`echo ${doi_mdcbaseurl} | sed -e 's/:/\\:/'`
bin/asadmin create-jvm-options "\"-Ddoi.mdcbaseurlstring=${doi_mdcbaseurl_esc}\""
if [ ! -z "${doi_dataciterestapiurl}" ]; then
bin/asadmin delete-jvm-options "-Ddoi.dataciterestapiurlstring=https\://api.test.datacite.org"
doi_dataciterestapiurl_esc=`echo ${doi_dataciterestapiurl} | sed -e 's/:/\\\:/'`
bin/asadmin create-jvm-options "-Ddoi.dataciterestapiurlstring=${doi_dataciterestapiurl_esc}"
fi
5 changes: 2 additions & 3 deletions conf/docker-aio/entrypoint.bash
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
#!/usr/bin/env bash
export LANG=en_US.UTF-8
#sudo -u postgres /usr/bin/postgres -D /var/lib/pgsql/data &
sudo -u postgres /usr/pgsql-9.6/bin/postgres -D /var/lib/pgsql/data &
cd /opt/solr-7.7.2/
sudo -u postgres /usr/bin/pg_ctl start -D /var/lib/pgsql/data &
cd /opt/solr-8.8.1/
# TODO: Run Solr as non-root and remove "-force".
bin/solr start -force
bin/solr create_core -c collection1 -d server/solr/collection1/conf -force
Expand Down
2 changes: 1 addition & 1 deletion conf/docker-aio/prep_it.bash
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ n_wait=5
cd conf/docker-aio
./0prep_deps.sh
./1prep.sh
docker build -t dv0 -f c7.dockerfile .
docker build -t dv0 -f c8.dockerfile .
# cleanup from previous runs if necessary
docker rm -f dv
# start container
Expand Down
Loading

0 comments on commit 2feb4db

Please sign in to comment.