Merge pull request #4935 from IQSS/develop

Merge v4.9.2

README.md

@@ -1,4 +1,4 @@
-Dataverse®
+Dataverse® 
 ===============
 
 Dataverse is an [open source][] web application for sharing, citing, analyzing, and preserving research data (developed by the [Data Science and Products team](http://www.iq.harvard.edu/people/people/data-science-products) at the [Institute for Quantitative Social Science](http://iq.harvard.edu/) and the [Dataverse community][]).
@@ -17,6 +17,8 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 
 [![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg?raw=true "Dataverse Project")](http://dataverse.org)
 
+[![Waffle.io - Columns and their card count](https://badge.waffle.io/IQSS/dataverse.svg?columns=all)](https://waffle.io/IQSS/dataverse)
+
 [![Build Status](https://travis-ci.org/IQSS/dataverse.svg?branch=develop)](https://travis-ci.org/IQSS/dataverse) [![Coverage Status](https://coveralls.io/repos/IQSS/dataverse/badge.svg?branch=develop&service=github)](https://coveralls.io/github/IQSS/dataverse?branch=develop)
 
 [dataverse.org]: https://dataverse.org

conf/docker-aio/1prep.sh

@@ -22,5 +22,6 @@ mvn package
 cd scripts/installer
 make clean
 make
+mkdir -p ../../conf/docker-aio/dv/install
 cp dvinstall.zip ../../conf/docker-aio/dv/install/
 

conf/docker-aio/readme.txt → conf/docker-aio/readme.md

@@ -1,43 +1,59 @@
+# Docker All-In-One
+
 First pass docker all-in-one image, intended for running integration tests against.
 Also usable for normal development and system evaluation; not intended for production.
 
-Quickstart:
+### Requirements:
+ - java8 compiler, maven, make, wget, docker
+
+### Quickstart:
  - in the root of the repository, run `./conf/docker-aio/prep_it.bash`
  - if all goes well, you should see the results of the `api/info/version` endpoint, including the deployed build (eg `{"status":"OK","data":{"version":"4.8.6","build":"develop-c3e9f40"}}`). If not, you may need to read the non-quickstart instructions.
  - run integration tests: `./conf/docker-aio/run-test-suite.sh`
 
-Requirements:
- - java compiler, maven, make, wget, docker
-
 ----
 
-More in-depth documentation:
+## More in-depth documentation:
 
 
-Initial setup (aka - do once):
+### Initial setup (aka - do once):
 - `cd conf/docker-aio` and run `./0prep_deps.sh` to created Glassfish and Solr tarballs in `conf/docker-aio/dv/deps`.
 
-Per-build:
-- `cd conf/docker-aio`, and run `./1prep.sh` to copy files for integration test data into docker build context; `1prep.sh` will also build the war file and installation zip file
-- build the docker image: `docker build -t dv0 -f c7.dockerfile .`
+### Per-build:
 
-- Run image: `docker run -d -p 8083:8080 -p 8084:80 --name dv dv0` (aka - forward port 8083 locally to 8080 in the container for glassfish, and 8084 to 80 for apache); if you'd like to connect a java debugger to glassfish, use `docker run -d -p 8083:8080 -p 8084:80 -p 9010:9009 --name dv dv0`
+> Note: If you encounter any issues, see the Troubleshooting section at the end of this document.
 
-Note: If you see an error like this... `docker: Error response from daemon: Conflict. The container name "/dv" is already in use by container "5f72a45b68c86c7b0f4305b83ce7d663020329ea4e30fa2a3ce9ddb05223533d". You have to remove (or rename) that container to be able to reuse that name.` ... run something like `docker ps -a | grep dv` to see the container left over from the last run and something like `docker rm 5f72a45b68c8` to remove it. Then try the `docker run` command above again.
+#### Setup
 
-Another troubleshooting note: `empty reply from server` or `Failed to connect to ::1: Cannot assign requested address` tend to indicate either that you haven't given glassfish enough time to start, or your docker setup is in an inconsistent state and should probably be restarted.
+- `cd conf/docker-aio`, and run `./1prep.sh` to copy files for integration test data into docker build context; `1prep.sh` will also build the war file and installation zip file
+- build the docker image: `docker build -t dv0 -f c7.dockerfile .`
 
-- Installation (integration test): `docker exec -it dv /opt/dv/setupIT.bash`
+- Run image: `docker run -d -p 8083:8080 -p 8084:80 --name dv dv0` (aka - forward port 8083 locally to 8080 in the container for glassfish, and 8084 to 80 for apache); if you'd like to connect a java debugger to glassfish, use `docker run -d -p 8083:8080 -p 8084:80 -p 9010:9009 --name dv dv0`
 
-(Note that it's possible to customize the installation by editing `conf/docker-aio/default.config` and running `docker exec -it dv /opt/dv/install.bash` but for the purposes of integration testing, the `setupIT.bash` script above works fine.)
+- Installation (integration test): `docker exec -it dv /opt/dv/setupIT.bash` 
+  (Note that it's possible to customize the installation by editing `conf/docker-aio/default.config` and running `docker exec -it dv /opt/dv/install.bash` but for the purposes of integration testing, the `setupIT.bash` script above works fine.)
 
 - update `dataverse.siteUrl` (appears only necessary for `DatasetsIT.testPrivateUrl`): `docker exec -it dv /usr/local/glassfish4/bin/asadmin create-jvm-options "-Ddataverse.siteUrl=http\://localhost\:8084"`
 
-Run integration tests: 
+#### Run integration tests: 
 
 First, cd back to the root of the repo where the `pom.xml` file is (`cd ../..` assuming you're still in the `conf/docker-aio` directory). Then run the test suite with script below:
 
 `conf/docker-aio/run-test-suite.sh`
 
 There isn't any strict requirement on the local port (8083, 8084 in this doc), the name of the image (dv0) or container (dv), these can be changed as desired as long as they are consistent.
 
+### Troubleshooting Notes:
+
+* If Dataverse' build fails due to an error about `Module` being ambiguous, you might be using a Java 9 compiler.
+
+* If you see an error like this: 
+ ```
+ docker: Error response from daemon: Conflict. The container name "/dv" is already in use by container "5f72a45b68c86c7b0f4305b83ce7d663020329ea4e30fa2a3ce9ddb05223533d"
+ You have to remove (or rename) that container to be able to reuse that name.
+ ``` 
+    run something like `docker ps -a | grep dv` to see the container left over from the last run and something like `docker rm 5f72a45b68c8` to remove it. Then try the `docker run` command above again.
+
+* `empty reply from server` or `Failed to connect to ::1: Cannot assign requested address` tend to indicate either that you haven't given glassfish enough time to start, or your docker setup is in an inconsistent state and should probably be restarted.
+
+* For manually fiddling around with the created dataverse, use user `dataverseAdmin` with password `admin`.

conf/docker-aio/testscripts/install

@@ -15,7 +15,7 @@ export SMTP_SERVER=localhost
 export MEM_HEAP_SIZE=2048
 export GLASSFISH_DOMAIN=domain1
 cd scripts/installer
-cp pgdriver/postgresql-8.4-703.jdbc4.jar $GLASSFISH_ROOT/glassfish/lib
+cp pgdriver/postgresql-42.2.2.jar $GLASSFISH_ROOT/glassfish/lib
 #cp ../../conf/jhove/jhove.conf $GLASSFISH_ROOT/glassfish/domains/$GLASSFISH_DOMAIN/config/jhove.conf
 cp /opt/dv/testdata/jhove.conf $GLASSFISH_ROOT/glassfish/domains/$GLASSFISH_DOMAIN/config/jhove.conf
 cp /opt/dv/testdata/jhoveConfig.xsd $GLASSFISH_ROOT/glassfish/domains/$GLASSFISH_DOMAIN/config/jhoveConfig.xsd

conf/docker/build.sh

@@ -1,15 +1,16 @@
-#!/bin/sh
+#!/bin/bash
 # Creates images and pushes them to Docker Hub.
 # The "latest" tag under "iqss" should be relatively stable. Don't push breaking changes there.
 # None of the tags are suitable for production use. See https://github.com/IQSS/dataverse/issues/4040
-# To interate on images, push to custom tags or tags based on branch names or a non-iqss Docker Hub org/username.
+# To iterate on images, push to custom tags or tags based on branch names or a non-iqss Docker Hub org/username. Consider trying "internal" to push to the internal Minishift registry.
+
 # Docker Hub organization or username
 HUBORG=iqss
 # The most stable tag we have.
 STABLE=latest
 #FIXME: Use a real flag/argument parser. download-files.sh uses "getopts" for example.
 if [ -z "$1" ]; then
-  echo "No argument supplied. For experiments, specify \"branch\" or \"custom my-custom-tag\" or \"huborg <USERNAME/ORG>\". Specify \"stable\" to push to the \"$STABLE\" tag under \"$HUBORG\" if your change won't break anything."
+  echo "No argument supplied. For experiments, specify \"branch\" or \"custom my-custom-tag\" or \"huborg <USERNAME/ORG>\" or \"internal\". Specify \"stable\" to push to the \"$STABLE\" tag under \"$HUBORG\" if your change won't break anything."
   exit 1
 fi
 
@@ -37,14 +38,29 @@ elif [ "$1" == 'huborg' ]; then
     TAG=$STABLE
     echo "We'll push to the Docker Hub organization or username you specified: $HUBORG."
   fi
+elif [ "$1" == 'internal' ]; then
+  echo "Building for internal Minishift registry."
+  TAG=$STABLE
 else
   echo "Unexpected argument: $1. Exiting. Run with no arguments for help."
+  TAG=$STABLE
   exit 1
 fi
-echo Images will be pushed to Docker Hub org/username \"$HUBORG\" with the tag \"$TAG\".
+echo Images are being built for registry org/username \"$HUBORG\" with the tag \"$TAG\".
+#
+# Build dataverse-solr
+#
 # Use "conf" directory as context so we can copy schema.xml into Solr image.
 docker build -t $HUBORG/dataverse-solr:$TAG -f solr/Dockerfile ../../conf
-docker push $HUBORG/dataverse-solr:$TAG
+if [ "$1" == 'internal' ]; then
+  echo "Skipping docker push because we're using the internal Minishift registry."
+else
+  # FIXME: Check the output of "docker build" and only push on success.
+  docker push $HUBORG/dataverse-solr:$TAG
+fi
+#
+# Build dataverse-glassfish
+#
 # TODO: Think about if we really need dataverse.war because it's in dvinstall.zip.
 cd ../..
 mvn clean
@@ -73,5 +89,20 @@ fi
 # We'll assume at this point that the download script has been run.
 cp ../../downloads/weld-osgi-bundle-2.2.10.Final-glassfish4.jar dataverse-glassfish
 docker build -t $HUBORG/dataverse-glassfish:$TAG dataverse-glassfish
-# FIXME: Check the output of `docker build` and only push on success.
-docker push $HUBORG/dataverse-glassfish:$TAG
+if [ "$1" == 'internal' ]; then
+  echo "Skipping docker push because we're using the internal Minishift registry."
+else
+  # FIXME: Check the output of "docker build" and only push on success.
+  docker push $HUBORG/dataverse-glassfish:$TAG
+fi
+#
+# Build init-container
+#
+cp ../../scripts/installer/postgres-setup dataverse-glassfish/init-container
+docker build -t $HUBORG/init-container:$TAG dataverse-glassfish/init-container
+if [ "$1" == 'internal' ]; then
+  echo "Skipping docker push because we're using the internal Minishift registry."
+else
+  # FIXME: Check the output of "docker build" and only push on success.
+  docker push $HUBORG/init-container:$TAG
+fi

conf/docker/dataverse-glassfish/.gitignore

@@ -2,3 +2,4 @@ glassfish-4.1.zip
 weld-osgi-bundle-2.2.10.Final-glassfish4.jar
 dvinstall.zip
 dataverse.war
+init-container/postgres-setup

conf/docker/dataverse-glassfish/Dockerfile

@@ -1,11 +1,9 @@
-FROM centos:7.2.1511
+FROM centos:7
 MAINTAINER Dataverse ([email protected])
 
 COPY glassfish-4.1.zip /tmp
 COPY weld-osgi-bundle-2.2.10.Final-glassfish4.jar /tmp
 COPY default.config /tmp
-COPY dvinstall.zip /tmp
-
 # Install dependencies
 #RUN yum install -y unzip
 RUN yum install -y \
@@ -27,20 +25,23 @@ ENV POSTGRES_USER "dvnapp"
 ENV RSERVE_USER "rserve"
 ENV RSERVE_PASSWORD "rserve"
 
+COPY dvinstall.zip /tmp
+
 #RUN ls /tmp
 #
 RUN find /tmp
 #
 #RUN exitEarly
 
 # Install Glassfish 4.1
-
-RUN cd /tmp \ 
+RUN cd /tmp \
     && unzip glassfish-4.1.zip \
     && mv glassfish4 /usr/local \
     && cd /usr/local/glassfish4/glassfish/modules \
     && rm weld-osgi-bundle.jar \
     && cp /tmp/weld-osgi-bundle-2.2.10.Final-glassfish4.jar . \
+    && cd /tmp && unzip /tmp/dvinstall.zip \
+    && chmod 777 -R /tmp/dvinstall/ \
     #FIXME: Patch Grizzly too!
     && echo "Done installing and patching Glassfish"
 
@@ -57,14 +58,21 @@ RUN chgrp -R 0 /usr/local/glassfish4 && \
 
 #RUN exitEarlyBeforeJq
 RUN yum -y install epel-release
-RUN yum install -y jq 
+# RUN rpm --rebuilddb 
+RUN yum -y install jq 
 
-# Install jq
-#RUN cd /tmp \  
-#    && wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 \
-#    && mv jq-linux64 /usr/local/bin \
-#    && chmod +x /usr/local/bin/jq-linux64 \
-#    && ln -s /usr/local/bin/jq-linux64 /usr/local/bin/jq
+#JHOVE
+RUN cp /tmp/dvinstall/jhove* /usr/local/glassfish4/glassfish/domains/domain1/config
+
+
+#SETUP JVM OPTIONS 
+ARG DOCKER_BUILD="true" 
+RUN echo $DOCKER_BUILD
+RUN /tmp/dvinstall/glassfish-setup.sh
+###glassfish-setup will handle everything in Dockerbuild
+
+##install jdbc driver 
+RUN cp /tmp/dvinstall/pgdriver/postgresql-42.2.2.jar /usr/local/glassfish4/glassfish/domains/domain1/lib
 
 # Customized persistence xml to avoid database recreation
 #RUN mkdir -p /tmp/WEB-INF/classes/META-INF/ 
@@ -80,7 +88,6 @@ RUN yum install -y jq
 #COPY config-dataverse /root/dvinstall
 
 #RUN cd /root/dvinstall && ./config-dataverse
-
 COPY ./entrypoint.sh /
 #COPY ./ddl /root/dvinstall
 #COPY ./init-postgres /root/dvinstall
@@ -90,8 +97,6 @@ COPY ./entrypoint.sh /
 #COPY ./setup-irods.sh /root/dvinstall
 COPY ./Dockerfile /
 
-VOLUME /usr/local/glassfish4/glassfish/domains/domain1/files
-
 EXPOSE      8080 
 
 ENTRYPOINT ["/entrypoint.sh"]

conf/docker/dataverse-glassfish/entrypoint.sh

@@ -109,29 +109,26 @@ if [ "$1" = 'dataverse' ]; then
     fi
 
     GLASSFISH_INSTALL_DIR="/usr/local/glassfish4"
-    cd $GLASSFISH_INSTALL_DIR
-    cp /tmp/dvinstall.zip $GLASSFISH_INSTALL_DIR
-    unzip dvinstall.zip
-    cd dvinstall
+    cd /tmp/dvinstall
     echo Copying the non-interactive file into place
     cp /tmp/default.config .
     echo Looking at first few lines of default.config
     head default.config
     # non-interactive install
     echo Running non-interactive install
     #./install -y -f > install.out 2> install.err
-    ./install -y -f
+    ./install -y -f 
 
 #    if [ -n "$DVICAT_PORT_1247_TCP_PORT" ]; then
 #        ./setup-irods.sh
 #    fi
 
     # We do change the Solr server in Minishift/OpenShift, which is
     # the primary target for all of the work under conf/docker.
-    echo -e "\n\nRestarting Dataverse in case Solr host was changed..."
-    /usr/local/glassfish4/glassfish/bin/asadmin stop-domain
-    sleep 3
-    /usr/local/glassfish4/glassfish/bin/asadmin start-domain
+    # echo -e "\n\nRestarting Dataverse in case Solr host was changed..."
+    # /usr/local/glassfish4/glassfish/bin/asadmin stop-domain
+    # sleep 3
+    # /usr/local/glassfish4/glassfish/bin/asadmin start-domain
 
     echo -e "\n\nDataverse started"
 

conf/docker/dataverse-glassfish/init-container/Dockerfile

@@ -0,0 +1,17 @@
+FROM centos:7
+MAINTAINER Dataverse ([email protected])
+
+### init-container is an Init Container for glassfish service in OpenShift or other Kubernetes environment
+# This initContainer will take care of setting up glassfish
+
+# Install dependencies
+RUN yum install -y \
+    nc \
+    perl \
+    postgresql \
+    sha1sum 
+
+COPY postgres-setup /
+
+ENTRYPOINT ["/postgres-setup"]
+CMD ["dataverse"]

conf/docker/dataverse-glassfish/init-container/default.config

@@ -0,0 +1,16 @@
+HOST_DNS_ADDRESS	localhost
+GLASSFISH_DIRECTORY	/usr/local/glassfish4
+ADMIN_EMAIL	 
+MAIL_SERVER	mail.hmdc.harvard.edu
+POSTGRES_ADMIN_PASSWORD	secret
+POSTGRES_SERVER	dataverse-postgresql-0.dataverse-postgresql-service
+POSTGRES_PORT	5432
+POSTGRES_DATABASE	dvndb
+POSTGRES_USER	dvnapp
+POSTGRES_PASSWORD	secret
+SOLR_LOCATION	dataverse-solr-service:8983
+TWORAVENS_LOCATION	NOT INSTALLED
+RSERVE_HOST	localhost
+RSERVE_PORT	6311
+RSERVE_USER	rserve
+RSERVE_PASSWORD	rserve