Merge pull request #8585 from IQSS/develop

Version v5.10.1

conf/docker-aio/0prep_deps.sh

@@ -4,9 +4,9 @@ if [ ! -d dv/deps ]; then
 fi
 wdir=`pwd`
 
-if [ ! -e dv/deps/payara-5.2021.5.zip ]; then
+if [ ! -e dv/deps/payara-5.2021.6.zip ]; then
 	echo "payara dependency prep"
-	wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.5/payara-5.2021.5.zip  -O dv/deps/payara-5.2021.5.zip
+	wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.6/payara-5.2021.6.zip  -O dv/deps/payara-5.2021.6.zip
 fi
 
 if [ ! -e dv/deps/solr-8.11.1dv.tgz ]; then

conf/docker-aio/c8.dockerfile

@@ -24,7 +24,7 @@ COPY disableipv6.conf /etc/sysctl.d/
 RUN rm /etc/httpd/conf/*
 COPY httpd.conf /etc/httpd/conf 
 RUN cd /opt ; tar zxf /tmp/dv/deps/solr-8.11.1dv.tgz
-RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2021.5.zip ; ln -s /opt/payara5 /opt/glassfish4
+RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2021.6.zip ; ln -s /opt/payara5 /opt/glassfish4
 
 # this copy of domain.xml is the result of running `asadmin set server.monitoring-service.module-monitoring-levels.jvm=LOW` on a default glassfish installation (aka - enable the glassfish REST monitir endpoint for the jvm`
 # this dies under Java 11, do we keep it?

doc/release-notes/5.10.1-release-notes.md

@@ -0,0 +1,96 @@
+# Dataverse Software 5.10.1
+
+This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### Bug Fix for Request Access
+
+Dataverse Software 5.10 contains a bug where the "Request Access" button doesn't work from the file listing on the dataset page if the dataset contains custom terms. This has been fixed in PR #8555.
+
+### Bug Fix for Searching and Selecting Controlled Vocabulary Values
+
+Dataverse Software 5.10 contains a bug where the search option is no longer present when selecting from more than ten controlled vocabulary values. This has been fixed in PR #8521.
+
+## Major Use Cases and Infrastructure Enhancements
+
+Changes and fixes in this release include:
+
+- Users can use the "Request Access" button when the dataset has custom terms. (Issue #8553, PR #8555)
+- Users can search when selecting from more than ten controlled vocabulary values. (Issue #8519, PR #8521)
+- The default file categories ("Documentation", "Data", and "Code") can be redefined through the `:FileCategories` database setting. (Issue #8461, PR #8478)
+- Documentation on troubleshooting Excel ingest errors was improved. (PR #8541)
+- Internationalized controlled vocabulary values can now be searched. (Issue #8286, PR #8435)
+- Curation labels can be internationalized. (Issue #8381, PR #8466)
+- "NONE" is no longer accepted as a license using the SWORD API (since 5.10). See "Backward Incompatibilities" below for details. (Issue #8551, PR #8558).
+
+## Notes for Dataverse Installation Administrators
+
+### PostgreSQL Version 10+ Required Soon
+
+Because 5.10.1 is a bug fix release, an upgrade to PostgreSQL is not required. However, this upgrade is still coming in the next non-bug fix release. For details, please see the release notes for 5.10: https://github.com/IQSS/dataverse/releases/tag/v5.10
+
+### Payara Upgrade
+
+You may notice that the Payara version used in the install scripts has been updated from 5.2021.5 to 5.2021.6. This was to address a bug where it was not possible to easily update the logging level. For existing installations, this release does not require upgrading Payara and a Payara upgrade is not part of the Upgrade Instructions below. For more information, see PR #8508.
+
+## New JVM Options and DB Settings
+
+The following DB settings have been added:
+
+- `:FileCategories` - The default list of the pre-defined file categories ("Documentation", "Data" and "Code") can now be redefined with a comma-separated list (e.g. `'Docs,Data,Code,Workflow'`).
+
+See the [Database Settings](https://guides.dataverse.org/en/5.10.1/installation/config.html#database-settings) section of the Guides for more information.
+
+## Notes for Developers and Integrators
+
+In the "Backward Incompatibilities" section below, note changes in the API regarding licenses and the SWORD API.
+
+## Backward Incompatibilities
+
+As of Dataverse 5.10, "NONE" is no longer supported as a valid license when creating a dataset using the SWORD API. The API Guide has been updated to reflect this. Additionally, if you specify an invalid license, a list of available licenses will be returned in the response.
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the [5.10.1 Milestone](https://github.com/IQSS/dataverse/milestone/102?closed=1) in Github.
+
+For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email [email protected].
+
+## Installation
+
+If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.10.1/installation/). Please also contact us to get added to the [Dataverse Project Map](https://guides.dataverse.org/en/5.10.1/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org) if you have not done so already.
+
+## Upgrade Instructions
+
+0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.10.1.
+
+If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user.  
+
+In the following commands we assume that Payara 5 is installed in `/usr/local/payara5`. If not, adjust as needed.
+
+`export PAYARA=/usr/local/payara5`
+
+(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell)
+
+1\. Undeploy the previous version.
+
+- `$PAYARA/bin/asadmin list-applications`
+- `$PAYARA/bin/asadmin undeploy dataverse<-version>`
+
+2\. Stop Payara and remove the generated directory
+
+- `service payara stop`
+- `rm -rf $PAYARA/glassfish/domains/domain1/generated`
+
+3\. Start Payara
+
+- `service payara start`
+
+4\. Deploy this version.
+
+- `$PAYARA/bin/asadmin deploy dataverse-5.10.1.war`
+
+5\. Restart payara
+
+- `service payara stop`
+- `service payara start`

doc/sphinx-guides/source/api/sword-atom-entry.xml

@@ -27,8 +27,8 @@
    <dcterms:coverage>United States</dcterms:coverage>
    <dcterms:coverage>Canada</dcterms:coverage>
    <!-- license and restrictions -->
-   <dcterms:license>NONE</dcterms:license>
-   <dcterms:rights>Downloader will not use the Materials in any way prohibited by applicable laws.</dcterms:rights>
+   <dcterms:license>CC0 1.0</dcterms:license>
+   <!--<dcterms:rights>Downloader will not use the Materials in any way prohibited by applicable laws.</dcterms:rights>-->
    <!-- related publications -->
    <dcterms:isReferencedBy holdingsURI="http://dx.doi.org/10.1038/dvn333" agency="DOI" IDNo="10.1038/dvn333">Peets, J., &amp; Stumptown, J. (2013). Roasting at Home. New England Journal of Coffee, 3(1), 22-34.</dcterms:isReferencedBy>
 </entry>

doc/sphinx-guides/source/api/sword.rst

@@ -65,6 +65,8 @@ Differences in Dataverse Software 4 from DVN 3.x lead to a few minor backward in
 
 - The Service Document will show a single API Terms of Use rather than root level and Dataverse collection level Deposit Terms of Use.
 
+- As of Dataverse Software 5.10, ``NONE`` is no longer supported as a license.
+
 New features as of v1.1
 -----------------------
 
@@ -80,7 +82,7 @@ New features as of v1.1
 
 - "Contributor" can now be populated and the "Type" (Editor, Funder, Researcher, etc.) can be specified with an XML attribute. For example: ``<dcterms:contributor type="Funder">CaffeineForAll</dcterms:contributor>``
 
-- "License" can now be set with ``dcterms:license`` and the possible values are "CC0" and "NONE". "License" interacts with "Terms of Use" (``dcterms:rights``) in that if you include ``dcterms:rights`` in the XML, the license will be set to "NONE". If you don't include ``dcterms:rights``, the license will default to "CC0". It is invalid to specify "CC0" as a license and also include ``dcterms:rights``; an error will be returned. For backwards compatibility, ``dcterms:rights`` is allowed to be blank (i.e. ``<dcterms:rights></dcterms:rights>``) but blank values will not be persisted to the database and the license will be set to "NONE".
+- "License" can now be set with ``dcterms:license`` and the possible values determined by the installation ("CC0 1.0" and "CC BY 4.0" by default). "License" interacts with "Terms of Use" (``dcterms:rights``) in that if you include ``dcterms:rights`` in the XML and don't include ``dcterms:license``, the license will be "Custom Dataset Terms" and "Terms of Use" will be populated. If you don't include ``dcterms:rights``, the default license will be used. It is invalid to specify a license and also include ``dcterms:rights``; an error will be returned. For backwards compatibility, ``dcterms:rights`` is allowed to be blank (i.e. ``<dcterms:rights></dcterms:rights>``) but blank values will not be persisted to the database and the license will be set to "Custom Dataset Terms".
 
 - "Contact E-mail" is automatically populated from dataset owner's email.
 
@@ -143,9 +145,9 @@ Dublin Core Terms (DC Terms) Qualified Mapping - Dataverse Project DB Element Cr
 +-----------------------------+----------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
 |dcterms:coverage             |              otherGeographicCoverage         |              |  General information on the geographic coverage of the Dataset.                                                                                             |
 +-----------------------------+----------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|dcterms:license              |                   license                    |              |  Set the license to CC0 (default in a Dataverse installation for new Datasets), otherwise enter "NONE" and fill in the dcterms:rights field.                |                
+|dcterms:license              |                   license                    |              |  Set the license. Alternatively, use the dcterms:rights field instead.                                                                                      |
 +-----------------------------+----------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|dcterms:rights               |                 termsofuse                   |              |  If not using CC0, enter any terms of use or restrictions for the Dataset.                                                                                  |
+|dcterms:rights               |                 termsofuse                   |              |  If not using dcterms:license, enter any terms of use or restrictions for the Dataset.                                                                      |
 +-----------------------------+----------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
 |dcterms:isReferencedBy       |             publicationCitation              |              |  The publication (journal article, book, other work) that uses this dataset (include citation, permanent identifier (DOI), and permanent URL).              |
 +-----------------------------+----------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+

doc/sphinx-guides/source/conf.py

@@ -65,9 +65,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '5.10'
+version = '5.10.1'
 # The full version, including alpha/beta/rc tags.
-release = '5.10'
+release = '5.10.1'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/sphinx-guides/source/developers/dev-environment.rst

@@ -79,15 +79,15 @@ On Linux, install ``jq`` from your package manager or download a binary from htt
 Install Payara
 ~~~~~~~~~~~~~~
 
-Payara 5.201 or higher is required.
+Payara 5.2021.6 or higher is required.
 
 To install Payara, run the following commands:
 
 ``cd /usr/local``
 
-``sudo curl -O -L https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.5/payara-5.2021.5.zip``
+``sudo curl -O -L https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.6/payara-5.2021.6.zip``
 
-``sudo unzip payara-5.2021.5.zip``
+``sudo unzip payara-5.2021.6.zip``
 
 ``sudo chown -R $USER /usr/local/payara5``
 

doc/sphinx-guides/source/installation/config.rst

@@ -2506,6 +2506,14 @@ Each set of labels is identified by a curationLabelSet name and a JSON Array of
 
 ``curl -X PUT -d '{"Standard Process":["Author contacted", "Privacy Review", "Awaiting paper publication", "Final Approval"], "Alternate Process":["State 1","State 2","State 3"]}' http://localhost:8080/api/admin/settings/:AllowedCurationLabels``
 
+If the Dataverse Installation supports multiple languages, the curation label translations should be added to the ``CurationLabels`` properties files. (See :ref:`i18n` for more on properties files and internationalization in general.)
+Since the Curation labels are free text, while creating the key, it has to be converted to lowercase, replace space with underscore.
+
+Example::
+
+  standard_process=Standard Process
+  author_contacted=Author contacted
+
 .. _:AllowCustomTermsOfUse:
 
 :AllowCustomTermsOfUse
@@ -2576,3 +2584,18 @@ For example:
 ++++++++++++++++++++++++++++++++
 
 When set to ``true``, this setting allows a superuser to publish and/or update Dataverse collections and datasets bypassing the external validation checks (specified by the settings above). In an event where an external script is reporting validation failures that appear to be in error, this option gives an admin with superuser privileges a quick way to publish the dataset or update a collection for the user. 
+
+:FileCategories
++++++++++++++++
+
+Overrides the default list of file categories that is used in the UI when adding tags to files. The default list is Documentation, Data, and Code.
+
+This setting is a comma-separated list of the new tags.
+
+To override the default list with Docs, Data, Code, and Workflow:
+
+``curl -X PUT -d 'Docs,Data,Code,Workflow' http://localhost:8080/api/admin/settings/:FileCategories``
+
+To remove the override and go back to the default list:
+
+``curl -X PUT -d '' http://localhost:8080/api/admin/settings/:FileCategories``

doc/sphinx-guides/source/installation/installation-main.rst

@@ -100,10 +100,6 @@ The Dataverse Software uses JHOVE_ to help identify the file format (CSV, PNG, e
 
 .. _JHOVE: http://jhove.openpreservation.org
 
-**A note about Payara-5.2021.5:** as of this writing there exists a logging configuration bug in Payara-5.2021.5. Any change to the logging configuration results in the contents of ``/usr/local/payara/glassfish/domains/domain1/config/logging.properties`` getting clobbered. In the absence of a proper logging configuration, Payara logs a number of WELD INFO entries on Payara launch, then proceeds to attempt to update its logging configuration approximately twice a second. This will result in unnecessarily junked-up system logs.
-
-This bug appears to be triggered in new Payara installations during the Dataverse installation routine. As the ``default-logging.properties`` file doesn't match the ``logging.proprties`` file distributed with Payara, we are offering :download:`a copy of it here<../_static/installation/files/usr/local/payara5/glassfish/domains/domain1/config/logging.properties>`. Simply stopping Payara, replacing the file, and starting Payara should correct the issue.
-
 Logging In
 ----------
 

doc/sphinx-guides/source/installation/prerequisites.rst

@@ -44,7 +44,7 @@ On RHEL/derivative you can make Java 11 the default with the ``alternatives`` co
 Payara
 ------
 
-Payara 5.2021.5 is recommended. Newer versions might work fine, regular updates are recommended.
+Payara 5.2021.6 is recommended. Newer versions might work fine, regular updates are recommended.
 
 Installing Payara
 =================
@@ -55,8 +55,8 @@ Installing Payara
 
 - Download and install Payara (installed in ``/usr/local/payara5`` in the example commands below)::
 
-	# wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.5/payara-5.2021.5.zip
-	# unzip payara-5.2021.5.zip
+	# wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.6/payara-5.2021.6.zip
+	# unzip payara-5.2021.6.zip
 	# mv payara5 /usr/local
 
 If you intend to install and run Payara under a service account (and we hope you do), chown -R the Payara hierarchy to root to protect it but give the service account access to the below directories:

doc/sphinx-guides/source/user/tabulardataingest/excel.rst

@@ -1,10 +1,32 @@
 Excel 
 +++++++
 
+Microsoft Excel files (XLSX format).
+
 .. contents:: |toctitle|
 	:local:
 
-Excel files (**New** in Dataverse Software 4.0!)
+Supported Formats
+------------------
+
+Only the newer XLSX Excel files are supported. We are not planning to add support for the old-style, binary XLS files.
+
+Limitations
+-----------
+
+If an Excel file has multiple sheets, only the first sheet of the file will be ingested. The other sheets will be available when a user downloads the original Excel file. To have all sheets of an Excel file ingested and searchable at the variable level, upload each sheet as an individual file in your dataset.
+
+Ingest Errors
+-------------
+
+You may encounter ingest errors after uploading an Excel file if the file is formatted in a way that can't be ingested by the Dataverse software. Ingest errors can be caused by a variety of formatting inconsistencies, including:
+
+* line breaks in a cell
+* blank cells
+* single cells that span multiple rows
+* missing headers
 
-Note: only the "new", XLSX Excel files are supported. We are not planning to add support for the old-style, binary XLS files.
+Example Data
+------------
 
+`An example of an Excel file that successfully ingests <https://github.com/IQSS/dataverse-sample-data/blob/master/data/dataverses/dataverseno/datasets/tabular-sample-data/files/Tabular_Sample_Data.xlsx>`_ is available in the Dataverse Sample Data GitHub repository.