Merge pull request #3955 from IQSS/develop

v4.7

CONTRIBUTING.md

@@ -1,6 +1,8 @@
 # Contributing to Dataverse
 
-Thank you for your interest in contributing to Dataverse! We welcome contributions of ideas, bug reports, usability testing, documentation, code, and more!
+Thank you for your interest in contributing to Dataverse!  We are open to contributions from everyone. You don't need permission to participate, just jump in using the resources below.  If you have questions, reach out to us on the [#dataverse IRC channel][], and hang around a while, as it may take time for community members to de-idle.
+
+We aren't just looking for developers, there are many ways to contribute to Dataverse.  We welcome contributions of ideas, bug reports, usability research/feedback, documentation, code, and more!
 
 ## Ideas/Feature Requests
 
@@ -26,7 +28,7 @@ An issue is a bug (a feature is no longer behaving the way it should) or a featu
 
 Before submitting an issue, please search the existing issues by using the search bar at the top of the page. If there is an existing issue that matches the issue you want to report, please add a comment to it.
 
-If there is no pre-existing issue, please click on the "New Issue" button, log in, and write in what the issue is (unless it is a security issue which should be reported privately to [email protected]). Someone on the Dataverse development team will appropriately tag and assign it to a member of the Dataverse development team.
+If there is no pre-existing issue, please click on the "New Issue" button, log in, and write in what the issue is (unless it is a security issue which should be reported privately to [email protected]).
 
 If you do not receive a reply to your new issue or comment in a timely manner, please email [email protected] with a link to the issue.
 
@@ -51,12 +53,11 @@ The source for the documentation at http://guides.dataverse.org is in the GitHub
 
 Before you start coding, please reach out to us either on the [dataverse-community Google Group][], the [dataverse-dev Google Group][], [IRC][] (#dataverse on freenode), or via [email protected] to make sure the effort is well coordinated and we avoid merge conflicts.
 
-We will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request.
-
-We hope you find the Developer Guide at http://guides.dataverse.org/en/latest/developers helpful.
+Please read http://guides.dataverse.org/en/latest/developers/version-control.html to understand how we use the "git flow" model of development and how we will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request.
 
-After making your pull request, your goal should be to help it advance through our kanban board at https://waffle.io/IQSS/dataverse . If no one has moved your pull request to the code review column in a timely manner, please reach out. Thanks!
+After making your pull request, your goal should be to help it advance through our kanban board at https://waffle.io/IQSS/dataverse . If no one has moved your pull request to the code review column in a timely manner, please reach out. We maintain a list of [community contributors][] so please let us know if you'd like to be added or removed from the list. Thanks!
 
 [dataverse-community Google Group]: https://groups.google.com/group/dataverse-community
 [dataverse-dev Google Group]: https://groups.google.com/group/dataverse-dev
 [IRC]: http://chat.dataverse.org
+[community contributors]: https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing

PULL_REQUEST_TEMPLATE.md

@@ -4,7 +4,7 @@ Welcome! New contributors should at least glance at [CONTRIBUTING.md](/CONTRIBUT
 
 ## Related Issues
 
-- connects to [#issue number]: [issue title]
+- connects to #ISSUE_NUMBER: ISSUE_TITLE
 
 ## Pull Request Checklist
 

README.md

@@ -1,20 +1,36 @@
 Dataverse®
 ===============
 
-Dataverse is an open source web application for sharing, citing, analyzing, and preserving research data (developed by the [Data Science and Products team](http://www.iq.harvard.edu/people/people/data-science-products) at the [Institute for Quantitative Social Science](http://iq.harvard.edu/)).
+Dataverse is an [open source][] web application for sharing, citing, analyzing, and preserving research data (developed by the [Data Science and Products team](http://www.iq.harvard.edu/people/people/data-science-products) at the [Institute for Quantitative Social Science](http://iq.harvard.edu/) and the [Dataverse community][]).
 
-Institutions and organizations can choose to install the Dataverse software for their own use. 
-In this case, the institution will be responsible for maintaining the application; installing upgrades, 
-setting up backups and data replication as needed. Dataverse 4.0 is now released and used by the Harvard Dataverse: [dataverse.harvard.edu](http://dataverse.harvard.edu/). If you'd like to first test it, you can use our demo site: [demo.dataverse.org](http://demo.dataverse.org).
+[dataverse.org][] is our home on the web and shows a map of Dataverse installations around the world, a list of [features][], [integrations][] that have been made possible through [REST APIs][], our development [roadmap][], and more.
 
-For more general information about Dataverse please visit [dataverse.org](http://dataverse.org). 
+We maintain a demo site at [demo.dataverse.org][] which you are welcome to use for testing and evaluating Dataverse.
 
-The Dataverse code is *open-source* and *free*. 
+To install Dataverse, please see our [Installation Guide][] which will prompt you to download our [latest release][].
 
-Dataverse releases are available for download from https://github.com/IQSS/dataverse/releases and installation instructions can be found in the [Installation Guide](http://guides.dataverse.org/en/latest/installation/).
+To discuss Dataverse with the community, please join our [mailing list][], participate in a [community call][], chat with us at [chat.dataverse.org][], or attend our annual [Dataverse Community Meeting][].
+
+We love contributors! Please see our [Contributing Guide][] for ways you can help.
 
 Dataverse is a trademark of President and Fellows of Harvard College and is registered in the United States.
 
 [![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg?raw=true "Dataverse Project")](http://dataverse.org)
 
-[![Build Status](https://travis-ci.org/IQSS/dataverse.svg?branch=master)](https://travis-ci.org/IQSS/dataverse) [![Coverage Status](https://coveralls.io/repos/IQSS/dataverse/badge.svg?branch=master&service=github)](https://coveralls.io/github/IQSS/dataverse?branch=master)
+[![Build Status](https://travis-ci.org/IQSS/dataverse.svg?branch=develop)](https://travis-ci.org/IQSS/dataverse) [![Coverage Status](https://coveralls.io/repos/IQSS/dataverse/badge.svg?branch=develop&service=github)](https://coveralls.io/github/IQSS/dataverse?branch=develop)
+
+[dataverse.org]: https://dataverse.org
+[demo.dataverse.org]: https://demo.dataverse.org
+[Dataverse community]: https://dataverse.org/developers
+[Installation Guide]: http://guides.dataverse.org/en/latest/installation/index.html
+[latest release]: https://github.com/IQSS/dataverse/releases
+[features]: https://dataverse.org/software-features
+[roadmap]: https://dataverse.org/goals-roadmap-and-releases
+[integrations]: https://dataverse.org/integrations
+[REST APIs]: http://guides.dataverse.org/en/latest/api/index.html
+[Contributing Guide]: CONTRIBUTING.md
+[mailing list]: https://groups.google.com/group/dataverse-community
+[community call]: https://dataverse.org/community-calls
+[chat.dataverse.org]: http://chat.dataverse.org
+[Dataverse Community Meeting]: https://dataverse.org/events
+[open source]: LICENSE.md

doc/sphinx-guides/source/_static/api/user-add.json

@@ -0,0 +1,8 @@
+{
+  "firstName": "Lisa",
+  "lastName": "Simpson",
+  "userName": "lsimpson",
+  "affiliation": "Springfield",
+  "position": "Student",
+  "email": "[email protected]"
+}

doc/sphinx-guides/source/_static/installation/files/var/www/dataverse/branding/custom-footer.html

@@ -0,0 +1,38 @@
+    <style>
+    	/* NOTE: You can use the custom CSS configuration to move this CSS your own stylesheet. */
+
+    	/* Overwrite default footer styles to make room at the bottom of the browser page for our custom footer... */
+    	html, body {
+            height: 100%;
+        }
+        #content.container {
+         	min-height: calc(100% - 210px);
+            margin-bottom: -142px; 
+        }
+        #content.container:after {
+        	content: "";
+        	display: block;
+        }
+        #footer {
+			color: #a9a9a9; margin-top:0px; padding-bottom:0px; bottom:-150px; position: relative;
+		}
+
+		/* Place custom footer just above default footer at the bottom of the page... */
+        #custom-footer, #content.container:after {
+            height: 142px; 
+        }
+        #custom-footer {
+    		background: #414141; color: #a9a9a9; min-height: 100px; padding: 1.25em 0 100px 0;
+		}
+		#custom-footer a {
+			color: #fff;
+		}
+    </style>
+    <div id="custom-footer">
+    	<div class="container">
+    		<p>SAMPLE FOOTER TEXT</p>
+    		<p class="small">
+		        <a href="https://dataverse.org" title="SAMPLE LINK" target="_blank">SAMPLE LINK</a>
+		    </p>
+    	</div>
+    </div>

doc/sphinx-guides/source/_static/installation/files/var/www/dataverse/branding/custom-header.html

@@ -0,0 +1,35 @@
+    <style>
+        /* NOTE: You can use the custom CSS configuration to move this CSS your own stylesheet... */
+
+        /* Overwrite default navbar styles to make room at the top of the browser page for our custom header... */
+        .navbar-fixed-top {
+            top:50px !important;
+        }
+        #dataverse-header-block {
+            padding-top: 150px;
+        }
+
+		/* Place narrow custom header at the top of the page... */
+        #custom-header {
+            background: #414141;
+            min-height: 44px;
+            position: fixed;
+            right: 0;
+            left: 0;
+            z-index: 930;
+            padding-top: 8px;
+        }
+        #custom-header a.cst-hdr-link {
+            line-height: 32px;
+            color: #BAC5C6;
+            font-weight: bold;
+        }
+    </style>
+    <div id="custom-header">
+        <div class="container">
+            <p class="text-right">
+                <a class="pull-left" href="https://dataverse.org" title="SAMPLE IMAGE LINK" target="_blank"><img src="/var/www/dataverse/branding/sample.png" alt="SAMPLE IMAGE"/></a>
+                <a class="cst-hdr-link" href="https://dataverse.org" title="SAMPLE HEADER LINK" target="_blank">SAMPLE HEADER LINK</a>
+            </p>
+        </div>
+    </div>

doc/sphinx-guides/source/_static/installation/files/var/www/dataverse/branding/custom-homepage.html

@@ -0,0 +1,13 @@
+
+    <style>
+        /* NOTE: You can use the custom CSS configuration to move this CSS your own stylesheet... */
+
+        /* Custom CSS for the custom homepage... */
+        div.jumbotron h1 {color:#C55B28;}
+    </style>
+
+    <div class="jumbotron">
+        <h1>Hello, world!</h1>
+        <p>Welcome to our Dataverse.</p>
+        <p><a class="btn btn-primary btn-lg" href="/dataverse/root" role="button">Browse Data</a></p>
+    </div>

doc/sphinx-guides/source/_static/installation/files/var/www/dataverse/branding/custom-stylesheet.css

@@ -0,0 +1,6 @@
+/* Add any CSS necessary for your custom content */
+
+.navbar-default {
+	background-color: gold;
+	background-image: none;
+}

doc/sphinx-guides/source/admin/geoconnect-worldmap.rst

@@ -1,10 +1,11 @@
 Geoconnect and WorldMap
 =======================
 
-.. contents:: :local:
-
 One of the optional components listed under "Architecture and Components" in the :doc:`/installation/prep` section of the Installation Guide is `Geoconnect <https://github.com/IQSS/geoconnect>`_, a piece of middleware that allows Dataverse users to create maps in `WorldMap <http://worldmap.harvard.edu>`_ based on geospatial data stored in Dataverse. For more details on the feature from the user perspective, see the :doc:`/user/data-exploration/worldmap` section of the User Guide.
 
+.. contents:: |toctitle|
+	:local:
+
 Update "mapitlink"
 ------------------
 

doc/sphinx-guides/source/admin/harvestclients.rst

@@ -1,8 +1,9 @@
 Managing Harvesting Clients
 ===========================
 
-.. contents:: :local:
-
+.. contents:: |toctitle|
+	:local:
+
 Your Dataverse as a Metadata Harvester
 --------------------------------------
 

doc/sphinx-guides/source/admin/harvestserver.rst

@@ -1,7 +1,8 @@
 Managing Harvesting Server and Sets
 ===================================
 
-.. contents:: :local:
+.. contents:: |toctitle|
+  :local:
 
 Your Dataverse as an OAI server
 -------------------------------

doc/sphinx-guides/source/admin/index.rst

@@ -10,7 +10,7 @@ This guide documents the functionality only available to the Dataverse Admin ("N
 
 These "superuser" tasks are managed via the new page called the Dashboard. A user logged in as a Dataverse Admin will see the Dashboard link rendered in the upper right corner of every Dataverse page. 
 
-Contents:
+**Contents:**
 
 .. toctree::
 

doc/sphinx-guides/source/admin/metadataexport.rst

@@ -1,7 +1,8 @@
 Metadata Export
 ===============
 
-.. contents:: :local:
+.. contents:: |toctitle|
+	:local:
 
 Automatic Exports
 -----------------

doc/sphinx-guides/source/admin/timers.rst

@@ -3,10 +3,10 @@
 Dataverse Application Timers
 ============================
 
-.. contents:: :local:
-
 Dataverse uses timers to automatically run scheduled Harvest and Metadata export jobs. 
 
+.. contents:: |toctitle|
+	:local:
 
 Dedicated timer server in a Dataverse server cluster
 ----------------------------------------------------

doc/sphinx-guides/source/admin/troubleshooting.rst

@@ -3,10 +3,11 @@
 Troubleshooting
 ===============
 
-.. contents:: :local:
-
 This new (as of v.4.6) section of the Admin guide is for tips on how to diagnose and fix system problems. 
 
+.. contents:: |toctitle|
+	:local:
+
 Deployment fails, "EJB Timer Service not available"
 ---------------------------------------------------
 

doc/sphinx-guides/source/api/apps.rst

@@ -5,7 +5,8 @@ The introduction of Dataverse APIs has fostered the development of apps that are
 
 The apps below are open source, demonstrating how to use Dataverse APIs. Some of these apps (and others) are built on :doc:`/api/client-libraries` that are available for Dataverse APIs.
 
-.. contents:: :local:
+.. contents:: |toctitle|
+	:local:
 
 Javascript
 ----------

doc/sphinx-guides/source/api/client-libraries.rst

@@ -5,6 +5,9 @@ Currently there are client libraries for Python, R, and Java that can be used to
 
 Because Dataverse is a SWORD server, additional client libraries exist for Java, Ruby, and PHP per the :doc:`/api/sword` page.
 
+.. contents:: |toctitle|
+	:local:
+
 Python
 ------
 
@@ -15,7 +18,7 @@ https://github.com/IQSS/dataverse-client-python is the offical Python package fo
 R
 -
 
-https://github.com/IQSS/dataverse-client-r is the official R package for Dataverse APIs.
+https://github.com/IQSS/dataverse-client-r is the official R package for Dataverse APIs. The latest release can be installed from `CRAN <https://cran.r-project.org/package=dataverse>`_.
 
 It was created by `Thomas Leeper <http://thomasleeper.com>`_ whose dataverse can be found at https://dataverse.harvard.edu/dataverse/leeper
 

doc/sphinx-guides/source/api/dataaccess.rst

@@ -1,12 +1,11 @@
 Data Access API
 ===============
 
-.. contents:: :local:
-
-
 The Data Access API provides programmatic download access to the files stored under Dataverse. 
 More advanced features of the Access API include format-specific transformations (thumbnail generation/resizing for images; converting tabular data into alternative file formats) and access to the data-level metadata that describes the contents of the tabular files. 
 
+.. contents:: |toctitle|
+   :local:
 
 Basic File Access
 -----------------

doc/sphinx-guides/source/api/index.rst

@@ -6,19 +6,11 @@
 API Guide
 =========
 
-We encourage anyone interested in building tools to
-interoperate with the Dataverse to utilize our 
-APIs. In 4.0, we require to get a token, by simply registering for a Dataverse account, before using our APIs 
-(We are considering making some of the APIs completely public in the future - no token required - if you use it only a few times).
-
-Rather than using a production installation of Dataverse, API users are welcome to use http://demo.dataverse.org for testing.
-
-Please note that the APIs in this guide are shipped with the Dataverse software itself but additional APIs are available if you install the "miniverse" application from https://github.com/IQSS/miniverse and give it read only access to your production Dataverse database. http://dataverse.org/metrics is powered by miniverse.
-
-Contents:
+**Contents:**
 
 .. toctree::
 
+   intro
    sword
    search
    dataaccess

doc/sphinx-guides/source/api/intro.rst

@@ -0,0 +1,56 @@
+Introduction
+============
+
+We encourage anyone interested in building tools that interoperate with Dataverse to utilize our APIs. The Dataverse community has supplied :doc:`client-libraries` for Python, R, and Java and we are always interested in helping the community develop libraries for additional languages. The :doc:`apps` section links to open source Javascript, PHP, Python, and Java code that you can learn from while developing against Dataverse APIs.
+
+.. contents:: |toctitle|
+    :local:
+
+How This Guide is Organized
+---------------------------
+
+We document the Dataverse API in four sections:
+
+- :doc:`sword`: For depositing data using a standards-based approach rather than the :doc:`native-api`.
+- :doc:`search`: For searching dataverses, datasets, and files.
+- :doc:`dataaccess`: For downloading and subsetting data.
+- :doc:`native-api`: For performing most tasks that are possible in the GUI.
+
+We use the term "native" to mean that the API is not based on any standard. For this reason, the :doc:`search` and :doc:`dataaccess` could also be considered "native" and in the future we may reorganize the API Guide to split the :doc:`native-api` section into "Datasets API", "Files" API, etc.
+
+Authentication
+--------------
+
+Most Dataverse APIs require the use of an API token. Instructions for getting a token are described in the :doc:`/user/account` section of the User Guide.
+
+There are two ways to pass your API token to Dataverse APIs. The preferred method is to send the token in the ``X-Dataverse-key`` HTTP header, as in the following curl example::
+
+    curl -H "X-Dataverse-key: 8b955f87-e49a-4462-945c-67d32e391e7e" https://demo.dataverse.org/api/datasets/:persistentId?persistentId=doi:TEST/12345
+
+Throughout this guide you will often see Bash shell envionmental variables being used, like this::
+
+    export API_TOKEN='8b955f87-e49a-4462-945c-67d32e391e7e'
+    curl -H "X-Dataverse-key: $API_TOKEN" https://demo.dataverse.org/api/datasets/:persistentId?persistentId=doi:TEST/12345
+
+The second way to pass your API token is via an extra query parameter called ``key`` in the URL like this::
+
+    curl "https://demo.dataverse.org/api/datasets/:persistentId?persistentId=doi:TEST/12345&key=$API_TOKEN"
+
+Use of the ``X-Dataverse-key`` HTTP header form is preferred because putting the query parameters in URLs often results in them finding their way into web server access logs. Your API token should be kept as secret as your password because it can be used to perform any action *as you* in the Dataverse application.
+
+Testing
+-------
+
+Rather than using a production installation of Dataverse, API users are welcome to use http://demo.dataverse.org for testing.  
+
+Support
+-------
+
+If you are using the APIs for an installation of Dataverse hosted by your institution, you may want to reach out to the team that supports it. At the top of the Dataverse installation's home page, there should be a form you can fill out by clicking the "Support" link.
+
+If you are having trouble with http://demo.dataverse.org or have questions about the APIs, please feel free to reach out to the Dataverse community via https://groups.google.com/forum/#!forum/dataverse-community .
+
+Metrics
+-------
+
+APIs described in this guide are shipped with the Dataverse software itself. Additional APIs are available if someone at your institution installs the "miniverse" application from https://github.com/IQSS/miniverse and gives it read only access to a production Dataverse database. http://dataverse.org/metrics is powered by miniverse.