v13.11.3

13.11.3

v13.11.2

13.11.2

New logging + dat doctor!

• update dat-node ebde553
• disable doctor tests temporarily dbbd21e
• standard 461b646
• add debug 4bfff17
• use new dat-log 76a536e
• use the new dat-doctor bfad640
• exit if the user has requested a specific port (#954) 9ac2d07
• update auth commands for new diffy 0e8a19c
• Update neat log to use new diffy (#1010) c1dde60
• update dependencies (#1009) 928b8b0
• Fix path to dat install script on github e24d8d3
• Show message for update, replaces #852 (#968) 86a0101
• New installer method (#955) b227a47
• Create issue_template.md (#972) 1591bec
• show messsage if invalid dir name on createDir (#889) ba5e34a
• add latest to travis 5d7a9a3
• #930 replaced / with path.sep (#962) f19cbcc
• fix tests, pin standard fb7b39a

v13.10.0

13.10.0

v13.9.2

13.9.2

v13.9.1

13.9.1

v13.8.2

13.8.2

Users' Choice! Backup your 🗝 too.

13.8.0

With this release, we are adding an exciting feature that really showcases how powerful Dat is, selective sync. Using the CLI you can now specify which files you want to download either with an option or using the .datdownload file. dat sync will download and keep updated on the selected files. This means you can put large datasets into Dat but have control over what files you download where.

All the features for this release:

• Selective Sync (#834)
• Key management (#828)
• Commands run faster via lazy required modules (#821)

And thanks to @G-Ray for fixing the .gitignore that was wrong but I kept forgetting to fix!

Selective Sync

We heard some feedback from many users, including Sloan Digital Sky Survey, that selective sync would be nice for very large dat archives. This new feature adds:

1. dat clone <key> --empty : create an empty archive.
2. dat sync <dir> --select=<remote-file>,<remote-file>,<remote-file>: download only selected files
   OR
3. dat sync <dir> --select-from-file=<local-file-list.txt> will parse the file and download all files listed (separated by newlines). Default is .datdownload

For example, check out this short video for a demo.

1. Clone an "empty Dat

To start selective sync, you download an empty copy be using the --empty option.

❯ dat clone datproject.org/jhand/cat-mapper ./cat-pictures --empty

Dat successfully created in empty mode. Download files using pull or sync.

This creates a .dat directory and downloads the metadata.

2. Sync the files you want

Once you have the dat folder created, there are two ways to choose which files to sync.

You can specify the files directly in sync:

~/Desktop
❯ cd cat-pictures 

~/Desktop/cat-pictures
❯ dat sync --select=cat-1.jpg,cat-2.jpg,dat.json
dat v13.7.0
Syncing: 2 files (7.6 MB)

(Note: we are working on calculating the download progress. You can tell it is done when the download speed goes to 0.)

You can also create a .datdownload file inside your dat with the filenames or directories:

# .datdownload file
cat-1.jpg
cat-2.jpg
dat.json

When you run dat sync (without any options), Dat will only download the files or directories in the .datdownload.

We are really excited about getting this feature released! We hope it'll help users managing large datasets like the Sloan Digital Sky Survey team.

Backup and Import your keys

Managing your secret keys is a vital feature for Dat. Without the secret key, you cannot update a dat you've created. However, it is important that the keys do not get accidentally shared so we know we need to get this right.

We've started implementing some basic key management. With this release, you can:

• dat keys - prints out dat key and discovery key
• dat keys export - prints out secret key (does not remove ownership from existing dat)
• dat keys import - prompts user for secret key to make a dat writable.
  These are still somewhat advanced key features but will at least make it possible to backup or transfer ownership of a dat.

v13.7.0

This release goes along with a new release of datproject.org v1.2.0. When you dat publish the metadata will be cached so users can always view the file list for a dat even when you go offline!

We also added a throttling option and improved a few things under the hood.

Changelog

Added

• Throttling - sometimes Dat goes too fast, so you can limit the upload + download speeds. (#806)
• Publish metadata to registry when publishing (#812)

Changed

• Use dat-node http support directly (#817)

Fixed

• Use npm package for registry testing.

Publish all the things! Full Dat registry support.

The Dat project registry is here! This release has a pretty short changelog with a big potential. As part of our grant from the Knight Foundation, we are creating a public registry for dats. You can now sign up, publish your dats, and use the datproject.org registry via the CLI!

• Explore some of the existing dats. datproject.org caches data and we have our example dats shared on our server to ensure higher availability.
• Create an account an publish your dats!

Why is dat publish so cool?

We are super excited about dat publish and its potential to create a decentralized Github for data. Dat's persistent identifiers, built in version control, and live updates mean you only have to "publish" a dat once, unlike git or npm. Dat's distributed network means this data can be stored anywhere after it is published, unlike Dropbox or Google Drive, which store all your data on their servers.

What happens when I publish?

• dat publish associates your dat link with your username and your dat name. This allows users to download it, such as dat clone datproject.org/jhand/cli-demo.
• datproject.org does not store your dat when you publish. We cache data when you visit a dat online, but only if it available at that moment.

The registry not storing data is a bit counterintuitive compared to npm or git, which have centralized servers hosting your repos. For someone to download your data (or view it online) you have to be running dat share, dat sync, or sharing your dat in Dat Desktop.

This is a new concept to many users more used to centralized services and may cause some learning pains. We do this will give users more control over where their data goes and how it is stored. We're working on ways to make hosting dats easier in the future. Our view is that moving from decentralized to centralized is much easier than the other direction.

More Registries!

Anyone can run a Dat registry and you can publish the same dat to multiple registries. Registries with different categories of data or with unique methods of discovering public data will allow for a broad set of uses. Registries can also create distributed hosting and backup services for published dats.

Changelog

Added

• Improved support for public Dat registries (#794)
• Add unpublish command