Is it possible to only download a fraction size of the dataset?

[mariusjohan]

I want to download the natural questions dataset in a Google Colab notebook, however when running the standard download function it fails because I lack disk space. Is it possible to download the dataset in fractions?

[mariosasko]

Hi! Currently, the natural_questions dataset doesn't support streaming, so the entire dataset must be downloaded. Hopefully, we can make this dataset streamable soon. Also, one of our main future goals is to optimize downloading to download only the files needed for the split when calling load_dataset("some_dataset", split=split) to save some disk space.

[lakshmi-speak]

one of our main future goals is to optimize downloading to download only the files needed for the split when calling load_dataset("some_dataset", split=split) to save some disk space.

Is this already implemented?

[mariosasko]

Not yet, but it should be soon!