03-Data_Access2_Subsetting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subsetting ICESat-2 Data with the NSIDC Subsetter\n",
    "### How to Use the NSIDC Subsetter Example Notebook\n",
    "This notebook illustrates the use of `icepyx` for subsetting ICESat-2 data ordered through the NSIDC DAAC. We'll show how to find out what subsetting options are available and how to specify the subsetting options for your order.\n",
    "\n",
    "For more information on using `icepyx` to find, order, and download data, see the [ICESat-2_NSIDC-DAAC_DataAccess Tutorial](https://github.com/ICESAT-2HackWeek/data-access/blob/master/notebooks/ICESat-2_NSIDC-DAAC_DataAccess.ipynb) or `icepyx`'s complimentary [ICESat-2_DAAC_DataAccess_Example Notebook](https://github.com/icesat2py/icepyx/blob/master/doc/examples/ICESat-2_DAAC_DataAccess_Example.ipynb).\n",
    "\n",
    "Questions? Be sure to check out the FAQs throughout this notebook, indicated as italic headings.\n",
    "\n",
    "#### Credits\n",
    "* tutorial notebook by: Jessica Scheick and Amy Steiker\n",
    "* original notebook by: Jessica Scheick and Zheng Liu\n",
    "* some source material: [NSIDC Data Access Notebook](https://github.com/ICESAT-2HackWeek/ICESat2_hackweek_tutorials/tree/master/03_NSIDCDataAccess_Steiker) by Amy Steiker and Bruce Wallin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _What is SUBSETTING anyway?_\n",
    "\n",
    "Anyone who's worked with geospatial data has probably encountered subsetting. Typically, we search for data wherever it is stored and download the chunks (aka granules, scenes, passes, swaths, etc.) that contain something we are interested in. Then, we have to extract from each chunk the pieces we actually want to analyze. Those pieces might be geospatial (i.e. an area of interest), temporal (i.e. certain months of a time series), and/or certain variables. This process of extracting the data we are going to use is called subsetting.\n",
    "\n",
    "In the case of ICESat-2 data coming from the NSIDC DAAC, we can do this subsetting step on the data prior to download, reducing our number of data processing steps and resulting in smaller, faster downloads and storage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import packages, including icepyx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ImportError",
     "evalue": "cannot import name 'icesat2data' from 'icepyx' (/Users/rominapiunno/opt/anaconda3/envs/icepyxTut/lib/python3.7/site-packages/icepyx/__init__.py)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-d38d543b1783>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0micepyx\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0micesat2data\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mipd\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mxarray\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mxr\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpandas\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mImportError\u001b[0m: cannot import name 'icesat2data' from 'icepyx' (/Users/rominapiunno/opt/anaconda3/envs/icepyxTut/lib/python3.7/site-packages/icepyx/__init__.py)"
     ]
    }
   ],
   "source": [
    "from icepyx import icesat2data as ipd\n",
    "\n",
    "import numpy as np\n",
    "import xarray as xr\n",
    "import pandas as pd\n",
    "\n",
    "import h5py\n",
    "import os,json\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an icesat2data object and log in to Earthdata\n",
    "\n",
    "For this example, we'll be working with a land ice dataset (ATL06) for an area along West Greenland (Disko Bay). A second option for an atmospheric product that uses profiles instead of the ground track (gt) categorization is also provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a = ipd.Icesat2Data('ATL06',[-55, 68, -48, 71],['2019-02-22','2019-02-28'], \\\n",
    "                           start_time='00:00:00', end_time='23:59:59')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment and run the code in this cell to use the second variable subsetting suite of examples,\n",
    "# with the beam specifier containing \"profile\" instead of \"gt#l\"\n",
    "\n",
    "# region_a = ipd.Icesat2Data('ATL09',[-55, 68, -48, 71],['2019-02-22','2019-02-28'], \\\n",
    "#              start_time='00:00:00', end_time='23:59:59')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.earthdata_login('username','email')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discover Subsetting Options\n",
    "\n",
    "You can see what subsetting options are available for a given dataset by calling `show_custom_options()`. The options are presented as a series of headings followed by available values in square brackets. Headings are:\n",
    "* **Subsetting Options**: whether or not temporal and spatial subsetting are available for the dataset\n",
    "* **Data File Formats (Reformatting Options)**: return the data in a format other than the native hdf5 (submitted as a key=value kwarg to `order_granules(format='NetCDF4-CF')`)\n",
    "* **Data File (Reformatting) Options Supporting Reprojection**: return the data in a reprojected reference frame. These options will be supported in the future for gridded ICESat-2 L3B datasets, once those are produced and made available.\n",
    "* **Data File (Reformatting) Options NOT Supporting Reprojection**: data file formats that cannot be delivered with reprojection\n",
    "* **Data Variables (also Subsettable)**: a dictionary of variable name keys and the paths to those variables available in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "region_a.show_custom_options(dictview=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, spatial and temporal subsetting based on your initial inputs is applied to your order unless you specify `subset=False` to `order_granules()` or `download_granules()` (which calls `order_granules` under the hood if you have not already placed your order) functions. Additional subsetting options must be specified as keyword arguments to the order/download functions.\n",
    "\n",
    "Although some file format conversions and reprojections are possible using the `format`, `projection`, and `projection_parameters` keywords, the rest of this tutorial will focus on variable subsetting, which is provided with the `Coverage` keyword. Note that as of June 2020, cryospheric ICESat-2 L3B datasets are not yet available, therefore no projection options currently exist."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _Why do I have to provide spatial bounds to icepyx even if I don't use them to subset my data order?_\n",
    "\n",
    "Because they're still needed for the granule level search. Spatial inputs are usually required for any data search, on any platform, even if your search parameters cover the entire globe.\n",
    "\n",
    "The spatial information you provide is used to search the data repository and determine which granules might contain data over your area of interest. When you use that spatial information for subsetting, it's actually asking the NSIDC subsetter to extract the appropriate data from each granule. Thus, even if you set `subset=False` and download entire granules, you still need to provide some inputs on what geographic area you'd like data for."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About Data Variables in an Icesat2Data object\n",
    "There are two possible variable parameters associated with each ```icesat2data``` object.\n",
    "1. `order_vars`, which is for interacting with variables during data querying, ordering, and downloading activities. `order_vars.wanted` holds the user's list to be submitted to the NSIDC subsetter and download a smaller, reproducible dataset.\n",
    "2. `file_vars`, which is for interacting with variables associated with local files [not yet implemented].\n",
    "\n",
    "Each variables parameter (which is actually an associated Variables class object) has methods to:\n",
    "* get available variables, either available from the NSIDC or the file (`avail()` method/attribute).\n",
    "* append new variables to the wanted list (`append()` method).\n",
    "* remove variables from the wanted list (`remove()` method).\n",
    "\n",
    "Each variables instance also has a set of attributes, including `avail` and `wanted` to indicate the list of variables that is available (unmutable, or unchangeable, as it is based on the input dataset specifications or files) and the list of variables that the user would like extracted (updateable with the `append` and `remove` methods), respectively. We'll showcase the use of all of these methods and attributes below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ICESat-2 data variables\n",
    "ICESat-2 data is natively stored in a nested file format called hdf5. Much like a directory-file system on a computer, each variable (file) has a unique path through the heirarchy (directories) within the file. Thus, some variables (e.g. `'latitude'`, `'longitude'`) have multiple paths (one for each of the six beams in most datasets). \n",
    "\n",
    "To increase readability, some display options (2 and 3, below) show the 200+ variable + path combinations as a dictionary where the keys are variable names and the values are the paths to that variable.\n",
    "\n",
    "### Determine what variables are available\n",
    "There are multiple ways to get a complete list of available variables.\n",
    "\n",
    "1. `region_a.order_vars.avail`, a list of all valid path+variable strings\n",
    "2. `region_a.show_custom_options(dictview=True)`, all available subsetting and data transformation options, as you ran above\n",
    "3. `region_a.order_vars.parse_var_list(region_a.order_vars.avail)`, a dictionary of variable:paths key:value pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "region_a.order_vars.avail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By passing the boolean `options=True` to the `avail` method, you can obtain lists of unique possible variable inputs (var_list inputs) and path subdirectory inputs (keyword_list and beam_list inputs) for your dataset. These can be helpful for building your wanted variable list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.avail(options=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _Why not just download all the data and subset locally? What if I need more variables/granules?_\n",
    "\n",
    "Taking advantage of the NSIDC subsetter is a great way to reduce your download size and thus your download time and the amount of storage required, especially if you're storing your data locally during analysis. By downloading your data using `icepyx`, it is easy to go back and get additional data with the same, similar, or different parameters (e.g. you can keep the same spatial and temporal bounds but change the variable list). Related tools (e.g. [`captoolkit`](https://github.com/fspaolo/captoolkit)) will let you easily merge files if you're uncomfortable merging them during read-in for processing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building your wanted variable list\n",
    "\n",
    "Now that you know which variables and path components are available for your dataset, you need to build a list of the ones you'd like included in your dataset. There are several options for generating your initial list as well as modifying it, giving the user complete control over the list submitted.\n",
    "\n",
    "The options for building your initial list are:\n",
    "1. Use a default list for the dataset with the `defaults=True` keyword argument (not yet fully implemented across all datasets. Have a default variable list for your field/dataset? Submit a pull request or post it as an issue on [GitHub](https://github.com/icesat2py/icepyx)!)\n",
    "2. Provide a list of variable names\n",
    "3. Provide a list of profiles/beams or other path keywords, where \"keywords\" are simply the unique subdirectory names contained in the full variable paths of the dataset. A full list of available keywords for the dataset is displayed in the error message upon entering `keyword_list=['']` into the `append` function (see below for an example) or by running `region_a.order_vars.avail(options=True)`, as above\n",
    "\n",
    "**Note: all datasets have a short list of \"mandatory\" variables/paths (containing spacecraft orientation and time information needed to convert the data's `delta_time` to a readable datetime) that are automatically added to any built list. If you have any recommendations for other variables that should always be included (e.g. uncertainty information), please let us know!**\n",
    "\n",
    "Examples of using each method to build and modify your wanted variable list are below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.wanted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "region_a.order_vars.append(defaults=True)\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reminder: the keywords available for this dataset are shown in the error message upon entering a blank keyword_list, as seen in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(keyword_list=[''])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifying your wanted variable list\n",
    "\n",
    "Generating and modifying your variable request list, which is stored in `region_a.order_vars.wanted`, is controlled by the `append` and `remove` functions that operate on `region_a.order_vars.wanted`. The input options to `append` are as follows (the full documentation for this function can be found by executing `help(region_a.order_vars.append)`).\n",
    "* `defaults` (default False) - include the default variable list for your dataset (not yet fully implemented for all datasets; please submit your default variable list for inclusion!)\n",
    "* `var_list` (default None) - list of variables (entered as strings)\n",
    "* `beam_list` (default None) - list of beams/profiles (entered as strings)\n",
    "* `keyword_list` (default None) - list of keywords (entered as strings); use `keyword_list=['']` to obtain a list of available keywords\n",
    "\n",
    "Similarly, the options for `remove` are:\n",
    "* `all` (default False) - reset `region_a.order_vars.wanted` to None\n",
    "* `var_list` (as above)\n",
    "* `beam_list` (as above)\n",
    "* `keyword_list` (as above)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.remove(all=True)\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples (Overview)\n",
    "\n",
    "Below are a series of examples to show how you can use `append` and `remove` to modify your wanted variable list. For clarity, `region_a.order_vars.wanted` is cleared at the start of many examples. However, multiple `append` and `remove` commands can be called in succession to build your wanted variable list (see Examples 3+)\n",
    "\n",
    "There are two example tracks. The first is for land ice (ATL06) data that is separated into beams. The second is for atmospheric data (ATL09) that is separated into profiles. Both example tracks showcase the same functionality and are provided for users of both data types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------\n",
    "### Example Track 1 (Land Ice - run with ATL06 dataset)\n",
    "\n",
    "#### Example 1: choose variables\n",
    "Add all `latitude` and `longitude` variables across all six beam groups. Note that the additional required variables for time and spacecraft orientation are included by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(var_list=['latitude','longitude'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 2: specify beams and variable\n",
    "Add `latitude` for only `gt1l` and `gt2l`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.remove(all=True)\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var_dict = region_a.order_vars.append(beam_list=['gt1l', 'gt2l'], var_list=['latitude'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 3: add/remove selected beams+variables\n",
    "Add `latitude` for `gt3l` and remove it for `gt2l`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(beam_list=['gt3l'],var_list=['latitude'])\n",
    "region_a.order_vars.remove(beam_list=['gt2l'], var_list=['latitude'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 4: `keyword_list`\n",
    "Add `latitude` and `longitude` for all beams and with keyword `land_ice_segments`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(var_list=['latitude', 'longitude'],keyword_list=['land_ice_segments'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 5: target a specific variable + path\n",
    "Remove `gt1r/land_ice_segments/longitude` (but keep `gt1r/land_ice_segments/latitude`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.remove(beam_list=['gt1r'], var_list=['longitude'], keyword_list=['land_ice_segments'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 6: add variables not specific to beams/profiles\n",
    "Add `rgt` under `orbit_info`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(keyword_list=['orbit_info'],var_list=['rgt'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 7: add all variables+paths of a group\n",
    "In addition to adding specific variables and paths, we can filter all variables with a specific keyword as well. Here, we add all variables under `orbit_info`. Note that paths already in `region_a.order_vars.wanted`, such as `'orbit_info/rgt'`, are not duplicated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(keyword_list=['orbit_info'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 8: add all possible values for variables+paths\n",
    "Append all `longitude` paths and all variables/paths with keyword `land_ice_segments`.\n",
    "\n",
    "Similarly to what is shown in Example 4, if you submit only one `append` call as `region_a.order_vars.append(var_list=['longitude'], keyword_list=['land_ice_segments'])` rather than the two `append` calls shown below, you will only add the variable `longitude` and only paths containing `land_ice_segments`, not ALL paths for `longitude` and ANY variables with `land_ice_segments` in their path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "region_a.order_vars.append(var_list=['longitude'])\n",
    "region_a.order_vars.append(keyword_list=['land_ice_segments'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 9: remove all variables+paths associated with a beam\n",
    "Remove all paths for `gt1l` and `gt3r`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "region_a.order_vars.remove(beam_list=['gt1l','gt3r'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 10: generate a default list for the rest of the tutorial\n",
    "Generate a reasonable variable list prior to download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.remove(all=True)\n",
    "region_a.order_vars.append(defaults=True)\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------\n",
    "### Example Track 2 (Atmosphere - run with ATL09 dataset commented out at the start of the notebook)\n",
    "\n",
    "#### Example 1: choose variables\n",
    "Add all `latitude` and `longitude` variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(var_list=['latitude','longitude'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 2: specify beams/profiles and variable\n",
    "Add `latitude` for only `profile_1` and `profile_2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.remove(all=True)\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var_dict = region_a.order_vars.append(beam_list=['profile_1','profile_2'], var_list=['latitude'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 3: add/remove selected beams+variables\n",
    "Add `latitude` for `profile_3` and remove it for `profile_2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(beam_list=['profile_3'],var_list=['latitude'])\n",
    "region_a.order_vars.remove(beam_list=['profile_2'], var_list=['latitude'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 4: `keyword_list`\n",
    "Add `latitude` for all profiles and with keyword `low_rate`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(var_list=['latitude'],keyword_list=['low_rate'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 5: target a specific variable + path\n",
    "Remove `'profile_1/high_rate/latitude'` (but keep `'profile_3/high_rate/latitude'`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.remove(beam_list=['profile_1'], var_list=['latitude'], keyword_list=['high_rate'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 6: add variables not specific to beams/profiles\n",
    "Add `rgt` under `orbit_info`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(keyword_list=['orbit_info'],var_list=['rgt'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 7: add all variables+paths of a group\n",
    "In addition to adding specific variables and paths, we can filter all variables with a specific keyword as well. Here, we add all variables under `orbit_info`. Note that paths already in `region_a.order_vars.wanted`, such as `'orbit_info/rgt'`, are not duplicated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(keyword_list=['orbit_info'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 8: add all possible values for variables+paths\n",
    "Append all `longitude` paths and all variables/paths with keyword `high_rate`.\n",
    "Simlarly to what is shown in Example 4, if you submit only one `append` call as `region_a.order_vars.append(var_list=['longitude'], keyword_list=['high_rate'])` rather than the two `append` calls shown below, you will only add the variable `longitude` and only paths containing `high_rate`, not ALL paths for `longitude` and ANY variables with `high_rate` in their path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.append(var_list=['longitude'])\n",
    "region_a.order_vars.append(keyword_list=['high_rate'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 9: remove all variables+paths associated with a beam\n",
    "Remove all paths for `profile_1` and `profile_3`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.remove(beam_list=['profile_1','profile_3'])\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 10: generate a default list for the rest of the tutorial\n",
    "Generate a reasonable variable list prior to download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.remove(all=True)\n",
    "region_a.order_vars.append(defaults=True)\n",
    "pprint(region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying variable subsetting to your order and download\n",
    "\n",
    "In order to have your wanted variable list included with your order, you must pass it as a keyword argument to the `subsetparams()` attribute or the `order_granules()` or `download_granules()` (which calls `order_granules` under the hood if you have not already placed your order) functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.subsetparams(Coverage=region_a.order_vars.wanted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, you can put the `Coverage` parameter directly into `order_granules`:\n",
    "`region_a.order_granules(Coverage=region_a.order_vars.wanted)`\n",
    "\n",
    "However, then you cannot view your subset parameters (`region_a.subsetparams`) prior to submitting your order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_granules()# <-- you do not need to include the 'Coverage' kwarg to\n",
    "                             # order if you have already included it in a call to subsetparams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = './download'\n",
    "region_a.download_granules(path) # <-- you do not need to include the 'Coverage' kwarg to\n",
    "                             # download if you have already submitted it with your order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _Why does the subsetter say no matching data was found?_\n",
    "Sometimes, granules (\"files\") returned in our initial search end up not containing any data in our specified area of interest. This is because the initial search is completed using summary metadata for a granule. You've likely encountered this before when viewing available imagery online: your spatial search turns up a bunch of images with only a few border or corner pixels, maybe even in no data regions, in your area of interest. Thus, when you go to extract the data from the area you want (i.e. spatially subset it), you don't get any usable data from that image. Note that as of June 2020 this error message produced by the NSIDC subsetter is currently being modified to reflect this behavior as more of an informational warning as opposed to an error. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check the variable list in your downloaded file\n",
    "\n",
    "Compare the available variables associated with the full dataset relative to those in your downloaded data file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put the full filepath to a data file here. You can get this in JupyterHub by navigating to the file,\n",
    "# right clicking, and selecting copy path. Then you can paste the path in the second set of quotes below.\n",
    "fn = '/home/jovyan/' + 'data-access/notebooks/download/177534727/processed_ATL06_20190225121032_09020203_003_01.h5'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check the downloaded dataset\n",
    "Get all `latitude` variables in your downloaded file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "varname = 'latitude'\n",
    "\n",
    "varlist = []\n",
    "def IS2h5walk(vname, h5node):\n",
    "    if isinstance(h5node, h5py.Dataset):\n",
    "        varlist.append(vname)\n",
    "    return \n",
    "\n",
    "with h5py.File(fn,'r') as h5pt:\n",
    "    h5pt.visititems(IS2h5walk)\n",
    "    \n",
    "for tvar in varlist:\n",
    "    vpath,vn = os.path.split(tvar)\n",
    "    if vn==varname: print(tvar) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compare to the variable paths available in the original data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_vars.parse_var_list(region_a.order_vars.avail())[0][varname]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}