Add cloud utilities for multirows, datalinks #22
+844
−0
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With the goal of breaking the other PR #18 into smaller pieces. The code here processes the cloud information, and it uses the download functions in PR #21 (so that should be approved first).
The main parts:
_process_json_column
,_process_cloud_datalinks
and_process_ucd_column
. These return a dictionary elements: e.g. {'url':..} forprem
or {'uri':.., 'bucket_name':.., 'key': ...} foraws
that can be passed to the relevant download functions.get_data_product2
. This is similar to what we have now but handles datalinks correctly, and multipleRecord
's (QueryResult
) andRow
's (Table
). Once review is done, the name should beget_data_product
.get_data_product
takes a product (pyvo Record or astropy Row/Table) and aprovider
(prem, aws etc) and a returns aProviderHandler
instance (a list that has adownload
method).get_data_product2
takes also another optionmode
if the user wants to selectjson
,datalink
(others can be added).ProviderHandler
has also aget_links
method that can be used to figure out the url/uri in case the user want to use the cloud cutout for example._fornax.py
so not to interfere withfornax.py
, but once review is done, the first should replace the latter.An example use case is: