🛑 setup_harvest failed: KeyError

[ooi-data-bot]

Overview

KeyError found in setup_harvest task during run ended on 2023-02-11T21:23:33.375388.

Details

Stream name: CE07SHSM-SBD11-04-VELPTA000-telemetered-velpt_ab_dcl_instrument
Flow name: stream_ingest
Flow run: b5bb1527-2390-4b2f-97c6-27d9951155fc
Task name: setup_harvest
Error type: KeyError
Error message: 'id'

Traceback

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 876, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/prefect/utilities/executors.py", line 467, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/home/runner/micromamba-root/envs/harvester/lib/python3.8/site-packages/ooi_harvester/pipelines/stream/tasks.py", line 232, in setup_harvest
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/ooi_harvester/producer/__init__.py", line 47, in fetch_streams_list
    streams_df, _ = get_ooi_streams_and_parameters(filtered_instruments)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/ooi_harvester/metadata/__init__.py", line 41, in get_ooi_streams_and_parameters
    streams_list = compile_instrument_streams(instruments)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/ooi_harvester/metadata/utils.py", line 157, in compile_instrument_streams
    streams_list = map_concurrency(fetch_streams, instruments, max_workers=50)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/ooi_harvester/utils/compute.py", line 20, in map_concurrency
    data = future.result()
  File "/srv/conda/envs/notebook/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/srv/conda/envs/notebook/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/srv/conda/envs/notebook/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/ooi_harvester/utils/conn.py", line 190, in fetch_streams
    newst.update(get_stream(stream["stream"]))
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/ooi_harvester/utils/conn.py", line 110, in get_stream
    "stream_id": stream_dict["id"],
KeyError: 'id'