Update the Runtime to work with the Berkeley schema (meta issue) 📚

[eecavanna]

note: planned "full freeze" of main branch on 2024-09-18, after which the goal is to merge berkeley into main.

---

This is a "meta issue" that represents the overall task of updating the Runtime to work with the Berkeley schema.

Terminology

• The Runtime:
  • Everything in the nmdc-runtime repository
• The Berkeley schema:
  • Version 11 of the nmdc-schema PyPI package (which is built from the berkeley-schema-fy24 repo)
• The legacy schema:
  • Versions <= 10 of the nmdc-schema PyPI package (which is built from the nmdc-schema repo)

Branches

I created a branch named berkeley that can act as our "main" branch for Berkeley-specific code.

If it turns out we can add Berkeley schema support directly to the main branch without breaking support for the legacy schema, we can delete the berkeley branch.

Constituent Tasks

When creating a GitHub Issue for a constituent task, please add a link to it to one of the lists below, depending upon whether you consider it to be a blocker for deploying an MVP of the NMDC system that uses the Berkeley Schema.

Blockers

• berkeley: Validate or remove _translate_omics_processing function (in gold_translator.py) that returned an nmdc.OmicsProcessing #586 (@sujaypatil96)
• berkeley: Synchronize Swagger UI documentation with standalone api_gui.md how-to guide #646 (@eecavanna)
• berkeley: Dagster job ensure_alldocs fails with AssertionError  #690 (@dwinston, @PeopleMakeCulture)
• berkeley: API endpoint /data_objects/study/{study_id} not returning correct results #689
• berkeley: Fix failing tests in test_gold_api_ops.py #678 (@sujaypatil96)
• berkeley: Merge main branch into berkeley branch #691 (@eecavanna)
• berkeley: The setup code of four tests in test_endpoints.py is failing (while populating alldocs) #703
• berkeley: Collection alldocs is missing one document from each upstream collection #708 (@eecavanna)

Non-blockers

• berkeley: Validate or remove _translate_omics_processing function (in neon_soil_translator.py) that returned an nmdc.OmicsProcessing #587 (@sujaypatil96)
• berkeley: Validate or remove _translate_omics_processing function (in neon_benthic_translator.py) that returned an nmdc.OmicsProcessing #588 (@sujaypatil96)
• berkeley: :param name: annotations for API endpoint are visible on Swagger UI #698
• berkeley: Replace designated_class with type (in ncbi_xml_utils.py script) #630 (@sujaypatil96)
• berkeley: Validate or remove _translate_omics_processing function (in neon_surface_water_translator.py) that returned an nmdc.OmicsProcessing #589 (@sujaypatil96)

Uncategorized

• Migrations: Implement notebook that runs all Berkeley schema migrators #519 (@eecavanna)
• berkeley: Update GHA workflow to publish container image from berkeley branch #582 (@eecavanna)
• berkeley: Fix type hint issue preventing Runtime from booting with Berkeley schema #578 (@eecavanna)
• berkeley: Fix test failing due to Biosample JSON data being invalid #580 (@aclum)
• berkeley: Update GHA workflow to publish container image from berkeley branch #582 (@eecavanna)
• berkeley: Fix failing test: tests/test_api/test_metadata.py::test_ensure_data_object_type (DataObject lacks type field) #590 (@aclum)
• berkeley: Update minter to use first typecode when pattern contains multiple typecodes #592 (@eecavanna)
• berkeley: Update activity related endpoints #593 (@PeopleMakeCulture)
• berkeley: Dagster job claim_and_run_apply_changesheet_jobs fails to authenticate with Mongo server #594 (@eecavanna)
• berkeley: Dagster does not insert data into Mongo despite json:submit returning success response #595 (@dwinston & @PeopleMakeCulture @eecavanna)
• berkeley: Fix failing tests in GOLD, NEON, and submission portal translators #600 (@PeopleMakeCulture)
• berkeley: Refactor _get_instrument_name in nmdc_runtime/site/translation/gold_translator.py #622 (@sujaypatil96)
• berkeley: Document expectations around merging main into berkeley branch over time #626 (@eecavanna)
• berkeley: Replace Activity with DataGeneration (in referential integrity checker notebook) #628 (@PeopleMakeCulture & @dwinston)
• berkeley: Replace OmicsProcessing with DataGeneration (in referential integrity checker notebook) #629 (@PeopleMakeCulture & @dwinston)
• berkeley: Update notebook based upon assumption that type is required (referential integrity checker notebook) #632 (@PeopleMakeCulture & @dwinston)
• berkeley: Allow developer to switch between ORCID production and sandbox via environment variable #636 (@eecavanna)
• berkeley: ORCID_JWK dictionary is not compatible with ORCID sandbox environment #638 (@eecavanna)
• berkeley: Update nmdc-schema dependency to latest pre-release (11.0.0rc20) #649 (@eecavanna)
• berkeley: Change workflow_executions endpoints to be planned_process #653 (@dwinston)
• berkeley: Update submission_portal_translator.py for Berkeley rollout #655 (@pkalita)
• berkeley: Function ensure_schema_collections_and_alldocs is incompatible with Berkeley schema #658 (@dwinston)
• berkeley: Restore hyperlinks to "Metadata" section and modernize Runtime description #685
• berkeley: Test test_submit_workflow_activities is being skipped and is incompatible with Berkeley schema #683 (@aclum)
• berkeley: Restore hyperlinks to "Metadata" section and modernize Runtime description #685 (@eecavanna)
• berkeley: Tests in 3 files are being skipped by mistake #687 (@eecavanna)

[dwinston]

#619 introduces a new test which will need to have type added to tests/files/test_changesheet_insert_study_doi.tsv in order to pass.

h/t @aclum

[eecavanna]

Someone asked me how often I expected the main branch to be merged into the berkeley branch and I realized I never announced or documented my expectation about that. My expectation was that main be merged into berkeley whenever there was something in main that people wanted to exist in berkeley, even if that happened multiple times a day. I want to keep those main-to-berkeley merges small.