Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ETL refactor #1594

Open
godexsoft opened this issue Aug 12, 2024 · 0 comments
Open

ETL refactor #1594

godexsoft opened this issue Aug 12, 2024 · 0 comments
Labels
enhancement New feature or request refactoring Code improvements without logical changes research testability Makes code testable and/or adds unit-tests
Milestone
2.3

Comments

@godexsoft
Copy link
Collaborator

godexsoft commented Aug 12, 2024 •
edited
Loading

Summary

Many parts of the ETL need improvements. Separate issues will be created and linked here for each sufficiently big component.

Subtasks

  1. Node awareness abstraction - ETL node awareness abstraction #1595
  2. Extractor - ETL ng extractor #1596
  3. Loader - ETL ng loader #1597
  4. Backfill - Backfill #66
  5. Plugins - ETL ng plugins support #1598
  6. Move NFT/bookdir - ETL ng NFT and book dir #1599

Motivation

ETL is one of the last parts of Clio that are still not refactored to be testable and covered by unit tests.
The ETL also does not support certain planned functionality that is too complicated to implement on top of the existing architecture.

Solution

Refactor/rewrite ETL
@godexsoft godexsoft added enhancement New feature or request testability Makes code testable and/or adds unit-tests refactoring Code improvements without logical changes research labels Aug 12, 2024
This was referenced Aug 12, 2024
Open
Open
Open
Open
Open
@godexsoft godexsoft added this to the 2.3 milestone Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request refactoring Code improvements without logical changes research testability Makes code testable and/or adds unit-tests
Projects
Clio
Status: 📋 Backlog
Milestone
2.3
Development

No branches or pull requests
1 participant
@godexsoft