Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The KGE Archive needs to handle *huge* graph file sets #33

Open
RichardBruskiewich opened this issue May 6, 2021 · 1 comment
Open
Assignees
Labels
September 2021 Relay September 2021 Relay bug fix / feature addition

Comments

@RichardBruskiewich
Copy link
Collaborator

The system needs to handle sizable graphs, but Relay may test this assumption. Real world graph sizes:

  • Node normalization KG: 23.18 GB
  • RTX2: 106.13 GB
  • Semmeddb: ~ 12 GB
  • SRI Reference KG 0.2: ~ 12 GB
  • New Robokop: 112.22 GB
@RichardBruskiewich
Copy link
Collaborator Author

RichardBruskiewich commented Jul 26, 2021

Some general progress already made: multi-part uploads, etc.

We probably should implement the direct URL file transfers (Issue #15) plus programmatic and CLI modes (Issue #16; with streaming) to facilitate the transfer of huge files.

Note that the KGEA has background uploading of files. What if a huge file upload needs to be cancelled (issue #37)? What should happen?

@RichardBruskiewich RichardBruskiewich added the September 2021 Relay September 2021 Relay bug fix / feature addition label Sep 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
September 2021 Relay September 2021 Relay bug fix / feature addition
Projects
None yet
Development

No branches or pull requests

2 participants