Refactor： use [fusio](https://github.com/tonbo-io/fusio) to implement the storage layer and support multiple storage #163
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KKould
added
enhancement
ethe
force-pushed
the
refactor/fusio
branch
from
d6ab5d4
to
3ed1231
Compare
… the storage layer and support multiple storage
KKould
force-pushed
the
refactor/fusio
branch
from
3ed1231
to
ba73c59
Compare
KKould
force-pushed
the
refactor/fusio
branch
3 times, most recently
from
f05f65f
to
c2e1ed0
Compare
KKould
force-pushed
the
refactor/fusio
branch
from
c2e1ed0
to
d3c42eb
Compare
KKould
force-pushed
the
refactor/fusio
branch
from
c4fd8e7
to
09bc312
Compare
|
Read: +--------------------------------------------+----------+---------+
| | tonbo | rocksdb |
+=================================================================+
| random range reads | 136978ms | 37863ms |
|--------------------------------------------+----------+---------|
| random range reads | 134460ms | 37787ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 53292ms | 37680ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 53218ms | 37731ms |
|--------------------------------------------+----------+---------|
| random range projection reads (4 threads) | 38840ms | 38093ms |
|--------------------------------------------+----------+---------|
| random range projection reads (8 threads) | 37563ms | 37933ms |
|--------------------------------------------+----------+---------|
| random range projection reads (16 threads) | 34760ms | 36319ms |
|--------------------------------------------+----------+---------|
| random range projection reads (32 threads) | 30681ms | 36181ms |
+--------------------------------------------+----------+---------+ |
|
Write: +-------------------------------+--------+---------+
| | tonbo | rocksdb |
+==================================================+
| bulk load | 6433ms | 9550ms |
|-------------------------------+--------+---------|
| individual writes | 8243ms | 10656ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8227ms | 10640ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7771ms | 10674ms |
|-------------------------------+--------+---------|
| batch writes | 8058ms | 7137ms |
|-------------------------------+--------+---------|
| batch writes (4 threads) | 7822ms | 7110ms |
|-------------------------------+--------+---------|
| batch writes (8 threads) | 7706ms | 7171ms |
|-------------------------------+--------+---------|
| removals | 3118ms | 3823ms |
+-------------------------------+--------+---------+ |
|
Read: +--------------------------------------------+----------+---------+
| | tonbo | rocksdb |
+=================================================================+
| random range reads | 138091ms | 37846ms |
|--------------------------------------------+----------+---------|
| random range reads | 135251ms | 37909ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 53065ms | 38053ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 53213ms | 37889ms |
|--------------------------------------------+----------+---------|
| random range projection reads (4 threads) | 38564ms | 38157ms |
|--------------------------------------------+----------+---------|
| random range projection reads (8 threads) | 34929ms | 38028ms |
|--------------------------------------------+----------+---------|
| random range projection reads (16 threads) | 34649ms | 36626ms |
|--------------------------------------------+----------+---------|
| random range projection reads (32 threads) | 31311ms | 36543ms |
+--------------------------------------------+----------+---------+ |
|
Write: +-------------------------------+--------+---------+
| | tonbo | rocksdb |
+==================================================+
| bulk load | 6406ms | 9470ms |
|-------------------------------+--------+---------|
| individual writes | 8351ms | 10466ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8168ms | 10689ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7694ms | 10589ms |
|-------------------------------+--------+---------|
| batch writes | 8009ms | 7102ms |
|-------------------------------+--------+---------|
| batch writes (4 threads) | 7700ms | 7165ms |
|-------------------------------+--------+---------|
| batch writes (8 threads) | 7707ms | 7130ms |
|-------------------------------+--------+---------|
| removals | 3115ms | 3792ms |
+-------------------------------+--------+---------+ |
KKould
force-pushed
the
refactor/fusio
branch
from
32319c3
to
0df8149
Compare
|
Read: +--------------------------------------------+----------+---------+
| | tonbo | rocksdb |
+=================================================================+
| random range reads | 129046ms | 38181ms |
|--------------------------------------------+----------+---------|
| random range reads | 129076ms | 38144ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 52355ms | 38120ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 53120ms | 37673ms |
|--------------------------------------------+----------+---------|
| random range projection reads (4 threads) | 40735ms | 37473ms |
|--------------------------------------------+----------+---------|
| random range projection reads (8 threads) | 40130ms | 37336ms |
|--------------------------------------------+----------+---------|
| random range projection reads (16 threads) | 37260ms | 35875ms |
|--------------------------------------------+----------+---------|
| random range projection reads (32 threads) | 32191ms | 35816ms |
+--------------------------------------------+----------+---------+ |
|
Write: +-------------------------------+--------+---------+
| | tonbo | rocksdb |
+==================================================+
| bulk load | 6444ms | 9445ms |
|-------------------------------+--------+---------|
| individual writes | 8330ms | 10657ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8201ms | 10652ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7787ms | 10590ms |
|-------------------------------+--------+---------|
| batch writes | 8021ms | 7150ms |
|-------------------------------+--------+---------|
| batch writes (4 threads) | 7601ms | 7122ms |
|-------------------------------+--------+---------|
| batch writes (8 threads) | 7579ms | 7130ms |
|-------------------------------+--------+---------|
| removals | 3109ms | 3825ms |
+-------------------------------+--------+---------+ |
|
Read: +--------------------------------------------+----------+---------+
| | tonbo | rocksdb |
+=================================================================+
| random range reads | 133243ms | 37035ms |
|--------------------------------------------+----------+---------|
| random range reads | 130852ms | 36974ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 53094ms | 36895ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 53073ms | 36839ms |
|--------------------------------------------+----------+---------|
| random range projection reads (4 threads) | 39900ms | 36908ms |
|--------------------------------------------+----------+---------|
| random range projection reads (8 threads) | 39510ms | 36935ms |
|--------------------------------------------+----------+---------|
| random range projection reads (16 threads) | 36956ms | 35581ms |
|--------------------------------------------+----------+---------|
| random range projection reads (32 threads) | 32172ms | 35633ms |
+--------------------------------------------+----------+---------+ |
|
Write: +-------------------------------+--------+---------+
| | tonbo | rocksdb |
+==================================================+
| bulk load | 6466ms | 9430ms |
|-------------------------------+--------+---------|
| individual writes | 8207ms | 10425ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8141ms | 10445ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7797ms | 10493ms |
|-------------------------------+--------+---------|
| batch writes | 7944ms | 7097ms |
|-------------------------------+--------+---------|
| batch writes (4 threads) | 7592ms | 7108ms |
|-------------------------------+--------+---------|
| batch writes (8 threads) | 7357ms | 7140ms |
|-------------------------------+--------+---------|
| removals | 3070ms | 3806ms |
+-------------------------------+--------+---------+ |
ethe
reviewed
Sep 27, 2024
ethe
approved these changes
Sep 27, 2024
ethe
reviewed
Sep 27, 2024
src/serdes/num.rs
Outdated
| buf | ||
| }; | ||
| async fn decode<R: Read + Unpin>(reader: &mut R) -> Result<Self, Self::Error> { | ||
| let bytes = reader.read_exact(vec![0u8; size_of::<Self>()]).await?; |
crwen
reviewed
Sep 27, 2024
ethe
reviewed
Sep 27, 2024
| use crate::fs::FileProvider; | ||
|
|
||
| pub trait Executor: FileProvider { | ||
| pub trait Executor { |
There was a problem hiding this comment.
Executor has to provide the local file system implementation, use tokio executor + monoio filesystem should cause panic.
|
Read: +--------------------------------------------+----------+---------+
| | tonbo | rocksdb |
+=================================================================+
| random range reads | 136761ms | 37490ms |
|--------------------------------------------+----------+---------|
| random range reads | 133886ms | 37780ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 53961ms | 37422ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 53031ms | 37231ms |
|--------------------------------------------+----------+---------|
| random range projection reads (4 threads) | 39582ms | 37312ms |
|--------------------------------------------+----------+---------|
| random range projection reads (8 threads) | 39151ms | 37213ms |
|--------------------------------------------+----------+---------|
| random range projection reads (16 threads) | 36138ms | 35554ms |
|--------------------------------------------+----------+---------|
| random range projection reads (32 threads) | 32139ms | 35607ms |
+--------------------------------------------+----------+---------+ |
|
Write: +-------------------------------+--------+---------+
| | tonbo | rocksdb |
+==================================================+
| bulk load | 6454ms | 9538ms |
|-------------------------------+--------+---------|
| individual writes | 8333ms | 10662ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8254ms | 10723ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7868ms | 10701ms |
|-------------------------------+--------+---------|
| batch writes | 7958ms | 7171ms |
|-------------------------------+--------+---------|
| batch writes (4 threads) | 7613ms | 7272ms |
|-------------------------------+--------+---------|
| batch writes (8 threads) | 7443ms | 7282ms |
|-------------------------------+--------+---------|
| removals | 3102ms | 3828ms |
+-------------------------------+--------+---------+ |
crwen
approved these changes
Sep 27, 2024
|
Read: +--------------------------------------------+----------+---------+
| | tonbo | rocksdb |
+=================================================================+
| random range reads | 129198ms | 39229ms |
|--------------------------------------------+----------+---------|
| random range reads | 116346ms | 39126ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 49997ms | 40353ms |
|--------------------------------------------+----------+---------|
| random range projection reads | 49972ms | 40300ms |
|--------------------------------------------+----------+---------|
| random range projection reads (4 threads) | 39654ms | 40491ms |
|--------------------------------------------+----------+---------|
| random range projection reads (8 threads) | 38564ms | 40539ms |
|--------------------------------------------+----------+---------|
| random range projection reads (16 threads) | 35719ms | 38945ms |
|--------------------------------------------+----------+---------|
| random range projection reads (32 threads) | 31509ms | 39040ms |
+--------------------------------------------+----------+---------+ |
|
Write: +-------------------------------+--------+---------+
| | tonbo | rocksdb |
+==================================================+
| bulk load | 6464ms | 9537ms |
|-------------------------------+--------+---------|
| individual writes | 8315ms | 10911ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8208ms | 10880ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7857ms | 10801ms |
|-------------------------------+--------+---------|
| batch writes | 7967ms | 7227ms |
|-------------------------------+--------+---------|
| batch writes (4 threads) | 7634ms | 7216ms |
|-------------------------------+--------+---------|
| batch writes (8 threads) | 7640ms | 7236ms |
|-------------------------------+--------+---------|
| removals | 3161ms | 3855ms |
+-------------------------------+--------+---------+ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.