Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make --no-mmap calls still use parallelism when filesizes are large #361

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Make --no-mmap calls still use parallelism when filesizes are large #361

wants to merge 4 commits into from

Commits on Nov 6, 2023

  1. Improve performance of --no-mmap mode by enabling multithread. 

    This uses a double buffer of 1MiB each, reading to one buffer while
hashing the other in parallel. This is around 2x as fast as hashing
singlethreadedly on my machine (ryzen 2600) with an in memory benchmark.

This is still 2x slower than using memmap.
    @ultrabear
    ultrabear committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    1322974 View commit details
    Browse the repository at this point in the history

  2. add comments explaining thought processes

    @ultrabear
    ultrabear committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    e297112 View commit details
    Browse the repository at this point in the history

  3. remove accidentally left in profile fiddling

    @ultrabear
    ultrabear committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    0b3f843 View commit details
    Browse the repository at this point in the history

  4. add testcase that hash_reader_parallel gives correct results

    @ultrabear
    ultrabear committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    03e0949 View commit details
    Browse the repository at this point in the history