Faster sparse reachability

[markopy]

This replaces sparse_mutual_reachability with a version which is 137 times faster for my datasets. The algorithm is essentially identical but it operates on the CSR matrix directly, omitting conversion to LIL and back which is both very slow and uses a lot of unnecessary memory.

The code supports both float32 and float64 natively which allows for more memory savings if double precision is not required. Additionally we save another unnecessary copy by passing overwrite=True to csgraph.minimum_spanning_tree.

Lastly the user can use overwrite=True when calling hdbscan to indicate the distance matrix they pass can be modified in place, saving yet another copy.

Overall memory usage can be cut to 1/4th which makes it possible to deal with very large distance matrices. I have successfully clustered graphs with 10M nodes and 1B edges.

[pep8speaks]

Hello @markopy, Thank you for submitting the Pull Request !

• In the file hdbscan/hdbscan_.py:

Line 38:63: W504 line break after binary operator
Line 46:1: E402 module level import not at top of file
Line 48:1: E302 expected 2 blank lines, found 1
Line 52:1: E101 indentation contains mixed spaces and tabs
Line 52:1: W191 indentation contains tabs
Line 52:6: E128 continuation line under-indented for visual indent
Line 53:1: E101 indentation contains mixed spaces and tabs
Line 64:1: E101 indentation contains mixed spaces and tabs
Line 64:1: W191 indentation contains tabs
Line 64:16: E128 continuation line under-indented for visual indent
Line 66:1: E101 indentation contains mixed spaces and tabs
Line 127:25: E128 continuation line under-indented for visual indent
Line 177:25: E128 continuation line under-indented for visual indent
Line 213:25: E128 continuation line under-indented for visual indent
Line 250:25: E128 continuation line under-indented for visual indent
Line 316:25: E128 continuation line under-indented for visual indent
Line 327:101: E501 line too long (104 > 100 characters)
Line 360:1: E101 indentation contains mixed spaces and tabs
Line 360:1: W191 indentation contains tabs
Line 361:1: W191 indentation contains tabs
Line 362:1: E101 indentation contains mixed spaces and tabs
Line 482:76: W291 trailing whitespace
Line 483:1: E101 indentation contains mixed spaces and tabs
Line 483:1: W191 indentation contains tabs
Line 484:1: E101 indentation contains mixed spaces and tabs
Line 640:1: E101 indentation contains mixed spaces and tabs
Line 640:1: W191 indentation contains tabs
Line 640:10: E128 continuation line under-indented for visual indent
Line 641:1: E101 indentation contains mixed spaces and tabs
Line 641:13: E127 continuation line over-indented for visual indent
Line 681:1: E101 indentation contains mixed spaces and tabs
Line 681:1: W191 indentation contains tabs
Line 682:1: E101 indentation contains mixed spaces and tabs
Line 841:76: W291 trailing whitespace
Line 842:1: E101 indentation contains mixed spaces and tabs
Line 842:1: W191 indentation contains tabs
Line 844:1: E101 indentation contains mixed spaces and tabs
Line 974:5: E303 too many blank lines (2)
Line 1005:5: E303 too many blank lines (2)
Line 1041:5: E303 too many blank lines (2)
Line 1074:18: E128 continuation line under-indented for visual indent
Line 1089:18: E128 continuation line under-indented for visual indent
Line 1090:18: E128 continuation line under-indented for visual indent
Line 1110:72: W504 line break after binary operator
Line 1111:76: W504 line break after binary operator
Line 1112:77: W504 line break after binary operator
Line 1113:75: W504 line break after binary operator