Keyvi - the short form for "Key value index" - defines a special subtype of the popular key value store (KVS) technologies. As you can imagine from the name, keyvi is an immutable key value store, therefore an index not a store. Keyvi's strengths: high compression ratio and extreme scalability. So if you need online read/writes keyvi is not for you, however, if your use case is mostly reads and infrequent writes you might be interested in checking keyvi out.
This is the continuation of cliqz-oss/keyvi. Keyvi was initially developed at Cliqz by Hendrik Muhs and others. For more information, please refer to https://github.com/cliqz-oss/keyvi
Precompiled binary wheels are available for OS X and Linux on PyPi. To install use:
pip install keyvi
The core part is a C++ header-only library, but the TPIE 3rdparty library needs to be compiled once. The commandline tools are also part of the C++ code. For instructions check the Readme file.
For the python extension of keyvi check the Readme file in the python subfolder.
- Howtos
- Compiling Dictionaries/Indexes
- Python version of keyvi
- Crashcourse
- Using python keyvi with EMR (mrjob or pyspark)
If you like to go deep down in the basics, keyvi is inspired by the following 2 papers:
- Sparse Array (See Storing a Sparse Table, Robert E. Tarjan et al. http://infolab.stanford.edu/pub/cstr/reports/cs/tr/78/683/CS-TR-78-683.pdf)
- Incremental, which means minimization is done on the fly (See Incremental Construction of Minimal Acyclic Finite-State Automata, J. Daciuk et al.: http://www.mitpressjournals.org/doi/pdf/10.1162/089120100561601)
keyvi is licenced under apache license 2.0, see licence for details.
In addition keyvi uses 3rdparty libraries which define their own licence. Please check their respective licence. The 3rdparty libraries can be found at keyvi/3rdparty.