Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I was surprised to see LevelDB (https://code.google.com/p/leveldb/) was missing from the list of storage solutions you tried, because it seems optimal for your use-case. Were you aware of it?


I'm not sure about the optimal use-case match. Sparkey is for "mostly static" datasets where on disk structures are generated by a batch process and pushed to servers providing read only access to the data to consumers.

leveldb on the other hand, supports concurrent writes and provides features to handle data consistency and cheap gradual reindexing.


Also Sparkey seems to work well with bittorrent/rsync distribution. I recall spotify use bittorrent to distribute files to their servers.


According to GitHub, last commit (which is also initial checkin) is two years old.


The project is hosted at Google Code. Last commit was Aug 21: https://code.google.com/p/leveldb/source/detail?r=748539c183...


There is also the basho fork of LevelDB (https://github.com/basho/leveldb) which adds a few nice things.

If you're interested in the internals of LevelDB, I strongly recommend watching this talk http://www.youtube.com/watch?v=vo88IdglU_8 (slides here https://speakerdeck.com/basho/optimizing-leveldb-for-perform...)


He means the last commit on sparkey


Right, this is actually the first commit - it's just that the history was squashed before publish - we had to remove sensitive Spotify specific things in it, and it seemed easiest to just do a big squash.


Why do you think it seems optimal for their use case?

A quick glance over LevelDB's features gives me the impression that its bulk-write performance would not be sufficient.


The problem is, that so many projects for so many things exists. It is hard to find the matching project, thats why many people invent the wheele again.


LevelDB has very few good use cases and this certainly isn't one of them. http://symas.com/mdb/microbench/ http://symas.com/mdb/hyperdex/




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: