This is pretty much the direction of all specialty database providers (MongoDB, Redis) are moving into. Everyone wants to be able to query datasets across enterprise. I bet is Redis Stack will soon querying against support Parquet or ORC files. Then connector to traditional databases like PG, MySQL, Oracle, etc.. Throw in some AI modeling. In a way, this is a marketing play to stay relevant in the enterprise space.
Redis as a pure in-memory store (caching) engine just doesn't make as much money as "analytic system". I don't have a problem with this as they have to make money somehow.
What a crazy mess of services, so sad to see. redis was such a beautiful, elegant and pragmatic code base that was extremely effective at what it does. It looks like they are trying their hardest to make it more enterprisy to squeeze every last cent out.
Open source has changed a lot recently. Most people/companies do open source to make money. The initial phase is just to make the software popular enough so that a money-making business can be built around it.
Frankly, I don't see anything bad with it. Those who don't want to spend money get it for free and those who can afford will pay for it.
Is it? I mean, how? The core Redis software is still open source as it's always been, with the same (well, improved over time) functionality, whereas the closed-source for-profit components are not even 'paywalled features' but clearly separate higher-layer products building on the core (visualisation, ML, time series, full-text search, graph structures, &c). To me this is a paradigmatic example of well-done open-core.
I don't see how it's 'toxic' to express dissatisfaction with mission creep in a piece of software. It's a totally normal, reasonable criticism, and from the author's point of view is far more useful than their staying silent.
It doesn't change Redis. redis-server will stay simple as it always been, redis-stack-server extends redis-server with addition functionality for those who need more low latency data access more than Redis K/V.
If you read the license carefully you'll notice it's much more permissive than AGPL, and allows you as an end user (as long as you're not developing a database) to do with it whatever you want.
Yea I agree, I love redis.
I have to admit seeing this group of services was a little strange, but your comment about enterprise money seems on the nose.
We feel like the most important part required from a low latency DB, Redis "Vanilla" and Redis Enterprise already cover.
Now, we can leverage this amazing platform to provide a real low latency multi model DB on top of it.
We tried to keep it as vendor natural as we could, for example in the Graph API we picked the openCypher query language and in the JSON API we used JSONPath.
Is there any example of this ever working in practice in the history of computing? As in, we built a large, stable service with datastore X, but we were forward looking enough to make sure that we only "developed to an interface" and we successfully swapped to datastore Y without significant code changes?
The project I'm currently working uses AWS DynamoDB as primary DB. Since it's proprietary, we ensure the AWS-specific APIs are contained in small and specific part of the codebase.
For instance, to create a "User" we need the PutItem API from DynamoDB [1]. Similarly, to retrieve a "User", there's the GetItem API [2].
Instead of making references to these APIs all over our codebase, we have a single `db-interface` module, which implements `get_user` and `create_user` functions. Each of these functions has an interface: they expect specific arguments with corresponding types. This interface is modeled against our data domain, not DynamoDB data domain quirks.
Inside the `db-interface` module, we implement the conversion from our data domain to Dynamo's. We also have general-purpose functions, like `create_item` and `get_item`, so that `get_user` and `create_user` are pretty much wrappers and only serve the purpose of defining the interface for interacting with the "User" data object.
The rest of our code only interacts with our internal interface, never with DynamoDB APIs directly.
If we were to switch database, we only need to re-implement the general-purpose functions (e.g. `create_item`, `get_item`).
May sound like a lot of work, but it took only a handful of days to implement the entire interface mapping everything we needed from DynamoDB.
Hope this helps clarifying a bit the application of this concept (develop for an interface, not an implementation) in the context of interacting with databases.
I understand the concept, I'm just asking if anyone has ever actually done it. DynamoDB has various guarantees, features, edge cases etc that you have to be aware of in your app code, and any DB that you want to switch to will have a different set of guarantees, features, edge cases etc that you will have to support throughout your app. So it's more than just an interface.
That's interesting. Have you tested this with another DB? The design makes a lot of sense, but I feel like it's the kind of thing that's hard to get right unless you actually test it. At the company I currently work, they switched DB some time ago. They already used interfaces but even that was not enough to make the switch, due to some differences in how the DBs worked.
If you can map the differences in your code and your interface does not leak any implementation detail, you won't have problems.
But if the current database has features that can't be mapped to the new one, the interface won't save you.
Example: DynamoDB has a "time-to-live" feature [1]. You add a specific property to any item with a timestamp and it will auto-delete this item at the timestamp determined.
Few databases will provide this out of the box. If you choose to use it and later need to migrate, you'll have to implement your own kind of "TTL monitor" to perform the same task. The interface can't possibly save you from this.
But this would be an issue whether you decide to use a central interface or not, anyway...
Thanks, that "time-to-live" feature is a great exemple of what I was talking about. At work if I remember correctly it was something about not being able to read a write just after making it, and having to wait one second. I guess that's the kind of things where you either need people to be very careful when developing the implementation, or have people that have already been burned by that to think about that.
The only thing I hate about Redis is that all data has to stay in memory...
Well I know that's obvious, but Redis is nearly the perfect solution in terms of performance. Unlike disk-based databases, Redis has consistently optimal performance on the order of millions of queries per second. And horizontal scaling is trivial with Redis Cluster (it involves running multiple Redis instances and many Redis service providers don't support it - I hope to have an easier way of running cluster though).
Given that memory is cheap it's quite plausible to store an entire database in a single Redis instance. I do hope that Redis finds its way to replace conventional databases. I think we just need an asynchronous `FLUSH` command, which blocks until the next `fsync`, and that should get us closer to ACID compliance.
I wish product would be small sharp tools that does something well
and sticks to it, iterating on making it faster,more stable,
better documented, easier to setup and configure.
Instead a lot of product start out that way and then if it becomes
popular scope creep rears it ugly head and it is extended to be
mediocre at many things.
I find far too many developers once they start investing time in
a product they want it to do all sorts of other things that there
are allready tools that do it well.
I have used Redis TimeSeries a few years ago and I found it easy to use and very performant for my needs. I think it is still chugging along at my last employer. The development team is already familiar with Redis and therefore found TimeSeries simple and easy to use. None of us wanted to use ( and learn) a dedicated TimeSeries database API and business did not want an expanded footprint to maintain. So Redis TimeSeries was an excellent solution for us
Around two months ago I was excited to learn about RedisJSON, only to get to know that there is no ARM-based Docker image available for installing it on a Raspberry Pi.
Probably some combination of knowledge, experience, time and interest.
Frankly it sounds like kind of an intimidating ask. Do you have special knowledge that makes this seem simple, or is it actually not challenging once you really get all the pieces involved? I'd love to be confident that I could see a statement like the one you replied to and immediately execute on it, so if you have any tips to share please do!
Well, to start, have you tried to send information in their issue tracker?
Maybe my answer sounded more harsh in tone than what I meant, sorry for that.
I red the message as "Hey they didn’t cover my specific case!", and I felt like this was unfair to ask an opensource project to cover each corner case upfront and not even consider a lacking feature as something that can be reported or maybe even contributed back.
Yes, it came off sharp, it would have discouraged me from contributing. Sometimes people don't really get that the process is open to all and need to be invited, I think it can be good to try and see stuff like this as an opportunity to extend that invitation.
I think you misread this one because it is basically "excitement about new thing -> implied disappointment at not getting to use new thing", and I can't find an ask in that.
The redisgraph functionality is really cool, it makes it an interesting choice over SQLite for small data science projects. The search functionality also is helpful for data science projects.
Does anyone know what this RSAL license’s impact is for internal tools (e.g log analysis)?
RSAL 1.1(b): "[Licensor hereby grants to You a ... license ... to:] use the Software, or your Modifications, only as part of Your Application, but not in connection with any Database Product that is distributed or otherwise made available by any third party"
As long as this log analysis is part of the production environment of your application, and you don't resell it as a tool, that should be fine.
I don't understand why there is timeseries solution there, usually people who use timeseries work with many Gigas of storage, are we gonna pay for RAM to do that?
First it all depends on your latency requirements and TP, if you need your TimeSeries DB to support high ingest TP and low query latency a RAM based is almost the only way you can go.
Second, many Gigas is what you get from any small size server...
And, on top of that Redis/TimeSeries supports an almost linear scale out which allows your memory to increase dynamically according to your needs.
Last, Redis Enterprise (& Cloud) add Redis on Flash support extending your RAM to your local Flash storage.
I have run Redis clusters, both with and without Redis on Flash, with hundreds of GBs of data. If you need the performance, it's doable and not prohibitively expensive.
You're right, we would love to provide a much better experience in Ruby it was just a matter of priorities...
BTW, all the modules already have ruby based clients developed by the community.
To use the extra features that come with Redis Stack, you need a client that supports those features. According to the site posted, the Ruby client doesn't support these features.
So it appears that you are the one that is missing something.
> So it appears that you are the one that is missing something.
They were asking what library support was missing for that person's use case (which your question answers, but I think the snark is based on a misunderstanding).
Redis Stack is not Open Source, so that wouldn't be possible. AWS would have to reimplement all of the functionality, make their implementation Open Source, and keep adding new features to keep up.
Redis as a pure in-memory store (caching) engine just doesn't make as much money as "analytic system". I don't have a problem with this as they have to make money somehow.