Every time I’m assigned to a project that uses a document database
“So how are you guys handling all your related data?”
Finds collection of massive JSON documents containing all the related data
“Oh boy.”
What’s the problem with that? In my previous team, we had a structure with four levels of nesting where we only ever needed to query the first two levels. At first we used Postgres with normalized tables, but it was just slow as hell. Switching to MongoDB actually made our performance issues vanish.
Of course it all depends on what kinds of queries you need to run, but I don’t think that large JSON documents are necessarily a problem.
They’re talking about relations between data. For example, when you delete a user, you may also want to delete their stored data.
To some degree, this is less of a problem with document databases, because they don’t force you to chop your data into small parts like relational databases do (e.g. you can have lists of that user’s stored data as part of the JSON document). But you will likely still need some relations at some point.
Chances are you have a layer in your application code which ensures these relations that way.
Which is fine in my opinion. With relational databases, there’s also often some relations which you cannot model in the database.
But yeah, it requires somewhat more software architecture awareness, to not lump the relation checking logic into general application logic. And you can’t connect a second application to that database, without having to implement the relations another time or at least pulling them out into a shared library.In my previous team, we had a structure with four levels of nesting
Those are rookie numbers.
But it’s webscale
I still suggest piping data to /dev/null during meetings to see who gets the joke
Oh man, do i have the product for you
Haha, they have a careers page.
Licensing
Pick any. We license it this way then.
*The old Facebook ReactJS non-compete license
*University of Utah Public License
*Apple Public Source License v1.x
*AT&T Public License
*The JSON License
*The Occulus Rift License
*TrueCrypt License 3.0
Wow, that video is 15 years old!
This is kinda absolute BS at this point, though.
Mongo has acid transactions, and has for years now. Although this is only within the same database, there are plenty of dbms (including rdbms) that don’t support cross-database transactions.
Mongo also, since time immemorial, has had “write concern” to ensure that it’s written to disk (to the journal) before the transaction is completed.
This post is very timely because I was just introducing some new people to Mongo earlier this week and led off with “Now you might still hear people say ‘mongo is trash, it’s not even ACID compliant!’ but those people are dumb… it’s had that for years and years and is just another DBMS at this point (but not relational)”
… the last part also answers the other reply to this post. Yes.
So is it just another database software at this point, then?
NoSQL has always been a niche use case thing.
For some stuff, no ACID is no problem. They have their place. What I’m more suspicious of is things like Google offering distributed databases that they pretend as if they could break the CAP theorem.
What’s ACID?
Atomicity: either all parts of the transaction complete, or all parts of the transaction don’t complete; there’s no “partly complete” state
Consistency: the state of the database after a transaction is stable; all “downstream” effects (e.g. triggers) of the query are complete before the transaction is confirmed.
Isolation: concurrent transactions behave the same as sequential transactions
Durability: a power failure or crash won’t lose any transactions
Traditionally, ACID is where relational databases shine.
You’ve gotten good answers from other folks but I’ll provide a ELI5:
Basically a set of rules in the database to make sure that it is immediately consistent.
NoSQL databases offer eventual consistency in exchange for speed so they are generally not considered to be ACID compliant.
Most traditional databases (MySQL, postgresql, etc.) are.
There are a couple of emerging companies that try to tackle speed for traditional databases. CockroachDB offers a postgress-based database that scales more like NoSQL while still offering ACID transactions.
TiDB is a similar company but for MySQL.
Not all NoSQL databases are the same. Neo4j is acid compliant, and lightning fast for complex relationships that relational databases struggle with.
https://en.m.wikipedia.org/wiki/ACID
Atomicity (something happens in its entirety or not at all), consistency (database is always in a valid state — if the database has constraints, they will always be honored), isolation (transactions don’t step on each other), durability (complete transaction is complete even if there’s a power failure).
Not a database expert, my parenthetical explanations may need work.
And yet my Uni treats it like the biggest thing in existence. Meanwhile I’ve never used anything other than RDBS and Redis (only for cache), neither in private nor at work.
MongoDB is huge though for all the wrong reasons, businesses think that just because it’s JS, they can just have frontend devs - sorry, they are “fullstack” now - doing DBA work.
I worked as one of two NoSQL DBAs for a Fortune 50 finance company, and there is a ton of CV-driven development going on giving NoSQL a bad name. Most use cases don’t need NoSQL. And for those which do, NoSQL is almost always harder to implement than simple SQL based RDBMSs.
Jumping in this, bingo. JavaScript only shops scare the fuck out of me.
why is my deploy process so slow? ©_©=> 500k npm packages
Just wail til they become AI-generated-JavaScript-only shops. They’re gonna be vibing like the Tacoma Narrows Bridge.
It always depends on the context… My current job is 100% on Elasticsearch and I’m not missing transactions at all.
Sharded RDBS gets you very very far from my experience at least.
Definitely, and I’m saying that while my jobs were mostly on NoSQL and I love doing it.
If you need to run queries that aggregate big amounts of data in a reasonable time and cost, you’ll need something built for it. For example, with a column oriented file format instead of the row oriented file format found in traditional relational databases
And the key word “big” here is far bigger than most engineers need to deal with. Hell, most supposed “big data” problems I’ve seen people try to tackle are small enough to fit the whole database into memory.
My point is more that 90% of use cases don’t need that, and for those that do, you can’t just slap eg. Cassandra at it and pretend it’s a relational database.
If that is what it takes to get these kick-ass benchmarks.
Amazing video
I never understood why people compare nosql to rdbms. They are entirely different systems with different use cases.
Where you neee data consistency and need to always get the same results to a query go with a structured rdbms. Where you need speed over all of that (and there are real use cases for this) then nosql is for you. Using both is of course a likely result too.
There’s of course a lot of other considerations. But they’re different tools for different situations.
I think it’s because the early marketing and hype compared NoSQL to rdbms. At the beginning they were all “hey man, don’t schemas suck? Isn’t it a pain having to migrate your data? Sometimes you just wanna cram shit somewhere, go fast, break things, and your DBA is a jackass! MongoDB”
And people, at that time, were either like “what the fuck?” and continue to not trust it to this day, or “hell yeah brother!” and then put everything into Mongo and were surprised when it lost some data or got into a corrupted state, or at least were surprised the first time they thought “huh, I really wish there was some consistency to all this data…”
So yeah, I think MongoDB didn’t come into the scene as “I’m a new kinda thing that has niche uses” it came on as “hey pussy, why are you still using your dad’s DB. Are you afraid?” and people still carry that in their hearts
Mongodb is not actually faster. Postgres still beats it in any benchmark that matters.
Nothing is ever actually schema-less. There is merely explicit and implicit schemas. If you don’t want to bother encoding the schema as proper columns and instead want the schema to remain implicitly encoded in JSON, Postgres’
jsonbcolumns do a better job of that than any NoSQL database does.This. Why would almost every web app use a 50 yo language designed for accountants to store like 10kB of data on a remote disk? If you’re not in the field, it should be astronomically confusing
Because nerds like to be smarter and “more efficient”.
God forbid you fuck around when PHP though!
What about PostgreSQL?
I do use postgres but only as an rdb provider. I thought while it supports json data as a type, does it provide for all of the other advantages of nosql databases for their use case?
Ultimately I feel like the best solution is to have a single database provider that could do both fully. I’m not sure it’s really there yet. But halpt to be told I’m wrong. I’ve not really needed that myself for my projects.
The query speed isn’t quite there but I would say it’s close enough for a lot of purposes, especially with proper indexing. And JSON column fields are indexable. Two things I’ve used Postgres’ JSON functionality for are:
1.) Storing unstructured data. 2.) Storing structured data that would exceed the table column limit.
In both cases, I’ve typically needed to extract the relevant data from the JSON records to either be stored in another table of turned into a materialized view so live query performance on the JSON columns was not that important.
Fair enough! Thanks for the detailed write-up
Or does writes to S3 via LSM
