The major cost factors in software development are developer wages.
It’s all about how you spend the time you have. Don’t build stuff that you can get off shelf. You’re not going to write a better database, even if you think you can. You may get great performance for the current system on day one (after spending a huge amount of time and effort on developing it, perhaps with zero value created) but over the lifetime of the database you’ll incur huge costs that you probably can’t even fully foresee. The details here are not clear enough, but working from first principles (“I want to build an Evernote-like app”) I can’t imagine an experienced developer suggesting you should write a database (cache/file system).
The best software development is mundane: glue together what’s there, buy the resources you need to get sufficient performance. Switch a component if you need something to be faster, re-architect key parts, and buy the new components off the shelf too. It’s fun to build new stuff with custom algorithms you work out, but instead of that you can go home at 5pm, try building a database in your spare time and learn why you shouldn’t, and still have space to relax.
As Sam Altman wrote recently, the productivity tip that most people are astoundingly ignorant of is: choose what you do carefully.
It’s telling that this article does not begin: “we had a bottleneck that could not be solved.”
The story people should take away from the article isn't "Microsoft wrote their own database for their note-taking app, so should I for mine!", it's that "Microsoft launched and made wildly popular a note-taking app using standard tools, then learned their problem space including subtle requirements about the deployment environment, and THEN wrote their own database to solve real pain-points they had in practice."
On the flip side - building just enough software to solve the problem at hand can sometimes run circles around assembling things using off the shelf components. This is especially true if something like speed / memory / correctness / optimality is important.
In each of the above fields there are categories of software where you can run circles around general solutions by writing something custom and yes some of those involve writing your own database.
Also just about any software that handles business workflow. A common tradeoff in development is dynamic vs efficiency. For example, in a database, if you can define columns for properties you know about, you will have a much faster database than if you created a bunch of "Thing" tables with "ThingProperty" relationships.
PATIENT
=======
PatientId
FirstName
LastName
vs
THING
=====
Id
ThingTypeId
THINGPROPERTY
=============
ThingPropertyId
ThingId
PropertyName
PropertyValue
Storing it doesn't take too much of a noticeable hit, but any queries on "Thing" tables get expensive real fast.
I’d take exception to embedded software and robotics, where reuse and safety usually take precedence over performance, although I’d agree that a minority of such applications may be good examples.
Apart from that, I agree with your list but I’d consider them to be very specialised domains that are not like the vast majority of software development, especially building conventional apps like OneNote (a document database).
Thoroughly agree. There are very rare times where you need to write your own low level, high complexity system for a high level application, but ... they’re really, really rare nowadays. Almost always it’s developers practicing resume driven development.
I’ve never worked with anyone who tried to implement their own database, but I have worked with people who implemented their own network protocols, JS SPA frameworks and service discovery layers, and they were all really bad ideas.
It gets even worse when said person leaves the company, their system is still critically important and full of bugs, and has the inevitable little-to-no documentation. Other devs have to maintain this mess, and nobody truly understands it.
It’s all about how you spend the time you have. Don’t build stuff that you can get off shelf. You’re not going to write a better database, even if you think you can. You may get great performance for the current system on day one (after spending a huge amount of time and effort on developing it, perhaps with zero value created) but over the lifetime of the database you’ll incur huge costs that you probably can’t even fully foresee. The details here are not clear enough, but working from first principles (“I want to build an Evernote-like app”) I can’t imagine an experienced developer suggesting you should write a database (cache/file system).
The best software development is mundane: glue together what’s there, buy the resources you need to get sufficient performance. Switch a component if you need something to be faster, re-architect key parts, and buy the new components off the shelf too. It’s fun to build new stuff with custom algorithms you work out, but instead of that you can go home at 5pm, try building a database in your spare time and learn why you shouldn’t, and still have space to relax.
As Sam Altman wrote recently, the productivity tip that most people are astoundingly ignorant of is: choose what you do carefully.
It’s telling that this article does not begin: “we had a bottleneck that could not be solved.”