Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

...and same credentials apparently also - there are lots of things that could have prevented something like this.


Yeah this stuck out to me as well. To be honest if they ARE using the same creds in prod as they are in dev, this is the first thing they should fix because it would have easily prevented the whole mess.

I'll give them the benefit of the doubt and suggest that it wasn't the case and that it's unlikely that a dev or OPs person was tunnelling through a bastion of some kind to run the script.

If they pulled their credentials from a store and populated the config object that way, it's very possible someone actually loaded the production credentials by mistake into the development secrets file. The CI/pipeline system has permissions/network access to deploy to any environment, hence how it ran the script to drop the table.

I'm purely speculating another alternative to the much worse case of the tunnelling scenario outline.


could also be that they just have an if staging/else somewhere for the credentials (but of course this isn't a counterpoint to the root of your point - the script wiping the database shouldn't be using anything that does something like this, and you probably shouldn't do it at all)


Keep going, I'm writing this down!


Take the "drop database" bit (on the production database) away from your developers, too.

As well as pretty much every other privilege they don't legitimately need to use on a daily basis -- which, for prod, should be most of them (quite possibly including "delete").

If or when they really need to delete a ton of rows all in one go, they can be given a (temporary) set of credentials that they can use to do that, once, and which are then revoked immediately afterwards -- after another set of eyes reviews the script they've written to actually perform the operation, of course.

Basic best practices aren't difficult or some secret thing that only the experts know about. It is necessary to actually implement and follow them, however!

Sure, it can be a pain in the ass sometimes. Will it be worth it when it eventually saves your ass one day, "after a couple of glasses of red wine"? Absolutely.


As a developer at all the jobs I've had, I've never even had access to the production database, full stop - only the server admins did. If I needed something from prod I'd go through them. I don't even consider it inconvenient.


The distance between the developer and the production database is directly proportional with the size of the company. The same is true for the amount of paperwork needed for even the simplest things. For some company sizes a feedback cycle measured in days is not good enough and may kill the business.

There is a whole lot of things that could be applied which would have prevented this, before needing to hire a separate server admin as an interface to the production database. :D


that's true, I totally forgot about SQL permissions, there's so many failsafes for this




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: