> We’ve heard your feedback on our communications during this incident. You want more transparency, more in-depth information, and fewer “we are working on it” posts.
Well, those, and:
1. Speed. It took days for heroku customers to be told about this.
2. Customers sign up at "heroku.com", the platform is called "Heroku", the CLI is "heroku", everything's heroku, so don't send emails from a parent company (Salesforce), send them from "Heroku".
3. Unambiguous info on what customers need to do. I had to guess based on HN comments whether config vars were accessed. Config vars are 100x more sensitive than code. Comms should be unambiguous and complete, and if incomplete for any reason, explain that (e.g. we don't know yet).
4. I still don't know whether having 1 Github Deploy on my Heroku account allowed unauthorized access to all heroku applications on my heroku account (i.e. those using other deploy methods, like `git push heroku main`). Were all my apps' repositories able to be accessed, or just the one(s) deployed via Github Deploys?
5. I still don't know whether unauthorized access was gained to all other GitHub respositories on my GitHub account, i.e. the repos that aren't heroku apps.
These said, I still really appreciate that security incidents happen and aren't easy to deal with, and there's no obligation for anyone at a profitable company to actually care about semi-captive customers, so thanks to Heroku for the efforts; it's genuinely appreciated.
> 2. Customers sign up at "heroku.com", the platform is called "Heroku", the CLI is "heroku", everything's heroku, so don't send emails from a parent company (Salesforce), send them from "Heroku".
They're working on something called "Project Periwinkle" that is intended to remove all Heroku branding and make everything Salesforce branded. Periwinkle being a colour between blue (salesforce) and purple (heroku). No more Heroku signups, you'll need a Salesforce account to use it. No more free tier either.
Heroku has been in the process of being sunset for years now. New features have been banned for years. Only "keep the lights on" projects are allowed. Not that they could do anything with the skeleton crew they have running the platform.
I work for a non-Heroku tech vendor, and I have always spoken highly of Heroku to my customers from my past experience of going all-in on Heroku, and receiving a top quality of support from Heroku back in 2017-2019. I'm going to point folks to the recent Hacker News threads now.
It is difficult for me to understand why Salesforce is not aware of the strength of the Heroku brand among experienced technology workers, and how much they have destroyed that brand in the last 2-3 years.
The amount of Salesforce fluff in the post is quite palpable. There is a clear lack of control with the leaders in Heroku judging off this post because of the salesforce transition.
This project Periwinkle sounds awful. Basically thats the end of using Heroku for us. If it remains like this its something to judge from.
There really aren't any Heroku leaders anymore. I think anyone director level or higher is Salesforce—not Heroku.
It's worth mentioning that this isn't a result of Salesforce acquiring Heroku. That happened 12 years ago when Heroku was next to nothing. Salesforce gets credit for investing in and making Heroku. Why they ultimately have decided to give up on it I have no idea. I hear it's because salespeople had a hard time understanding how to sell it which seems like a strange reason to give up entirely.
This happens after every single Salesforce acqusition. If you think you're different, compare how many new internal promotions there are, how many senior management have left and how many Salesforce transplants there are.
Everyone thought they were the darling child, at least for the first 18 months.
> because salespeople had a hard time understanding how to sell it which seems like a strange reason to give up entirely.
Yet unfortunately from my experience with large orgs and sales, the sales people get all the control and freely shit on the people actually making the software they sell.
Periwinkle never materialized. Neither did Shinrai, or a number of other projects that people brought up in hn threads about this incident. It’s clear that Heroku has lost a ton of great talent and momentum due to questionable business decisions but some of the most controversial fail to ever make it past codename status.
It's on ice. As you say it's also not the first time they've attempted something like this.
Eventually they'll get there, but customers should know that's the direction. They'll be users of "Compute Cloud" writing APEX instead of Procfiles someday.
> New features have been banned for years. Only "keep the lights on" projects are allowed.
That's what it's looked like from the outside -- that no features were for whatever reason(s) no longer going to happen. But still dismaying to hear it from the inside in those terms.
You've gotta go way back on the Heroku Changelog to find anything that isn't a language version upgrade or feature removal: https://devcenter.heroku.com/changelog
The interesting thing is that heroku still is so at the top of what it does, as far as developer UX and ease and reliability. It's hard to describe exactly what i mean, but I know plenty of other people agree. Other things do other things better -- but it's only in the past year or two that some things have started to come close or equal.
I don't know what's going on exactly. Those who set up heroku for the first ~5 years somehow did such a good job that they could coast for another 5-10 and still stay on top of what they were on top of.
I was a long time Heroku user/lover (~10 years) for both personal and company projects. I've recently moved to Render[1], and so far it's been great. I haven't been using them long enough to put it through it's paces, but I'd certainly say it's worth a look. Other alternatives I've heard good things about are Fly.io[2] and Porter[3].
Checked out render.com in the context of side projects that I want on the air but don't expect anyone to ever use (I have so many on heroku, who got rewarded for their generosity with many thousands of dollars of business from me), which means I'm willing to pay a few dollars but not to pay per-app. Looks great except that they delete your free databases after 90 days, making it completely useless for this use case. Their announcement blog post says they plan to remove this limitation "early 2022", but it hadn't happened yet.
I next looked at fly.io, which seem ok for the first two apps, assuming they use the same database. If I ever want more than two apps, it seems I will need to start hosting different apps together, which is the opposite of the headache-free experience I'm looking for.
Porter runs on my own cloud account, so I can't trust it to not cost too much.
Maybe I can get a Kubernetes cluster somewhere (DigitalOcean?) and deploy all my small apps to it, but it sounds like a headache.
Dokku on top of a Digital Ocean droplet is a pretty cheap and easy option if you're OK with a single-server solution (i.e. small side project) and you're looking for something Heroku-like.
What about DO App Platform? I was thinking of trying that next. I’ve read a lot about slow build times and random build failures, but at the same time they at least seem to be actively developing the service.
Elsewhere in this thread is a link to a blogpost comparing it to doing things manually with GCP and quickly dismissing it as extremely limited, so I didn't delve further.
I have paid apps and free apps (staticman comment processing, and very simple apps I create while following learning tutorials). This project sounds bad! I’ve been slowly exploring alternatives, and was about to abandon my search due to demands on my time. Guess I need to keep exploring.
This all somehow seems unsurprising. If anyone has tried to use Heroku in easily the last year, the number of times you get failed builds over really trivial things is noticeable.
I haven’t experienced a failed build in a few years.
I did try a heroku competitor recently and my builds failed. And there was no detailed log to show why. So I couldn’t troubleshoot it and I immediately gave up!
As an aside, I was really hoping this post was going to be “we read that one competitor’s blog post last week clearly detailing all the areas where we can improve, and we HEAR you.”
“I have a lifelong enthusiasm for developers and the experience they have building software together”
And then drops a link to contact them, via LinkedIn…
LinkedIn is the polar opposite of GitHub. It’s the worst example of social media, from its news feed, to spam invites. And it’s broken every rule in the “be a good netizen” play book, from constant spam, to slurping your email contacts and surveillance to the extreme.
I struggle to imagine a developer saying “I’d like to contact xxx, and I’d love to do it via LinkedIn”
Why not drop your email? Or a GitHub profile with a public email, and readme containing other contact methods, would have been more dev centric.
I agree with you about LinkedIn. I also would have agreed with you about how silly that is, until I started working with giant banks and other huge finance industry people. Many of the devs there legit do use LinkedIn. A number of people actually blog and post updates and stuff on LinkedIn. It was really surprising. Startup culture and big corporate tech, despite using many of the same technologies, are miles apart culturally.
Yes, surprisingly many of the devs used LinkedIn for blog posts. The type of blog posts that they did were somewhat different as well. It was normal to see posts that had to do with engineering culture and how to structure an engineering department, things like that. They would also often talk about technology from a high level. For example "why I think vert.x is the future of Java high performance programming", or something like that.
I assume it’s so that they can mine your LinkenIn connection data and make it available to their sales teams to upsell you and people you know. I’d imagine they would end up storing all that data in some kind of web-based CRM, possibly sold as a SAAS product.
I guess it depends on what "developer friendly" means. If it means 'easiest method for a developer to contact me' that's email. It's ubiquitous. Everyone has at least one. It's easy.
If it means 'easiest method for only developers to contact me' that would probably be some documented REST API.
Email address and / or GitHub. No one's saying he needs to show off what a great programmer he is through his contact methods, just don't use a site that's infamously bad for dark patterns and recruiter spam.
As a former herokai, communication was always the #1 point of discussion internally. This idea that they "can" do better is a half truth. They DID do better. I leave it as an exercise to the reader to determine what the limiting resource was here.
By the way, how many senior devs and cofounders are left at Heroku Bob? Why doesn't it show up at dreamforce anymore?
(Former) Salesforce employee here, long enough to have been there a couple of years before the Heroku acquisition. I can't let this stand without a comment.
Heroku never seemed to want to integrate, but instead be the "cool kids", those who just do not have to worry about the enterprisy stuff such as automated backups, DR, high availability, enabling Java, etc. Everything they did was great, everything "Salesforce" sucked, yet they never made any money before the acquisition. The acquisition was a waste of money in first place (IMHO).
Trying to integrate with SFDC was a mistake. The product never fit inside of the company. Of course their investment made Heroku successful over the first few years (and eventually highly profitable as I understand) but they should've just spun it off and let it succeed without trying to figure out how to "enterprise" it.
What we knew how to do at Heroku was build a great platform for developers to launch apps. We never claimed to know how to make that model work for enterprise. In fact we were awful at trying.
Former Heroku employee here: everything you list happened within the first year of acquisition.
You’re correct that the lack of real integration with SFDC was a large part of the problem though. Though as far as SFDC acquisitions go Heroku was cheap. More expensive ones had far worse outcomes even with integration.
> Heroku never seemed to want to integrate, but instead be the "cool kids", those who just do not have to worry about the enterprisy stuff such as automated backups, DR, high availability, enabling Java, etc.
Seems like Salesforce got exactly what they bought. Not sure why they’d be surprised about it.
I knew the writing was on the wall for Heroku a few years back. I had asked a group of Salesforce sales reps about Heroku Connect (two-way sync between postgres and salesforce - amazing!), and they had no absolutely no idea what it was. They checked back with their team and weren't able to get any information. We basically couldn't buy it.
That's crazy! I work with Salesforce reps a lot for some pre-sales bids and they've been pushing connect (both salesforce connect and heroku connect) for the last few years. Cost is in the range of 5k/m or so for salesforce connect (from what I recall).
Heroku is a one of kind platform, many tried to replicate, none have come close, the ability to setup a whole app with a few clicks on dynos and add-ons is great. The price was always salty $50/month for 1gb RAM shared CPU, and in the few past years no newly released features comes to mind. As a customer I have no idea what this security issue impact, the communication has been poor as this post does not clarify much, you have to click a status post link to see a feed of what's going on.
Their docs are really copy paste for 99.9% applications. Whoever in that company made the decision to invest in docs was right on the money. It made switching to them a much easier choice. https://render.com
Update: Migrating the simplest possible Rails app from free-tier to free-tier didn't work on Render (deploys fail, no logs to be found), so I'm bailing out to try Fly.
To any render people reading this, it appears to have been an issue with the migration plugin and the Heroku-18 stack. Manually deploying the app worked fine.
Thanks for the heads up and good to hear you were able to deploy the app manually. We might deprecate the migrator because it's hard to keep up with all the Heroku stack and buildpack configs and because manual migrations are relatively straightforward.
> Whoever in that company made the decision to invest in docs was right on the money.
Also the reason Django has always been so approachable (particularly in the early days, as it is orders of magnitude more complex now) and therefore popular
Older still, IMHO also why mIRC scripting was so much fun despite its shortcomings. The help file had literally everything you could want to learn about the language and it almost begged to be read. It's how I got into coding
I hadn't heard of Render prior to this incident, but now I see it recommended in almost every related thread. Of course, Heroku's free tier had always worked well enough for my hobby apps, so I had no reason to search for alternatives. Looking forward to giving Render a try.
That isn’t a very charitable take. It’s been 24x7 for them, the acknowledgement is most welcome, in my opinion. I’m not sure waiting until Monday 9AM Pacific just to avoid these type of comments would be the right choice. The internet is 24x7 after all.
With what intention? Honest question, I don't see a strategic advantage in posting this on a Friday. More of a disadvantage, actually, with less visibility.
The point is less visibility/criticism. Since this is confirming pretty bad news (the attackers had access to the environment variables.) The concept of a Friday night news dump is a fairly well-worn, but basically true, trope.
> With what intention? Honest question, I don't see a strategic advantage in posting this on a Friday. More of a disadvantage, actually, with less visibility.
The intention presumably is to be able to say they communicated this while (intentionally) limiting the amount of awareness that's spread about this.
Fail it's because I am new; Succeed then it's my superior capability; plus, the learning rate during crisis is at least 10 times higher than the peaceful time, let alone the nerves and mental fortitude where peaceful time would never train one for.
Let’s give Bob a chance. I worked with him at AWS and prior. He is very committed to the customer, hence his post. We all try to communicate the best we can, so please try to support him in his new role.
I launched several startups on Heroku over the last decade +, and feel they have gravity on trust. Some of the best devs I have worked with.
> We also wanted to address a question regarding impact to environment variables. While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets.
Where do I find those notifications? I only got a single email, stating I should "reset my user password as mentioned in our previous notification". Nothing prior, nothing since, and their password reset form shows nothing but "Internal Server Error". I've been getting some details from HN but I think I'm missing a lot of the picture since they obviously don't send everyone every email.
> and the threat actor did not access the encryption key necessary to decrypt config var secrets.
If the threat actor had access to any of the systems that use the key, they may not have needed to. Even this statement isn't clear that they couldn't have done it, but suggests that they don't think it's true...
They got hacked about a month ago and have been extremely cagey about what exactly was hacked ever since.
Turns out the master database with encrypted username/passwords got leaked and encrypted environment variables were also leaked but it was like pulling teeth to get them to answer whether or not these happened or even admit that it might have been possible. Presumably more than this was also leaked but so far they haven't said anything on that. Env vars were the biggest concern on everyone's mind.
They gave the absolute least amount of information over the longest period they could muster.
The problem wasn't the hack really, it was the lack of transparency in the response.
I should've said "hashed" not encrypted passwords. But the env vars are the real problem. They haven't categorically dismissed the hacker somehow getting access to the actual environment variables either. Only said there isn't evidence of that happening.
If it comes out that the hacker did get to unencrypted env vars I think it's game over for Heroku. Nobody should trust them with sensitive data.
Maybe that’ll change, there’s a new Sheriff in town as the saying goes? Bob was previously General Manager of AWS Elastic Kubernetes (EKS) which seemed to have a good reputation and to engage with its customers pretty well.
>The Heroku team and their colleagues have worked around the clock, including nights and weekends
Can someone more familiar with an event like this tell me what they are working so hard on? I imagine securing the vulnerable service and resetting various credentials doesn't take that much work.
You have to make sure the attackers aren’t still in your network, you have to get them out if they are, you have to fully scope out what they messed with, and restore anything that’s plausibly connected. Even in the best case with all the right monitoring systems in place there is a lot of manual work involved from owners of the various different effected services and just managing the overall response adequately. I would say the amount of effort involved is comparable to managing a novel, ongoing SEV1.
Additionally there are legal concerns, for both regulatory compliance and for prepping for inevitable litigation. Those concerns aren’t necessarily a blocker for service restoration, but it really depends on the systems involved. If service could be restored by rebooting a system, for example, but that system also has data related to customers’ (and their customers’) PII and it might have been accessed by the attacker, then you need to make sure it’s all properly preserved forensically first, so that you can comply with regulations regarding breach notifications. The forensic analysis could then happen, but it’s definitely a “measure twice, cut once” situation with lots of lawyers involved (they won’t understand the systems, but they’ll make you explain everything so they can make decisions about risk; and they are in charge).
Also, generally, it’s a “fog of war” scenario, where you can have so many unknowns to work through in a compressed time period, and sometimes there’s an active attacker and they get a vote, too.
One quick example: combing through logs which can go back months or years can be expensive in time, especially if those are stored in something like Glacier and need to be pulled back into a system which permits analysis.
Proper incident response involves a lot more than just "securing the vulnerable service and resetting various credentials" and it will in this case probably involve a lot of different teams.
For starters, you have to document everything. At the very minimum the legal team for the company should insist on this, if no other measures, just as a CYA move.
If an attacker had broad access, it's entirely possible an all hands on deck approach is required to help identify (and document) what systems were compromised. Yes, you definitely want a team working on patching the hole ASAP. You also need a team hunting for any possible persistence. Another team probably involving standing up brand new "safe" systems and failing over client systems over to those running the patched software. While that happens yet more people may start doing audits of what was compromised on the original systems.
I've seen incidents where 50-80 people were pulled in to work on an incident at a company of about 150 employees. Depending on how well-funded your SoC is they can cut that number down substantially.
I moved over to Render and cancelled Heroku. I didn't mind it before because it was reliable (if not expensive) but this Github fiasco made me move. Happy to get rid.
Still weeks away from enabling the GitHub integration again, you can still manually deploy via the heroku CLI.
Our team realised heroku was going to take weeks and we’ve replicated our “review app” development workflow with GitHub actions that clone apps on PR, push code and rebuild them on push and destroy them on PR close. It’s not as seamless as the heroku GitHub integration but it’s good enough for now.
Same way we’ve done for years, via our CD (circle CI), which uses the Heroku CLI
We’ve been lucky that this entire event has, theoretically, not touched us as we never connected GitHub. That may change as more information comes to light.
We’re still strongly considering moving to AWS, and are in the process of getting quotes from vendors.
We're looking at a simple Lift & Shift to AWS. We're building our next product on AWS, so makes sense to move everything there, and the agency that we have doing the 24 hours monitoring runs a bunch of rails apps on their AWS stack so they've got everything in place to do it pretty simply.
GitHub should insist on it being converted to a GitHub App before they allow it to be re-enabled.
That way org admins can see the requested permissions and control exactly which repos are permitted. GitHub OAuth apps are an absolute nightmare to audit or control.
So I looked at their product offerings and the Wikipedia page about Heroku and I'm still left scratching my head as to what it actually even is. They basically rent out other cloud computer providers' time at a higher price and make you use some kind of shitty middleware? Is that all this actually is? Please help me to understand what in the hell this even is.
You broadly have it right, but it's not "shitty" middleware, despite their problems the actual process of launching an app on Heroku is stellar and was especially groundbreaking 10 years ago. It would provision a server, install dependencies, configure, then launch your app just by pushing a git repository to them.
They were the gold standard for rails deployments back in the day when rails was popular—which was important since rails was actually pretty hard to host on a server compared to say PHP.
Well, those, and:
1. Speed. It took days for heroku customers to be told about this.
2. Customers sign up at "heroku.com", the platform is called "Heroku", the CLI is "heroku", everything's heroku, so don't send emails from a parent company (Salesforce), send them from "Heroku".
3. Unambiguous info on what customers need to do. I had to guess based on HN comments whether config vars were accessed. Config vars are 100x more sensitive than code. Comms should be unambiguous and complete, and if incomplete for any reason, explain that (e.g. we don't know yet).
4. I still don't know whether having 1 Github Deploy on my Heroku account allowed unauthorized access to all heroku applications on my heroku account (i.e. those using other deploy methods, like `git push heroku main`). Were all my apps' repositories able to be accessed, or just the one(s) deployed via Github Deploys?
5. I still don't know whether unauthorized access was gained to all other GitHub respositories on my GitHub account, i.e. the repos that aren't heroku apps.
These said, I still really appreciate that security incidents happen and aren't easy to deal with, and there's no obligation for anyone at a profitable company to actually care about semi-captive customers, so thanks to Heroku for the efforts; it's genuinely appreciated.