"All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding,"
This is false. AWS infrastructure is vastly more secure than almost all company data centers. AWS has a rule that the same person cannot have logical access and physical access to the same storage device. Very few companies have enough IT people to have this rule. The AWS KMS is vastly more secure than what almost all companies are doing. The AWS network is vastly better designed and operated than almost all corporate networks. AWS S3 is more reliable and scalable than anything almost any company could create on their own. To create something even close to it you would need to implement something like MinIO using 3 separate data centers.
> AWS infrastructure is vastly more secure than almost all company data centers
Secure in what terms? Security is always about a threat model and trade-offs. There's no absolute, objective term of "security".
> AWS has a rule that the same person cannot have logical access and physical access to the same storage device.
Any promises they make aren't worth anything unless there's contractually-stipulated damages that AWS should pay in case of breach, those damages actually corresponding to the costs of said breach for the customer, and a history of actually paying out said damages without shenanigans. They've already got a track record of lying on their status pages, so it doesn't bode well.
But I'm actually wondering what this specific rule even tries to defend against? You presumably care about data protection, so logical access is what matters. Physical access seems completely irrelevant no?
> Very few companies have enough IT people to have this rule
Maybe, but that doesn't actually mitigate anything from the company's perspective? The company itself would still be in the same position, aka not enough people to reliably separate responsibilities. Just that instead of those responsibilities being physical, they now happen inside the AWS console.
> The AWS KMS is vastly more secure than what almost all companies are doing.
See first point about security. Secure against what - what's the threat model you're trying to protect against by using KMS?
But I'm not necessarily denying that (at least some) AWS services are very good. Question is, is that "goodness" required for your use-case, is it enough to overcome its associated downsides, and is the overall cost worth it?
A pragmatic approach would be to evaluate every component on its merits and fitness to the problem at hand instead of going all in, one way or another.
Physical access is pretty relevant if you could bribe an engineer to locate some valuable data's physical location, then go service the particular machine, copy the disk (during servicing "degraded hardware"), and thus exflitrate the data without any traces of a breach.
Physical access and logical root access can't hide things form each other. It takes both to hide an activity. If you only have one, then the other can always be used to uncover or detect in the first place, or at least diagnose after.
1. big clouds are very lucrative targets for spooks, your data seem pretty likely to be hoovered up as "bycatch" (or maybe main catch depending on your luck) by various agencies and then traded around as currency
2. you never hear about security probems (incidents or exposure) in the platforms, there's no transparency
I think it's a very relevant bar, though. The top level commenter made points about "a business of just about any size", which seems pretty exactly aligned with "most corporate stuff".
If you don't want your data to be accessible to "various agencies", don't share it with corporations, full stop. Corporations are obliged by law to make it available to the agencies, and the agencies often overreach, while the corporations almost never mind the overreach. There are limitations for stuff like health or financial data, but these are not impenetrable barriers.
I would just consider all your hosted data to be easily available to any security-related state agency; consider them already having a copy.
That depends where it's hosted and how it's encrypted. Cloud hosts can just reach into your RAM, but dedicated server hosts would need to provision that before deploying the server, and colocation providers would need to take your server offline to install it.
Colocated / Dedicated is not Cloud, AFAICT. It's the "traditional hosting", not elastic / auto-scalable. You of course may put your own, highly tamper-proof boxes in a colocation rack, and be reasonably certain that any attempt to exfiltrate data from them won't be invisible to you.
By doing so, you share nothing with your hosting provider, you only rent rack space / power / connectivity.
At least you can get your own /48, at least if you're under RIPE.
You should only do it if you expect to multihome though, or you're doing some experimentation that absolutely needs a PI address. Please don't pollute the default-free zone just for no reason.
There's much variation by jurisdiction. Eg US based big-cloud companies would seem more risky here if you're from a country with traditionally less invasive (and less funded) spooks.
4. we keep hitting hypervisor bugs and having to work around the fact that your software coexists on the same machine with 3rdparty untrusted software who might in fact be actively trying to attack you. All this silliness with encrypted memory buses and the various debilitating workarounds for silicon bugs.
So yes, the cloud is very secure, except for the very thing that makes it the cloud that is not secure at all and has just been papered over because questioning it means the business model is bust.
Most corporations (which is the vast majority of cloud users) absolutely don't care about spooks, sadly enough. If that's the threat model, then it's a very very rare case to care about it. Most datacenters/corporations won't even fight or care about sharing data with local spooks/cops/three letter agencies. The actual threat is data leaks, security breaches, etc.
If I remember right, the attacker’s AWS employment is irrelevant - no privileged AWS access was used in that case. The attacker working for AWS was a pure coincidence, it could’ve been anyone.
one of my greatest learnings in life is to differentiate between facts and opinions- sometimes opinions are presented as facts and vice-versa. if you think about it- the statement "this is false" is a response to an opinion (presented as a fact) but not a fact. there is no way one can objectively define and defend what does "real technical understanding" means. the cloud space is vast with millions of people having varied understanding and thus opinions.
so let's not fight the battle that will never be won. there is no point in convincing pro-cloud people that cloud isn't the right choice and vice-versa. let people share stories where it made sense and where it didn't.
as someone who has lived in cloud security space since 2009 (and was founder of redlock - one of the first CSPMs), in my opinion, there is no doubt that AWS is indeed superiorly designed than most corp. networks- but is that you really need? if you run entire corp and LOB apps on aws but have poor security practices, will it be right decision? what if you have the best security engineers in the world but they are best at Cisco type of security - configuring VLANS and managing endpoints but are not good at detecting someone using IMDSv1 in ec2 exposed to the internet and running a vulnerable (to csrf) app?
when the scope of discussion is as vast as cloud vs on-prem, imo, it is a bad idea to make absolute statements.
Great points. Also if you end up building your apps as rube goldberg machines living up to "AWS Well Architected" criteria (indoctrinated by staff lots of AWS certifications, leading to a lot of AWS certified staff whose paycheck now depends on following AWS recommended practices) the complexity will kill your security, as nobody will understand the systems anymore.
about security, most businesses using AWS invest little to nothing in securing their software, or even adopt basic security practices for their employees
having the most secure data center doesn't matter if you load your secrets as env vars in a system that can be easily compromised by a motivated attacker
so i don't buy this argument as a general reason pro-cloud
This exactly, most leaks don't involve any physical access. Why bother with something hard when you can just get in through an unmaintained Wordpress/SharePoint/other legacy product that some department can't live without.
It’s like putting something in someone’s desk drawer under the guise of convenience at the expense of security.
Why?
Too often, someone other than the data owner has or can get access to the drawer directly or indirectly.
Also, Cloud vs self hosted to me is a pendulum that has swung back and forth for a number of reasons.
The benefits of the cloud outlined here are often a lot of open source tech packaged up and sold as manageable from a web browser, or a command line.
One of the major reasons the cloud became popular was networking issues in Linux to manage volume at scale. At the time the cloud became very attractive for that reason, plus being able to virtualize bare metal servers to put into any combination of local to cloud hosting.
Self-hosting has become easier by an order of magnitude or two for anyone who knew how to do it, except it’s something people who haven’t done both self-hosting and cloud can really discuss.
Cloud has abstracted away the cost of horsepower, and converted it to transactions. People are discovering a fraction of the horsepower is needed to service their workloads than they thought.
At some point the horsepower got way beyond what they needed and it wasn’t noticed. But paying for a cloud is convenient and standardized.
Company data centres can be reasonably secured using a number of PaaS or IaaS solutions readily available off the shelf. Tools from VMware, Proxmox and others are tremendous.
It may seem like there’s a lot to learn, except most problems they are new to someone have often been thought of a ton by both people with and without experience that is beyond cloud only.
Usually the larger the company and the more mission critical the product: the worse the implementation.
Twitch source code (which, I guess counts as Amazon already), Disney leaks- and my own experience working with very large companies. (Nokia, Ubisoft, Facebook, Activision/Blizzard).
Your comment tells me you have never read any of AWS many documents about how they engineer their components. They put an huge amount of effort into it. AWS is much more reliable that Azure. They have built the largest and most reliable storage system in the world with S3. AWS has stated that some customers have S3 buckets using over 1 million hard drives. Netflix relies heavily on AWS for its streaming services. Lyft runs its ride-sharing platform on AWS. Capital One migrated its entire infrastructure to AWS. Slack relies on AWS for its messaging platform. GE utilizes AWS for industrial IoT (Internet of Things) solutions, predictive maintenance, and data analytics. Twitch streams video to 31 million viewers from AWS.
AWS is an industry leader in using formal methods and automated reasoning to prove the security and reliability of critical software and detect insecure configurations
One of the ways the NSA and security services get so much intelligence on targets isn't by direct decryption of what they are storing in data or listening in.
A great deal with their intelligence is simply metadata intelligence. They watch what you do. They watch the amount of data you transport. They watch your patterns of movement.
So even if eight of us is providing direct security and encryption in the sense of what most security professionals are concerned with key strength etc etc etc, Eddie of us still has a great deal about of information about what you do, because they get to watch how much data moves from where to where and other information about what those machines are
Isn’t it more like leasing in a public property? Meaning it is yours as long as you are paying the lease? Analogous to renting an apartment instead of owning a condo?
I would just like to point out that most of us who have ever had a job at an office, attended an academic institution, or lived in rented accommodation have kept stuff in someone else’s desk drawer from time to time. Often a leased desk in a building rented from a random landlord.
Keeping things in someone else’s desk drawer can be convenient and offer a sufficient level of privacy for many purposes.
And your proposed alternative to using ‘someone else’s desk drawer’ is, what, make your own desk?
I guess, since I’m not a carpenter, I can buy a flatpack desk from ikea and assemble it and keep my stuff in that. I’m not sure that’s an improvement to my privacy posture in any meaningful sense though.
It doesn’t have to be entirely literal, or not literal at all.
A single point of managed/shared access to a drawer doesn’t fit all levels of data sensitivity and security.
I understand this kind of wording and analogy might be triggering for the drive by down voters.
A comment like the above though allows both people to openly consider viewpoints that may not be theirs.
For me it shed light on something simpler.
Shared access to shared infrastructure is not always secure as we want to tell ourselves. It’s important to be aware when it might be security through abstraction.
The dual security and convenience of self-hosting IaaS and PaaS even at a dev, staging or small scale production has improved dramatically, and allows for things to be built in a cloud agnostic way to allow switching clouds to be much easier. It can also easily build a business case to lower cloud costs. Still, it doesn’t have to be for everyone either, where the cloud turns to be everything.
A small example? For a stable homeland - their a couple of usff small servers running proxmox or something residential fibre behind a tailscale or cloudflare funnel and compare the cost for uptime. It’s surprising how much time servers and apps spend idling.
Life and the real world is more than binary. Be it all cloud or no cloud.
> Keeping things in someone else’s desk drawer can be convenient and offer a sufficient level of privacy for many purposes.
Too torture a metaphor to death, are you going to keep your bank passwords in somebody else's desk drawer? Are you going to keep 100 million people's bank passwords in that drawer?
> I guess, since I’m not a carpenter, I can buy a flatpack desk from ikea and assemble it and keep my stuff in that. I’m not sure that’s an improvement to my privacy posture in any meaningful sense though.
If you're not a carpenter I would recommend you stay out of the business of building safe desk drawers all together. Although you should probably still be able to recognize that the desk drawer you own, that is inside your own locked house is a safer option then the one at the office accessible by any number of people.
If you have something physical of equivalent value to 100 million people's bank passwords, you may well not want to risk keeping it in a desk drawer at all, and instead want to look into renting a nice secure drawer from someone else to keep it in. That would be a safety deposit box.
Which I would argue is rather more like what cloud providers offer than 'someone else's desk drawer' is.
The other part is that when us-east-1 goes down, you can blame AWS, and a third of your customer's vendors will be doing the same. When you unplug the power to your colo rack while installing a new server, that's on you.
It's not always a full availability zone going down that is the problem. Also, despite the "no one ever got fired for buying Microsoft" logic, in practice I've never actually found stakeholders to be reassured by "its AWS and everyone is affected" when things are down. People want things back up and they want some informed answers about when that might happen, not "ehh its AWS, out of our control".
OTOH, when your company's web site is down you can do something about it.
When the CEO asks about it, you can explain why its offline and more importantly what is being done to bring it back.
The equivalent situation for those who took a cloud based approach is often... ¯\_(ツ)_/¯
The more relevant question is whether my efforts to do something lead to a better and faster result than my cloud providers efforts to do something. I get it - it feels powerless to do nothing, but for a lot of organizations I’ve seen the average downtime would still be higher.
I worked in IT for a state government and they had a partial outage of their Exchange server that lasted over 2 weeks. It triggered a full migration to Exchange online.
With the cloud, in a lot of cases you can have additional regions that incur very little cost as they scale dynamically with traffic. It’s hard to do that with on-prem. Also many AWS services come cross-AZ (AZ is a data center), so their arch is more robust than a single Colo server even if you’re in a single region.
Cross region from on-prem to the cloud for a website is easy. In fact, as long as you don't buy into "cloud native" ("cloud lock-in"?), it's probably more cost effective than two on-prem regions or two cloud regions.
Being able to choose from so many different Availability Zones in so many different regions is one of the best things about AWS. Combined with sophisticated routing strategies that Route 53 supports allows for some very effective designs.
KMS access is granted by either environment variables or by authorizing the instance itself. Either way, if the instance is compromised, then so is access to KMS. So unless your threat model involves preventing the government from looking at your data through some theoretical sophisticated physical attack, then your primary concerns are likely the same as running a box in another physically secure location. So the same rules of needing to design your encryption scheme to minimize blowout from a complete hostile takeover still apply.
An attacker gaining temporary capability to encrypt/decrypt data through a compromised instance is painful. An attacker gaining a copy of a private key is still an entirely different world of pain.
Painful is an understatement. Keys for sensitive customer data should be derived from customer secrets either way. Almost nobody does that though, because it requires actual forethought. Instead they just slap secrets in KMS and pretend it's better than encrypted environment variables or other secrets services. If an attacker can read your secrets with the same level of penetration into your system, then it's all the same security wise.
There are many kinds of secrets that are used for purposes where they cannot be derived from customer secrets, and those still need to be secured. TLS private keys for example.
I do disagree on the second part - there’s a world of a difference whether an attacker obtains a copy of your certificates private key and can impersonate you quietly or whether they gain the capability to perform signing operations on your behalf temporarily while they maintain access to a compromised instance.
It's all unencrypted secrets from perspective of an attacker. If they somehow already have enough access to read your environment variables, then they can definitely access secrets manager records authorized for that service. By all means put secrets management in a secondary service to prevent leaking keys, but you don't need a cloud service to do that.
It's now been two years since I used KMS, but at the time it seemed little more than S3 API interface with Twitter size limitations
Fundamentally why would KMS be more secure than S3 anyway? Both ultimately have the same fundamental security requirements and do the same thing.
So the big whirlydoo is KMS has hardware keygen. im sorry, that sounds like something almost guaranteed to have nsa backdoor, or has so much nsa attention it has been compromised.
If your threat model is the NSA and you’re worried about backdoors then don’t use any cloud provider?
Maybe I’m just jaded from years doing this, but two things have never failed me for bringing me peace of mind in the infrastructure/ops world:
1. Use whatever your company has already committed to. Compare options and bring up tradeoffs when committing to a cloud-specific service(ie. AWS Lambdas) versus more generic solutions around cost, security and maintenance.
2. Use whatever feels right to you for anything else.
Preventing the NSA from cracking into your system is a fun thought exercise, but life is too short to make that the focus of all your hosting concerns
I guess since this is Hacker News, I shouldn’t be surprised that there are a bunch of commenters who are absolutely certain they and their random colo provider will do a better job of defeating the almighty NSA than AWS.
You won’t even know when they serve your Colo provider with a warrant under gag order, and I’m certain they’ll be able to bypass your own “tamper-proof” protections.
Soo..... you're saying that KMS hardware key generation isn't that great anyway...
so, again, why bother with KMS? What does it offer?
My point about the hardware was asking why KMS hardware key generation has any real value vs a software generated key, and then why bother with KMS and its limited secret size, and you access KMS with a policy/security user or role that can be used equally to lock down S3?
If the NSA is part of your threat model then good luck. I'm not sure any single company could withstand the NSA really trying to hack them for years. The threat of possible NSA backdoors is not a reasonable argument against a cloud provider as the NSA could also have backdoors in every CPU AMD and Intel and AWS makes.
You can securely store your asymmetric key for signing, but if I remember correctly the logs are pretty useless, basically you just know the key was used to make a signature, no option to log the signature or additional metadata, which would help auditing after an account/app compromise.
Taking for granted all these points. How many businesses out there actually need this kind of security/scalability, compared to how many use cloud services and pay extra cost for something they don't need?
From a critical perspective, your comment made me think about the risks posed by rogue IT personnel, especially at scale in the cloud. For example, Fastmail is a single point of failure as a DoS target, whereas attacking an entire datacenter can impact multiple clients simultaneously. It all comes down to understanding the attack vectors.
Cloud providers are very big targets but have enormous economic incentive to be secure and thus have very large teams of very competent security experts.
That doesn't sum up my comments in the thread. A rogue actor in a datacenter could attack zillions of companies at the same time while rogue actors in a single company only once.
AWS hires the same cretins that inhabit every other IT department, they just usually happen to be more technically capable. That doesn't make them any more or less trustworthy or reliable.
This is false. AWS infrastructure is vastly more secure than almost all company data centers. AWS has a rule that the same person cannot have logical access and physical access to the same storage device. Very few companies have enough IT people to have this rule. The AWS KMS is vastly more secure than what almost all companies are doing. The AWS network is vastly better designed and operated than almost all corporate networks. AWS S3 is more reliable and scalable than anything almost any company could create on their own. To create something even close to it you would need to implement something like MinIO using 3 separate data centers.