Almost double that, as we didn't hit the tier for the lower $0.05 pricing. We also used some of their more expensive locations for some storage too, like Singapore - the 23TB was spread between different regions.
Yes. AWS is that awesome with the number of different and sophisticated services that they offer. Azure is probably the only other cloud that comes close. DigitalOcean is cheap, unreliable, and the amount of services being offered can be counted on the fingers of my one hand.
I can talk about the issues there are with DO on a daily basis. Once you are past MVP, your business needs a better home.
I would love to hear more about the issues you’ve experienced with DO. We have been using them more and more lately and have had a much better experience there than on AWS, GCP, or Linode.
That's a bad analogy but yes, AWS is a vast portfolio of services to solve your problems. If all you need are some bare-metal servers and nothing else then you really don't need AWS in the first place and are paying a premium for stuff you're not using.
They have a vast portfolio of services to solve resume driven architectures problem. The scale at which you outgrow 2 modern high end load balanced boxes is something 99% of the projects will never achieve. There are legitimate use cases for cloud services but what majority of projects are using them for is not it. Reminds me of service at FB that issues and processes data from roughly several hundred million requests per day runs on a single box and is a regular python app. 99% of startups would implement same thing as some data processing pipeline with kafka, spark , hadoop or whatever else is "absolutely critical" to have to process 1/1000 the size of data.
And comparing AWS to GCP is like comparing Hubble telescope to James Webb telescope :)
Recently I have moved my infra to a more hybrid architecture where all complex services run on major cloud providers while stateless compute services run on a cheap vultr vms. Result is quite nice!
1. Spin up 50 AWS Lightsail instances[0] for parallelism
2. In each instance download from S3 and upload to B2.
S3 to any AWS service in the same region is free[1]. $5 Lightsail instances come with 2TB of data transfers each, so 50 of them can easily handle 23TB. The whole transfer can be done within a few hours so the total computing cost is less than $10 ($5 / 30 * 50 = $8.3). Total data retrieval cost for S3 is ($0.0007 per GB) * 23,000GB = $16.1.
[1] "Transfers between S3 buckets or from Amazon S3 to any service(s) within the same AWS Region are free." according to https://aws.amazon.com/s3/pricing/
LightSail has (or at least had) a hard limit of 20 instances. They also have a soft limit of 2 instances, after which you must request an upgrade to a higher limit. I had to submit a support request explaining my intended usage. It took a week to get approved.
The stated reason for these limits is to avoid unexpectedly large bills. But I suspect that it's also to prevent crazy-ass strategies for getting around bandwidth costs.
Lightsail instances have terrible bandwidth throughput. It has high limits because it's all overbooked low-priority traffic shunted off their network as soon as possible.
Great writeup on data migrations. I was wondering whether you did a comparison for this method vs using AWS snowball[1] to export S3 Data and B2 Fireball[2] to ingest it.
We looked briefly at the snowball and fireball, but wanted to do this as quickly as possible, whilst keeping the process entirely transparent to our users. It was also an excuse for our team to get intimately familiar with the B2 API, since it's not compatible with S3.
If we were to consider another large migration like this, physical media would probably be the way to go.
I evaluated moving 2PB with snowball v. putting in 10g/100g links. The issue with snowball ( i started a company that did what snowball did and shut it down (failed )) and other fedex/RAID solutions is that you have 3 transfers. You think the LAN transfer will be quick - but you're generally rate-limited by the systems more than you are by the bandwidth-delay product. If you're in a high traffic DC area, it's pretty easy to get temp bw or install circuits to carry that. 10g for 2pB is 18 days of transfer - which sounds like a lot - but that's 5 days of transfer on each site, 1 day of setup, and 1 week of shipping. Those numbers aren't real but they're close.
So, snowball works in a lot of area, but like so many AWS products, it works if you adapt to it.
pigz/scp/zstd works extremely fast in line.
In your case you're pulling from S3 to another object store.
I moved ~1PB from one S3 region to another. "Why not use replication," they asked. That only works if it's turned on when you upload the object - another fine-print 'gotcha' in the easy AWS service. Then you get into rate-limits. In 2010 I asked AWS if I could spin up 1000 servers to test something - nope - elastiticy at that level is for the big boys.
Now I work for a large cloud company and we still run into elasticity.
To move the 1PB from one S3 region to another we spun up hundreds of spot instances (oh, we were compressing and glacierizing it too) and built a perl/mysql batch job "s3 get | zstd | s3 put" process and parallelized it. One thing nice about S3 is it pulls the md5 hash - unless multipart, in which case it's the hash of the hash, oh yeah.... So you should split it in advance if you want to verify the hash (more fine print).
Worked great. Good for you for sharing this project, very cool.
"10g for 2pB is 18 days of transfer - which sounds like a lot - but that's 5 days of transfer on each site, 1 day of setup, and 1 week of shipping."
I can confirm this, to some degree.
We have larger customers with 20 or 40 or 80 TB of data to bring into rsync.net and everyone is always very interested in physical delivery, which we offer, but it's always easier to nurse along a 20-30 day transfer than ship JBODs around.
As long as you have a transfer mechanism that can be resumed efficiently (such as zfs send) and you don't have terribly bad bandwidth, we always counsel to just run the very long transfer. It does help that we are inside he.net in one location and two hops away from their core in two other locations and we can just order 10gb circuits on a days notice ... because he.net rocks.
As a physicist I've always found the name "elastic scaling" funny. If it's elastic in the physical sense, it means that the energy required to grow to some size is quadratic (or higher) in the size. The marketing meaning is "easy scaling", but the physical meaning is "really hard scaling".
E.g. compare a soap bubble versus a bubble gum bubble. It's a lot easier to scale up the soap bubble, which is not elastic.
It's a very good observation, and I think it's more than just a funny aside. The word 'elastic' connotes increasing resistance as the cluster grows, but this is a false intuition. From AWS's POV 'resistance' to adding a node is generally small, fixed, and, in general, independent of cluster size. I suspect this is what makes cloud computers in general, and EC2 in particular, such a cash-cow.
Moreover it turns out that elasticity is a very valuable quality of a cluster for most workloads; we want this intuition to be true, that our cluster meets resistance as it grows, in the sense that it will shrink when the workload decreases. This matches our economic intuition, too. We want this so much we have to build another software layer to make this happen - e.g., k8s.
We did a Snowball transfer of 150TB (mostly media files) from our on prem DC. Cost is one thing we really failed to plan on. You're charged per day you have the snowball (in our case 3 for 2 separate DCs).
During the transfer, the AWS sync constantly failed due to random issues which drove up the total time to transfer the files. Something like having a tilde (~) in the filename will totally break the sync. You really need to keep track of where it failed. We were constantly trying to craft additional rules into our sync logic to catch the 'gotchas'.
Another point you alluded to was the Etag/MD5sum that's stored in AWS. Pretty useful if you know how to use it...
It appears to be an S3 clone. Competing via lower price and less micro charges. Highlights compared to Digital Ocean and Wasabi:
DO/Wasabi: minimum $5/month, but great deals compared to AWS/GCE otherwise.
B2: First 10GB storage free (probably not including bandwidth).
For a side project or startup looking for its first storage option, B2 seems compelling. But something important: is it a drop in replacement? Is the api accessible on your platform?
Their API is not S3 compatible. There are a bunch of client libraries on various platforms for it though.
They have 1GB free of outgoing bandwidth per day. If you put them behind Cloudflare then you effectively get your bandwidth for free. So half a cent per GB per month is all you way. Plus your time building out the integration instead of S3.
I'm guessing the payback time on this is fairly long by the time you factor in the cost of moving all the backups. S3 One Zone is pretty competitive against Backblaze.
Cloudflare is betting on most of their customers serving HTML not large uncachable blobs, that "Bandwidth Alliance" will disappear pretty quickly at any sort of scale.
I can't speak for Cloudflare, but we've talked to them pretty extensively about the whole project, and even did a case study about our use-case with the bandwidth alliance, and as we switched cloud providers. Things may change in the future of course, but they very much encouraged what we were doing. https://www.cloudflare.com/case-studies/nodecraft-bandwidth-...
Would you consider open sourcing the micro service you wrote to perform the migration? I could see it being helpful to others interested in migrating from S3 to B2.
This request can be disregarded. I’m going to explore extending s3proxy for the same purpose (migration and backfill of disparate object storage systems through an abstraction layer).
If this is purely backup data, wouldn't glacier be a better fit than either S3 or B2?
Glacier is already cheaper than B2 and has the advantage of storing data redundantly across datacenters. And glacier deep archive is 4 times cheaper than that.
They're "backups" in the sense of storing customer data, but realistically they're more like instance snapshots for various customer game servers. They're created and restored multiple times a day for every customer as they hop between games (accessed very frequently), so Glacier wouldn't be a good fit.
B2 didn't used to be multiple ADs, or at least not multiple ADs over a wider geographic area, like AWS's regions. I'm not sure if that's changed at all. It used to be why B2 was able to be cheaper. You were paying for less durability and availability.
S3 storage is redundant over an entire region. Entire availability zones can go down (extremely improbable) without your data being put at risk.
It, therefore, struck me as odd to see:
> Due to S3 and B2 being at least nearly equally* accessible, reliable, available, as well as many other providers, our primary reason for moving our backups now became pricing.
but then ...
> * Science is hard, blue keys on calculators are tricky, and we don’t have years to study things before doing them
WTF? They're trying to be joking here I hope, but that comes across somewhat "We don't give a crap about our customer's data".
I was trying to interject some humor into a dry process. When I asked a couple peers from my storage background, "how would you do this project" they both said the same thing: 1) form a working group 2) do a study 3) test 4) 90 days later, do an analysis. Epic process.
We care a lot, but that's beyond our scope. This was an all-hands reviewed process, and like machines that are built to be ultimately reliable including: hospital generators, human-rated spacecraft equipment, airplane engines for single-engine remote location flying (PT6), our process was detuned speed-wise specifically, for overall service reliability and minimal impact.
We tested, we analyzed, we saw weird failures (python core dump?) in simple code, and followed up on all of those before proceeding. We agonized.
In the end we were satisfied, because our customers couldn't tell anything changed, and our CSR's got zero new tickets. Not bragging at all, relieved of course, but, we care about the CSR's load and work experience too.
Thanks for checking the article out, and let me know if we can show you more about our service.
I was pleasantly surprised to see that the previous rather crazy retrieval pricing model (price based on the peak retrieval bandwidth you used at any time in the current month) was replaced by a straight price-per-GB model a couple years back. That made my life a lot easier when pricing up Glacier vs B2 for a client.
I understand your main concern for moving was pricing. Developer hours also cost money. It seems like you had to invest much more developer hours than you would have if you gradually moved over the course of one month or so. (probably a week?)
We had to pick something that was "good enough" for compression time/size, as well as easy for our customers to download/view if they wish, on any OS. Zip being supported in every popular operating system, and the average user using Windows being able to right click -> unzip, was the primary reason for choice.
There are of course significantly faster and more efficient compression formats like LZ4 which would be ideal if we were solely using the data internally in managed environments, but we offer these backups as downloads to our users, some of which aren't very technically inclined and still need to be able to access the files easily.
Unfortunately... yes. They have Europe coming "really soon" when we were talking to them which we're excited about, but it's one location in the west coast of the US currently.
We do use Cloudflare, but a lot of the instance backups we store are multiple GBs - Cloudflare doesn't cache those. Not to mention uploads from regions like Singapore can be very slow all the way to the US.
It's not a deal-breaker for us, but we're very much looking forward to when they can support more regions.
https://aws.amazon.com/s3/pricing/
I'd be pretty nervous about hitting the delete button after the data transfer.