In Germany the app was developed by research institutes. Then canceled. Then SAP picked it up and charged 10 million Euros for what is essentially a few 1000 lines of UI code. The German Telekom got the same amount, just „because“ and will additionally charge 2-3 million per month mostly for the hotline (with limited capacity).
So the total cost will be 50-100 millions. Despite the fact that most technical work was already done by Apple and Google for free.
I‘m not even talking about the fact that my dad got a new phone to run it - sold by T-Mobile (Telekom) of course.
Not sure where you get the 1000 lines from. That's ridiculous. `cloc` counts 16k lines of Kotlin for the Android app and 22k lines of Swift for the iOS app. That does not even include other aspects like the server side code or support documentation. Also it's not only the app that had to be build: It required all the notification infrastructure and coordination to deliver test result to the app. This is really one of the very few big government projects where I can't really complain about the costs. There was only one attempt at getting this done right and in time. They managed both.
You're right that the parent comment is exaggerating, but 20 million for a project of this size is still a bit crazy. The same app cost Austria "only" 2 or 3 million, IIRC.
Plus, I just don't understand all the NIH. Why isn't this kind of stuff being developed in one place and then forked by the respective countries instead of everyone creating their own tiny island?
That said, I don't have any major gripes with how the code turned out, just with the cost.
Yep, am positively surprised how the product turned out. Plus the launch also went well I think, including a comic manual and even an AMA on reddit; I didn't read of any glitches or anything bad either.
I really can’t understand this for a COVID-19 contact tracing app. You should find out how to give feedback on this to make them change availability to global.
It boggles my mind that I have to remind many app publishers that we live in a globalized world and that it’s a natter of a few minutes work to just select the countries to make the app available in. More so for free apps where the publisher doesn’t have to worry about taxes and payments.
Even for apps that are not localized to English or other languages, the publishers should prefer to get them out to as many people as possible in the whole wide world who may be visiting or on a long stay in a specific country.
In this case, the app developer can surely include help text that explains where the app is meant to be used or would be useful (it’s also not like people with a German App Store account living in some other country would be able to use this app effectively).
Many times it has happens due legal restrictions that developer has in place in his org . It is worse when the publisher is a government body or contracted by one
Sadly, no. It's been left to the states, and almost none of the states are planning on using apps[0].
This boggles my mind. Sure, it will take a reasonable effort to get widespread use of these apps. But the impact would be huge. Even low rates of app use have a real chance of lowering the reproduction rate significantly, and reasonably broad usage bring it less than 1.0.
The technology is designed in the right way, where anonymous data is being collected with no app required. People can download the app right after they get a positive test and report themselves positive retroactively. Same on the other side, you can download the app and know how much contact you've had with positive cases retroactively.
I'm no longer surprised the US federal government is doing nothing to push this. But I expected more out of the states.
Where I come from (.au) we paid Boston Consulting $8mil to copy/paste the source code from the Singapore governments open sourced implementation.
The government was then talking it up with "If enough people download the app, we'll let you go to football games again!!!"
Current media reports say none of our local health departments have found _any_ contacts using it that they hadn't already found via manual contact tracing.
(I think 100 apps+backends for $10mil is a bit optimistic, if those backends are required to be HA and support a significant percentage of a countries population. As a single datapoint, where I work has many iOS+Android+backend app projects for gov departments that serve tens to hundreds of thousands of users that cost them high 5 or low 6 figures plus 4 figures a month support/hosting, but if they wanted me to support the 60% of the population (~15mil users) that our CovidSafe app was targeting it'd most likely come to more like mid to high 6 figures, and certainly 5 figures a month to host/support it. So I'd have said maybe 20 apps realistically for that price. Still just as fraudulent a rorting of taxpayer money though...)
I honestly think this will be a net loss financially for SAP and Telekom. To get it done so quickly in parallel to the contact tracing API being just developed will have required a team of their best developers. The lost opportunity cost for this alone will be huge. Then it is not just development, but also QA, community management (git hub issues are in the hundreds), legal, project management, etc. I think the primary reason SAP and Telekom even offered to participated at all was political pressure.
The loss in public reputation if they had failed was not worth even participating.
Also, having worked in a corporate setting would have told you that you can't launch anything for less than 1m EUR/USD, can't launch two things for less than 5m and 3 for less than 20m.
I do work in a corporate setting and have done so, as a consultant, for years. I know the numbers very well, both from what I and others charge and what large companies actually budget.
If you would look at what the app actually has to do to „handle the pandemic“ you‘d find most apps would be more complex.
Same for what you call concurrent users. Those users have no identity at all and essentially need to anonymously download a bunch of static files which are the same for everyone from a bucket periodically. Put a CDN in front. Done.
Infrastructure for the one we have here in .au, which started out as a copy/paste from the .sg opentrace app's git repo, is minuscule.
The only backend touches that apps do is a few login/register APIs when you first launch it, and if you ever test positive and agree to send your data it sends a big json blob to an AWS Lambda nodejs function that pretty much just uploads the json to S3.
I don't recall how complex the login/register stuff is, but I don't remember thinking "That's gonna need heroics to register 10 or 20 million users quickly" when reviewing the code.
Simpler really. No need to get the Facebook SDK, DoubleClick SDK, and Cambridge Analytics plugins set up and configured with your payment details correct and tested...
(And the MSS/GRU/NSA/Mossad will all insert their code into your app bundles without you needing to do anything...)
Nope, it's not 1000 lines of UI code. It is a project with lots of pressure (timeline, politics, ...) and two years of maintenance. Including two apps (iOS and Android) multiple Backend systems and a concept, project management (coordination between government, testing laboratories, ...) looking at the team size (educated guess based on GitHub stats) the hourly rate is on the higher side, but it's still low for a government contract and in case it has effect (which is still not proven) peanuts compared to the financial cost of COVID.
I said a few 1000 and I actually browsed the code on Github when it came out.
There were just six weeks to develop it.
And the concepts where developed before by public research institutes for a tiny fraction of that price (not included).
(If you insist, here is a small calculation. Let‘s say we charge a whopping 500 Euros per hour per engineer. In other words an engineer that costs 1 million per year. And we have 6 weeks time which was all they had. And let’s take 20 people for that. And let‘s add 500.000 for some overhead. Then that‘s still just about 3 million. And I‘ve used insane numbers for a product that trivial.)
First mistake: you calculate 6 weeks of dev time ignoring that SAP got 11.5M including maintenance for 2 years. Certainly amount of work in two years time is lower than initial dev work, but not zero.
Secondly you are only taking in "development work", but such a contract entails other things
- certainly there is a penalty if they deliver too late
- there is legal risk to cover, if they deliver crap in some legal sense they are liable
- probably SAP had other contracts which they had to postpone
- count in all the lawyers writing the contract :-D
And yes, they certainly made a profit out of this and yes a proper biding procedure might be better, but given the timelines and the transparency I think it's ok.
Okay, and the additional 1.5 million should suffice to „maintain“ an app that will not need or get any new features.
Let‘s assume another million for lawyers „writing the contract“. It‘s still far off.
Additionally let‘s remember the 20 million is just for the development. It does not include running the thing. That‘s what the other 50+ million are for.
I don‘t agree. I run a business myself. I charge plenty. I know how much DAX size companies charge as well. The numbers don‘t add up, and this is especially true for something that could have been done pro bono by SAP in the current situation where pretty much everyone tries to do whatever they can.
I can‘t reply to marvion below so I put it here: The total initial cost is 20 million. As I said SAP gets half. These „operation“ costs by Telekom do not actually include any operations. The actual operation (including the hotlines) is estimated at 2-3 millions per month. But it actually depends, it‘s not a flat fee.
Operations for two years are actually not 20 but 40M. But this includes two phone hotlines, thus actual labor cost. (And by then the development costs are close to irrelevant already ...)
> but such a contract entails other things - certainly there is a penalty if they deliver too late - there is legal risk to cover,
... celebratory cocaine for the entire sales team, new yacht bonuses for the entire executive team, another few mil into the CTO's golden handshake fund, and a brown paper bag full of unmarked notes for the gov procurement guy. Poor bastards probably barely broke even on the project - certainly not enough to pay out any bonuses to the devs or project managers...
20M for the boost of Germany's worldwide status is a drop in the ocean. The money was well spent, Germany did not embarrass themselves by penny pinching. It needed to work, and it appears to. Compare this to the UK who is still flapping around.
It doesn‘t work great. I actually run it on my phone. It has silly bugs, despite the simplicity.
Additionally, for that price you could have easily have 3 independent teams develop it and take the best one and still be cheaper and better.
I hope someone will challenge this in court. I don‘t think they can get away without a call for bids which should have been done (across Europe, as the law requires).
There is this contact tracing framework in iOS and Google Play Services and that causes a few errors. Some users get a message like "not possible in your region" and some get some random error codes about rate limits and such errors, which bubble up from those frameworks and can only be fixed there.
Yeah. It‘s the framework which does pretty much all of the interesting technical work, if I may say so. It does the communication with other phones and determines if there was a contact and generates, stores and maintains the tokens and given the server data figures out matches and so on. It has a tiny, simple to use API on top.
It was jointly developed by Apple and Google. For free.
That's what I thought, it seems that the entire project is basically a fancy wrapper around the Exposure Notification apis provided by Apple and Google.
There's some kind of cognitive dissonance around your idea that these companies deserve so much compensation for designing a solution that relies so heavily (entirely) on these features, and yet be free from responsibility for serious bugs that arise from this decision.
I mean, if there was a public tender there's little to complain, if there wasn't then it seems right to, even if it's "a drop in the ocean".
And frankly nobody thinks "wow, Germany has a working COVID app" since almost everyone else has it too. Even Italy, amongst a ton of screw ups, has managed to produce a perfectly fine open source contact tracing app.
Yeah, I don’t get all these people criticizing the cost. Also, when a government is dealing with a flagship corporation of their country, cost is a secondary or tertiary concern anyways. It’s a subsidy in part, and it’s also just that $10m for a nation state government to get it right the first time is just so so so inexpensive. I think SAP let them off easy this time, if anything.
An illegal subsidy. Public contracts require a bidding process, usually across Europe.
It‘s unfair that smaller companies don‘t get a chance.
I reject the idea that Germany should prop up its biggest companies, excluding what really is the backbone of the German economy (which works very differently from the US one) and also excluding our partners across all of Europe.
I think the UK's decision to switch to Google and Apple's framework was actually the right decision, although it's all rather moot and a waste of money as I can't see any contact tracing app work if nobody downloads it (see France).
If one of these apps leads to avoidance of a handful of serious cases (by identifying clusters earlier and thus reducing spread) it has paid for itself. An ICU bed costs _at least_ 2000 euro a day (that's for a normal one, ongoing operational costs only; rush-job new ICU capacity is presumably more expensive); it doesn't take too many averted hospitalisations to pay off one of these apps.
Obviously, it would have been better if the German app has been cheaper. But rush jobs are expensive, and the cost involved is pretty minimal relative to the cost of managing the pandemic writ large. If it's even slightly effective, it was worth it.
I think a lot of people could have made a good working app for a lot less. But on the other side: if SAP and Telekom had failed no one would have blamed the government.
And if the both had failed their reputation would have suffered a lot. Well it's not the best as it is, so well.
They have a top team and delivered on time. Thats rare in software development.
And 20 million are nothing compared to the tax break which costs about 3 trillion.
For the government it's money well spend in two german flagship companies.
My guess most of it was insurance money in case something went wrong, so these companies would have taken the fall.
Don't bother, HN users LOVE expensive government IT projects because it artificially inflates their own self worth. Remember those 20 lines of python you wrote before lunch was really hard and worth millions. You are a ROCKSTAR!
Hey developers of German Corona Warn-App. Just wanted to say even I have installed it, not because I'm convinced it'll help too much (the verdict is still open I guess) but because it's a good project result given the constraints, and there's no doubt an effort was made to deliver the best possible privacy, so I didn't want to be the one boycotting it or something.
The total cost of the app (over a 2-year timeline) is about the same as 1-2 days of Covid testing in Germany. If it helps to reduce the spread of the pandemic it will be absolutely worth it.
I think what's worse is that I suspect the whole project won't be all that useful, and a drop in the bucket compared to society-wide (enforced) change in behavior.
From everything I've read, this kind of app 1) is only really useful if enough people actively use it, and 2) have serious issues with false positives because of inherent positional inaccuracy (or inability to be accurate without sucking battery).
In regards to 1, in my country at least, unless using the app would be mandatory, I know many more people who will refuse to use it or be too lazy to use it. And I'm pretty sure hell will freeze over before the government can make usage (and owning a smartphone) mandatory.
So, false positives may not matter in some conditions. Ireland is currently reporting anything from 5 to 30 cases per day, doing anything from 3000 to 10000 tests per day, and has capacity for at least 15000 tests per day. If the app doubles the number of tests being done and finds a few new cases, that's fine; it's still a lot better than NOT finding those cases.
These apps would, granted, be far less useful in places with major ongoing outbreaks and inadequate testing infrastructure; lockdowns should be used to bring numbers down to the level where this sort of thing is workable.
(1) is probably a greater problem. Ireland has about 30% uptake, which probably isn't enough.
Yeah, after I wrote my comment I figured that perhaps I overstated how useless the app would be. I imagine even with relatively low uptake it'll have /some/ effect. Even a little would help, I suppose.
Relative to other COVID-related economic damage this is peanuts. It's acceptable to throw a few tens of millions at a wall and hope that something sticks.
> The German government says its app cost 20 million euros ($22.7 million) to develop and will require 2.5 million to 3.5 million euros per month to operate. It’s available in German and English, with Turkish and other languages to follow.
I cannot speak for Germany, but I have a bit of experience in how public health projects are funded in the US. Usually, the set of incentives are misaligned for maintaining products.
Research institutes are typically funded through contract grants, where you as a principle investigator are supposed to be using the money for a research project. Obviously there are exceptions, but if you want to make a large name for yourself in academia, doing maintenance projects isn't the way to get yourself promoted; all the flashy rewards are in publishing novel cutting edge ideas in high ranking journal.
I am not an expert on how things are in Germany, but if they're anything similar to the US experience, I'm not surprised at all that they would fail due to the misaligned incentives.
Because it was a mess. A privacy mess. Also because the fact that supporting iOS well requires using the Apple framework was ignored. And also lot‘s of politics.
So the total cost will be 50-100 millions. Despite the fact that most technical work was already done by Apple and Google for free.
I‘m not even talking about the fact that my dad got a new phone to run it - sold by T-Mobile (Telekom) of course.
And hardly anyone complained.