Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Ask HN: How does Apple achieve both secrecy and quality for a release?
99 points by billti on April 18, 2022 | hide | past | favorite | 87 comments
I know there are several counterexamples, but in general I'm impressed that Apple manages to keep new products/features under wraps until a major event, and then the product is in stores a couple weeks later with pretty high quality (user experience, reliability, minimal major bugs, etc.) Can anyone who has first-hand knowledge shed some light on how this is achieved with what I assume is quite limited user testing to contain leaks. Even working on products with multiple public betas and customer feedback sessions, it's hard to hit a high quality bar with a product release.


This is from conversations with friends at Apple over the years. Apple employees: please confirm/complicate!

- Secrecy does actually get in the way. It's not great when you aren't aware of large features or products that will impact your work.

- Apple hires people who have great product design instincts, and even end-users know Apple's foundational design principles. However, lack of feedback from end-users does occasionally bite them (see: aborted Safari redesign)

- Apple takes QA very seriously and invests in high quality QA engineers. QA is not just "a step in the process" or a marginalized outsourced group.

- It's a culture of accountability, not committees. Every product and feature has a DRI (Directly Responsible Individual). When something isn't working out, the first question is "who's the DRI on this?"

- Most of the organization is highly siloed by function. This allows units to focus entirely on their objectives, which can be productive. But it makes cross-functional collaboration rare. In my opinion, this is why certain aspects of Apple's ecosystem feel disjointed, missing, or don't hang together holistically as well as they could.


> It's a culture of accountability, not committees. Every product and feature has a DRI (Directly Responsible Individual). When something isn't working out, the first question is "who's the DRI on this?"

This seems to be in contrast to the tenet of blameless postmortems [1] adopted at Google et al. Does this culture lead to blaming said responsible individual, or is the feedback seen as constructive?

[1] https://sre.google/sre-book/postmortem-culture/


That isn’t quite right. When you are appointed DRI at Apple it means you are responsible for getting that problem solved & you have the authority to move whatever mountains are in your way. I saw more people blamed for not supporting the DRI properly than I ever saw the DRI getting blamed for failing.

You usually don’t get that type of appointment unless leadership is confident you’re the right person to be successful at it anyway.


> This seems to be in contrast to the tenet of blameless postmortems [1] adopted at Google et al.

While teams may vary, in my experience postmortems seek to understand where in the process a breakdown occurred, rather than fault individuals.


Blame and responsibility can be orthogonal concepts.

An SRE may be blameless for a bug but they may also be the DRI for getting an incident resolved.


More about Directly Responsible Individuals (DRI) from gitlab-- It's an interesting organization concept imo. [1]

From the article-- Of course, when things do go wrong, it's also the DRI who (usually) takes the fall as was the case when Scott Forestall, then iOS senior vice president, was forced to resign after he "refused to sign the letter apologizing" for Apple's infamously error-laden Maps app redesign in 2011.

https://about.gitlab.com/handbook/people-group/directly-resp...


Is he the one directly responsible for the Silicon Valley "It's Apple Maps Bad" scene?

https://www.youtube.com/watch?v=tVq1wgIN62E


My employer stopped promoting QA and the best ones all went to Apple.


Makes you wonder if modern product practices and buzzwords (agile, ux, mvp, etc) is all nonsense

Just compare the quality of products and failure rate of the competition...

Ex: https://news.ycombinator.com/item?id=31069434


Some of it is just misapplied. Agile is a continuous improvement process, not an innovation process. If Apple had shipped an "MVP" smartphone in 2005, they never would have iterated their way to the iPhone we know.

But some of those buzzwords are perfectly appropriate for companies operating at a smaller scale. Apple has the ability to spend a huge amount of time and effort up-front to get the product right before it's ever announced. They're perfectly happy to spend 3 years on a product no one has seen.

Smaller companies [believe they] have to generate value faster, so they push out MVPs with the hope they can find product-market fit and iterate their way to success. For better or worse (often worse), that kind of post-launch iteration is just not in Apple's DNA. Look how little the Apple Watch software has changed in 8 years.

I don't think everyone should spend as long "perfecting" a product as Apple does, but I do think most companies could save a huge amount of time and money by doing more up-front research and design, to make sure they're building the right thing.


I don't think it is about research, design or experimentation to get product market fit but technical expertise. Just look at how Apple is organized compared to the competition - technical people lead: https://news.ycombinator.com/item?id=31052647

Proof: The majority of today's winning or great products were started by technical people. Yet the industry feels the need to systematically discover "the next Steve Jobs" with MBAs, PMs, methodologies and buzzwords to justify the success of the next innovations...

Not denying its usefulness, some of these might help but seems like too much buzz


> Apple takes QA very seriously and invests in high quality QA engineers

Willing to bet this is worth 80% of the value to the task, especially if your QA testers can write enough code to have good instincts about why something might break & control for a test environment / create novel test situations.


Could you please give an example what could be better holistically / what other company does it well?


Siri is the biggest example, as a product which crosses SW/HW/services boundaries across nearly every device they make. Apple owns the whole stack down to the silicon, yet Siri is still not where it needs to be. Looking in from the outside, I suspect their organizational structure is standing in the way.


Siri is not where it needs to be because Apple refuses to mine user data to enrich it.

They also are very hesitant to allow researchers to publish their breakthroughs which makes recruitment very hard. Although this is changing

https://machinelearning.apple.com/


Yeah, I hear that. Apple's privacy position makes Siri a bigger challenge, but I don't think that alone explains why Siri is so far behind. They can do a LOT on-device these days.


I have trouble with Siri setting a timer, telling me the weather, or letting me turn on and off individual lights (it used to do this). Data mining might be some of it but I don't see how that is an issue with a timer.


I have set timers with Siri 100's of times. It's practically the only thing I do with it. Hasn't failed me once.

Just yesterday, though, I said "Hey, Siri, navigate home" and that literally got into a failed state. The first time I did it it told me I didn't have a "Home" contact, so I made it. Didn't work when I was in the car yesterday, but it did just now. So, I agree, there are issues with Siri.


And there's also the issue that it doesn't seem to have any concept of priors. Every now and then I give Siri (and Maps) a try, and it's always something like, I ask it to "navigate to [name of local business]" and it starts navigating to a business with the same name ... but 2000 miles away. Even though it does have that business on the local map...


Context is what’s missing. A dictionary of recently referred to entities. Most recent person, most recent place. Also things like current location. Am I home, am I moving, etc


Siri is great for setting timers, making calls, and changing the volume when my hands are busy.

The biggest “WTF” moment was when I was using Siri in Russian, and told it to “lower the volume” (as I’d done hundreds of times in English). It lowered Siri’s volume, but nothing else (and told me as much).

Never did figure out how to adjust my music volume with Siri in Russian.


Sounds like a genuinely very impressive setup.


> When something isn't working out, the first question is "who's the DRI on this?"

Wow. Every company in SF Bay area I worked at stressed that they don't look for who's to blame, but on how to solve the issue.


Responsible Engineers are a thing in hardware engineering.

It’s less of making scapegoat to blame, and rather a person who has been empowered to call the shots on their product, and entrusted with knowing their product or domain inside -out like the back of their hand. If something goes wrong, you ask for the RE/DRI so you can get the person who is supposed to have all the answers, well, answer.


I think there is a clear difference between being responsible for something, and being blamed for something. There may also be overlap. My experience has been the anti blame culture (which is a good thing) has been twisted into a lack of ownership and responsibility in some orgs (which is a bad thing).


In many cases, the first step in "how to solve the issue" is "who is responsible for this?"

It doesn't have to have anything to do with blame.


Yes, as in who knows this component more in-depth than anyone else in the company and is best able to solve any problem with it?


The intent is not (or shouldn't be) to find a scapegoat—it's so others know WHO exactly is going to help resolve the issue.

That said, from what I've heard, culture at Apple varies heavily by group and manager. There are plenty of horror stories!


I kinda took it as more of a find the person immediately, understand the situation and why they made those choices, then find a path forward. It probably can be used nefariously, but I'll give them the benefit of the doubt.


If you want to solve an issue, it makes sense to reach out to the person who is responsible for the domain the issue is in. Nothing in the sentence you quoted implied any amount of blame.


It is relatively simple: strong Non-Disclosure Agreements (NDAs).

One of my previous employers worked with Apple a couple of years ago to ensure the stability and security of Apple Arcade, and every employee, even the janitor, needed to sign an extensive NDA to ensure the secrecy of the project.

We kept “the secret” for six (6) months or so, and at one point we almost lose the contract because someone mentioned the project’s name during a phone call, at home, while a family member was listening, and that person told someone else, and that someone apparently talked about it somewhere and Apple found out.

Fortunately, the CEO managed to, somehow, keep the contract.

Even inside Apple, they have to sign NDAs when you need to talk with a different team about a new product, project, feature, etc. I asked about it during recent interviews (January 2022) to join a team that is supposed to have massive cross-team work with the entire engineering organization.


I read this funny article: https://www.wsj.com/articles/apple-big-a-the-fruit-company-w...


And yet we have had an avalanche of leaks about their car project (and even the fact that it existed, then didn't, and now apparently exists again). This plus all the product leaks makes me think we should be giving the leakers a lot more credit.


> strong Non-Disclosure Agreements (NDAs).

https://www.computerworld.com/article/2468894/cops-raid-home...


The short answer? $2T can buy a lot of secrecy and extensive QA.

The long answer is I don't think either of these statements are necessarily true? Apple products usually leak before their official launch. And as for quality while generally ok... Apple has had some poor launches including: Apple Maps, Safari on Windows, butterfly keyboard mechanisms. And other issues are user unfriendly product design like the mouse that has as charging port on the bottom rendering it unusable. And still more quality issues stem from lack of transparency like Battery Gate.

I personally think Apple has a strong fanbase that overlooks these issues and is quick to forget


> I personally think Apple has a strong fanbase that overlooks these issues and is quick to forget

Apple has a strong core of two users, in my opinion

- people who couldn't give a shit less about the actual software, the UX, or whatever else it provides, they just know that it can take photos, browse the internet, and that Apple products are associated to "premium" and they want to be premium. This trickles down to the younger generation through their favorite movie stars, singers, tiktok influencers all use Iphones, so they want to use Iphones. The "rich" kids at school have Iphones and if they see you have the green bubble instead of the blue Imessage bubble in group chat, you'll get made fun of. (I just had this exact scenario happen with one of my friend's daughter, who took my old Iphone 8+ off my hands to replace their Galaxy)

- users who have grown to realize all consumer tech is essentially trash, and recognize the marginal utility that Apple products uniquely offer: namely the fact that their products have resaleability, have a pretty well connected ecosystem, and at the end of the day, a physical store near you that you can take your product to.

I am in the latter boat, having used PCs my entire life, modding custom setups, rolling my own linux distros, installing the latest de-googled android OS, logitech, the 40 brands of headphones and headsets, and all of that fun (miserable) stuff. You could tell me right now the latest X1 carbon is 994234% faster than my M1 mac, with a graphics card capable of running the latest AAA games, and I wouldn't give a shit. I just don't give a shit anymore. I want something that will work 99% of the time, and let me do my job wherever. Apple products have done that for me much more often than when I relied on whatever other amalgamations of Dells/Samsungs/Lennovos/Blackberrys/etc, whether its getting frustrated with wifi connectivity, not having my passwords or photos sync across devices, or fumbling with bluetooth headsets and devices from god knows how many vendors and companies. Give me an Iphone, a Mac, some Airpods and I will deal with whatever minor inconveniences like having to charge my mouse upside down.


I couldn't agree more. As I've gotten older I care far less about being on the bleeding edge or pushing a system to its limits and more about just being able to sit down and work. I don't have the time or energy to configure a linux DE and keep it updated/working and Windows isn't something I care to ever touch again if I can help it. Mac gives me a unix-y (FreeBSD) environment that feels sane and most of the time it "just works".

I think a lot of people write off the (G)UI/UX of macOS as toy-like but one thing I've also learned is I like looking at pretty things. Sue me. I like my apps to look nice, I like my UI to look nice, and I like when it all feels connected/consistent. Of course there are outliers but far fewer than when I look over the wall at linux/windows. It really comes down to (for me) "where do you want to spend your time?". For myself I want to spend it producing and doing the things I love, not endlessly configuring or fighting my OS, macOS lets me do that. It's not perfect and I won't pretend it is, but it's leaps and bounds ahead of the competition when it comes to things that matter to me.

One quick anecdote, when my sisters went to college my dad got them Macs (after being a windows-only household growing up and him only starting to using Apple products after I did) and he still tells me how happy he was with that decision. He will say "Do you know how many times I got called to do tech support? Zero, not once". He and I both had the "tech support" role in our friends/family groups so we both knew what a windows computer would have resulted in, we had lived it for other relatives in and out of college, heck I rebuilt a ton of friends computers when I was in college due to virus/malware/slow/etc. The people that would buy a new $400-$500 dollar bargain bin laptop every 1-2 years in college when their previous one fell apart yet would make fun of the cost of Apple computers drove me up a wall seeing how I had the same laptop all 4 years and it cost ~$1K (and it felt great to carry and looked good to boot).

Do I believe that you can game better on windows or customize linux to be your personal nirvana? Absolutely, but neither of those things matter to me, I just want to be able to sit down and produce, not be digging in config files to figure out why my graphics card isn't working.


Thank for articulating what has been in my head for years but I couldn't express as well as you. This is how I think a lot Apple users feel. I came from the same background as you and ended up in the same conclusion.

>You could tell me right now the latest X1 carbon is 994234% faster than my M1 mac, with a graphics card capable of running the latest AAA games, and I wouldn't give a shit.

You could have your cake and eat it too as I hear the M1s are pretty competitive with most PC laptops.


It's just not something that can be articulated, I am afraid, because admittedly, there is a degree of subjectivity to it. It's the value of being frustrated and hassled the least amount of possible by low grade consumer tech which is intruding into more aspects of our lives each passing moment. This isn't saying Apple products aren't frustrating, but the combination of using their products together is less frustrating and less hassle than the alternatives.


> - users who have grown to realize all consumer tech is essentially trash, and recognize the marginal utility that Apple products uniquely offer: namely the fact that their products have resaleability, have a pretty well connected ecosystem, and at the end of the day, a physical store near you that you can take your product to.

This is a great point. A couple of my mates have had issues with their laptops and had to talk to the manufacturer, send it across the world at their own cost and wait forever to get it fixed. I've had plenty of issues with my macbook pro and it's not covered by warranty anymore, but the people in the store are happy to fix it for free out of goodwill and the repair is always fast.


> user unfriendly product design like the mouse that has as charging port on the bottom rendering it unusable.

That was a feature according to Apple not a bug.


No it wasn't and it's pretty obvious it wasn't. The Magic Trackpad 2 and Magic Keyboard released at the exact same time as the Magic Mouse 2 yet both actually let you use them wired, not just charging but actually with signals going over USB and not Bluetooth.

If Apple had a serious hard-on for getting rid of cables as is claimed from everyone's desk they would have done the same thing to those devices. It's much more likely they just wanted to recycle the old Magic Mouse chassis, parts and tooling and this was the only way.


The most famous bad launch might be the "you're holding it wrong" antennagate. Which has a clear line to the secrecy.


Fear. They make a big deal about punishing leakers to make people fear what will happen if they leak.

Also control. Here is a story I heard from an iPad launch partner, who got an iPad before it was announced:

The device arrived in a large box with little plungers sticking out to press the buttons and the screen. You couldn't actually see the device inside, so there was no way to know what buttons were on it, what its dimensions or weight were, etc. Furthermore the box had to be kept in a locked room with strict access controls. Anyone who had access to the room had to be under draconian NDA that included severe personal penalties for leaking anything (like enough to financially ruin you). Also everything was watermarked so you couldn't release photos without them knowing exactly where it came from.

The box was taken back and replaced with a standard consumer model after the launch. Most people in the company weren't even aware they were a launch partner until after the announcement.


For those too young to remember (or who may have just missed the news when it happened, since it wasn't mentioned yet in this thread), in 2010 the iPhone 4 was leaked before the release, physically, an employee left it in the bathroom of a bar. Someone found it, tried to return it to Apple, they didn't take it seriously and so this person turned to Gizmodo to make a quick buck: https://www.fastcompany.com/1621516/iphone-4-leak-saga-start...

They haven't had a leak that big ever since, but occasionally things have gotten leaked (specs, components being used, etc) from China who does most of the manufacturing.


My (opinionated) take on how this is possible:

- Apple has practically unlimited money

- The company can afford the best of the planet's talent

- The companies business is ethical and therefore the top layer of talent hesitant to join ad-based companies pick Apple naturally

- The company is highly design driven, which means it's the perfectionists making most decisions (roughly speaking)

All this seems to have a cultural impact that optimizes for quality without much a/b testing.


It's also infamous for relatively low pay and a culture of brutal overworking, and during the jobs years, seemingly a fear of the CEO.


They also do not invest in their workforce the way other Bay Area companies do. For example, they do not really encourage engineers to grow outside hired abilities and make it almost impossible to find time to work on anything except the Radars (tasks) put in front of you. Plenty of managers would proudly say out-loud they would prefer someone go build a new skillset outside of Apple and then come back. (This is jokingly referred to as your "next tour of duty" as many engineers actually do come back.)

They also make it challenging to switch teams internally, you basically compete for the same role an external candidate would, and you interview as if you were an external candidate.

All of that makes it very easy to get burnt out and so either you are someone bought into the "cult" of Apple and couldn't see yourself working anywhere else, or you peace-out once the rose-colored glasses come off.


low pay?


Compared to other FAANG. They will match high offers, but will not go out of their way to set particularly high comps.


I Lolled.

Highly ethical ? They're China's lapdogs (Google left China). Also they're the ones who don't want to pay employees for the time they spend being searched, nor acknowledge when their platform or their customers have a virus.

Design-driven ? They're the ones with keyboards with a 30% failure rate and $500 repair cost. (They're PR and financials driven).

They certainly have unlimited money, and have shown they're not afraid to sue, and bully, or reward compliance.


They market ethics, and like most things they market, the claims are often misleading or exaggerated.

That said, of the companies who market their ethics, Apple seems to actually _do_ more than most.


Google didn’t leave China for any ethical reason.


Their cult graphic designer following in the 90's and 00's and insistence on top spec hardware has helped them enormously because that cult group following is intrinsically linked to the media sector and thats helped then enormously because I've never known of anyone to bad mouth Apple as a result, but Google and Microsoft, thats different. They became the premium product manufacturer in IT.


Apple has extensive indoor private test facilities. Like strip malls, but each shop is a testing ground. Imagine being paid to go hang out at the mall all day. Yea you’ll sign an NDA for that.

In addition to the NDAs, Apple tracks employee data very closely (e.g. iCloud) to mitigate leaks.

The major downside to this approach is the sample size cannot be “internet scale” and it will be biased towards what Apple thinks it knows. That’s why things like Apple Maps and Siri had such poor roll-outs.


> In addition to the NDAs, Apple tracks employee data very closely (e.g. iCloud) to mitigate leaks.

Intriguing! Could you elaborate more on this?


Apple is the biggest company on earth and out of the FAANG companies, they are the most prestigious place to work. Their market cap is on it's way to 10 Trillion. There's very little incentive for people to jeopardize this job and give up their stock in the company.


Definitely not the most prestigious FAANG to work at. Perhaps if you're a hardware designer/engineer? Certainly not for software.


They all make some decent software and some shit software.


What's the most prestigious?


> Apple is the biggest company on earth

Hardly.

It is number six on this list: https://en.wikipedia.org/wiki/List_of_largest_companies_by_r...

And does not even appear on this list: https://en.wikipedia.org/wiki/List_of_largest_employers

> they are the most prestigious place to work

Eh, that doesn't seem to be a universally accepted point of view.


And yet, it's number 1 on this list, which is linked from both of the lists you linked: https://en.wikipedia.org/wiki/List_of_public_corporations_by...


I don't know why you are downvoted. I agree. Personal experience says that Apple is not a great place to work at for PMs. I am curious to hear other folks chime in to round out the discussion.


Revenue is not a measure of success. It’s about how much you keep - profit.


I think GP was talking about market cap, not number of employees.


"Apple is the biggest company on earth and out of the FAANG companies"

A pointless metric given that over the last 5+ years, Google/Apple/Amazon have had and lost that spot back and forth. It's a vanity metric and in terms of the actual business, is an abstraction.

"they are the most prestigious place to work."

By what criteria? If you are talking about by patents and research filed, it would be IBM (which is also the longest running firm at over a century), if its hardware one might be more impressed by SpaceX, if....etc. "Prestige" is another vanity abstraction, and isn't even prone to a metric.

"Their market cap is on it's way to 10 Trillion. There's very little incentive for people to jeopardize this job and give up their stock in the company."

If that were as true as you state, people wouldn't leave for startups at the rate that they do. Startups wouldn't be as common in general if these big 'prestige' firms were such a thing, but the valley only in the last dozen years or so has fallen to that as 'common wisdom' when the actual habits are a lot more mundane -- NDAs and it's a small valley so people who can't keep a secret don't get hired elsewhere. That was the case in the 90s when the big players were HP, IBM, MSFT, Sun, SGI,and folks bounced from firm to firm, it was the case a decade later when folks started moving into what became the dotcom wave........


The only Apple product I use is MacOS so perhaps you have a different perspective, but their operating system releases are of shockingly poor quality. My workplace has a blanket ban on installing the latest OS releases because of all the problems they cause. And of course this ban isn't enforced by group policies because that's not even possible in this ecosystem, so they end up having to deal with the fallout regardless.


"Don't buy/install the first version of any Apple product" has been a universal law since the 90s. The M1 is the only release I can remember that bucks that.


Have you used Windows recently?


Yes. I don't know what that has to do with MacOS quality...?


You can block devices enrolled in a MDM from upgrading to new macOS versions.


I'll never forget the time I walked into Cupertino wearing Google Glass when they were close to releasing the iPhone 6 (big screen one) and I was escorted off the property. Lol


One aspect I haven't seen mentioned in other comments is that secrecy and control is part of the core Apple culture, in a positive way. Out of all the big tech co's, Apple has the strongest brand and marketing. They have built an image on good design and high quality products (this is the perception, even if you dispute the reality). They have carefully built this image over the years by releasing products very deliberately, not focused on the tech or specs inside but instead focused on the product as a whole.

Controlling the narrative _requires_ secrecy during development. Every person who joins Apple today is well aware of the culture and how it has contributed to Apple's success. They have every incentive to maintain that secretive culture to continue the success.


I think that the secrecy at Apple is ridiculous. I interviewed for them and they couldn't even tell me they were working on the launch of a CI/CD product. Like, come on. That's not exactly spaceship tech.


I think part of it is that, sure, leaked iPhone spec's will always get some attention, but it's nowhere near as big a deal as the first few iPhones or other new product launches. News of the next minor evolution of iPhone spec's probably aren't profitable enough to devote resources to fretting them out. I think we're past the point where (for established products) it's as big a deal as when Gizmodo got a hold of a pre release.


Rather than this being a special quality of Apple, I think the marketing strategy of a lot of other B2C brands has now involved """leaks""" to enthusiast circles for a long time, probably since """evangelists""" and """influencers""" became subjects of pop consumer culture --- so I think that Apple leaking not less but the others just leaking intentionally more info ahead of release to fuel the hype is an important factor at play here.

I don't wish to comment on the "and quality" part in your question though.


The leaks generally aren’t intentional: https://www.macrumors.com/2021/07/16/apple-warns-leakers-to-...


They don't do as well with hardware, due to lack of wide testing: https://twitter.com/marcan42/status/1504449580595040258

> Since they want to keep new hardware support a secret, they work on it in parallel on another branch / whatever. So all the betas only include a subset of the changes going into the real release. And then when the -RC drops, they merge major changes, especially around boot/HW.


Interests align - those who keep the secrets are better off for keeping them (not just for punishment of breaking NDAs, but the fruits (pun intended) of revealing something wonderful).


Apple TV+ has a show about this.


https://en.wikipedia.org/wiki/Severance_(TV_series)


Ya, very r/oddlyspecific.


Although they have secrecy, it's nothing compared with the years 1999-2008. After Jobs, the company practically couldn't hold any secret. We know in advance how the next iPhone will be, all the components, even things like mac OS or other hardware is easily known in advance. To me, the beginning of the end it was with the iPhone 4 era, that's when the company started to get careless with her movements and way of operating


Microsoft does not have this culture of secrecy but makes it practically impossible to report issues. I don’t see why Apple would do worse than them.


I don’t think they do. Hardware specs/features are almost always broadly known well before the announcement.


To do that in software is one thing. To pull that off in Cars is quite crazy.


Money




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: