Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Tangential, where can I learn more about shadow-traffic testing? Books, blogs, tools etc.


Cindy Sridharan often writes well about these topics.

https://copyconstruct.medium.com/testing-in-production-the-s...

For this one I'd start about half way down, under the heading

"Shadowing (also known as Dark Traffic Testing or Mirroring)"

Unfortunately the terminology is a bit fragmented - shadowing, mirroring, teeing, dark traffic (ick), ad nauseam.

Whatever you call it, it's often pretty high overhead to add the infrastructure, unless you're already using some sort of "service mesh" (Envoy/Istio/Caddy/whatever) that supports it. Even then, if you're dealing specifically with a storage system then there can be some thorny issues - idempotent vs. non-idempotent vs. destructive requests, requests which require other objects (files/objects or directories/buckets) to exist or be in specific states, etc. I'm not going to pretend it's easy.

If you can do it, though, it can be an incredibly valuable tool. There ain't nothing like the real traffic, baby. ;) My favorite feature, which I alluded to earlier, is that you can shadow traffic from a larger production cluster onto a smaller shadow cluster and give it a serious stress test. All sorts of bugs tend to fall out that way. The one thing you can't really catch, even with a good shadow, is interactions with other services - including things like permissions or quotas. But if those are the only things you have to shake out in true production, you're doing well.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: