I may be wrong, but I seem to remember a post or comments here about this topic. It was some simple sysctl or /proc settings that tuned the TCP stack to avoid the hug of death.
If you didn't apply it, the TCP stack would be the bottleneck and/or drop connections on you. I remember it being more than the just the 'listen' backlog parameter.
Does anyone have a good succinct reference or tips on this? It is not a problem I will have, but I was curious about this today.