This is the primary reason that you shouldn't make hiring decisions based on the test/problem results alone.
The interview problem should only serve to be a conversation catalyst. If they got some help with the problem, outsourced the problem or simply googled prepackaged solutions to the problem then they are going to have a really hard time talking through the context of how they solved the problem.
What you're testing for here is communication skills and, if possible, a really basic datapoint that may indicate their technical ability. The latter is very subjective and incredibly difficult to test for but the former is arguably a lot more important if they are going to be working as a part of a team.
Totally agree, conversation catalyst is a great way to phrase it too. This is very much the approach I've taken having had very mixed outcomes from trying to do just one thing or the other. I find using a simple, 100% self contained task that should take a few hours and then using that to kick start discussion has provided the best way for us to hire and screen people. We've found using a task that is very simple to get a working solution but provides a lot of scope for edge cases or growth to be effective. It allows the candidate to not blow a whole weekend on something for the sake of it but gives us obvious directions to take the conversation.
Communication skills seem to be held in much lower regard despite them being the key thing for effective day to day work. I can manage / handle working with someone who needs more time than a "rockstar" or hasn't memorized TAOCP if they'are able to communicate effectively. Outside of very specific roles, non-trivial software development is a team effort. It's usually extremely obvious within 15 minutes if a candidate genuinely gets what they've coded or is winging it.
You and GP both bring up a really mind-opening view into this. "Conversation catalyst" indeed, sorry but I'm definitely stealing that one.
> Outside of very specific roles, non-trivial software development is a team effort.
Couldn't agree more, and I have to say I'm rather baffled that so few recruitment processes are fine-tuned for this. Or maybe I should say that the image one easily gets is so. I myself have had pretty well balanced interviews that also tried to assess things like this, but I've always kind of considered myself just lucky.
When one takes your points into consideration, I can definitely see how having the take-home assignments offer value above the usual whiteboarding+bullshit bingo.
The interview problem should only serve to be a conversation catalyst. If they got some help with the problem, outsourced the problem or simply googled prepackaged solutions to the problem then they are going to have a really hard time talking through the context of how they solved the problem.
What you're testing for here is communication skills and, if possible, a really basic datapoint that may indicate their technical ability. The latter is very subjective and incredibly difficult to test for but the former is arguably a lot more important if they are going to be working as a part of a team.