Oh, I'm not about to claim that these licensing exams people in other professions take once and then never worry about again (bar exam, medical boards, etc.) compare in difficulty to a typical SWE interview.
But, here me out here:
Suppose the CA bar exam were changed in format to be more like a SWE interview. That is, instead of being a 2 day affair consisting of 5 essays of 60 minutes apiece, a 90-minute performance test[0], and 200 multiple choice questions, let's shrink it down to a format that fits in one day and under 5 hours.
Now, let's nix the performance test right away, because it would be incredibly difficult to shoehorn in any kind of real, practical task in under 90 minutes.
According to [1], it looks like 100 multiple choice questions are allocated 3 hours worth of time. So, basically, our cut down format could be 100 multiple choice and 2-3 essays. So, that's about half the amount of testing that's done currently, more or less.
Now, if you have half the amount of testing available, you have a choice: you can cover half as many areas of law, you can cut down the depth of coverage in each area, or some combination of the two. In any case, what you've done is increase the variability in the test. In other words, passing is now more likely to be influenced by exactly which version of the test you get.
If there's an area of law you're weak in (say, bird law), it might not even be covered. Conversely, if you're a real expert in bird law, you won't get as much of a chance to shine in the new format. That means our new format both allows somewhat more marginal candidates to pass, and gives up some sensitivity in detecting people who are really, really good.
So, in summary, the new format omits any sort of practical task, increases the variability of the test, increases the probability that marginal candidates will get through, and decreases the ability to distinguish truly excellent candidates.
Seems to me that's sounding more and more like a typical day-long SWE interview, isn't it?
But, here me out here:
Suppose the CA bar exam were changed in format to be more like a SWE interview. That is, instead of being a 2 day affair consisting of 5 essays of 60 minutes apiece, a 90-minute performance test[0], and 200 multiple choice questions, let's shrink it down to a format that fits in one day and under 5 hours.
Now, let's nix the performance test right away, because it would be incredibly difficult to shoehorn in any kind of real, practical task in under 90 minutes.
According to [1], it looks like 100 multiple choice questions are allocated 3 hours worth of time. So, basically, our cut down format could be 100 multiple choice and 2-3 essays. So, that's about half the amount of testing that's done currently, more or less.
Now, if you have half the amount of testing available, you have a choice: you can cover half as many areas of law, you can cut down the depth of coverage in each area, or some combination of the two. In any case, what you've done is increase the variability in the test. In other words, passing is now more likely to be influenced by exactly which version of the test you get.
If there's an area of law you're weak in (say, bird law), it might not even be covered. Conversely, if you're a real expert in bird law, you won't get as much of a chance to shine in the new format. That means our new format both allows somewhat more marginal candidates to pass, and gives up some sensitivity in detecting people who are really, really good.
So, in summary, the new format omits any sort of practical task, increases the variability of the test, increases the probability that marginal candidates will get through, and decreases the ability to distinguish truly excellent candidates.
Seems to me that's sounding more and more like a typical day-long SWE interview, isn't it?
---
[0]: Interestingly, this is essentially a work sample test, which is the thing proven to correlate best with job performance: https://en.wikipedia.org/wiki/Performance_test_(bar_exam)
[1]: https://www.tjsl.edu/academics/bar-prep/california-bar-exam