Magnus Carlsen talks a lot about the physical stamina you need to play chess, a purely mental game. Niels Bohr was famously a passionate footballer, who played at club level in Denmark.
Maths/Physics now need a full time career just to reach the current frontier of knowledge, you have to build on existing ground, Athletes must somehow take time to learn this knowledge, and by the time they have learned it, they are not as physically fit as they once were.
Roger Bannister, first person to run under 4 minutes for the mile, was a neurologist. He considered his work in medicine to be more important than running the first sub-4 minute mile.
Just because a person does not use a thing does not mean that he/she is not capable of doing so. Additionally, to quote my favorite math teacher, what one fool can do, so can any other. The issue is our most limited resource: time. I’ve spent a ridiculous amount of time doing things that further no ability. As such, I’m less than mediocre at both sport and and work. Becoming a good athlete requires an insane investment of time into diet, exercise, training, and study of the sport not only modern but past. To claim that an athlete is dumb is to make a prescriptive claim about what knowledge is valuable. The athlete has an intuitive understanding of physics, tactics, and strategy that others do not. The athlete may also have some formal scholarly knowledge on the topic as well. The physicist would not be capable of matching the athlete in use of those same physics with his/her own body without the same training regimen and time use; and should they swap time spent, each would begin to become the other.
Edwin Hubble, while not a Nobel prize winner, was a basketball player, football, baseball player, and pugalist. He also did track and field and waterpolo polo while studying Law at Oxford .
Alternatively, why is not a single successful scientist in great physical shape?