Personally I wouldn't let servlets off that easily. One of the basic design choices they made was one request handling function per class (the service method, delegating to doGet/doPost/etc for http requests). As a result, most applications actually needed another framework on top of servlets to do routing based on request URL and mapping on to some model of server side components, like MVC. See for example struts, spring, webwork and a host of other java frameworks that were around at the time Rails appeared on the scene. At the time, Rails' routes file and having classes grouping related controller methods together was a clear demonstration of what the servlet API had got wrong.
Heck, the architects even designed it as a completely generic request/response framework with http as just one possible protocol. When http is your 99.9% use case, that just smacks of over-engineering.
I certainly don't want to be one to defend Java from criticisms of overengineering. But on this point, I can't blame them much.
The Servlet spec was finalized more than 15 years ago, when CGI scripts were the mainstream server-side tech and browsers were circa version 3. The domain "google.com" wasn't registered until 3 months later. I think Servlets were reasonably well done for the time. The web was obviously dominant, but FTP was still widely used, and it certainly wasn't clear that newer protocols wouldn't be important as well. Rails didn't turn up until 8 years later.
One could certainly blame the java folks for stopping there, but there wasn't a lot of advantage to be gained from redoing the foundations in a way that forced everybody to rework all of their existing servlet code. And as you say, the community was making solid progress on their own, so it's probably better that Sun didn't interfere.
Oh, yeah - absolutely. The servlet API was very much of its time, and for several years it seemed to be a good solution. I think it's only in retrospect that it became more clear that there were key challenges that everyone faced in building a web application that the servlet API, and J2EE more broadly, didn't solve. For me that was the thing Rails demonstrated: a full stack solution for building web applications that actually tackled the real problems (like routing requests from URLs to controller methods). And I think the major changes to J2EE since have been in large part a reaction to Rails showing what problems a web framework should actually try to solve.
Heck, the architects even designed it as a completely generic request/response framework with http as just one possible protocol. When http is your 99.9% use case, that just smacks of over-engineering.