The problem you described has a very simple solution. Grant more permits to increase the number of brand new housing units added to the market each year. The rate at which SF adds housing units given demand is absurd. In fact, you'd probably have many more small business opportunities in SF if the city were willing to add residential units faster.
Which is why the financial district was wiped off the map during the 1989 earthquake, whereas lower units such as those in the marina survived without problems?
The Marina was wiped off the map in the 89 earthquake. It's built entirely on landfill and suffers from terrible liquefaction. The buildings there suffered bad structural damage and many people who lived in the Marina at the time moved elsewhere because of all the damage, making room for all the yuppies that took over that neighborhood.
So long as the foundation is bedrock, there is nothing preventing the building of highrises in San Francisco. We have the technology.
There is actually tons of demand in silicon valley but development there is mostly illegal. Mountain View recently rejected Google's request for mixed use zoning in the Googleplex which IMO is just gratuitous nimbyism (it's not even in anyone's back yard!). This is probably related to so many silicon valley types living in SF instead of anywhere near their work.
