Probably never, except maybe during the period when only white male landowners could vote and so the "we" was a much smaller and wealthier group. Voter turnout is pretty high these days though.

https://www.electproject.org/national-1789-present