Generally, I try to avoid downvoting when there's been an honest attempt to add to/start a discussion, or when I merely disagree with someone.

So the only things I generally downvote are literal spam (this makes up the vast, vast majority of content I downvote, however small that number is) and deliberate misinformation.