I didn’t downvote your comment, but I’ll explain why I don’t agree with it.

We should upvote and downvote based on whether a comment contributes or doesn’t contribute to the discussion. If I disagree with a comment but still find it thought provoking or worth reading, then I’ll probably upvote. If I agree with a comment that adds nothing worthwhile to the discussion then I won’t upvote and might even downvote. If a comment adds no value to the discussion then there’s nothing to be gained by replying to it.