I agree with the anti-piling on sentiment. If an answer is already below zero then I'll not vote it any further. I'll try to leave a comment explaining why I don't think it's a useful answer. Indeed, if I feel that an answer is an honest attempt to answer the question, then even if it's wrong I'll not vote it down, trusting rather to comments and voting for other answers to differentiate. I tend to reserve my down votes for answers that are clearly pointless and don't even have the defence of being an honest attempt.
Of course, I'm only one person so if I don't vote against something that doesn't stop a few others voting against it. But if I leave a comment saying, "This doesn't answer the question because ..." then I think that people are inclined to vote up the comment rather than vote down the answer.
So to encourage friendly behaviour, I'd encourage everyone to leave comments as much as possible - friendly comments, but to the point. That way, there's a middle path for those that want to register that they don't like the answer, but also don't like the whole "voting down" part of the site.
tex.SX seems like quite a friendly place so far, I'd like to keep it that way. I think that tex is a bit unique in that we don't need any barriers to keep time-wasters out. On the whole, time-wasters won't come here. So I'd like us to be friendly by default and only use the negative aspects of the site if necessary.
On MathOverflow, a common thing is for people to start meta discussions on questions or answers where they feel that they'd be inclined to vote against it but they aren't sure, so we discuss it a bit first and that tends to soften things a little. (That is a part that I feel really does work better with a genuine forum than the SE architecture, but that's a battle for another day.)