I know that we should not downvote all bad questions/answers. So in what case should we downvote those questions/answers?
First separate the question into two categories: Posts on Meta and posts on the Main site.
Voting on meta doesn't influence your reputation and therefore people might be more willing to downvote, especially questions. Regardless, voting on meta is considered different from voting on the main site since the posts deal with content on the main site. This includes things like a feature-request, a discussion or support about certain functionality. Voting is often used as a form of support for/against a proposal. The proposal might be well-written and show a lot of research effort, but you're against it.
Voting (including accepting) is the main way to rank content on the site. Posts with higher scores are considered more useful than others. So, have that in mind when voting. You have 3 choices: Upvote, Downvote, No vote. Downvote if the content is bad and doesn't benefit the community; don't feel bad about it, it's your choice. Upvote if you feel the opposite. Abstain if you're indifferent or otherwise.
The current spread of votes from the top 10 voters on the main site points to 0.4% downvotes (1,510) compared to 99.6% upvotes (356,448), or one downvote for every 200 upvotes.
Downvoting is associated with reputation loss (there used to be reputation loss when downvoting questions as well). So people tend to (should!) think twice about why.
As far as I know, this is completely up to you. Of course, the understanding is that you might downvote a question if
"This question does not show any research effort; it is unclear or not useful".
and an answer if
"This answer is not useful."
In case of a question, the first part on the research effort can be judged rather objectively. However, whether or not a question or answer is useful is very subjective IMHO. I guess the idea is that, since there are many users, on average useful questions will receive a higher score than not so useful ones, whereby "useful" is an opinion that emerges after averaging over many users.
Another question is what this might be good for. Probably the idea is that highly voted posts are more visible, and a highly voted answer may be the one which is endorsed by more users than not so highly voted answers. In practice, of course, this model turns out to be far from perfect. For instance, posts with a rather popular tag
such as get read by more users, and may thus receive more votes than posts under a more "exotic" tag. (According to this analysis
tikz-pgf is not exceptionally strong in drawing votes. I wrote these lines before I knew the data, and now I feel I need to avoid singling out
tikz-pgf here. The question why so many users feel that
tikz-pgf is particularly good at attracting votes is very interesting, but I do not think I can answer it. In any case the data seem to indicate that the votes depend on the tag. So the score of a post has, at best, a relative meaning.) Yet it is probably hard to come up with a better scheme.
I personally downvote very rarely, and if I do, it is in cases of severe academic dishonesty, i.e. someone trying to advertize the post of another user as their own.
I also agree with @moewe's comment below. In particular, downvotes should IMHO be cast with care.
My main message is that it is up to you.
Bottom-line: vote as you think is right. If everyone does that, an average opinion may emerge that could be somewhat useful, at least on a tag by tag basis.