20

The following is taken from What are the review queues, and how do they work?:

The review queues (also known as review tasks) contain posts that possibly need community attention, as determined by the system or other community users. You are shown these posts, one at a time, and you "review" them. Like flagging, there are badges for this.

So, what constitutes a "review"? It depends on the queue. The queues have one consistent option, though — the "skip" option. This permanently skips the post (you won't see it in the same review queue again) and is useful if you're not sure what to do.

Each user has their own copy of the queue — if you review a post in a queue, it is not necessarily removed from that queue for other users. It gets removed for all users after a few more reviews by other users (the exact amount depends upon the queue and review action done).

Badge-hunters may be prone to falsely review posts in these queues without really considering their content. This despite there being mention of the "skip" button "if you're not sure what to do". Reviewing without actually reviewing is called robo-reviewing.

StackOverflow attempts to pro-actively perturb robo-reviewers/-reviewing by so-called review audits - tests to see whether reviewers are actually invested in the process. However, this is not active on TeX - LaTeX.

How should the community deal with robo-reviewers?

13
  • 4
    The review audit is active on Math.SE; I believe it's good, although some times I completely disagree with the “Stop and Listen” message: I found some of those audits worthy of closing or not closing, while other members had already had different opinions. It's normal in a community. On the other hand, some of the audits are really trivial, especially in the Suggested Edits queue.
    – egreg
    Commented Jul 23, 2014 at 22:19
  • @egreg: Agreed. There has been some mention of the possibility to "flag a review" (either in addition to/as an alternative to review audits).
    – Werner Mod
    Commented Jul 23, 2014 at 22:22
  • I'm sorry to admit, but this is a robo-review.
    – Werner Mod
    Commented Jul 24, 2014 at 3:20
  • @Werner: Unfortunately, the link to the review leads to a question, which has been deleted (flag for being a of low quality?) in the meantime.
    – user31729
    Commented Jul 24, 2014 at 6:26
  • @ChristianHupfer It was obvious spam. :(
    – egreg
    Commented Jul 24, 2014 at 7:09
  • @egreg: Most probably it was (obvious) spam, but I can't tell, because I didn't see the post at all.
    – user31729
    Commented Jul 24, 2014 at 12:17
  • @ChristianHupfer This is the first sentence: “Raising is of course the Bred 11s RICE method.” The rest you can imagine, as there are three or four attempts a week like this (today two) and you probably have been able to see one.
    – egreg
    Commented Jul 24, 2014 at 12:22
  • 1
    @egreg: Let me guess: "ling"? I have flagged about 20 spams of that sort already in the last months, so I am familar with that posts. So "No Action needed" was a tiny bit too relaxed ;-) Thanks for providing information.
    – user31729
    Commented Jul 24, 2014 at 12:31
  • 3
    @ChristianHupfer: Here is a picture of the review. My points are: (1) it was obvious spam; (2) it was reviewed before the post was deleted; (3) similar behaviour is found for other posts (which may not be spam), pointing towards more than just a fluke occurrence.
    – Werner Mod
    Commented Jul 24, 2014 at 14:26
  • 2
    I don't like the sound of the automated screening even though it seems to me that the system doesn't always work very well without it. I would probably be even less likely to review if I expected to undergo random tests with no appeal!
    – cfr
    Commented Jul 25, 2014 at 2:34
  • Again: tex.stackexchange.com/review/first-posts/48476
    – egreg
    Commented Jul 25, 2014 at 8:58
  • @egreg Yes. I know. And if the only options are (1) status quo or (2) the automated screening thing, then I agree that (2) is probably the better option. But that doesn't mean I have to like it. In particular, thinking (2) > (1) is perfectly consistent with thinking that both (1) and (2) are pretty yucky.
    – cfr
    Commented Jul 25, 2014 at 21:04
  • 1
    Discussed elsewhere as well: meta.stackexchange.com/search?q=robo+reviewers
    – Jason C
    Commented Aug 4, 2014 at 21:14

5 Answers 5

Reset to default
11

While only some of the larger sites have automated review checking, the moderators are in a position to suspend review privileges for users if necessary. Thus if you believe there is a pattern of behaviour to be addressed, raising a meta question about it may be appropriate. Ideally, such a question won't 'out' an individual but will rather highlight for example questions/answers where the mods should investigate the review history. We have access to some data about reviews that is not public, and can look over this before making a judgement and if necessary implementing a ban. I hope it's clear that an answer to such a meta question will be of the form 'We will take a look' but not much else! As such, it may also be appropriate to ping one of the mods in the chat room, again being careful not to give details of the user of concern.

8

I don't like the automated version of screening every detail on the site, though I see the need for it clearly. But

We should be able to discuss this issue with robo-reviewers instead of ignoring the individuals and handling them super indirectly with a major code implementation.

We had this discussion mostly with Papiro and others but they don't care. I don't think any implementation would fix anything unless they change their own behavior.

My major dislike about robo-reviewing is that whenever they form a symbiosis with a mass-editor who doesn't have enough points, it's very boring to see edits are coming in, reviewed and accepted before you even see the question and front page is filled with 3 years old questions with a comma or a letter changed in the name of an edit.

I would love to transfer my rep points(which again are useless) to them if they wish to stop their nonsense. Other than this annoyance I'm perfectly fine with any badge-craving behavior of any user though I still think they are stupid colored dots.

3
  • 1
    Discussing this issue with robo-reviewers could either be done by the community, or by moderators. There exists no formal way as of now, other than finding a post that the users has had interaction with and commenting (yes, you can invite them to chat, but if they don't have an account for that, it's not possible). Moderators have a super-ping (I don't know how else to describe it), which could be used to send a message to a user's inbox directly...
    – Werner Mod
    Commented Jul 24, 2014 at 19:01
  • 5
    I've sometimes tried to actually review one of these but it has disappeared before I've finished reading and I realised that people must either be extremely fast readers or non-readers to be removing things from the queue that quickly. I tend to look at the queues now only if extremely bored and seriously trying to procrastinate to avoid something I'm meant to be doing! Sometimes it is OK but more often than not, things disappear before I do anything with them so I don't see much point.
    – cfr
    Commented Jul 25, 2014 at 2:22
  • 4
    Let's allow the mods to review-kick or review-ban someone.
    – yo'
    Commented Jul 27, 2014 at 15:54
5

1. Ignore

I will suggest just to ignore such childish behaviour. Any harm the robo-reviewers may do is negligible and do not hurt anybody.

Actually, I earned two badges one of the last days for something I did not know exist. First, I accepted an minimal edit somebody had done (obviously I reviewed the question according to the system). Thereafter, I had to make some silly edits in two answers I had downvoted by mistake, just to be able to correct my mistakes.

2. Remove badges for reviewing

EDIT: After seeing the comments, I suggest removing the badges you can earn for reviewing answers. Serious users do not care for badges and will continue to review without them. Robo-viewers will not robo-view without badges, because the badges are the reason for robo-viewing. Sounds like a win-win for me.

Observation: I have obviously stepped on some badge-collector's toes, since I have deserved three downvotes for this answer.

13
  • 3
    I'm not sure that this is the best approach. The review queues are there for a reason, and robo-reviewing defeats the purpose in two different ways. First, the users that are robo-reviewing are not using the queues for their intended purpose, and second, as @cfr noted users who want to actually use the queues for their intended purpose are less interested in doing so because of robo-reviewers.
    – Adam Liter
    Commented Jul 28, 2014 at 18:16
  • 2
    -1. The intention of the queues is to improve the site. If a new user need to be welcomed, maybe directed the right way and his post edited, you want the post to appear in the First Post queue and be treated appropriately, not skipped with No Action Needed. Similarly, if a vandalism appears, you don't want thhe suggested edit to be aproved. And so on for all the queues. If someone misuse them repeatedly and heavily, it is harmful.
    – yo'
    Commented Jul 28, 2014 at 23:18
  • @tohecz But you can be assured that such vandals are very pleased if they see that their behaviour annoy people and that somebody use a lot effort to fight against them. Just ignore them, you cannot beat them. Then they will go away. But of course, if you want to spend energy on this, just continue.
    – Sveinung
    Commented Jul 29, 2014 at 8:53
  • @tohecz And I totally disagree that the behaviour is harmful. Who do they harm? They get more reputation that they deserve, but so what. On the other hand, downvoting a serious answers just because you disagree, that is harmful behaviour, in my opinion.
    – Sveinung
    Commented Jul 29, 2014 at 20:55
  • 3
    @Sveinung They do harm: (1) everyone who wants to use a solution if the post is vandalized, (2) everyone who reads the first page if tons of posts are meaninglessly bumped, (3) every new user if his first post needs some work which is not done. Should I continue? I honestly don't give a poo about the 2 rep points or some funny badges the robo-reviewers or whoever receive, I just care about the quality of the site.
    – yo'
    Commented Jul 30, 2014 at 7:38
  • @tohecz And as I said, if you want to spend energy on robo-viewers, just continue use your energy on that. Since robo-viewers are after badgets and reputation, why not disable the badgets and reputation for reviewing? And still I is of the opinion that you should accept that somebody has a different opinion than you.
    – Sveinung
    Commented Jul 30, 2014 at 8:03
  • 1
    @Sveinung Notice, please, that it's not the robo-reviewers I spend time with, it's the harmful things that happen which concern me. And I accept your opinion, I just quite strongly disagree.
    – yo'
    Commented Jul 30, 2014 at 8:28
  • @tohecz Well, how typical. I think both of you have a strong point which is we ignored up to this point the weirdness and it was exactly what you wrote, childish. Then as tom mentioned this started to creep up on us. So the childish thing is reaching its adult phase and becoming annoying little by little.
    – percusse
    Commented Jul 30, 2014 at 21:14
  • 2
    @Sveinung: I would also say that they robo-reviewers do some harm: Regardlessly passing every post (question/answer) without editing/commenting/retagging/flagging when necessary. Of course, anybody of us just clicked too fast, me too, perhaps, because the post was not read completely. On the other hand, there are users here just keen on the gold badges, they are already beyond the 2 reputation credits for each edits, so they can't get reputation any longer, just badges. And there are 18 badges to earn for them ;-) (Well, it would be difficult for Late answers;-))
    – user31729
    Commented Jul 30, 2014 at 22:05
  • @ChristianHupfer But then a better solution is to remove the badgets for reviewing. Serious users do not care of badges and will review without them. Robo-viewers will not robo-view without badgets. Sound like a win-win for me.
    – Sveinung
    Commented Jul 31, 2014 at 21:33
  • @tohecz From your arguments, I will conclude that the harm the robo-viewers do to the system is negligently, which actually further convince me that the correct response is just to ignore them.
    – Sveinung
    Commented Jul 31, 2014 at 21:56
  • 2
    Just a detail before it turns sour, On meta, downvotes represent agree/disagree (though we also tend to not downvote here out of habit) so they might not be even users who can review.
    – percusse
    Commented Aug 3, 2014 at 9:52
  • 4
    @percusse: Off-topic comment by me: I don't use a downvote here on meta as a statement of disagreement as well as on the main site. In my point of view, posts should be only downvoted why they they are offensive or of low quality. Disagreement can be stated in a comment or a follow-up-answer to someone else's answer and keeping the voting neutral.
    – user31729
    Commented Aug 4, 2014 at 7:32
3

I'll just briefly say some ideas I have concerning the improvement of the review quality:

  • Comments on reviews: Each suggested edit would have a comment thread, allowing the reviews to be commented. These comments would be received by the reviewers and possibly the editor (but I'm not sure whether that would make a good idea).

  • Review flags: This has been discussed before.

  • Review agreement/disagreement: The ability to disagree with a review result; such suggested edit would make it back into the queue for more users to state their opinion.

  • The ability to somehow address the reviewer directly: this would be likely similar to comments on reviews. I would as well welcome the ability to address the editor directly.

Most of these work only for the Suggested Edits queue. The two other problematic queues (First Posts and Low Quality Posts) would use some completely different approach I think.

Disclaimer: These are very rough ideas. My intention is to open a discussion over the possibilities, not to give definite answers. As well, it's almost 1am just now...

3
  • 4
    i've experienced a different wish -- to be able to "pend" looking at a questioned item, since i don't have enough information to make a reasoned judgment, and want to come back later ... but i want to see the rest of the reviews in the queue. "skip" takes it out of the queue forever. maybe i should make this a "feature request"?
    – barbara beeton
    Commented Jul 24, 2014 at 13:02
  • @barbarabeeton: There is a way to complete a review after it has been skipped: You can either remember the URL (copy it and paste it later), or view the review queue history, find the review you're interested in and click on the review choice by another user (that takes you to the same URL). This should work for skipped entries where a review is still required, but not for those where sufficient reviews already exist. Well, you'll be able to see it, but not review it.
    – Werner Mod
    Commented Jul 24, 2014 at 14:31
  • @Werner -- thanks. i did enter a new question, but this seems like a reasonable approach. (i'm sure i've tried that before, but didn't get the result i was hoping for, so forgot about it.) you might want to add it as an answer to my new question, and if nothing more enticing turns up in a few days, i'll accept it.
    – barbara beeton
    Commented Jul 24, 2014 at 14:40
3

What I'll propose is a little bit different of the review tests as they exist now. What I've noticed is that the post is cleared from the review queue when it's seen one time.

What I can be done it's to modify the system and let the post on the review for 2 or 3 review, or for X minutes AFTER the first review. In this case if robo-reviewers make a different choice it's not count as a review to earn badges.

The major problem from this idea is that the review queue will be bigger and so the possibility to earn badges too.

6
  • 1
    +1 I like the idea, it probably needs some work, but the idea is good. Note that the size of the queue is very rarely more than 3 on our site, so it wouldn't be a problem. As well, it shouldn't be a problem on bigger sites, since the post simply disappears from the queue after some time.
    – yo'
    Commented Jul 30, 2014 at 7:40
  • @tohecz : the queue size is not really a problem in my solution. The problem is more than the amount a review will be bigger and so the amount of review badges too. Also if this is implement another choice in the review task could be: "nothing to add to previous review" only avaible a review is already done on this post.
    – Romain Picot
    Commented Jul 30, 2014 at 9:46
  • @tohecz I thought part of the problem was that groups of robo-reviewers tend to all vote the same (incorrect) way e.g. 'no action needed' on obvious spam or edits which should be rejected. Aren't they already reviewed by multiple reviewers? Or is the idea that the same robo reviewer will see a review item 2 or 3 times and so may make a different choice? I guess this might be unlikely if such reviewers have preferred default choices ('no action needed' or whatever). And, unfortunately, I'd assume they mostly do...
    – cfr
    Commented Aug 6, 2014 at 1:03
  • 1
    @cfr The idea is that the item is removed from the queue only after some time (let's say 15 minutes), no matter how many people review it. Therefore the true reviewers who don't click F5 every 3 seconds catch the item as well and can vote in another way. The problem is that you don't want e.g. a suggested edit to wait 15 minutes for accepting or rejecting, so there's much more to think about.
    – yo'
    Commented Aug 6, 2014 at 7:39
  • @tohecz : waiting 15 min for a suggested edit is stupid yes. So time should be adapt depending on the review queue. When I look the item present in the review queue I think is the only one which can cause some problems in fact. The other one can stay this time. So for suggested edit you can said that X people have to review it or something else. Due to the rep level needed, not many robo-reviewers should be present.
    – Romain Picot
    Commented Aug 6, 2014 at 7:45
  • @tohecz Thanks. That makes sense. So there are multiple suggestions in this answer. In that case, I think the X minutes one is the interesting one...
    – cfr
    Commented Aug 6, 2014 at 11:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .