13

Stack Exchange recently started a collaboration project with the University of Melbourne, in an attempt to improve the automatic detection of duplicate questions.

More information on the project and a call for help from the community can be found here: Project Reduplication of Deduplication Has Begun!

In it, Tim Post mentions that Doris Hoogeveen (me) will post a message on the different meta sites to let you know that we're ready for help. This is it!

As a brief summary, the project is about finding ways to automatically determine if two questions are duplicates, and also to detect the opposite: falsely tagged duplicates. To validate our methods, we need manual judgements from people who really know the data. This is where we need your help. It is impossible to manually annotate all the data from Stack Exchange, and so we have applied several filters to make sure we only show you questions pairs with a high likelihood of being duplicates.

Participation is completely optional, and there's no minimum time requirement. If you are interested in participating in the project, please head over to the annotation interface, which can be found here: http://hum.csse.unimelb.edu.au/se-annotate/

You can log in with your Stack Exchange account, so no sign up is required. The site also contains some more information on the project, which might answer some questions you may have. And I'm also happy to answer them here.

Thank you!

17
  • 6
    I like the idea of this project and have tried a few cases on the site. I have a question though: one of the cases was a very general question A and a very specific question B, strictly a subset of A. On this site, it is standard policy that B would be marked as a duplicate of A. However, it doesn't mean that (as phrased in the study) both questions are duplicates! I marked the questions as "related, but not duplicates", because imo that was the only correct answer to the question asked, before realizing what just happened. Which answer would help you most for this kind of situations?
    – T. Verron
    Oct 27, 2016 at 14:08
  • 2
    Hi Doris. Tiny detail: It is TeX. Not TEX and not Tex ;-)
    – Johannes_B
    Oct 30, 2016 at 8:51
  • 1
    May i ask for a definition of a pair of duplicates according to your study? That would be helpful as the different communities handle duplicates differently. And even within the communities opinion vary on what a duplicate question is.
    – Johannes_B
    Oct 30, 2016 at 9:09
  • T. Verron, that's a difficult case, and it's a good example of a situation in which different people may make different annotation choices. That gives us information too though. If some people choose 'duplicate' and some people choose 'related', then it's clearly an edge case that needs to be looked at. I think you've made a good decision though.
    – Monozygotic
    Oct 30, 2016 at 22:11
  • Johannes_B, thanks! I'll fix it in the interface. Duplicate questions are semantically equivalent. They should express the same information need.
    – Monozygotic
    Oct 30, 2016 at 22:12
  • 'Semantically equivalent' and 'express the same information need' are not the same and I'm not sure I understand either in this context. As best I understand the first, no questions would count as duplicates. As best I understand the second, no questions would count as duplicates for different reasons.
    – cfr
    Nov 1, 2016 at 3:45
  • 1
    What information about me will you get if I use my SE login to access your site? I am generally very suspicious of people who want me to give them credentials I use on other sites. Can I choose not to use my SE login?
    – cfr
    Nov 1, 2016 at 3:53
  • I see it says that all you know is what you can get from SE's API using my ID, but I have no idea what that is and I'm presumably giving you access to my account details by giving your server my user id and password to authenticate. So I don't see how you can be getting only my ID. (I'm not saying you aren't being truthful but that I'd want more information before using the site.)
    – cfr
    Nov 1, 2016 at 3:59
  • I used the demo version. I'd like a button labelled 'The first of these questions is almost certainly not a duplicate of any other on the site. Please do not show me pairs including this question again.' Ditto for the second. At least 3 questions came up time and time and time again and not only is the answer always 'No", I can pretty much guarantee you it always will be 'No' regardless of how many pairs you put to me. Other than that, the pairings seemed pretty random, to be honest. I classified one pair as a duplicate, but I doubt I would close it as a duplicate.
    – cfr
    Nov 1, 2016 at 4:10
  • Hi crf, I would like you to use the same reasoning to label two questions as a duplicate as you would on this site. That way the obtained annotations are in line with the existing duplicate labels. The pairs that are shown are far from random. If you want to know how they are selected you can read this paper: people.eng.unimelb.edu.au/tbaldwin/pubs/sigir2016-webqa.pdf
    – Monozygotic
    Nov 1, 2016 at 6:26
  • I'm using the StackExchange API for login exactly for the privacy reasons you mentioned. This way I don't have to do any authentication, and so you don't have to give me any information. StackExchange tells me you are a suitable candidate to annotate question pairs and that's all I need to know. It is also the only way I have to verify that you have enough reputation points to do the annotation. You can read about the API here: api.stackexchange.com
    – Monozygotic
    Nov 1, 2016 at 6:27
  • The number of duplicate question pairs in the set to be annotated varies per site. This gives us insight in the differences between the sites, and how a duplicate detection system needs to behave for different sites. If none of the questions we show you are duplicates, then this is possibly one of the more difficult sites. If I had a perfect way of selecting potential duplicate question pairs, then my research wouldn't be necessary.
    – Monozygotic
    Nov 1, 2016 at 6:30
  • You should tell explicitly that a gold badge or more than 3000 reputation points are needed to contribute.
    – Clément
    Nov 1, 2016 at 14:23
  • Hi Clément, you're right. That information is in Tim Posts's post, but I should've added it to mine too.
    – Monozygotic
    Nov 1, 2016 at 22:48
  • Some technical comments: Every once in a while, the layout is strange when loading new questions: i.stack.imgur.com/kIhZB.png. If I reload the page, it looks OK again. [happens with firefox on mac osx 10.9]. As a creature of habit, it would be nice to get the same syntax colouring than on this site. This would help enhance the code readability.
    – samcarter_is_at_topanswers.xyz
    Nov 11, 2016 at 16:58

3 Answers 3

Reset to default
19

For me, your website does not show enough information to decide if they are duplicates

  1. comments are missing, and often crucial information and clarification is only revealed in comments

  2. answers are missing. I already miss them in the normal review queue for two reasons

    • to check if the other question actually has an answer that solves the current problem. [Sometimes even in the in the comments to answers]
    • I tend to define duplicates based on if the answer solves the current problem. There are cases that an answer solves different questions, which can obviously not be assessed from the question alone.

At least link to the original questions would greatly improve your site!

8
  • 2
    +1 for the bolded sentence alone, though the rest of the answer is definitely upvote-worthy also. :-)
    – Paul Gessler
    Oct 27, 2016 at 16:21
  • @PaulGessler Thanks! And also for correcting my typos!
    – samcarter_is_at_topanswers.xyz
    Oct 27, 2016 at 16:32
  • 2
    "Why are StackExchange URLs in the questions anonymised? If you want to know whether two questions already have a duplicate label on the forum, you can of course copy the title into Google, to find the question on the forums. Ideally though, we'd like you to make your judgement based on the text presented only. A link to a StackExchange URL may give you a clue about how to judge a particular pair. It might introduce a bias, which we'd like to try to avoid." (from their FAQ)
    – T. Verron
    Oct 27, 2016 at 16:41
  • @T.Verron Interesting information - thanks for finding it!
    – samcarter_is_at_topanswers.xyz
    Oct 27, 2016 at 16:55
  • They have a very strict definition of what is a duplicated, they have to be semantically equivalent.
    – Braiam
    Oct 27, 2016 at 18:21
  • 2
    I see someone already answered why StackExchange URLs have been anonymised. Duplicate questions are questions that are semantically equivalent, or, another way of looking at it, is that the information need expressed in them is the same. Of course there will always be some degree of subjectivity in a task like this, but that's alright. It we get multiple votes by different people, then we can work with that.
    – Monozygotic
    Oct 30, 2016 at 22:06
  • 1
    Hi everyone, thanks to everyone who has helped me in this project so far. I have improved the interface based on feedback from you. You can now see tags, the voting has directionality, and you can optionally see the comments and answers to questions. Your previous work has all been saved, so you will not have to start from scratch.
    – Monozygotic
    Nov 10, 2016 at 21:59
  • @Monozygotic Thanks a lot! I find the new features really helpful!
    – samcarter_is_at_topanswers.xyz
    Nov 11, 2016 at 16:49
8

Almost all question here deal about the document of a single user. The documents and needs of the users posting a question are as unique as the different users themselves.

That being said, our duplicate rate is much lower (c.f. Duplicates tolerance is huge in this site) than on other siites.

This is also caused by often multiple (small) details in questions.

I tested the site. Out of 13 possible duplicates, 2 were related, one was a duplicate. Some of the proposed dupicates were surprisingly different. For example displaying CJK versus a problem with xparse and math-mode.

3
  • I wondered what the logic was to some of the suggestions: they did seem to be more-or-less random.
    – Joseph Wright Mod
    Oct 29, 2016 at 17:31
  • This is exactly why we are doing this project! It is very difficult to automatically decide whether two questions are duplicates. We have used some simple methods and heuristics to limit the number of question pairs to show you, and to try and make them more likely to be duplicates, but it is far from perfect. That's what we would like to improve using the annotations we get from you. You can read about the methods of selection in this paper: people.eng.unimelb.edu.au/tbaldwin/pubs/sigir2016-webqa.pdf
    – Monozygotic
    Oct 30, 2016 at 22:09
  • @Monozygotic Thanks for the link, i'll take a closer look at it tomorrow.
    – Johannes_B
    Oct 30, 2016 at 22:15
0

The project is drawing to a close, and I am planning to take down the annotation interface this weekend.

Thank you very much to everyone who helped out, and thank you for all the valuable feedback and suggestions. Hopefully the collected annotations will lead to the development of better duplicate question detection systems, especially for StackExchange.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .