Community effort in fixing the double backslashes issue

Question

First things first, the background for this meta thread is the following issue reported by Enrico Gregorio (egreg) at the beginning of the year (January 11, 2017):

Double backslashes disappear from code

Since then, a lot of users (from high ranked to newbies) have devoted a significant part of their online times to manually fix the codes in thousands and thousands of questions and answers. At the risk of forgetting someone (please, forgive me if I do so), we the TeX.sx people would like to thank

• David Carlisle
• barbara beeton
• Moriambar
• karlkoeller
• Troy
• samcarter
• ShreevatsaR
• [add more names here]

and many other heroes that want to make good, correct, compilable code available to the entire community for the greater good.

Thank you very much!

A warm note to the Powers that Be: please, note the community effort to fix (or at least mitigate the effects of) the reported issue, which harmed a great deal of TeX knowledge gathered here. The answer took more than three months to arrive; in the meantime, we were desperately waiting for a feedback. I kindly ask to look at us more closely from now on, please. :)

Answer 1

A list of (probably) all posts that show this problem is returned by the following query

https://data.stackexchange.com/tex/query/695628/posts-probably-affected-by

Which returns 11554 postings, this shows all the pre-2014 postings on the site whose initial unedited markdown has a single backslash followed by four spaces, or spaces and a tab. This is a (very) small over-estimate of the posts corrupted. A few tens of posts had ASCII-art diagrams involving \␣␣␣␣ and a few less than that had \␣␣␣␣ in actual tex code.

So fixing this problem meant applying a regular expression replace of

 "\([^\\]\)\\\(␣␣␣␣\|␣*\t\)"

to

"\1\\\\\n␣␣␣␣"

This could have been applied in about a couple of minutes by anyone with write access to the back end.

---

---

Initially a different search was used in SEDE (and similar searches using code: in the live site web search)

http://data.stackexchange.com/tex/query/685412/bug-no-filter

which currently returns no results.

This shows all posts prior to 2014 with HTML with a single backslash followed by a tab or four spaces (tabs are are normalised to spaces in the SQL data dump HTML)

However in March of this year, after several posts had already been fixed before this data query was produced, this was showing 7322 posts, as noted in a comment under m0sa's answer.

As will be explained later, this missed over 4000 corrupted posts, but this was not realised at the time.

A purely automatic method of fixing these posts was complicated by the fact that there were some (less than a hundred) posts with legitimate use for backslash-four spaces. Typically "ASCII art" renditions of trees. For postings prior to 2014 as returned by the above query these have all been changed to use a form not returned by the query (using non-breaking space for ASCII Art or just formatting the tex code not to have four spaces after a single \) This was done to prevent the same postings re-appearing and requiring human intervention each time anyone tried to search remaining bug instances.

As Stackexchange failed to provide any tools to address this issue, the bulk of the questions were addressed by running variants of the above query, which provided links directly to the edit page of the affected post. Then copying the text to a buffer in an emacs text editor which had the required replacement function already defined. This function fixed the text and put the replacement text back in the operating system clipboard in a few keystrokes, which could then be pasted back into the browser edit window and saved.

The emacs lisp used query-replace so that the edits could be checked for false positive cases as described above, these had to be handled by hand, but for the bulk of cases a posting could be fixed in around 10 seconds. For ~7500 postings that is still 21 hours, and actual elapsed time was certainly greater than this as people slow down, or the site flagged issues with images in edited posts that required manual intervention, or any other reason.

The data-query and emacs based workflow described above was used by Moriambar and myself, Barbara Beeton who was the other person to do bulk edits used a variant using site search directly on the live site (apart from anything else having the alternative search helped fix some issues in our data queries). Many other people helped out by fixing smaller numbers of queries, in particular fixing their own posts that had been affected.

---

---

After the first version of this answer was published it was realised that the search on the live site or the data-explorer post table is checking the pre-rendered HTML, The SQL was modified to instead check the last edited entry in the posthistory table which records the underlying markdown.

This link does check the markdown (by looking in the post history)

http://data.stackexchange.com/tex/query/695037/bug-query-on-markdown

Initially produced 4573 missed posts, and after a couple of days when the SEDE dump was refreshed on the 18 July this was down to 790, and I believe these are now fixed, so it should be 0 next week.

This massive increase in productivity was due to ShreevatsaR providing a javascript solution replacing the need to use an external editor and allowing the whole operation to be done with two keystrokes per post, at around the site maximum rate of 5 seconds per post (the site locks you out if you post edits faster than that) Note that this throttling to 5 seconds would have been unnecessary had authorised access been used, and would have brought the total time down to seconds rather than hours) This was applied by ShreevatsaR, Moriambar and myself on subsets of the list returned by the SEDE query.

---

---

A list of everyone who edited a pre-2014 posting in 2017 is:

http://data.stackexchange.com/tex/query/685628/users-with-the-most-edits-of-old-posts-in-2017

It's worth noting that by far the largest number of edits is by "Community" with 3 times as many edits as the next person. These would (mostly) be edits to change URL from http to https. Note this was done as a silent back-end edit to the collection, without flagging each question as an edited question and raising to the top of the home page, and without requiring dozens of hours of manual volunteer work to make the change. It is a shame that Community could not have been coerced into fixing this issue. (Bulk editing the full list followed by a manual check for the few cases where the edit was not required would have been much quicker).

---

---

Note that the "2nd batch" of 4300 posts looked OK on the site but the underlying markdown was corrupt, so any edit to the question would put corrupted text with missing \\ into the edit box.

This issue is the reason for the related question on meta

‘double backslash + newline’ collapses to ‘single backslash’ when I hit ‘edit’

It is also the reason why the issue surfaced particularly strongly this year, three years after the initial corruption took place. Earlier this year Stackexchange edited all postings with images replacing http URL with https. On this site that would have re-generated thousands of old posts that were visibly OK but had corrupted markdown, so exposing thousands of corrupted postings on to the public site.

Answer 2

On behalf of the team, thank you very much to everyone who participated in fixing the backslash problem. It really was excellent, amazing work. Editing is always underappreciated, and this must have been all the duller to do by hand because it was looking at the same bit of code formatting over and over rather than cleaning up prose.

I apologize for our inability to do more to help you, and for our poor communication. In the spirit of "better late than never," here's a very short post-mortem. Several developers and CMs did spend a significant amount of time "behind the scenes" trying to identify the cause of the bug and create a fix. As has been mentioned, the errors were initially caused by a change on our end, but we could not track down the exact original details due to a combination of age and poor record-keeping. Regardless, we considered solving the problem by performing a mass edit on the backend, but every edit strategy we tested seemed to break as many posts as it fixed, and we erred on the side of not introducing new bugs. Not knowing the original list of affected posts hampered us there. Ultimately, of course, we did not solve the problem. We did learn a lesson about keeping better backups, which should help you as well as the rest of the network going forward.

I want to acknowledge that a big part of the problem here was how we were working: as I said earlier, it was behind the scenes. As such, you had no way of knowing whether we were working at all, or even that we cared. I often give the advice that a bad perception can be as damaging as a bad reality, and in this case I failed to heed it myself. We on the team have a tendency to think of ourselves as transparent—because that is our honest intent—while falling into the trap of waiting "just a little longer" until we have something more substantive to announce, when in many cases "just a little longer" turns into "a lot longer" or "never." We've become more aware of the problem recently, and are working on it. In short, we'll try to be more communicative with you about what steps we are taking as we are working, in the future.

At the suggestion of your moderator team, Stack Exchange will be making a contribution in TeX SE's honor to TUG 2018, the annual meeting of the TeX Users Group (TUG) which will be held in Rio de Janiero next year. (This is separate from/in addition to the annual donation we make to TUG that grants this community institutional membership benefits in that organization.) The exact details are still in the works but will be announced when possible.

As a more tangible show of gratitude, we will be sending some TeX SE "swag" to users who contributed to the editing effort. If that includes you, be on the lookout for an e-mail from Stack Overflow (or just e-mail us yourself, referencing this meta post).