What potential problems should be checked for in the great CommonMark switch?

Question

tex.sx is scheduled to undergo a code conversion to CommonMark tomorrow (10 June 2020); for details, see We're switching to CommonMark.

The last time an under-the-covers code conversion was performed, we lost a myriad double backslashes, which were converted silently to single backslashes. After many, many hours of manual labor, and some inspired identification of probable patterns that could be matched to locate corrupted postings, sanity was (mostly) restored. Here's a commentary on that experience: The CommonMark diary.

In an effort to make checking somewhat more rational, this question is an attempt to consolidate possible/probable situations that might be subject to auto-corruption. Please add your suggestions one per answer.

Answer 1

Copied and pasted from the edit window of https://tex.stackexchange.com/a/400900:

Pandoc doesn't like the option in brackets after the latex command. In order to make it work you have to use a recent version of pandoc that supports generic raw attributes. Now you can create a raw latex block around the offending command:

```{=latex}
\Opensolutionfile{ans}[ex1] 
```

Or inline:

`\Opensolutionfile{ans}[ans1]`{=latex}

Answer 2

Using double backticks to escape backticks in inline code, from https://tex.stackexchange.com/a/543571/134574

---

\textbackslash is just for typesetting. As the name says, it's a text command, so it just prints the character \ from the current font, and isn't of any use for anything other than that.

You are looking for:

\catcode`\\=12

---

The syntax for the \catcode primitive is \catcode<number>=<catcode>, and it sets the catcode of the character whose ASCII code is <number> to <catcode>. This means that you could also use:

\catcode 92=12

However it's a mouthful to remember all the ASCII codes, and your code becomes quite a lot less readable. To improve on that TeX allows you to specify a number using an “alphabetic constant”. To do that, the <number> should start with a `, followed by the character token you want to make a number of. Valid alphabetic constants are `a, `*, etc.

The only problem is that a catcode-0 character (here the backslash) doesn't produce a token, so neither:

\catcode`\=12
\catcode`\ =12

do what you want (the first one changes the catcode of =, and the second the catcode of ). To specify these characters, TeX allows you to escape the character with a backslash, so to specify the alphabeitc constant `\ you prefix it with another backslash: `\\, so:

\catcode`\\=12

---

Bonus, from https://tex.stackexchange.com/a/516660/134574, using three backticks to escape double backticks

---

As Mico noted, when two ` are used (like in ``should'') you usually don't want to apply the special formatting. To detect this case you can check that the argument of active-` is empty and then use the original ` character instead.

---

Or more generally, using n+1 backticks to escape n backticks:

`````````````````````````

Answer 3

Subscripts and superscripts, from https://tex.stackexchange.com/a/527538/134574 (the answer is huge, so I copied only a paragraph).

---

(\outer is a primitive, so \meaning\outer is \12o12u12t12e12r12). In this case, the code will see the \ and will do its thing with \outer^[note 1]. Later, it arrives at \umbrella and hits it with \meaning:

Answer 4

(also reported here)

Links to URLs with a ) in them will break. Consider this link to the Stack Exchange API documentation:

https://api.stackexchange.com/docs/questions-by-ids#order=desc&sort=activity&ids=349185&filter=!)rTkraPYPefwELKox66q&site=meta&run=true

If you try to link to it right now, it works, but after the update, it breaks:

_{(screenshot taken on Meta.SE which is already migrated)}

Answer 5

Code in lists, copied from https://tex.stackexchange.com/a/461589/134574

---

This is somewhat similar to this question.

The values in the fractions matter because the boxes of the characters have different sizes. The p has a descender below the baseline which the x doesn't, thus when you swap them, the box of the denominator get a little bigger and TeX uses a larger delimiter to make that fit.

You have a few possibilities to work around that (basically the same ones I listed in the linked question):

• You can use a fixed delimiter size (\bigg or \Big, for instance):

  $$\Bigl(\frac{x}{p}\Bigr)\Bigl(\frac{p}{x}\Bigr)$$
  $$\biggl(\frac{x}{p}\biggr)\biggl(\frac{p}{x}\biggr)$$
  

• You can \raise the p so that TeX won't try to use a larger box:

  $$\left(\frac{x}{\raise0.35ex\hbox{$p$}}\right)\left(\frac{p}{x}\right)$$
  

• or you can add an invisible p next to the x so that the delimiter used will be the larger one:

  $$\left(\frac{x}{p}\right)\left(\frac{p}{\vphantom{p}x}\right)$$
  

• Or you can change change TeX's \delimiterfactor (and \delimitershortfall, which I didn't show here) and let TeX adjust the delimiters accordingly:

  $$\delimiterfactor=790
  \left(\frac{x}{p}\right)\left(\frac{p}{x}\right)$$
  
  $$\delimiterfactor=970
  \left(\frac{x}{p}\right)\left(\frac{p}{x}\right)$$
  

Full code:

\documentclass{article}
\begin{document}
% \delimitershortfall=5pt % Default
% \delimiterfactor=901    % Default

$$\left(\frac{x}{p}\right)\left(\frac{p}{x}\right)$$

$$\Bigl(\frac{x}{p}\Bigr)\Bigl(\frac{p}{x}\Bigr)$$
$$\biggl(\frac{x}{p}\biggr)\biggl(\frac{p}{x}\biggr)$$

$$\left(\frac{x}{\raise0.35ex\hbox{$p$}}\right)\left(\frac{p}{x}\right)$$

$$\left(\frac{x}{p}\right)\left(\frac{p}{\vphantom{p}x}\right)$$

$$\delimiterfactor=790
\left(\frac{x}{p}\right)\left(\frac{p}{x}\right)$$

$$\delimiterfactor=970
\left(\frac{x}{p}\right)\left(\frac{p}{x}\right)$$
\end{document}

Output:

Answer 6

Code copied from https://tex.stackexchange.com/a/44765 which had double backslashes trashed in earlier update.

\documentclass[12pt,a4paper,oneside]{report} 
\usepackage[left=2.00cm,right=2.00cm,top=2.45cm,bottom=2.5cm]{geometry}   
\usepackage{amsmath,amssymb,amsthm,amscd,amstext}

\def  \st    {\backepsilon}
\DeclareMathOperator{\dom}{dom}
\DeclareMathOperator{\rng}{rng}

\begin{document}
\begin{proof}$ $
\vspace{-\abovedisplayskip}\vspace{-\baselineskip}
\begin{align*}
  \text{Let } y \in \rng{R^{-1}} &\iff \exists\; x \in Y \st (x,y) \in R^{-1}\\
  &\iff \exists\; x \in Y \st (y,x) \in R \hspace{15pc}\\
  &\iff y \in \dom{R}\\
\therefore \rng{R^{-1}} &= \dom{R}.
\qedhere
\end{align*}
\end{proof}
\end{document}

Answer 7

This is a situation reported as a possible problem in a response to the announcement of the CommonMark switch.

Here is a multiparagraph quote, wrapped around another multiparagraph quote. There are blank lines separating all paragraphs.

Here is a multiparagraph quote It is embedded in another multiparagraph quote, with a blank line between the paragraphs.

Here is the second paragraph of the embedded multiparagraph quote It is embedded in another multiparagraph quote, with a blank line between the paragraphs.

Here is the last paragraph of a multiparagraph quote, wrapped around another multiparagraph quote. There are blank lines separating all paragraphs.

The automatic formatting via the "quotes" icon has inserted an extra blank line above and below both quoted paragraph groups, i.e., an extra four blank lines.

Answer 8

Code copied from Problem with formatting tables with equal column widths which had double backslashes trashed in earlier update.

    \begin{table}[h]
        \caption{Distribution of mappings for the SLA parameter $\pi$.}
        \label{tab:distPi} \centering
        \begin{tabular*}{0.45\textwidth}{@{\extracolsep{\fill}}|c|c|c|}
            \hline
                \multicolumn{3}{|c|}{\textbf{Name of the SLA parameter $\pi$}} \\
            \hline
                \emph{Cost} & \emph{Charge} & \emph{Rate} \\
            \hline
                15\% & 15\% & 40\% \\
            \hline
            \hline
                \multicolumn{3}{|c|}{\textbf{Unit of the SLA parameter $\pi$}} \\
            \hline
                \emph{USD} & \emph{GBP} & \emph{YPI} \\
            \hline
                38\% & 2\% & 40\% \\
            \hline
        \end{tabular*}
        \vspace{-0.3cm}
    \end{table}

and this is how I've made the second table:

    \begin{table}[h]
        \caption{Distribution of mappings for the SLA parameter $\mu$.}
        \label{tab:distMu} \centering
        \begin{tabular*}{0.45\textwidth}{@{\extracolsep{\fill}}|c|c|c|c|c|c|}
            \hline
                \multicolumn{6}{|c|}{\textbf{Name of the SLA parameter $\mu$}} \\
            \hline
                \multicolumn{3}{|c|}{\emph{MemoryConsumption}} & \multicolumn{3}{|c|}{\emph{Consumption}} \\
            \hline
                \multicolumn{3}{|c|}{70\%} & \multicolumn{3}{|c|}{30\%} \\
            \hline
            \hline
                \multicolumn{6}{|c|}{\textbf{Unit of the SLA parameter $\mu$}} \\
            \hline
                \multicolumn{2}{|c|}{\emph{Mbit}} & \multicolumn{2}{|c|}{\emph{Gbit}} & \multicolumn{2}{|c|}{\emph{Tbit}} \\
            \hline
                \multicolumn{2}{|c|}{5\%} & \multicolumn{2}{|c|}{55\%} & \multicolumn{2}{|c|}{40\%} \\
            \hline
        \end{tabular*}
        \vspace{-0.3cm}
    \end{table}

Answer 9

emacs / auctex code, copied from https://tex.stackexchange.com/a/3859.

There's a few bits of junk in there as well, but most of it is focussed on this issue. It works by rewriting the commands tex-file and tex-view-file so that they can be modified according to options set by choosing an output type (which is actually more refined than just the output type) for the file.

So I just do M-x set-tex-output-type xe and I get my file compiled using xelatex.

;(server-start)
;(load "xdvi-search.el")

(setq latex-block-names '("theorem" "corollary" "proof" "lemma" "defn" "thm"))
;(setq tex-dvi-view-command
;          (if (eq window-system 'x) "xdvi" nil))
;(setq tex-dvi-print-command "dvips -Pmirkti *")
(setq tex-alt-dvi-print-command "dvips * -ta4 -o")

(defcustom tex-output-type "pdf"
  "*Default output type for TeX jobs."
  :type '(radio (const :tag "Postscript" "ps")
        (const :tag "Postscript (rv)" "psrv")
        (const :tag "PDF" "pdf")
        (const :tag "PDF (rv)" "pdfrv")
        (const :tag "Dvi" "dvi")
        (const :tag "Dvi (draft)" "dvid")
        (const :tag "XeLaTeX" "xe")
        (const :tab "LuaLaTeX" "lua")
        (const :tag "HTML" "html")
        (const :tag "XHTML" "xhtml")
        (const :tag "MathML" "xml")
        (const :tag "OpenOffice" "sxw")
        (const :tag "maruku" "maruku"))
  :group 'tex-run)

(defcustom tex-ext-alist
  '(("ps" . "ps")
    ("psrv" . "ps")
    ("pdf" . "pdf")
    ("pdfrv" . "pdf")
    ("dvi" . "dvi")
    ("dvid" . "dvi")
    ("xe" . "pdf")
    ("lua" . "pdf")
    ("html" . "html")
    ("xhtml" . "xhtml")
    ("xml" . "xml")
    ("sxw" . "sxw")
    ("maruku" . "xhtml"))
  "*alist of eventual file extensions")

(defcustom tex-command-alist
  '(("pdf" . "pdflatex")
    ("pdfrv" . "pdflatex")
    ("ps" . "latex")
    ("psrv" . "latex")
    ("dvi" . "latex")
    ("dvid" . "latex")
    ("xe" . "xelatex")
    ("lua" . "lualatex")
    ("html" . "mk4ht")
    ("xhtml" . "mk4ht")
    ("xml" . "mk4ht")
    ("sxw" . "mk4ht")
    ("maruku" . "maruku")
    )
  "*alist for processing file into various types of output")

(defcustom tex-start-options-alist
  '(("pdf" . "\\\\nonstopmode\\\\input")
    ("pdfrv" . "\\\\nonstopmode\\\\input")
    ("ps" . "\\\\nonstopmode\\\\input")
    ("psrv" . "\\\\nonstopmode\\\\input")
    ("dvi" . "\\\\nonstopmode\\\\input")
    ("dvid" . "-src-specials \\\\nonstopmode\\\\input")
    ("xe" . "\\\\nonstopmode\\\\input")
    ("lua" . "\\\\nonstopmode\\\\input")
    ("html" . "htlatex")
    ("xhtml" . "xhlatex")
    ("xml" . "mzlatex")
    ("sxw" . "oolatex")
    ("maruku" . "-m itex2mml")
    )
  "*alist for options that preceed the filename")


(defcustom tex-end-options-alist
  '(("pdf" . "")
    ("pdfrv" . "")
    ("ps" . "")
    ("psrv" . "")
    ("dvi" . "")
    ("dvid" . "")
    ("xe" . "")
    ("lua" . "")
    ("html" . "\"html,3,frames\"")
    ("xhtml" . "\"html,3,frames\"")
    ("xml" . "\"html,3,frames\"")
    ("sxw" . "\"html,3,frames\"")
    ("maruku" . "")
    )
  "*alist for options that postceed the filename")

(defcustom tex-post-command-alist
  '(("pdf" . "")
    ("pdfrv" . "")
    ("ps" . "dvips * -ta4 -o 2> /dev/null")
    ("psrv" . "dvips * -ta4 -o 2> /dev/null")
    ("dvi" . "")
    ("dvid" . "")
    ("xe" . "")
    ("lua" . "")
    ("html" . "")
    ("xhtml" . "")
    ("xml" . "")
    ("sxw" . "")
    ("maruku" . "")
    )
  "*alist for post-processing commands")

(defcustom tex-mid-ext-alist
  '(("ps" . "dvi")
    ("psrv" . "dvi")
    ("pdf" . "")
    ("pdfrv" . "")
    ("dvi" . "")
    ("dvid" . "")
    ("xe" . "")
    ("lua" . "")
    ("html" . "")
    ("xhtml" . "")
    ("xml" . "")
    ("sxw" . "")
    ("maruku" . "")
    )
  "*alist of mid-point file extensions if we do a two-stage process")


(defcustom tex-view-commands-alist
;  (cond 
;    ((eq window-system 'x)
     '(("pdf" .  "xpdf")
       ("pdfrv" .  "xpdf")
     ("ps" .  "gv -ad=~/.gv_spartan_right.dat")
     ("psrv" .  "gv -ad=~/.gv_spartan_right_rv.dat")
     ("dvi" . "xdvi")
     ("dvid" . "xdvi")
     ("xe" .  "xpdf")
     ("lua" .  "xpdf")
     ("html" . "firefox")
     ("xhtml". "firefox")
     ("xml". "firefox")
     ("sxw". "ooffice2")
     ("maruku" . "firefox")
     )
;    (t
;     '(("pdf" . "pdftotext * | cat -s")
;     ("ps" . "ps2ascii * | cat -s")
;     ("dvi" . "dvips * | ps2ascii | cat -s")
;     ("html". "lynx")
;     ("xhtml". "lynx")
;     ("xml". "lynx")
;     ("sxw". "false"))
;     )))
  "*alist for viewing the various types of output")

;(defcustom tex-dvi-to-ps-command
;  "dvips * -ta4 -o 2> /dev/null"
;  "*Command for conversion from dvi to ps."
;  :type 'string
;  :group 'tex-run)

(defun tex-font-lock-type-alist
  '(("pdf" . "")
    ("pdfrv" . "")
    ("ps" . "")
    ("psrv" . "")
    ("dvi" . "")
    ("dvid" . "")
    ("xe" . "")
    ("lua" . "")
    ("html" . "")
    ("xhtml" . "")
    ("xml" . "")
    ("sxw" . "")
    ("maruku" . "markdown-mode-font-lock-keywords")
    )
  "*alist for modifying font-lock defaults")


(modify-syntax-entry ?\\ "." tex-mode-syntax-table)

(defun set-tex-main-file ()
  "Set 'tex-main-file' to something appropriate from the current buffer."
  (interactive)
  (make-variable-buffer-local 'tex-main-file)
  (setq tex-main-file (expand-file-name (car (find-file-read-args
                          "(La)TeX main file name for buffer: " nil))))
)

(defun set-tex-output-type ()
  "Set the 'tex-output-type'."
  (interactive)
   (setq newtype
      (completing-read "TeX Output Type: " tex-ext-alist nil t)
      )
  (if (not (equal newtype "")) 
      (setq tex-output-type newtype)
    )
  )

(defun template ()
  "Insert template.tex at the appropriate place."
  (interactive)
  (insert-file-contents "~/tex/papers/useful/template.tex")
)

(defun end-document ()
  "Insert end-of-document commands at the point."
  (interactive)
  (insert "\\bibliography{arxiv,articles,books,misc}\n\n\\end{document}\n\n")
)

(defun find-end-document ()
  "Skips to the end of the current document."
  (interactive)
;  (goto-char (point-min))
  (search-forward "\\end{document}")
)

(defun my-tex-file ()
  "Modify \\[tex-file] to take into account the desired output type."
  (interactive)
  (let ((tex-command (concat
              (cdr (assoc tex-output-type tex-command-alist))
              " "
              (cdr (assoc tex-output-type tex-start-options-alist))
              " \* "
              (cdr (assoc tex-output-type tex-end-options-alist)))))
  (tex-file)
  )
  (if (< 0 (length (cdr (assoc tex-output-type tex-post-command-alist))))
      (let ((tex-mid-file (tex-append tex-print-file (concat "." (cdr (assoc tex-output-type tex-mid-ext-alist))))))
    (if (not (file-exists-p tex-mid-file))
        (error "No appropriate file could be found")
      (tex-send-command (cdr (assoc tex-output-type tex-post-command-alist)) tex-mid-file)
      )
    )
    )
  )


(defun my-tex-view ()
  "Modify \\[tex-view] to take into account the desired output type."
  (interactive)
  (let* ((tex-suffix (concat "." (cdr (assoc tex-output-type tex-ext-alist))))
     (tex-view-command (cdr (assoc tex-output-type tex-view-commands-alist)))
     (view-file-name (tex-append tex-print-file tex-suffix))
        test-name)
    (if (and (not (equal (current-buffer) tex-last-buffer-texed))
             (buffer-file-name)
             ;; Check that this buffer's printed file is up to date.
             (file-newer-than-file-p
              (setq test-name (tex-append (buffer-file-name) (concat "." (cdr (assoc tex-output-type tex-ext-alist)))))
              (buffer-file-name)))
        (setq view-file-name test-name))
    (if (not (file-exists-p view-file-name))
        (error "No appropriate file could be found to view.")
      (if (tex-shell-running)
          (tex-kill-job)
        (tex-start-shell))
      (tex-send-command
       tex-view-command view-file-name
       t))))

(define-key tex-mode-map "\C-c\C-n" 'find-end-document)
(define-key tex-mode-map "\C-c\C-d" 'end-document)

(define-key tex-mode-map [remap tex-file] 'my-tex-file)
(define-key tex-mode-map [remap tex-view] 'my-tex-view)

(define-key tex-shell-map [remap tex-file] 'my-tex-file)
(define-key tex-shell-map [remap tex-view] 'my-tex-view)

(defadvice server-visit-files (around save-buffers last activate)
    "Try to emulate gnuclient behaviour with emacsclient.
Works only for visiting one buffer at a time."
    (let* ((filen (car (car (ad-get-arg 0))))
           (buf (get-file-buffer filen))
           (this-buf-modified-p nil))
      ;;; the following is copied from server-visit-files, with
      ;;; a modification for the `verify-visited-file-modtime' test
      (if (and buf (set-buffer buf))
          (if (file-exists-p filen)
              ;;; if the file has changed on disk, reload it
              ;;; using `find-file-noselect'
              (if (not (verify-visited-file-modtime buf))
                  (progn
                    (find-file-noselect filen)
                    ;;; if user answered `no', reset modtime anyway
                    ;;; so that server-visit-files doesn't realize
                    ;;; the difference:
                    (set-visited-file-modtime)))
            ;;; if file exists no longer, we let server-visit-files
            ;;; deal with that
            t)
        (setq buf (find-file-noselect filen)))
      (setq this-buf-modified-p (buffer-modified-p buf))
      (set-buffer buf)
      (set-buffer-modified-p nil)
      ad-do-it
      (set-buffer-modified-p this-buf-modified-p)))


(require 'markdown-mode)

(defun markdown-font-locks ()
  "Set the font lock to that of markdown mode."
  (interactive)
  (set (make-local-variable 'font-lock-defaults)
       '(markdown-mode-font-lock-keywords))
  (set (make-local-variable 'font-lock-multiline) t)
  (setq font-lock-mode-major-mode nil)
  (font-lock-fontify-buffer)
  )

Answer 10

Copied from Error with pygmentize when using minted package.

Problem:

I get multiple undefined control sequences when using the package minted and I believe it is caused by pygmentize.

Background

I use the software Texpad and the active LaTeX distribution is set to:

• /usr/local/texlive/2016/bin/x86_64-darwin

I then check both python and pygmentize:

• which python results in /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python
• which pygmentize results in /opt/local/bin/pygmentize

I have MacPorts installed and I check the active version by using:

• port select --list python which results in python34 (active)

Question 1

This is where I get confused, shouldn't which python also point to /opt/local/bin/python?

Paths

I check my paths by doing nano ~/.bash_profile, which results in:

export PATH=/System/Library/Frameworks/Python.framework/Versions/2.7/bin:$PATH
export PATH=/usr/local/texlive/2016/bin/x86_64-darwin:$PATH
export PATH=/usr/local/bin:$PATH
export PATH=/usr/local/:$PATH

##
# Your previous /Users/Batman/.bash_profile file was backed up as /Users/Batman/.bash_profile.macports-saved_$
##

# MacPorts Installer addition on 2017-02-12_at_00:07:31: adding an appropriate PATH variable for use with MacPorts.
export PATH="/opt/local/bin:/opt/local/sbin:$PATH"
# Finished adapting your PATH environment variable for use with MacPorts.

Question 2

What is required given the above information in order for pygmentize to work?

Minimal Working Example (MWE):

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}

\usepackage{minted}

\usemintedstyle[php]{autumn}

\begin{document}
\begin{minted}{php}
<?php
    try {
        // Connect to SQL database
        $dsn = new PDO('mysql:host=servername;dbname=databasename', 'username', 'password');
    }
    catch (PDOException $e) {
        // Prints out error message
        echo 'Error: ' . $e->getMessage();
    }
?>
\end{minted}
\end{document}