URLs containing colons are escaped and do not resolve

Question

The problem

While writing my last answer I noticed that linking to external resources containing a colon does not work, since the colon is being escaped as %3a and thus mangled and the URLs do not resolve any longer.

Example

https://gitorious.org/context/context/source/b76759d73e6734cb014fe46da1f8521dcc9a8d71:metapost/context/base/mp-mlib.mpiv#L182-256

Analysis

I tried to link to external resources using the markdown syntax [description](URL). The URL contained unescaped colons. After having read the RFC - Uniform Resource Identifier I hold that colons do not have to be escaped and that this is a bug in the stackexchange code, rather than a fault on the external resources' side.

Colons are listed as reserved characters. The relevant section is:

URI producing applications should percent-encode data octets that correspond to characters in the reserved set unless these characters are specifically allowed by the URI scheme to represent data in that component. If a reserved character is found in a URI component and no delimiting role is known for that character, then it must be interpreted as representing the data octet corresponding to that character's encoding in US-ASCII.

Take this URL:

https://gitorious.org/context/context/source/b76759d73e6734cb014fe46da1f8521dcc9a8d71:metapost/context/base/mp-mlib.mpiv#L182-256

According to the RFC the colon belongs to the path part:

  foo://example.com:8042/over/there?name=ferret#nose
  \_/   \______________/\_________/ \_________/ \__/
   |           |            |            |        |
scheme     authority       path        query   fragment
   |   _____________________|__
  / \ /                        >       urn:example:animal:ferret:nose

Furthermore, the path is of type path-absolute, which requires the syntax path-absolute = "/" [ segment-nz *( "/" segment ) ] with segment-nz = 1*pchar and segment = *pchar and pchar = unreserved / pct-encoded / sub-delims / ":" / "@".

This means, the path can contain all unreserved characters, encoded characters or colons, if I'm not mistaken.

Workaround

For the time being an URL shortener can be used and the shortened URL can be included. However, this is generally not desirable.

Discussion

I don't know why stackexchange alters the link in the first place. As far as I know, the parentheses enclose an URL, which is only valid if has been properly encoded by the user. No further encoding is required.

EDIT: It was indeed a bug on gitorious side. And Marcin committed a fix today, which has already been released, so the link mentioned above resolves correctly.

Answer 1

I believe there's an answer on MSE that addresses the issue. The important bit is:

A colon and %3a are semantically identical in URLs per the RFC.

If you look at an example on Gaming (and read the comments), you'll see that the problem should actually be fixed by the site that serves pages with colons in the URL. This is actually a bug at Gitorious.org. If you notice, the server returns with 500 Internal server error:

Sorry, something went wrong

Gitorious encountered an server error. We are automatically notified of errors and will look into it. If the error persists beyond what's reasonable, let us know.

The reason for encoding seems to be that failing to do so breaks refernce-style links in Markdown. Your solution of using a link shortener seems the best workaround.