In T238285#5666483, @Vgutierrez wrote:

BTW, Checking RFC 3986, I'm not sure that https://ban.wikipedia.org/wiki/Mal:; is a valid URL where path = /wiki/Mal:;

considering that the RFC lists the semicolon as a reserved character (https://tools.ietf.org/html/rfc3986#section-2.2):

reserved    = gen-delims / sub-delims

gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
            / "*" / " " / "," / ";" / "="

The semi-colon being in this list is precisely what makes it a valid url, not an invalid one. URLs containing = or & in popular query string form, are not invalid either.

The significance of these characters in the spec is that they must never be liberally percent-encoded or decoded as part of some kind of normalization logic as they may be used for a special meaning. This in contrary to e.g. other ASCII characters which a browser or web server is allowed to proactively encode or decode based on its preferred form. For example, /P/P/, /P/P/ and /P/P/ can be considered equivalent (where P = P), but /P/P/ is definitely something else (where / = /).

The use case for ; being reserved is that CGI scripts often used ;a=foo;b=bar instead of the current convention of ?a=foo&b=bar as a way to pass query parameters.

For a url like https://example/foo/bar?x=y;z=1#zed the path is /foo/bar?x=y;z=1. But, whether the the slashes, question marks, ampersands or semi-colons actually have special significance in any given URI is entirely the concern of the underlying application. The traffic layers should make no assumptions about those. E.g. /Foo? is not equivalent to /Foo, and /Foo?a=1&b=2 is not equivalent to /Foo?b=2&a=1. While an empty query string and the order of params might be insignificant, that's not upto HTTP.

In T238285#5821312, @MusikAnimal wrote:

I ran into this when I was unable to block https://en.wikipedia.org/w/index.php?target=Printf("Herro World");&title=Special:Contributions. All the user links in revision histories, etc., use path-style parameters which produce a broken link such as Special:Block/Printf("Herro_World");. Until this is fixed, we can use AbuseFilter to disallow creation of usernames/pages with semicolons.

@MusikAnimal do we need a note here to get the title blacklist have the line removed when the fix is implemented?

Simple repro:

krinkle@people1002$ echo -e "Hello world.\n" > 'foo;'
krinkle@people1002$ cat foo\;
Hello world.

krinkle@people1002$ curl -I 'http://localhost/~krinkle/foo;'
HTTP/1.1 200 OK
Server: Apache
Last-Modified: Fri, 25 Sep 2020 20:44:12 GMT
Content-Length: 14
…

$ curl -I 'https://people.wikimedia.org/~krinkle/foo;'
HTTP/2 404 
date: Fri, 25 Sep 2020 20:44:46 GMT
server: Apache
age: 0
x-cache: cp2041 miss, cp2029 pass

I suspect this is likely caused by the same issue in our Varnish or ATS configuration and confirms that Apache can and does serve it without issue.

With the dupe merger, maybe we owe a status update here:

We're pretty sure this is a bug in Apache Traffic Server. There's some obscure and/or interesting things about how that bug came to be, and various HTTP standards, and why an HTTP server would even care about a ; in the first place, but that's all relatively-irrelevant. The bottom line is that these URLs should work as-is (without any need for percent-encoding), and we're pretty certain it's ATS that's breaking them. We have some backlogged followup with upstream ATS to do here I'm sure.

This issue is a thing in ruwiki as well. We have a user https://ru.wikipedia.org/wiki/Участник:Var_new_username_=_"Temich"; , their userpage is unnaccessable even if you encode it: https://ru.wikipedia.org/wiki/Участник:Var_new_username_=_"Temich"; .
Hovewer, I managed to access through /w/index.php: https://ru.wikipedia.org/w/index.php?title=Участник:Var_new_username_=_"Temich";

This seems to be the root cause of:
https://el.wikipedia.org/api/rest_v1/page/mobile-sections/Υπάρχουν_Χρυσόψαρα_Εδώ;

I have created a dedicated ticket at MWoffliner as this is not that rare:
https://github.com/openzim/mwoffliner/issues/1529

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all tickets that aren't are neither part of our current planned work nor clearly a recent, higher-priority emergent issue. This is simply one step in a larger task cleanup effort. Further triage of these tickets (and especially, organizing future potential project ideas from them into a new medium) will occur afterwards! For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

Note: Pages whose title contain multiple semicolons (;) are accessible even if they do end with semicolon
https://el.wikipedia.org/wiki/Από_πού_ερχόμαστε;_Τι_είμαστε;_Πού_πάμε;

https://en.wikipedia.org/wiki/;_ doesn't work either. https://en.wikipedia.org/w/index.php?title=; does though.

Some observations:

• https://en.wikipedia.org/wiki/; _ _ also redirects to the main page. Now that's just wrong, it doesn't end in a semicolon at all! It does have spaces and underscores though, characters that get stripped. This request results in a 301 Moved Permanently and directs to * https://en.wikipedia.org/wiki/; (which appears to be not URL-encoded) and predictably results in a 301 to the Main Page. So this is a double redirect.
• https://en.wikipedia.org/wiki/%3B; is an invalid title that ends in a semicolon but does not redirect to the main page.
• https://en.wikipedia.org/wiki/Special:Contributions/Example; actually does work, showing the contributions for User:Example, but probably shouldn't?
• https://en.wikipedia.org/wiki/This_page_does_not_exist; doesn't redirect and shows https://en.wikipedia.org/wiki/This_page_does_not_exist.
• https://commons.wikimedia.beta.wmflabs.org/wiki/;; works fine.
• https://commons.wikimedia.beta.wmflabs.org/wiki/;% is article ";%"
• https://commons.wikimedia.beta.wmflabs.org/wiki/%; is article "%"

@BBlack The last status update on this bug was ~18 months ago, and indicated the issue was an upstream bug and you were following up there, with a fallback to a WMF-specific patch if upstream got stuck. I see no indication there is any question this behaviour is a bug (cf. eg. Krinkle's comment above). It's also a problem that makes certain pages inaccessible on all projects, breaks contribution histories and other core features for certain users, and necessitates manually prohibiting a character in page and user names that is intended to be permitted.

IOW: an update would be appreciated.

Same problem with an article of WPJA. Whereas it works fine for human interface:

$curl -L 'https://ja.wikipedia.org/wiki/おでかけレスターれれれのれ(^^;'

It leads to a problem with the API:

$ curl -L 'https://ja.wikipedia.org/api/rest_v1/page/mobile-sections/おでかけレスターれれれのれ(^^;'
curl: (47) Maximum (50) redirects followed

In T238285#7981540, @Xover wrote:

@BBlack The last status update on this bug was ~18 months ago, and indicated the issue was an upstream bug and you were following up there, with a fallback to a WMF-specific patch if upstream got stuck. I see no indication there is any question this behaviour is a bug (cf. eg. Krinkle's comment above). It's also a problem that makes certain pages inaccessible on all projects, breaks contribution histories and other core features for certain users, and necessitates manually prohibiting a character in page and user names that is intended to be permitted.

IOW: an update would be appreciated.

Retagging with non-icebox Traffic to at least get an update

since this bug was reported back in 2019, our CDN stack has changed a little bit, we currently use HAProxy varnish ATS. Trying to fetch https://ban.wikipedia.org/wiki/Mal:; currently results in a 404 Bad title response and the same happens with https://ban.wikipedia.org/wiki/Mal:;

For https://ban.wikipedia.org/wiki/Mal:;, the URL reaches ATS unaltered and the trailing semi-colon gets removed and as a consequence the URL hitting mediawiki is https://ban.wikipedia.org/wiki/Mal:
For https://ban.wikipedia.org/wiki/Mal:; varnish translates /Mal:; to /Mal:; and again ATS removes the trailing semi-colon.

Considering that the semi-colon (;) is a reserved character according to https://www.rfc-editor.org/rfc/rfc3986#section-2.2. That section explicitly states:

URI producing applications should percent-encode data octets that
correspond to characters in the reserved set unless these characters
are specifically allowed by the URI scheme to represent data in that
component.

IMHO we should comply with RFC3986 and start using the encoded version (;) of the semi-colon and configure Varnish to stop decoding ; to ;. However since this collides with the feedback that @BBlack provided in this very same task on

In T238285#6518280, @BBlack wrote:

With the dupe merger, maybe we owe a status update here:

We're pretty sure this is a bug in Apache Traffic Server. There's some obscure and/or interesting things about how that bug came to be, and various HTTP standards, and why an HTTP server would even care about a ; in the first place, but that's all relatively-irrelevant. The bottom line is that these URLs should work as-is (without any need for percent-encoding), and we're pretty certain it's ATS that's breaking them. We have some backlogged followup with upstream ATS to do here I'm sure.

some internal discussion must happen to move forward with this task. I'll address this issue tomorrow during the Traffic team weekly meeting.

Change 882663 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] Possibly mitigate ATS bug with semicolon in Path

https://gerrit.wikimedia.org/r/882663

T261624 was merged here; in that ticket I asked:

On testing, I can see that it is no longer possible to create user names ending in semi-colons (at least on en.Wikipedia - not tested elsewhere); however the handling of legacy accounts with such names is, as demonstrated, sub-optimal. Perhaps we should force renames?

That question does not seem to have been addressed.

Change 882663 merged by BBlack:

[operations/puppet@production] Possibly mitigate ATS bug with semicolon in Path

https://gerrit.wikimedia.org/r/882663

With the merge above, I think this issue is at least mitigated for now. It's not a great long-term solution, but it should alleviate the user-facing side of this in practice.

In T238285#8549627, @Pigsonthewing wrote:

T261624 was merged here; in that ticket I asked:

On testing, I can see that it is no longer possible to create user names ending in semi-colons (at least on en.Wikipedia - not tested elsewhere); however the handling of legacy accounts with such names is, as demonstrated, sub-optimal. Perhaps we should force renames?

That question does not seem to have been addressed.

I suggested that the blacklist entry disallowing user names ending in semi-colons be removed: https://meta.wikimedia.org/wiki/Talk:Title_blacklist#T238285_workaround

Even if that doesn't happen, I think we haven't forced renames in the past, e.g. T254045: One can register a username with an equals sign in it (except when required for technical reasons, like the SUL renames some years ago).

Change 912312 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] Varnish/ATS semicolon workaround for Restbase

https://gerrit.wikimedia.org/r/912312

Change 912312 merged by BBlack:

[operations/puppet@production] Varnish/ATS semicolon workaround for Restbase

https://gerrit.wikimedia.org/r/912312