| cscott | |
| Apr 27 2023, 2:35 PM |
| F37094491: Screenshot 2023-06-05 at 14-58-48 REST API Documentation.png | |
| Jun 5 2023, 7:01 PM |
In order to determine who is using the /wikitext/to/lint and /html/to/wikitext endpoints, the plan is to selectively disable them for increasing periods of time (1 hr, 1 day, 1 week, etc) and monitor village pumps/phabricator to see if the user (presumably a community bot or tool of some sort?) surfaces.
This will should of course be announced on the appropriate mailing lists and technews beforehand.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | None | T324931 Clean up open RESTBase related tickets | |||
| In Progress | None | T262315 <CORE TECHNOLOGY> API Migration & RESTBase Sunset | |||
| Open | None | T344944 Move Parsoid endpoints out of RESTBase | |||
| In Progress | None | T374135 Migrate RESTbase page content endpoints | |||
| Open | MSantos | T334238 Create deprecation plan for public parsoid endpoints | |||
| Invalid | None | T335513 Selective outage of `/wikitext/to/lint` and `/html/to/wikitext` RESTBase endpoints |
I think the /wikitext/to/lint endpoint is used by https://en.wikipedia.org/wiki/User:PerfektesChaos/js/lintHint .. worth looking at that code. That gadget is used by editors for fixing lints and so we shouldn't be turning that off.
Errr, what? We have a process for API deprecations, and making announcements to determine usage patterns. There's no need to break things beforehand...
We do not, to my knowledge, have a process for deprecating REST APIs. We don't even have a clear way to indicate whether an API is stable for external use at all.
But of course, we shouldn't just break stuff without announcement. I updated the task description to make this explicit.
Sure we do, though I'd posit that the fact that it's a "REST" API is rather meaningless from an API consumer standpoint. You can quibble on whether it's an "approved policy", but we certainly have a process: https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmap#Deprecation_process
We don't even have a clear way to indicate whether an API is stable for external use at all.
Again, yes we do. https://www.mediawiki.org/wiki/API_versioning#End_point_stability
Example:
(of course, the lint endpoints are still marked as "experimental" which to me suggests that they weren't updated, rather than actually being experimental quality)
Linking for cross reference purposes: T287819: Add automatic temporary feature disabling to encourage migration away from long deprecated feature
Quick stats check: both endpoints are seeing about two requests per minute each. More than half come from a single IP sending fake user-agent header that starts with a duplicate "User Agent:" prefix. Perhaps we should block this client for failing to follow the User-Agent policy - if it did follow the policy, we would have a way to contact the people that run it.
Btw, we do know one user who we may want to reach out to: the endpoints both get one request from Googlebot roughly every 15 minutes.
Ok, right. In my mind, this process applies to the action API, which was known as "the API" at the time.
It is not obvious how this process would be applied to REST APIs, in particular the bit about "The old feature will report deprecation warnings". We should have a spec for this, e.g. following this draft, but we don't. We currently have no direct way to inform clients that use a REST API that this API is deprecated.
We don't even have a clear way to indicate whether an API is stable for external use at all.
Again, yes we do. https://www.mediawiki.org/wiki/API_versioning#End_point_stability
My point was that we have, at least in practice, failed to properly distinguish between APIs for internal use by clients we control, and APIs stable for external users. As far as I know, breaking changes have frequently been made to "stable" APIs in RESTbase, under the assumption that we were under no obligation to third party users. But perhaps I am wrong about this, I am relatively new to the world of RESTbase. Perhaps @cscott can shed some light on how this has been handled in practice.
I am very much for having better policies and specs, clear versioning and stability, well defined obligations, and proper communcation channels. But in reality, we don't have that. When we need to change or deprecate an API, we often have no good way to contact the users of that API.
If we announce the deprecation of an API but still see a considerable amount of traffic to that API, "selective outages" seem like a good way to generate attention. Better than a) keeping the deprecated stuff forever or b) just flipping the switch.
Someone should really clarify what we are talking about here.
API servers, endpoints, gateways apps, envoy, internal, external, public, the terminology is used all mixed and inconsistently. Ppl don't know what to expect. To quote discord: "i'm screaming pre-emptively [..] no docs on what's supposed to be the new endpoints"
Anyone screaming about this pre-emptively, please tell us about your use case here on this ticket :)
I was the one screaming pre-emptively on le Discord, previous comment about this on T334238#9119834. I didn't even know that this was the right task to make the comment on. 🤦
What's the replacement API module for this? Is there documentation?
The use case is getting lint errors given wikitext? e.g. https://gitlab.wikimedia.org/repos/mwbot-rs/delinter/-/blob/main/src/api.rs?ref_type=heads#L67
Actually, I was wrong. Lint was ported to core:
curl -d 'hello world' -X POST https://en.wikipedia.org/w/rest.php/v1/transform/wikitext/to/lint/X
Awesome! I just switched over my bot and in my limited testing it works the same. I'll look into updating documentation tomorrow.