Page MenuHomePhabricator
Create Task
Maniphest T264669

Move VE API from RESTBase into core.
Closed, ResolvedPublic1 Estimated Story Points
Actions

Description

Currently VisualEditor depends on RESTBase for performance, correctness (avoiding dirty diffs) and certain features (wikitext<->VE edit switching). After Parsoid was migrated to MediaWiki core, having RESTBase as a caching/stashing layer for VisualEditor no longer makes sense.

The goal of this task and it's subtasks it to support all RESTBase features needed by VisualEditor in MediaWiki core or VE extension.

VisualEditor depends on the following GET endpoints in RESTBase

We already have corresponding endpoints in MediaWiki core, but some features are missing. Missing features:

  • Language variant support T317019
  • Stash for edit T267990
  • Ability to either follow or suppress on-wiki redirects T301372

Additionally, VisualEditor depends on the following transform endpoints:

These endpoints don't need to be moved to restbase one-to-one, we might want to redesign how these work in process.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolveddanielT386326 REST: endpoints that return rendered page content should support all content models
 StalledNoneT324931 Clean up open RESTBase related tickets
 In ProgressNoneT262315 <CORE TECHNOLOGY> API Migration & RESTBase Sunset
 ResolvedMSantosT264669 Move VE API from RESTBase into core.
 ResolvedMSantosT265846 Turn RESTBase tests for /html and /transform endpoints into api-testing integration tests
 InvalidNoneT267980 Benchmark cached Parsoid HTML endpoints RESTBase vs Core
 ResolveddanielT267981 Implement /revision/{id}/html endpoint in core
 Resolved• PcheloloT267982 Extract cached Parsoid access to an internal service object
 Resolved• WDoranWMFT262597 MediaWiki Developer stores revision HTML in parser cache
 Resolved• PcheloloT269663 Implement parser output caching for /revision/{id}/html endpoint
 ResolvedNoneT267990 Support stashing in page/html and revision/html endpoints in MW core
 ResolvedNoneT267991 Simple per-IP ratelimit for HTML stashing
 ResolveddanielT268205 Store data-parsoid in parsoid ParserCache
 OpenNoneT308424 Determine http cache control and active purging for REST endpoints serving parsoid output
 DuplicateNoneT308684 Determine directory to place ParsoidStash logic
 ResolveddanielT308743 PageHTMLHandler: add support for getting fully annotated parsoid HTML.
 ResolveddanielT308744 PageHTMLHandler should return different eTag for different output modes
 ResolvedDAlangi_WMFT309017 Collect stats on stash and cache usage in the page/html and revision/html endpoints
 ResolveddanielT309549 Create end-to-end tests for the page/html and revision/html endpoints.
 InvalidNoneT262572 Use Parsoid-specific Parser Cache for Visual Editor
 InvalidNoneT262605 Parsoid REST API Client gets a speedy result
 DuplicateNoneT262593 MediaWiki Developer stores different HTML output for different language variants in the parser cache
 DuplicateNoneT262586 MediaWiki Developer stores parse metadata in parser cache
 Resolved• eprodromouT262604 Reader quickly reads a wiki page
 Resolved• PcheloloT263587 CAPEX for ParserCache for Parsoid
 Resolved• PcheloloT268234 Evaluate ETags in core HTML endpoints
 ResolvedMSantosT301370 Move transform endpoints from RESTBase to MediaWiki
 ResolveddanielT310467 Remove implementation of the transform endpoint from the parsoid extension
 ResolvedNoneT311477 Remove pagebundle support from public transform endpoint
 OpenNoneT311645 Split the TransformHandler class
 ResolveddanielT311819 Make the transform endpoint match ETags emitted by the page endpoint
 ResolveddanielT311867 Support redirects to private endpoints
 ResolveddanielT350661 Expose parsoid transformation API from MediaWiki core.
 ResolvedMSantosT301371 Preemptively warm caches for Parsoid output
 ResolvedcscottT311648 Allow WikitextContentHandler to use Parsoid for rendering wikitext
 ResolvedcscottT300190 Ensure that if 'useParsoid' flag is present in ParserOptions that we use Parsoid to parse the page.
 OpenNoneT310520 Parsoid content not compatible with `index.php?title=` URLs
 ResolvedDAlangi_WMFT327769 ParserOutputAccess should use a separate ParserCache instance for Parsoid output
 ResolvedDAlangi_WMFT308588 Do not cache fast parsoid output in ParserCache (exploration)
 ResolvedppelbergT308511 [SPIKE] Determine necessity of edit session continuity during data center switchovers
ResolveddanielT310710 Do not ignore HTTP conditionals in transform REST endpoint.
 ResolvedNoneT310904 Find out if Varnish is messing with ETags, and what to do about it.
 ResolveddanielT310878 Consolidate ParsoidHandler base class with HtmlOutputRendererHelper
 ResolvedDAlangi_WMFT311728 Gracefully handle non-wikitext pages when returning HTML from the page endpoints
 DuplicateNoneT312916 Parsoid: Migrate 3rd party clients from RESTbase to the MW core API
 DuplicateNoneT312917 Parsoid: Proxy old RESTbase URLs to the core endpoints
 ResolvedMSantosT314244 Parsoid: make roundtrip-test.js run as a Mocha test in CI
 ResolveddanielT317017 Make the core page/html endpoint return system message content when appropriate
 ResolvedNoneT318393 Parsoid: Maintain context for selser on the server side
 ResolvedNoneT310464 Make use of stashed data-parsoid mapping for html-to-wikitext transformation
 ResolveddanielT310489 Collect statistics on stash read failures
 ResolvedNoneT318395 Parsoid: use rendering from ParserCache for selser if the render ID matches.
 ResolvedNoneT318398 Parsoid: use current revision rendering to enable selser, when no stash key/etag is provided.
 ResolveddanielT318399 HtmlInputTransformHelper: collect statistics on selser context
ResolveddanielT320529 Configure VE backend to use Parsoid directly, instead of calling RESTbase
 ResolvedNoneT310376 Make VE Action API use Parsoid directly
 ResolvedNoneT310377 Make parsoid transformation API available internally
 ResolvedmatmarexT314565 ApiVisualEditor: Convert ApiParsoidTrait into a helper object.
 ResolvedPRODUCTION ERRORdanielT320703 VisualEditor can't save pages on de.wikipedia.beta
 ResolveddanielT320704 When VisualEditor saves an edit, it should use the same backend that the HTML was originally loaded from
 DuplicatexSavitarT320532 Create dual backend mode for VE API
 ResolveddanielT320534 Put Parsoid output into the ParserCache on every edit
 ResolveddanielT322427 Job for pre-emptive population of the ParserCache with Parsoid output
 ResolvedNoneT322672 Make ParsoidHandler::wt2html write to parser cache
 ResolvedovasilevaT326147 Stop fragmenting ParserCache entries for mobile frontend
Declined• nrayT328802 Devise Test cases for functional fallout of ParserCache fix
 ResolvedDAlangi_WMFT327164 Trigger ParsoidCachePrewarmJob on page views
 ResolvedssastryT328034 ParsoidCachePrewarmJob should set the useParsoid ParserOption
 ResolveddanielT341123 High insertion rate of ParsoidCachePrewarmJob causes substantial backlog
 ResolveddanielT329366 Enable WarmParsoidParserCache on all wikis
 ResolvedClement_GoubertT333528 Increase memory_limit for jobrunners to $wmgMemoryLimitParsoid
 ResolveddanielT333684 VE load latency degraded on hewiki
 ResolvedKrinkleT327162 RefreshLinksJob should update ParserCache
 ResolveddanielT320536 Ensure that stashing backend for the VE API has sufficient capacity
 ResolveddanielT309016 Determine storage requirements for stashing parsoid output for VE edits
 ResolveddanielT320531 Configure VE backend to use Parsoid directly on the beta cluster and testwiki
 ResolveddanielT320535 Put Parsoid output into the ParserCache on the beta cluster and testwiki
 ResolvedKCVelaga_WMFT338634 Calculate edit session length by platform and editing interface
 ResolveddanielT338638 [SPIKE] What – if any – user-facing regressions might RESTbase deprecation have caused
 ResolvedDAlangi_WMFT339227 Remove VRS code from VisualEditor
 Resolved• R_RanaT338922 Introduce maintenance script for warming cache with Parsoid Outputs

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ArielGlenn subscribed.Jan 5 2021, 7:29 AM
• Clarakosi closed subtask T265846: Turn RESTBase tests for /html and /transform endpoints into api-testing integration tests as Resolved.Feb 8 2021, 4:31 PM
• Pchelolo closed subtask T262572: Use Parsoid-specific Parser Cache for Visual Editor as Invalid.Feb 24 2021, 5:01 PM
• Pchelolo closed subtask T269663: Implement parser output caching for /revision/{id}/html endpoint as Resolved.Mar 2 2021, 3:27 PM
• WDoranWMF closed subtask T267981: Implement /revision/{id}/html endpoint in core as Resolved.Nov 15 2021, 12:56 PM
• DAbad mentioned this in T262315: <CORE TECHNOLOGY> API Migration & RESTBase Sunset.Dec 21 2021, 7:38 PM
• lbowmaker subscribed.Feb 8 2022, 7:01 PM
• Pchelolo updated the task description. (Show Details)Feb 9 2022, 3:53 PM
• Pchelolo updated the task description. (Show Details)Feb 9 2022, 3:56 PM
• Pchelolo updated the task description. (Show Details)Feb 9 2022, 4:09 PM
• Pchelolo reopened subtask T265846: Turn RESTBase tests for /html and /transform endpoints into api-testing integration tests as Open.Feb 9 2022, 4:21 PM
cscott subscribed.Feb 22 2022, 9:46 PM
daniel subscribed.Mar 15 2022, 3:25 PM
BPirkle mentioned this in T303822: Evaluate API Platform Tasks Related to RestBase Sunset.Mar 17 2022, 6:13 PM
daniel added a subscriber: Nikki.Apr 4 2022, 2:43 PM
• DAbad assigned this task to • nnikkhoui.Apr 5 2022, 1:12 PM
• DAbad added a project: API Platform.
• DAbad moved this task from Incoming to Must do now on the API Platform board.
• DAbad reassigned this task from • nnikkhoui to xSavitar.Apr 26 2022, 1:06 PM
• DAbad added subscribers: • nnikkhoui, • DAbad.

Derick will be taking over this from Nikki.

Restricted Application added a project: User-xSavitar. · View Herald TranscriptApr 26 2022, 1:06 PM
xSavitar reassigned this task from xSavitar to DAlangi_WMF.Apr 26 2022, 1:08 PM
xSavitar subscribed.
xSavitar moved this task from Backlog to Doing [WIP] on the User-xSavitar board.May 4 2022, 9:53 PM
• WDoranWMF added a project: Platform Team Workboards (MW Expedition).May 5 2022, 4:03 PM
• WDoranWMF moved this task from Unsorted pile to VE Backend pile on the Platform Team Workboards (MW Expedition) board.
Krinkle removed a subtask: T308684: Determine directory to place ParsoidStash logic.May 18 2022, 9:52 PM
ppelberg added a subtask: T308511: [SPIKE] Determine necessity of edit session continuity during data center switchovers .May 19 2022, 3:37 PM
daniel triaged this task as High priority.May 24 2022, 8:03 AM
daniel moved this task from VE Backend pile to Doing on the Platform Team Workboards (MW Expedition) board.
daniel added a project: Epic.May 24 2022, 8:13 AM
DAlangi_WMF changed the status of subtask T267990: Support stashing in page/html and revision/html endpoints in MW core from Open to In Progress.May 26 2022, 11:51 AM
DAlangi_WMF changed the status of subtask T301370: Move transform endpoints from RESTBase to MediaWiki from Open to In Progress.Jun 1 2022, 12:01 PM
gerritbot added a comment.Jun 2 2022, 6:32 PM

Change 802586 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] Rest: Remove {domain} param from TransformHandler

https://gerrit.wikimedia.org/r/802586

gerritbot added a project: Patch-For-Review.Jun 2 2022, 6:32 PM
BPirkle moved this task from Must do now to Backlog on the API Platform board.Jun 7 2022, 1:26 PM
daniel mentioned this in T310904: Find out if Varnish is messing with ETags, and what to do about it..Jun 17 2022, 4:20 PM
DAlangi_WMF closed subtask T308588: Do not cache fast parsoid output in ParserCache (exploration) as Resolved.Jun 27 2022, 2:04 PM
daniel closed subtask T310710: Do not ignore HTTP conditionals in transform REST endpoint. as Resolved.Jun 28 2022, 10:44 AM
daniel closed subtask T267990: Support stashing in page/html and revision/html endpoints in MW core as Resolved.Jun 29 2022, 10:55 AM
Eevans unsubscribed.Jul 1 2022, 2:26 PM
daniel raised the priority of this task from High to Unbreak Now!.Jul 27 2022, 10:22 AM
daniel lowered the priority of this task from Unbreak Now! to Needs Triage.
daniel triaged this task as High priority.
daniel reopened subtask T310710: Do not ignore HTTP conditionals in transform REST endpoint. as Open.Aug 4 2022, 10:47 AM
Nikki unsubscribed.Aug 5 2022, 7:55 AM
DAlangi_WMF closed subtask T310710: Do not ignore HTTP conditionals in transform REST endpoint. as Resolved.Aug 17 2022, 4:01 PM
MSantos subscribed.Aug 18 2022, 2:35 PM
daniel updated the task description. (Show Details)Sep 5 2022, 9:44 AM
VirginiaPoundstone moved this task from Backlog to Needs Grooming on the API Platform board.Sep 13 2022, 6:19 PM
VirginiaPoundstone moved this task from Needs Grooming to Radar on the API Platform board.Sep 13 2022, 6:34 PM
abi_ changed the status of subtask T317019: REST page/html endpoints should support language variants from Open to In Progress.Sep 28 2022, 8:08 AM
daniel added a subtask: T320529: Configure VE backend to use Parsoid directly, instead of calling RESTbase.Oct 11 2022, 4:07 PM
daniel closed subtask T310376: Make VE Action API use Parsoid directly as Resolved.Oct 18 2022, 2:53 PM
daniel closed subtask T318393: Parsoid: Maintain context for selser on the server side as Resolved.
daniel closed subtask T314244: Parsoid: make roundtrip-test.js run as a Mocha test in CI as Resolved.Oct 18 2022, 6:24 PM
daniel removed a subtask: T301372: Core Page REST API should follow redirects.Oct 19 2022, 3:02 PM
daniel removed a subtask: T317019: REST page/html endpoints should support language variants.
santhosh mentioned this in T323667: Use Parsoid in Mediawiki Core for Content Translation.Nov 23 2022, 6:50 AM
daniel added a project: RESTBase Sunsetting.Dec 6 2022, 3:35 PM
daniel moved this task from Unsorted to Doing on the RESTBase Sunsetting board.Dec 6 2022, 3:40 PM
daniel closed subtask T310878: Consolidate ParsoidHandler base class with HtmlOutputRendererHelper as Resolved.Dec 6 2022, 3:45 PM
daniel closed subtask T317017: Make the core page/html endpoint return system message content when appropriate as Resolved.Jan 9 2023, 9:25 AM
daniel closed subtask T311728: Gracefully handle non-wikitext pages when returning HTML from the page endpoints as Resolved.Jan 11 2023, 5:30 PM
xSavitar removed a project: User-xSavitar.Jan 27 2023, 12:57 PM
xSavitar unsubscribed.
DAlangi_WMF updated the task description. (Show Details)Jan 27 2023, 1:00 PM
daniel removed DAlangi_WMF as the assignee of this task.Feb 1 2023, 2:50 PM
daniel added a subscriber: DAlangi_WMF.
ppelberg closed subtask T308511: [SPIKE] Determine necessity of edit session continuity during data center switchovers as Resolved.Mar 20 2023, 3:50 PM
VirginiaPoundstone moved this task from Radar to RESTBase Deprecation Roadmap on the API Platform board.Apr 26 2023, 1:28 PM
VirginiaPoundstone edited projects, added API Platform (RESTBase Deprecation Roadmap); removed API Platform.
daniel closed subtask T320529: Configure VE backend to use Parsoid directly, instead of calling RESTbase as Resolved.Jun 15 2023, 2:12 PM
DAlangi_WMF changed the status of subtask T339227: Remove VRS code from VisualEditor from Open to In Progress.Jul 10 2023, 10:41 AM
Yaron_Koren mentioned this in T341131: Replace usage of VRS in non-WMF deployed extensions (case: VEForAll).Jul 10 2023, 1:49 PM
DAlangi_WMF closed subtask T339227: Remove VRS code from VisualEditor as Resolved.Jul 12 2023, 8:15 AM
Pppery removed a project: Patch-For-Review.Oct 10 2023, 5:32 AM
Pppery subscribed.

Anything left to do here?

ssastry reopened subtask T317018: Make HtmlOutputRendererHelper use ParserOutputAccess as Open.Oct 17 2023, 1:45 AM
Pppery removed subscribers: Pppery, • nnikkhoui, • Pchelolo.Oct 17 2023, 2:07 AM
ssastry removed a subtask: T317018: Make HtmlOutputRendererHelper use ParserOutputAccess.Oct 23 2023, 4:56 PM
MSantos closed subtask T301371: Preemptively warm caches for Parsoid output as Resolved.Thu, Nov 21, 12:04 PM
MSantos closed subtask T265846: Turn RESTBase tests for /html and /transform endpoints into api-testing integration tests as Resolved.Thu, Nov 21, 12:19 PM
MSantos closed subtask T267980: Benchmark cached Parsoid HTML endpoints RESTBase vs Core as Invalid.
MSantos closed subtask T301370: Move transform endpoints from RESTBase to MediaWiki as Resolved.Thu, Nov 21, 12:28 PM
MSantos closed this task as Resolved.Thu, Nov 21, 12:33 PM
MSantos claimed this task.
MSantos removed projects: Platform Team Workboards (MW Expedition), Platform Engineering Roadmap, Platform Engineering Roadmap Decision Making.

At this point, VE is already using Parsoid directly without the REST layer as per T320529: Configure VE backend to use Parsoid directly, instead of calling RESTbase. The remainder tickets here are not needed for the epic completion so I'm closing it. Please let me know in case I'm missing something.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits