Page MenuHomePhabricator

RFC: 2019 Process amendments
Closed, ResolvedPublic

Description

Motivation

The 2017 revision to the RFC process (diff) simplified things enormously. The barrier to entry is now low, with literally the only requirement being to create a task, and there's no unnecessary formalities required on the author's part.

But, I also believe this change hasn't helped RFC authors and stakeholders to be able to anticipate what will happen between now and approval.

Objective and scope

Take what we've learned in the past three years, and add a bit of structure back to our process.

  • It should be naturally clear what the stakeholders likely want to know about an RFC, who these stakeholders are for a given RFC, and at what point they need to know the things they want to know. (For example, it is not needed to have a solid plan and proof of concept at filing of the RFC.)
  • RFC authors should gain greater autonomy, and receive a quicker overall process by knowing what is happeniing now and what will happen next, without them needing to ask or be told by someone.

This is meant to be a retrospective and inform an interative change to our process. To constrain ourselves:

  • No new requirements for what's needed at intial RFC task creation.
  • No added new process overhead for RFC authors to follow.

Exploration

Initial thinking points
  • What are are questions that participants (esp. TechCom members) ask on almost every RFC? These are good things to consider adding into a boilerplate, FAQ, or other self-discoverable process.
  • What are the steps that participants expect to be followed on most RFCs (e.g. define and consult stakeholders before scheduling meetings)? We should make it hard to mistakenly think that a step has already happened. Also try to make internal triage work easier to rotate and transfer for TechCom internally.
  • What steps have we forgotten during past RFCs? What steps should have happened earlier or in a different order? We can amend the process in a small ways to optimise against such inefficiencies.
Proposal
Draft of the process page with the below changes applied: https://www.mediawiki.org/wiki/Requests_for_comment/Process/Draft

Notable changes:

  • Remove the Inbox and Backlog stages.

The Inbox was blocking authors early on, awaiting response from a weekly triage. Instead, there is now a task template that covers what TechCom would typically ask for, as well upfront documentation of what happens when and in what order.

Our previous process also did not account well for when the initial pieces of information (problem statement, resources) are not yet specified. It handled this with a Backlog column that was placed left from the Inbox on the workboard, as such it was effectively a negative (-1) phase.

Instead, when the task is filed (using the template) the process starts in "Phase 1: Define" where the focus is on these bits of information. And once filled out, the author moves it forward (no need for TechCom involvement). The same applies to "Phase 2: Resource". Authors are encouraged to proceed through the phases on their own whenever possible.

  • Replace "Under discussion" stage with "Explore" and "Tune" phases.

The exploration phase is when stakeholders are notified and asked for input. It is also where proposals should be drafted (doing that earlier is still possible just as well, but this is the point where it is explicitly focused on, and needed before moving forward). In the old process it was not clear when proposals are expected, and that uncertainty sometimes led to delays when RFCs would not be filed until after proposal(s) were already iterated upon elsewhere.

The tuning phase is where the authors and other participants collaborate and iterate on the proposal(s) until the requirements and raised concerns are addressed. This now also makes explicit mention of the Architecture Principles.

  • Last Call changed from "at least one week" to "two weeks".

Two weeks is what we've used in practice and I suggest we codify this to set a clearer expectation. If there is a reason to wait longer for some reason, I suggest we not start the Last Call yet. I've also added that no more than 2 RFCs should be on Last Call at any given time to ensure stakeholders have decent time to review and consider their impact.

Event Timeline

Here's a random idea:

  • Allow no more than 2 concurrent RFCs on last call. And, no more than 1 last call ending in the same week.

Reason being to make sure everyone can have the time, focus, and opportunity they need.

There is also the change that Marko proposed: require the RFC to be on phabricator, with the option to reference wiki pages; and require teh RFC discussion to be on phabricator, and discourage on-wiki discussion once discussion on phab has started.

Here are a few ideas I have:

  • provide a checklist for "when should I file an RFC"
  • provide a checklist for "what should be in an RFC document"
  • consider a last call with no feedback from outside TechCom failed. Positive feedback would be required for an RFC to be approved. This ensures that we solicit input from relevant stakeholders.
  • make explicit that it is TechCom's job to solicit input from relevant stakeholders. Presently, we usually just hope that relevant people would read the radar email and react.

It would nice if there was cleaner differentiation between RfCs blocked on lack of consensus, lack of interest, lack of resources to implement, or lack of time for discussion (as in, waiting for an IRC discussion to be scheduled, which can sometimes take a while).

Krinkle moved this task from P1: Define to Under discussion on the TechCom-RFC board.
Krinkle triaged this task as Medium priority.Mar 13 2019, 1:31 PM

Note to self:

In the future we may have a few boilerplate questions for the RFC author to answer on Phabricator before scheduling a requested IRC meeting. Once we've tried this out and refined it a bit, we can also document those publicly on the RFC process page for better discovery.

Depending on when that happens, it can either ride this amendment or be part of a later revision.

Another note based on a comment from @Milimetric in today's TC meeting.

Our process doesn't have clear path for how to close stalled RFCs. We are currently encouraging RFCs to be started very early on, before there is known resourcing or product support. The process currently states that TechCom can move these to the backlog where they reside until resourcing, product support and a problem statement exist.

The only paths to closing are:

  • Upon triage, when the RFC is out of scope (per TechCom's charter).
  • Via Last Call, if TechCom declines the problem statement and/or technical proposal (once either exists).
  • By the author if they withdraw the RFC.

This means feature ideas for which an RFC was started early on by an individual that doesn't own the affected product, remain open indefinitely in the backlog. See T91137 for an example.

Do we want to allow for closing an RFC in this stage? If so, on what grounds? And what about RFCs in this stage that are not old, do we still want to encourage use of the workboard for new product proposals in an early stage? (I'd say, yes, that's still desirable and useful to get involved early on).

Do we want to allow for closing an RFC in this stage?

Does not seem any different to me from closing tasks on the grounds that team X does not intend to work on it anytime soon... which has not been popular. Is there any practical benefit from closing them? You can hide workboard columns, replace the project tag etc.

@Tgr In your example "closing an RFC" is not the same as "closing a task tagged TechCom-RFC". If the task represents only an RFC, then closing it would mean closing the task, there is no reason not to. If the task in question tracks other work, or tracks a general intent for a product or feature, then "closing the RFC" would mean removing the tag (or moving to a certain column). In both cases, it can always be re-tagged or re-opened.

The question is, whether we want to allow for this at all, to "closing the RFC process" for an RFC in the backlog that did needs product approval but has no recent participation from a relevant product owner.

The added value of doing so would be to reduce workload of triaging the backlog for us. As well as for all other participants to have a clear view of what is happening and where they can participate and contribute.

Maybe we should just have an "attic" column.

Maybe we should just have an "attic" column.

1. Or a "no resources" one :P

The process should ensure that at least one TechCom member spends some time digging into the RFC before it goes to an IRC discussion or a Last Call. In the past, some of us usually did that at some point, but we don't presently have any mechanism to ensure this. I think we should.

TechCom is hosting a IRC meeting on this: 21 August at 2pm Pacific/23:00 CEST/21:00 UTC in #wikimedia-office

tagging so I know to leave a comment per discussion with @kchapman

Random thoughts about my experiences with the RFC process:

  • People showing up for the RfC meeting were wholly unprepared and wasted a LOT of the very short time slot asking basic comprehension questions that should have been addressed on the talk page before the meeting
  • Speaking of the meeting, 1 hour is seriously not enough time to have a cogent discussion about something as big as an RfC
  • The process to get an RfC through the track is very poorly defined. Ask 3 different people what you need to do in order to move the RfC to the next stage and you'll get 4 different answers
  • Forcing the RfC to live in a bajillion different places makes communication difficult. Why do we need to have discussion on phab but content on mw.org? Let's just use phab as a placeholder with a link to the real content, so that phab stuff can happen with that task, but all discussion lives on mw.org between the RfC page and the talk page
  • Ocotillos are pretty and huggable
  • Communicate with the stakeholders (aka include the RfC authors in this) BEFORE scheduling the IRC meeting, to ensure that a time is chosen that works for everyone. It's frustrating when IRC meetings are scheduled for things that I'm involved in at times that I'm literally unable to attend
  • I see above that you want to move all the discussion to phab instead. That is a massive mistake. Barrier to entry on phab is higher (not easy to create an account and contribute compared to on-wiki), and it's impossible to have multiple independent discussions on phab whereas it's super easy to have threaded discussions on mw.o

Based on my experience with two RfCs this year, two things stood out:

  1. People don't actually read the RfCs before discussing them. Depending on the level of their involvement in the discussion, this wouldn't necessarily be an issue, except:
    1. This is apparently the expected norm for the IRC meetings.
    2. Some parts of MediaWiki, nobody (at the WMF) is really familiar with it to begin with, and you can't have a meaningful discussion about something nobody understands, has read into, or knows what's actually being proposed. We explained, in detail, the use cases. We linked to dozens of examples of demand for the functionality. We included sample code and examples of usage, and described the limitations with the current software. We were then, in the meeting, asked questions we had already answered, in detail, in the RfC itself, and not realising the people asking perhaps hadn't actually read it to begin with, stumbled to try to come up with better, clearer explanations on the assumption that they had and it what was there wasn't good enough. Much of what was discussed, the questions and recommendations, simply made no sense in the context of what we were trying to resolve or proposing, and it was incredibly frustrating trying to get the discussion back on topic. In the end, we were suggested to do something that was not only not feasible as far as we could tell having looked into it and worked on and maintained the relevant software for years, but whose very infeasibility had basically been the entire premise of what the RfC was seeking to resolve from a technical standpoint.
    3. After a point, it's just downright disrespectful.
  2. A double standard between how WMF- and third-party-written RfCs are handled and move forward:
    1. WMF written RfCs, if nobody shows up to object, they move forward.
    2. Third-party written RfCs, if nobody comes out to supports it, no matter how much we've tried to reach out to potentially relevant parties, it's not good enough and requires more feedback in order to move forward. Even when it turns out the main reason we haven't gotten any meaningful feedback is because we're the ones most familiar with this part of MediaWiki and its ramifications to begin with. Bonus points when there are a few other people within the WMF who probably could meaningfully comment, but they don't want to get involved after someone from another team left some objections that as far as we can tell aren't even based on the proposal or sample code to begin with (and this person hasn't come back to explain any of it, either, despite being asked multiple times).

Basically the whole thing comes across incredibly WMF-centric, and it doesn't feel like there's any room left for actual... requesting comments. As a volunteer and third-party developer, I'm not even sure I see the point trying at all anymore, as things stand.

Some thoughts said to @kchapman during the meeting over PM

22:21:31 <RhinosF1> TechCom should be able to un-tag RfCs no-ones interested in etc. and/or decline the task.
22:22:08 <RhinosF1> and when things are stalled then there should be options to untag/decline or call attention to it
22:43:53 <RhinosF1> yes, RfCs should be all in one place :)
22:44:56 <RhinosF1> as brion said makes it easier to read and for everyone to find stuff and work out what's going on

(Note from WMTC19: Dan has the session notes, Daniel took pictures, they are in the TechCom folder)

debt renamed this task from RFC Process: 2019 amendments to Wikimedia Technical Conference 2019 - Unconference - RFC Process: 2019 amendments.Nov 14 2019, 10:43 PM
debt assigned this task to daniel.

Notes from session at Wikimedia Technical Conference 2019, Atlanta

Process: Everyone writes down: question about RFC process, hate about *, like about *, would like to change about *

Attending: Gergo, Moriel, Ricardo, Antoine, Stephen N., Joaquin, Leszek, Piotr, Niklas, Niharika, Daniel, Dan, Leon

neighbor pairing

  • (everyone can add their notes here later)

scary, long, time consuming, odd hours for timezone (Europe), alternate times are both too far outside working hours
clear, transparent, follow rules, clear way to decide what needs to be an RFC, decisions publicized better
guide for success, efficient, effective, avoid RFC hell (driving process forward)
third rotation meeting time
tech com meetings behind closed doors
open, community-oriented
need more outreach & more accessible materials
does TechCom consider it a good use of their time? (Could it be run by less skilled people with more time?)

Q: why only once in a while? should it run more often?
Hate: lacks programmatic (?) to ensure implementation follows
Like: Offers a point of sync between team / tech project
Change: Make it a governance system that drives a vision ( dedicated PM)

Q: What's its purpose?

		when do we do RfCs vs. not?
		Who makes decisions?
		How do we create an RfC for success
		Needed features - feels judgemental

Hate: No guidance; write RfC and "defend yourself"

		Navigate decisions / plans on your own
		Features are dependent on these & feeks uncertain and long & hostile

Like: Perspectives from different angles

		Having to think beyond the specific feature
		Encouraging high quality
		Pacing product overhauls
		Opportunity to get expert review

Change: Get guidance ahead of the process itself

		Positive attitude rather than feeling like judgement
		Take into account speed / iteration of product
		Look at new security process for inspiration

alternate neighbor pairing

guidance ahead of time, little more conscientious to constructive criticism, "let's make it happen" attitude, process is vague, meaning vague

  • lack of project managers before and after the RFC, prep so we have something solid when we get to the RFC, inclusivity is more theoretical

inclusive process

why is RFC process needed?, any alternatives?

  • too long, noise, instead of answers we get suggestions, prefer direction, end up with more options instead of fewer

lots of feedback from smart people
improve: too many cooks, be very clear on requirement for an RFC, discussion burnout

process is vague (purpose / how / who decides)
no shared understanding
lack of project management / guidance before & after the RfC
an inclusive process of expertise sharing

present aggregate

Group 1

  • process is vague, no shared understanding, are there any alternatives?
  • too long, lots of noise, lack of guidance from product before or after (took so long that everyone found another job)
  • open spirit, feedback from lots of smart people
  • clarify when RFC is needed, guidance on how to prepare for the process
    • have a product request - change fundamental something in core, need insight from expert people, going in don't know exactly what it involves, but attitude in the IRC meeting is very judgemental, which is fine to find problems but not fine if that's your first interaction with your idea. Better if we can find someone with an overview of the system to help us design the system in a way that it won't be ridiculed when it gets to the IRC meeting. Love to see more "let us help you set up for the RFC process" than "let's see if you idea holds up to lots of people poking holes at it".

Group 2

  • how can we be more successful and avoid abyss where they stay open forever, decide when something needs to be an RFC, can this be expanded to ToolForge tools
  • IRC meetings does not nurture by design, time consuming, state of RFCs is not always clear
  • like that the RFC process exists, somewhat transparent with IRC meetings
  • clear rules for what needs to be an RFC, clear next steps, require participants to agree that the problem exists and get feedback weighed based on how affected you are by the problem

Video meeting with Moriel et al on delayed notifications was good, an example of the guidance suggested above

Ricardo: wondering if there should be two states: pre-approval where you discuss and set up, then concrete RFC proposal where more criticism than guidance is welcome

Moriel: suggestion re: security team. Anti-harassment team starting to work with delicate things, went in talked to security. New process where they don't want to say no to a need, we want to identify risk and mitigate. Analysis phase would be very interesting for tech-com to do as a pre-RFC phase. Attitude is a lot more welcoming.

Daniel: agree with the attitude. Poking holes is unpleasant, maybe necessary, but should not be the start of it. Tech Com doesn't have much time to do too much guidance.

Moriel: we should fix this lack of time. As gatekeepers, you should have time to do that role.

Leszek: would rather throw out the process than have it be a productivity killer. Asked to go through the RFC process, and after initial exchange the task was just sitting there for 3 weeks, and we halted work while waiting and hearing nothing back.

Gergo: negative experiences were different. Makes me wonder if some things should not be RFCs. We used it for technical changes where we understood change very well, but connection between the RFC and the change was not very obvious. Usually a complete failure when we don't have the capacity to explain the change so that people in the RFC process could understand. Instead of RFC would prefer outreach and inclusive practices like technical writing to explain.'

Moriel: Tech Com needs to understand you're driving it, be explicit about expectations of who does what. Who drives, for example, can be very confusing and generating misunderstandings.

Antoine: for code review on landing code in production, had too many people proposing things, only handful of people able to approve. Instead of having a few people being the bottleneck, adding gerrit pre-commit workflow so people pile-up in a queue to guard production. The way I understand the complaint about the RFC process is you need guidance quickly on a product that you are looking to work on. Lots of preparation work that needs to happen. Tech Com is busy, so you could imagine that Tech Com people could focus more on RFCs, providing consulting services, have more time for guidance.

Daniel: lot of thinking about making the committee itself being more inclusive.

daniel renamed this task from Wikimedia Technical Conference 2019 - Unconference - RFC Process: 2019 amendments to RFC Process: 2019 amendments (also: Wikimedia Technical Conference 2019 - Unconference).Nov 15 2019, 8:11 PM
Krinkle renamed this task from RFC Process: 2019 amendments (also: Wikimedia Technical Conference 2019 - Unconference) to RFC Process: 2019 amendments.Nov 28 2019, 1:47 AM

Two ideas that came up at TechConf:

  1. (My own originally) introduce a bit of boilerplate to the task description for new RFCs, this would be optional to fill in initially (same as now) but required to proceed to the "Under discussion" stage. Basically same as now but with our most common questions / concerns documented upfront instead of being asked by me ad-hoc. This was well-received from what I could tell. But some of the other ideas I think are actually better, see point 2.
  1. Integrate the different requirements and expectations at the top level of the process itself (our workboard columns). So instead of having things like "Define problem, resourcing, and stakeholders" be form fields, they would be stages in the pipeline. We could still document the outcome of these stages in the task description, but it would be more hands-on, and it would be more clear in what order we do them, and they they are required. This also makes it easy for participants to see at a glance where we are at for a given RFC, and in the other direction (lacking today) to see which RFCs have reached a given stage.

For example:

  • Stage 0: Inbox.
  • Stage 1: Problem space. (Define the problem space, criteria, outcome.)
  • Stage 2: Resourcing & Stakeholders. (Ensure agreement from relevant product owner, confirm rough plan for implementation/maintenance, or document interest to volunteer-maintain the code and assign a code steward as fallback. At this stage we would also document from which stakeholders we require input from, e.g. CommRel/Legal/SRE/Third parties/MW devs, etc.). Example at T232563#5485923
  • Stage 3: Awareness & Solutions. (Make an explicit effort to reach out and hear from the wider developer community. This is also where we start gathering ideas for different ways it could be solved.)
  • Stage 4: Consensus & Tuning. (Second feedback round. Narrow down the solution. Verify with the stakeholders and other participants that concerns are addressed).
  • Stage 5: Last Call.

Tim mentioned in today's meeting we can use this Phab template as inspiration:
https://phabricator.wikimedia.org/maniphest/task/edit/form/46/.

Krinkle renamed this task from RFC Process: 2019 amendments to RFC: 2019 Process: amendments.Feb 19 2020, 9:52 PM
Krinkle renamed this task from RFC: 2019 Process: amendments to RFC: 2019 Process amendments.
Krinkle claimed this task.

Tim mentioned in today's meeting we can use this Phab template as inspiration:
https://phabricator.wikimedia.org/maniphest/task/edit/form/46/.

A quick way to prototype pre-filled task templates is to use https://tools.wmflabs.org/phabulous/ to construct a custom URL. Fancy forms like the one linked require custom code work from @mmodell as far as I know.

This incremental change is on Last Call until March 28th (notes).

In the past week I have reached out to a number of people, including those involved in the conversations that informed the current draft. I've heard back from most and so far it's been positive or neutral toward the specific draft we have on Last Call. I did receive a number of other considerations for what we can do more, and I'll definitely take that into the next iteration later this year.

@Krinkle / @daniel / @Milimetric: Thank you for proposing and/or hosting this session. This open task only has the archived project tag Wikimedia-Technical-Conference-2019.
If there is nothing more to do in this very task, please change the task status to resolved via the Add Action...Change Status dropdown.
If there is more to do, then please either add appropriate non-archived project tags to this task (via the Add Action...Change Project Tags dropdown), or make sure that appropriate follow up tasks have been created and resolve this very task. Thank you for helping clean up!

Oops, sorry. There are already sufficient tags on this task. Please ignore my previous comment!

This incremental change is on Last Call until March 28th (notes).

In the past week I have reached out to a number of people, including those involved in the conversations that informed the current draft. I've heard back from most and so far it's been positive or neutral toward the specific draft we have on Last Call. I did receive a number of other considerations for what we can do more, and I'll definitely take that into the next iteration later this year.

Can this be closed as resolved?

This hasn't been formally approved by TechCom, yet.