imo there is value in having both closed and open vocabularies for finding content. I'd hate to deal with proliferation of terms to find cargo plugins.

@epage I feel like this example is actually one in favour of removing the existing closed vocabularies.

For example, right now, there's one category for cargo plugins, which contains over 350 crates. This gives some indication that instead of cargo being a single category, it should be multiple. But with the existing vocabulary, this isn't an option, since we can't retroactively remove categories. The other option is to make categories have arbitrary nesting, which can be quite messy.

So, instead, crates can choose to add to the existing cargo plugins subtag (development-tools::cargo-plugins), but also to a new cargo plugins tag (cargo-plugins) which can be split into subtags (like cargo-plugins::testing) as a way of making it easier to identify cargo plugins by purpose. Crates that adopt this new convention can both identify themselves as cargo plugins while also differentiating based upon purpose, something that couldn't be done under the old model.

Sure, we could just spend hours fiddling with the categories as examples like this show up. But the point is that they will always show up, and I think that it'd be more ideal to decouple this fiddling from the actual technical standard, rather than it being an inherent part of it.

While the existing development-tools::cargo-plugins can be used as a tag in the future, we have no scaffolding in place to help these kind of canonical tags from cropping up in the future. Categories are already not that easy to discover and define in manifests, keywords are even worse off. While its not perfect, the closed vocabulary is filling an important role and I don't think we should remove it until we've filled that gap with a viable alternative.

Right. The point here is that this is suggesting that alternative.

Changing vocabulary is something that cannot and should not happen by decree. It has to happen organically in stages. Namely:

1. People identify that an existing category is not sufficient.
2. A small subset of people suggest a new category and start using it.
3. A larger group of people start adopting that category until it reaches critical mass.
4. The old category is officially replaced.

Since a crate can have multiple categories, this is something that's entirely doable without losing the old categorization. The difference here is that by making the canonicity of a category an emergent property (modified on the side of crates.io), and not one explicitly labelled (in Cargo.toml), we have the option of trying things out. Under the current system, we have either this bad option:

1. People identify that an existing category is not sufficient.
2. They make a PR adding a new one.
3. People slowly start using the new one.
4. The old category sits around forever, confusing everyone.

Or this bad option:

1. People identify that an existing category is not sufficient.
2. They add a keyword instead.
3. People start using the keyword, until everyone is using it.
4. A PR is made to instate the keyword as a category.
5. Now the community has to keep using both the keyword and the category until the category is used by everyone.
6. The old category still sits around forever.

This feels obvious to me, based upon the text of the RFC. Is it not? Have you read it?

This is not suggesting what I feel is needed: guard rails to help in working towards de facto standardized tags. The problem is with discoverability for crate authors. The RFC mentions noteworthy tags which helps with some of the problem but not all. We're asking crate authors to do SEO and giving them a limited tool that is managed by a small team of people (crate authors with sway and crates.io) and are putting it outside of their workflow. I think itd be good to do an analysis of how we are doing on categories and keywords and find ways to improve those as a testing bed for if we can handle exclusively open vocabulary.

The current state of categories and keywords in crates is poor, but OTOH I don't think the root cause is in what they're called, or in which toml field they're specified.

When you have a mix of blessed and arbitrary tags, you still have categories and keywords, only merged into a single namespace, and you lose control over the category namespace.

The current categories are an odd bunch, with some large gaps, and easily confused siblings.

• there's neuroscience, but no biology.
• there's a bunch of categories for aerospace. There's nothing for IoT or home automation.
• there's both localization and internationalization, which is a slim difference.
• there's no category for fonts, chat clients or chatbots, antivirus software, pentesting software.
• nearly everyone uses "parsing" when they should have used "parser-implementations".
• there's "compilers", which is a bit too specific for non-rust programming languages in general, interpreters, shell scripting.
• "algorithms" is an incredibly broad category. Lacks subcategories for hashing, random, search.
• developer-tools and command-line-utilities are also pretty broad (everything is CLI, from cat to TUI spotify clients)
• there's no place for applications with a GUI. "gui" is for toolkits, and categorizing by functionality often lacks specific categories.
• there's computer vision, but not machine learning.
• text-processing spans everything from markdown to nlp.
• there's no specific place for clients for specific web APIs/services (a webby equivalent of external-ffi-bindings). the web-programming category is quite broad.

Few crates actually select categories, so browsing by category on crates.io gives an incomplete picture. However, keywords are also a mess! They're also pretty sparse.

Dealing with synonyms and normalization is laborious. There's data-structure, data-structures, datastructures, ["data", "structure"]. Normalization of these is not as easy as it seems, e.g. kebab-case makes iOS into i-os and SSDs into ssd-s. Some names and acronyms appear plural, but aren't: bio != bios, hdf != hdfs. On lib.rs I have 4300 synonyms, 7000 categorization exceptions, 2000 associations between keywords and categories.

I've tried using StackOverflow's tag data, but they actually have very little data for tag synonyms, and their perspective is weirdly specific to their site, focused on SEO, .Net and webdev product names (webform -> winforms, jws -> java-web-start, ctrl-c -> copy-paste, span -> html). They maintain descriptions/wikis for their tags, so they can afford having ambiguous tag names.

Making good categories based on keywords is surprisingly hard. I thought it'd be easy to have everything categorized on lib.rs, but it's been an enormous time sink and endless whack-a-mole. "geo" is for geometry or geography, sometimes both. A crate tagged ["layout" "algorithm"] can be for GUI (widgets or window manager), or struct layout in rust-patterns or FFI, or be an app for visualising charts, or get lost in the bottomless pit of the generic "algorithm" tag. A crate tagged "database" can be a database engine, or a client for a general-purpose database, or a client for a dataset (postcode database), or a dataset itself (unicode property database). If you pick one broad tag, you have a mishmash. If you pick one specific tag, you have sparse data. If you try to pick combinations of tags, you have an endless list of missed cases, and exceptions to exceptions.

So overall I think this change is just rearranging fields, and improvement of categorization needs other work, that doesn't really require tags.

• Many crates have zero categories and zero keywords. They won't have any tags either. There needs to be some mechanism to nudge authors to add metadata to their crates.

• Authors seem to have trouble finding the right categories, and don't always pick the most appropriate ones. They seem to have difficulty understanding difference between similar categories, or difference between technical and general meaning of category names. acessibility is the worst example — it seems to be for "I can access something with this crate". IMHO they'd benefit from more guidance, e.g. an interactive category picker that explains the terms or asks clarifying questions. StackOverflow has an interactive picker for their tags with descriptions, plus crowdsourced moderation. Unprivileged SO users can't pick arbitrary tags, so they're really categories in disguise.

• crates.io would benefit from reorganizing and growing the categories in a systematic way. This is equally needed whether they're category slugs or blessed tags. Picking tags to bless is not that different from analyzing keywords to pick new categories.

I don't know how relevant it is, but the tag curation effort that immediately came to mind whilst reading this is the AO3 tag wranglers, as they seem to have a "moderation team" style group dedicated to the curation of aliases and hierarchy.

I realise now that there's probably something more equivalent in the Wikipedia categorisation moderation space, which might be more applicable due to the public nature of Wikipedia governance?

because categories cannot be removed, they also cannot be renamed or otherwise curated by the community.

Categories can be removed and renamed. crates.io could ignore existence of slugs it doesn't want, and create aliases for category slugs it wants to rename. On lib.rs I've completely deleted external-ffi-bindings, and renamed math to science::math.

we talked through this RFC in the crates.io team meeting today. I'll try to summarize:

• the team is in agreement that we should keep keywords as they are instead of introducing a new "tags" concept.
  • introducing a whole new concept means a lot of additional work for the crates.io team, but also on the cargo side. since the benefits from this new concept are relatively slim, we would prefer to avoid that.
  • it would be great if the RFC could be amended accordingly, so that we have a place to discuss the finer implementation details.
• the team is generally in favor of dropping categories, though some of the migration details will have to be discussed before we can move this RFC forward.
  • when people publish their crates, do we want to automatically convert the categories values to keywords for them? do we just ignore the categories?
  • how do we want to handle existing crates that only have categories, but no keywords? how many such crates are there?
  • do we need to increase the limit on the number of keywords per crate?
• one internal implementation detail that was not mentioned in the RFC: the crates.io codebase is currently using the ltree PostgreSQL extension to implement nested categories. when moving away from nested categories towards flat keywords our code to support this would become quite a bit more manageable :)
• regarding the work involved in curating the list of noteworthy tags vs. categories and any potential bias there: the major difference is that people can start to use keywords before they become noteworthy, while with categories they only become usable once the crates.io team has added them to the database. this makes it possible for us to retroactively promote keywords to be noteworthy after we have seen them actually used a lot by crate authors. this can avoid situations where e.g. https://crates.io/categories/science::neuroscience only has a single crate.
• with the crates.io team in favor of the change, it is up to the cargo team to raise any objections from their side. with the change of keeping keywords, and not introducing "tags", the work on the cargo side should be relatively small though.

@rust-lang/crates-io I hope I summarized this correctly. If not, please let me know :)

Thank you for the detailed feedback! I'll try to address these over the weekend and remove the new "tags" in favour of just keeping keywords.

In terms of migrating categories over, my thought process was that categories would be implicitly treated as keywords, but we can work out the details on that after the revisions.

Okay, the big revision is done, and tags are no more, replaced with keywords. Please let me know if anything seems off; I may have overzealously search-replaced "tags" with "keywords" in ways that might not make sense.

Subtags are now "commonly paired" keywords, which allow the crates.io team to create some kind of keyword structure without there actually having to be one. There are probably still details to flesh out, but that's what the RFC process is for.

Thanks to @Turbo87 we now have a completely revised version of this RFC which adds historical context and some more analysis. It's still probably going to be changed before the final version, but hopefully things are a lot clearer now!

Personally, I think this is still premature and we should be trying out some of the ideas mentioned here to vet them out before we commit to relying on them to make up for the loss of a closed vocabulary.

I definitely think that there's room for doing a "soft" rollout to these features after the RFC is merged, since they by their nature depend on the community using them to work. I don't think that even if we accepted this RFC as-is that we'd be locked into the approach, although I'm also not sure what the general policies are for the cargo and crates.io teams to experiment before an RFC is accepted.

I think that experimentation is especially important here since we don't know 100% how the community will respond to the changes, but the primary idea (replacing a closed vocabulary with a curated open one) is still there. What kind of vetting do you think should be required to make a change like this?