Before you start building with AI, make sure the product or feature that you have in mind requires AI, or would be enhanced by it.

AI is well-suited for applications like:

• Recommending different content to different users, such as movie suggestions
• Predicting future events, such as weather events or flight price changes
• Natural language understanding
• Image recognition

A rule or heuristic-based solution may be better when:

• Maintaining predictability is important
• Users, customers or developers need complete transparency
• People don’t want a task automated

See the User Needs chapter for more on when to use (or not) AI.

Because AI systems are probabilistic, your system will probably give an incorrect or unexpected output at some point.

This makes it critical that you help users calibrate their expectations about system functionality and output. Do this by being transparent about both its capabilities and limitations.

For example, indicating a prediction could be wrong may cause the user to trust that particular prediction less. However, in the long term, users may use or rely on your product more, because they’re less likely to over-trust your system and be disappointed.

AI as a technology has unlocked a wide array of new, and often exciting, products and services. But no matter how novel your use of AI, when explaining your AI-powered product to your users, focus primarily on conveying how it makes part of the experience better or delivers new value, versus explaining how the underlying technology works.

Run user studies to answer questions like:

• What do users need to know about how the system works to understand and use your product?
• Why is this knowledge useful to the users?

The level of technical detail needed in your explanation will vary depending on the product and users.

From early in the product development process, plan for the fact that your AI system will make bad predictions at some point. This is an important part of confirming that AI is the right technology for your project.

Think through the types of errors that your system could make, and their consequences. You should have an informed point of view about what's at stake for your user for a given error and expected impact of false positive and false negative predictions.

Plan to remediate for such errors using approaches like:

• Setting users expectations about your system with explanations
• Providing manual controls when the AI fails
• Offering high-touch customer support

AI-powered products can suffer without the right level of focus and resources on data from early on. We call such downstream effects “data cascades” and they can be hard to diagnose and detect until your product experience is impacted.

Good planning and scrutiny of your dataset can help you avoid issues downstream. Some actions that you can take include:

• Collect data in batches
• Embrace “noisy” data
• Plan for data maintenance
• Partner with domain experts

When designing your AI-powered product, evaluate the tradeoffs between having the set of recommendations your system gives to be broad (prioritizing recall) or accurate (prioritizing precision). This decision will have a big impact on the end user experience.

• Prioritize precision: If you are building a product in a high stakes domain (such as healthcare), and the risks stemming from an error are high, you will likely want to be more conservative with your model output, and only give recommendations for the things the system is most confident of. This means prioritizing precision.

• Prioritize recall: if your product is in a lower stakes domain and it’s not a problem for users to sort through a longer list of recommendations (for example, music recommendations), you might prefer to prioritize recall. This means your user will get a larger set of results, offering the chance for surprising or serendipitous discoveries, but some of them may be less relevant.

Many AI systems rely on user data to personalize their recommendations or optimize the system.

If you are collecting user data, proactively ask users for permissions early on, and make it easy for them to adjust their settings.

Over time, you might also need to remind your users of the settings that they’ve chosen so that they can update to match their current preferences.

When prompting users to set or review their permissions, explain what you’re using their data for and why, and make sure your explanations are easily understandable.

AI systems often require access to some data or input from their users to provide them with personalized recommendations.

Before asking new users what and how much data they are willing to share, give them the opportunity to test your system. For example, this can be done by offering an initial experimentation experience.

This will help new users who are eager to get started, or who don’t have time to fully consider and configure their preferences before diving in. It will also support users who are wary of sharing this information before they fully understand what the system offers in return.

Even after onboarding is complete, continue to make user actions and decisions reversible where possible. User preferences and context are always changing. Giving piecemeal undo options allows them to manipulate preferences recommendations without needing to resort to a nuclear reset option.

With AI-driven products, there can be a temptation to communicate the “newness” or “magic” of the system’s predictions through its UI metaphors.

However, unfamiliar UI touchpoints can make it harder for users to learn to use your system, potentially leading to degraded understanding of, or trust in, your product, no matter the quality of your AI output.

Instead, anchor new users with familiar UI patterns and features. This will make it easier for them to focus on the key task at hand, which is building comfort with, and calibrating their trust in, your system’s recommendations.

In high stakes domains or when users are new, it can be useful to provide users with contextual information from other people to help them determine the value of your recommendations. Sources of this kind of contextual information can include:

• third-party experts, whose information can help users vet the quality of your output
• relevant community groups, who can help users calibrate their trust through human-to-human connection

This additional context, in tandem, with explanations and confidence displays, can help set user expectations about the quality of your recommendation.

In some situations, you can help users gauge how much trust to put in the AI output with model confidence displays that explain how certain the AI is in its prediction, and the alternatives considered.

However, in other contexts, confidence displays can be challenging for users to understand.

If you decide to use them, test different types of displays early in the product development process to find what works best for your users.

When explaining recommendations from your AI system, focus on sharing the information that users need to make decisions and move forward. Don’t attempt to explain everything that’s happening in the system.

Often, the rationale behind a particular prediction is unknown or too complex to be summarized in a simple phrase or sentence. Users may also not want to be overwhelmed or distracted by superfluous explanations as they use your product.

The Explainability Trust chapter offers examples of different approaches for crafting succinct, user-friendly explanations, which include partial explanations, progressive disclosure and model confidence displays.

If you’d like to share longer or more detailed explanations of how the overall system works, do this outside of the active user flow, for example in marketing materials or onboarding content.

It can benefit your users and your product to provide deeper explanations of how your product works. This type of explanation can help users strengthen their understanding of how your system works, and how they can use it to best meet their needs. Having this level of understanding can also help them give you more robust feedback on your product.

Deeper explanations work best when shared outside core product flows, where they can be distracting, and can include:

• Marketing copy, for example on your website or product packaging
• Onboarding materials
• Help center materials or educational content explaining how your product works (an example is this 5 minute video of how Google Search works)

Remember to target these materials to their context, and to update them as your product evolves over time. For example, users have different goals when they are onboarding than when they are reading marketing materials.

Such explanations will be different from or in-the-moment explanations, which should be short and optimized for understanding.

When determining how much to automate your product flows, think about the stakes of your product, and the level of comfort that users may have with your type of product.

In low risk, well-established products, like content recommendation systems, you might choose to prioritize a more heavily automated product flow where user control is available but optional.

However, when onboarding to a new type of product, or in high-stakes situations, errors can be particularly problematic, and can corrode user trust and potentially cause dangerous situations. In such cases, design your system to give users more control over the system.

When your AI-enabled system behaves in a way that a user doesn’t expect or want, make sure that they have an option to share feedback. And, as much as possible, use that feedback to improve your model.

Feedback in AI systems can take a range of forms, including,

• Giving a thumbs up or thumbs down on a recommendation
• Hiding unwanted recommendations
• Flagging or reporting problematic recommendations
• More traditional feedback flows, where a user manually reports a problem through a form or other mechanism

Once a user gives feedback, acknowledge that you received it. If possible, let them know how the system will respond to the feedback.

Allow your users to supervise automation and take back control when needed, rather than automating across the board.

This is helpful to users in a number of ways, including:

• Building comfort in new or high stakes situations with the support of controls to override the system if needed.
• Learning how the system works as they exercise their controls. For example, by manually confirming an appointment time recommendation, they become more familiar with the system’s appointment recommendations.
• Giving them a way to complete their task when the system doesn’t work as intended.

Even in cases where users may not frequently exercise the option to take back control, it can be helpful to let them know that they have that option, and to help them build confidence in the system.

As you design your product, think critically about the balance of automation and control that you need to offer your users for them to use your product successfully.

In some cases, it may make sense to offer multiple “levels” of automation in your product (for example, partial automation and full automation), to help users with varying levels of comfort or familiarity with your product use it successfully.

In such cases, start users with the lowest level of automation, and progressively increase automation. Make sure that users can easily adjust their choice, including dialing up or down their level of automation, and that the steps between levels are small and land well.

Choosing the right level of automation depends heavily on your user and product needs and context. Research from Sheridan et al has demonstrated 10 levels of automation to consider.

When an AI system fails or gives users a poor prediction, the easiest path forward is often to let the user take over in a non-automated way.

Make it as easy and intuitive as possible for users to quickly pick up where the system leaves off. Give them all the information they need to take the reins:

• Awareness of the situation
• What they need to do next
• How to take their next action

In difficult or high-stakes situations, you may even need to redirect users to a human for extra support.

For supervised learning, accurate data labels are a crucial ingredient to achieve relevant ML output. Labels can be added through automated processes or by people known as labelers.

Labeling tools range from in-product prompts to specialized software. If you’re working with labelers, it’s worth investing time upfront in selecting or designing the tools, workflows, and instructions. The best way to do this is often in collaboration with the labelers themselves.

When labelers understand what you’re asking them to label, and why, and they have the tools to do so effectively, they’re more likely to label the data correctly. And as partners in the process, they can also help you improve your labeling tasks overall.

Develop a data maintenance plan early on, and monitor and maintain your data over time to detect issues proactively, rather than relying on corrective measures when something goes wrong.

Keep the following in mind for data maintenance:

• Similarities and differences between training and live data
• When the data was gathered
• Anything that may have changed since you first collected your training data

How old is your training data? What has changed since you collected it?

When you encounter labels that are “messy," unexpected, or hard to reconcile, don’t categorically discard them as “noisy." Take time to investigate whether issues with labeler tools, workflows, instructions, or overall data strategy may be leading to such issues with labels.

For example, say you’re training a model to flag toxic comments. Your labelers might apply different toxicity labels based on their personal experience, which can lead to discrepancies.

These disagreements in labels offer an opportunity to identify deeper data and/or labeling issues that you may need to address to ensure data quality.

As you develop your training dataset, don’t strive for something perfectly curated. Instead, allow some “noise” to make the data as similar as possible to the real-world data you expect to get from your users. This can help head off errors and poor quality recommendations once you release your model into the real world.

To do this, think about the types of data that you expect to get from your users, and then ensure that data is represented in your training set.

For example, for an image recognition system, consider the data you might get from you users. If it’s likely they will not have the time to take high-quality photographs and your model will have to work with blurry smartphone images, include blurry images in your training data.

When creating your own dataset, make time early on to observe a domain expert your product aims to serve — for example, watch an accountant analyze financial data, or a botanist classify plants. This can give you valuable insights about the types of data that they use to solve the problem your product is addressing.

To identify the right partners, work with user researchers, or others with experience in identifying and interviewing domain experts.

Aim for sustained relationships with domain experts throughout the project lifecycle (rather than one-off consultations), whenever possible.

Do you have domain experts that can help highlight data issues throughout the development lifecycle?