How do you incorporate feedback and learning from your strategic experiments and pilots?

The onus is on the strategic thinker to actively pursue avenues for expeditious, accurate, and two-sided feedback that can help build effective mental models. Constructing sound mental models is pivotal for facilitating swift and precise strategic decision-making under time pressure. Positive feedback serves as a robust reinforcement for commendable actions, while negative feedback functions as a corrective measure for suboptimal decisions. This iterative process, meticulously honed over time, culminates in the development of a formidable capability for strategic thinking in dynamic environments.

In my experience, it’s helpful to begin a feedback cadence with your stakeholders, especially as you develop a new capability into a real service or product. I find a two-week operations tempo works nicely as it gives the team enough time to incorporate feedback into the new idea while at the same showing incremental improvement to the stakeholder. Additionally, this type of ongoing interaction allows for increased communication which is never bad. Speaking with stakeholders about their needs, incorporating their feedback into your development process, and demonstrating its integration into your effort only increases the probability for success.

First, remember to do no harm. Think through the unintended consequences of the test. Attempt to experiment in a controlled environment where the participants in the research understand that they are part of building or perfecting a new service. Perhaps you can leverage a panel of your own customers that you manage. Going to market with a product that the company considers as having reached minimum viable product status with the intent of rapidly testing into a more perfected form may backfire and damage the product’s and company’s reputation should the MVP disappoint the product innovators and early adopters who will be first to buy the product and first to write reviews.

In my experience, successful tests have a couple key criteria: 1) They are simple. Spend the time getting something up that will test the core of your hypothesis but won't take considerable development effort. Spend the time making it look great. I worked with a client once who built a complicated pilot, shipped it and failed - but because it was overbuilt, it was hard to discern if users didn't want that product, or if the UX of the pilot was so bad they didn't even get it. 2) Test the most dangerous assumption you are making. For example, if you believe that there is a key differentiator that will make people drop your competition - then just make landing pages for THAT FEATURE. If you try to test too much, you can't progress...

Designing strategic experiments and pilots starts with setting clear, measurable objectives in line with an organization's goals. Key success metrics need to be identified to gauge the pilot's impact. Choosing a manageable yet representative pilot area is essential for focused testing. Formulating a hypothesis guides the experiment's structure and subsequent analysis. The experiment requires a detailed plan, including duration, resources, and data collection methods. Post-pilot, analyzing the data against set metrics and hypotheses helps uncover insights and determine if adjustments are needed. This process is not just about validating ideas but also about learning and adapting to inform broader strategic decisions.

For whatever experience, learning or finding you get in terms of feedback, ask these three questions: 1. Relevance: What is the impact on the relevance of our strategy. Is the strategy still right, or should we adapt? 2. Progress: What is the impact on progress. Are we still on schedule, can we move on or do we need to adapt? 3. Mood: What is the impact on people's mindsets and attitudes. Is everyone doing okay, or should we act and raise spirit?

Most organizations capture user feedback, which is quite helpful. However, the opportunity to also capture internal team feedback on the opportunities and challenges of creating and delivering products and services to the market is equally valid. I introduced the Retrospective Radar as a way to facilitate the capture of verbatim feedback from team members as qualitative data points. Because the Retrospective Radar slices the feedback into 5 segments (Start Doing, Stop Doing, Keep Doing, Do More Of, Do Less Of) and 3 dimensions (Team, Managers/Stakeholders, Executives), the content and context for the feedback is specific, targeted, and actionable. By parsing feedback thru Natural Language Processing, qualitative data can be quantified.

Learnings are important, feedback thus plays a pivotal role. Feedback should be used as a tool to achieve growth, hence it has be to assimilated, analysed and delivered with absolute clarity. Sharing feedback is an art, one has to deliver the findings while staying transparent and be empathetically connected, giving people enough datasets and time to improve on the findings. Work with teams to improve these findings.

Establish the order of communications from the beginning of the process. Remember, some executives may see more downside risk than upside gain from the experiments. Make sure you have fully briefed your own supervisor and they have committed their support to the transparency of the test. While it may be best to brief all stakeholders on the test’s outcome at the same time, you may find that you must order the roll out to give stakeholders with the most seniority or the most at risk an opportunity to understand the outcomes first. For you, however, never compromise your ethics in reporting the results of the experiments. Your credibility as strategist or researcher depends on stakeholders seeing you as a voice of truth.

Feedback should be shared iteratively throughout the pilot and scale-up periods to accelerate speed to market. Therefore, it's crucial for teams to exchange feedback effectively, informally, and constructively. Executives should foster a professional environment where constructive feedback is the norm, encouraging teams to approach feedback with a professional mindset, ensuring it is never taken personally.

Publishing the aggregated insights from collected feedback across teams/users allows for the quantification of words, phrases, sentiment, emotion, and keyword frequency via Natural Language Processing (NLP). This provides an objective way to go from ideas or complaints to actionable intelligence through the quantification of qualified, unstructured data as feedback.

In applying feedback and learnings from strategic experiments, I prioritize a reflective and adaptive strategy development. I conduct thorough reviews to extract key insights, focusing on both successes and areas for improvement. This involves actively seeking diverse perspectives and feedbacks, which I then analyze. The most valuable insights become part of the strategy planning, while ensuring that the strategy is continually refined and evolved based on real-world learnings. This cycle of experimentation, feedback, and improvements is the key to keep the strategy relevant and effective.

If you have done your tests correctly, you should be validating or invalidating assumptions with every test. This will allow you to continue with or without that assumption in developing your new test(s). When I consult with companies, we make a list of our most critical assumptions that usually have to do with the mindset of the customer and what it will take to convert them from our competition to us. After we test, we determine if we have validated or invalidated the assumption. Then we re-calibrate all the assumptions. Sometimes the results will prompt us to realize there is a new assumption we uncover (e.g. that we can REACH our potential customer base with ads, etc.)

Feedback should loop, not cascade because the purpose of feedback is more than sharing an opinion; it's the single most effective way to objectively evaluate metrics (numeric data) with feedback (qualitative data) to ask if the Strategy is valid and, at times, if even the directive/Goals/initiative is valid. Goal is set -> Strategy is chosen -> Deliverables are created -> Metrics and Feedback are evaluated -> repeat the cycle to see if the Goals and Strategies are valid and/or effective.

A mature organization can benefit from developing specific processes to systematize use of feedback from strategic experiments. The process's main aim is to assess viability of scaling up a disruptive solution at the end of a strategic experiment. If it is, then it is best to create an independent unit (or a spinout) for the potentially disruptive solution within the larger organization (with mature products/services). On the other hand, an early stage organization is relying on experiments a lot more than a mature organization for its survival. For such organization, feedback incorporation requires having an After Action Review process to move forward with an approach or pivot. The frequency of such process use will need to be high.

Am besten können Sie strategisches Lernen mit einer Veränderung in der kontinuierlichen SWOT-Analyse erkennen. Dazu sortieren Sie die Themen bitte nicht hart in die 4 Kategorien ein, sondern bewerten sie auf zwei senkrecht zueinander angeordneten Scoring-Skalen SW und OT von -10 bis 10. Die Veränderungen der visualisierten Themen-Position in einem so gestalteten Koordinatensystem, zeigt dann tatsächlich auch grafisch Ihre Lernkurve an.

Incorporating feedback and learning from strategic experiments and pilots is absolutely crucial for driving improvement and achieving success. I proactively gather feedback from all stakeholders involved, including team members, customers, and relevant parties. I meticulously analyze this feedback to identify both strengths and areas for improvement, enabling us to adjust our strategies, processes, and approaches for maximum impact. Fostering a culture of learning within the organization, I encourage open, candid discussions about what worked well and what didn't. Furthermore, I ensure thorough post-experimentation reviews and debriefings to capture valuable insights for future initiatives.