Experiment To Change
The team or wider organisation may readily identify that there are problems with its current way of working, that it wishes to improve its performance, or change its technology. Understanding the problem is inherently simpler than identifying a solution. Even if a potential solution is evident, we may be uncertain about its effectiveness. We experiment with changes to understand whether they will be effective and, if so, the extent of improvement we can realise.
In A Nutshell
Experimentation is at the heart of our empirical way of thinking about the world - it is one of the principles of anAgileMind. We seek to try things out - “try before you buy”. We seek to gather evidence that one change is more effective than another change.
The nature of an experiment is that its results are testable. We accept that the outcome of an experiment may prove that our ideas for change are wrong. This is not a failure. The experiment is a success because it has told us that what we believed a-priori to be true is demonstrated to be wrong. Equally the experiment may prove our belief is correct.
The evidence (positive or negative) helps us to achieve a wide consensus about what we should do next. This makes it easier to carry the experimental change into full implementation (assuming the hypothesis is proven); it also makes it easier to continue the conversation about future change.
An experiment is only a failure when we fail to collect sufficient data to judge whether the hypothesis has been proven or disproven. Experiments fail because they are poorly executed, not because the hypothesis is disproven.
Experiment Focus
Experimentation should be a widely used technique for evaluating all scales and types of change, including:
Organisation - Changes that affect the structure, roles, responsibilities, strategy, operation or governance of the organisation
Team - Changes that affect the ways of working within the team or ways of collaborating across a group of teams
Features - Deciding how new features can best be presented to users and implemented technically
Technology - Deciding whether and how to adopt new or changing technology into the platform used by a product
Defining The Experiment
Hypothesis
The hypothesis describes both the outcome and the nature of the change to be tested, perhaps in the form:
We believe that we can <achieve outcome> by <change to be tested>
For example
We believe we can increase the value delivered to our customers by collaborating more closely with customers as we prioritise our backlog
Indicators
We must also describe the indicators we expect to observe in the form:
We will know we have succeeded when <indicator statements>.
For example
We will know we have succeeded when customers routinely collaborate in backlog refinement activity and customer feedback given in sprint retrospectives improves.
Measures
Having defined the indicators, we must go further and define exactly what we are going to measure so that we confirm our indicators objectively.
In our example, we must define metrics related to routine collaboration and customer feedback.
Note, too, customer feedback is expected to improve - that is to change from some current level. What is that current level and how will it be measured?
Defining The Protocol
The protocol for an experiment describes the pre-conditions, boundary conditions and the method to be employed in performing the experiment.
Pre-Conditions
Must be met before the experiment can begin - they set the experiment up for success. For example we may ask our customers to commit to participating in some backlog refinement activity before the experiment begins.
Boundary Conditions
During the execution of the experiment we will need to work with other stakeholders who are not engaged in the experiment. Boundary conditions describe how collaboration with these other stakeholders will continue during the experimental period. For example if there is a Jedi Council that normally sets backlog priorities, we may still have to accept priority overrides from the organisation. How will this collaboration take place?
Method
The method is the recipe for performing the experiment. How are we to schedule “formal” backlog refinement? Are customers to be engaged on informal refinement activity? When do we start taking measurements? How do we capture customer feedback? When do we expect the experiment to end? Are there early escape steps if we see early signals of success or failure?
Performing The Experiment
To be successful with our experiment we must create the environment in which we can run the experiment. The pre-conditions describe some of what we must achieve. But we also need to ensure that all the stakeholders engaged in the experiment have a shared understanding of what is to be done and are fully committed to doing it. Where comparative metrics are to be used, we must take the time to establish our baseline data.
During the life of the experiment we follow the protocol. We look for the occurrence of boundary conditions and ensure that all stakeholders respond to these as the experiment definition describes. We look for opportunities to exit the experiment early, based on the data we are observing. If the hypothesis is proved (or disproved) early in the life of the experiment, we need to consider whether we should end the experiment. Where the hypothesis is proved we can decide how we integrate the features of our experiment into our every day work.
If the hypothesis is disproved we choose to avoid the cost of continuing with the experiment and revert to the status quo ante. We also decide what we do next? Are there other experiments that we want to try?, for example.
“Let’s Experiment!” I heard one leader cry. “But Why?” retorts another.
Maybe we find ourselves trying to do more with less, trying to keep pace with a competitor or even keep one step ahead. In all these cases, we will need to adapt and try new things - experimenting can be a systematic way of testing our assumptions and documenting our findings.