How to test a change to your event experience before rolling it out

Your workshop team wants to add a practical task card to the opening explanation. They think people will get started faster. The designer likes it, the facilitator likes it, and the first attendee you ask says it looks useful. Do you print cards for the whole conference?

Try the change in a limited, properly planned comparison first. Decide what it should improve, give the existing and proposed approaches a fair chance, and check the extra work and cost before making it permanent. Keep the experience people paid for intact throughout the test.

That sounds straightforward until you start arranging the comparison. Your busiest session has the strongest facilitator. Morning attendees are different from the people arriving after lunch. Friends share instructions. Someone changes the exercise halfway through because the new card seems promising. By the end, you have plenty of opinions and very little agreement about what happened.

A useful event experience test anticipates those problems. It won't remove every uncertainty, particularly if you run only a few events each year. It can still tell you whether a change is practical, whether the evidence supports trying it more widely, and what remains unproven before you commit the next budget.

Start with the decision you need to make

Write down the proposed commitment before choosing a test. Are you deciding whether to print a different instruction card, book another facilitator for every session, or rebuild the programme around a new format? Those decisions need different evidence because their costs and consequences are different.

For a small wording change, seeing people understand and use the instructions may justify another limited run. A permanent staffing commitment needs a clearer account of the improvement, its reliability and the work required. A format change that alters the advertised experience also needs a customer and production review before you can offer it.

Be specific about the problem. "Attendees aren't engaged" could mean they aren't starting the activity, aren't asking questions, leave early, or don't return next year. Each possibility suggests a different change. Look at delivery notes and attendee accounts until you can name the behaviour you actually want to improve.

Throughout this guide, we'll use an invented example. Northbank runs a paid professional conference with optional practical workshops. Its team has noticed that some participants spend the opening exercise asking what to do. The proposed change is a task card that breaks the same exercise into clearer steps. The workshop content, duration, equipment and advertised learning opportunity stay the same.

Northbank wants to know whether the card helps participants complete the opening exercise without individual explanation. That is a delivery question. It isn't a claim about the popularity of the speaker, future ticket sales or the value of the whole conference. Those might matter later, but combining them now would make the first test harder to interpret.

HM Treasury's Test and Learn guidance recommends testing uncertain assumptions before committing to wider delivery. Applied here, the uncertain assumption is that unclear instructions are causing the slow start and that a card can help. Northbank can investigate that without redesigning every workshop at once.

Before continuing, ask what you would do with each plausible result. If nothing would change your decision, you may be seeking reassurance rather than information. If a useful result would arrive after the print deadline, move the test earlier or change the commitment it informs. Evidence has to be available while you can still act on it.

Describe exactly what will change

Save both versions of the experience in enough detail that another member of the team could deliver them. For Northbank, that means the existing spoken opening, the task card, the exercise materials, the facilitator's instructions and the time allowed. Calling one version "the interactive approach" leaves too much room for interpretation.

The card can contain several instructions and a new layout. You don't have to reduce a coherent improvement to one changed word. What matters is that you can describe the complete difference and keep unrelated decisions stable. The result would concern the task card as delivered, rather than proving which sentence or graphic helped.

Check what staff will naturally do differently. A facilitator who hands out the card might also become more enthusiastic, give extra demonstrations or spend longer answering questions. If those changes are part of the intended permanent method, include them in its description and cost. Otherwise, explain what should stay consistent and practise it before the live comparison.

Northbank keeps the same opening explanation in both approaches. In the proposed version, participants also receive the step card. The team isn't simultaneously shortening the presentation, changing the exercise and introducing different equipment. If it wants to assess that larger package later, it can test and name that package honestly.

This distinction matters when a sponsor contributes something to the trial. A new networking format accompanied by free refreshments and extra hosts is a different offer from the format alone. You might want the whole arrangement, but you need to know whether the refreshments and hosts will be available and affordable on the next date.

Use your existing attendee journey map to check the moments around the proposed change. A smoother workshop opening may require a different arrival instruction, an earlier room reset or an extra handover between teams. Include those dependencies in the trial so you don't discover them after announcing a permanent change.

Agree who can alter the versions. Staff should always respond to a real attendee problem, but an improvement made during the comparison changes what you're studying. Record the time, the reason and the affected sessions. You can learn from the revised delivery without combining it invisibly with the original version.

Choose an outcome you can actually observe

A useful main measure describes a result that matters to the attendee and can be recorded consistently. Northbank chooses the share of assigned participants who complete the opening exercise by its scheduled twelve-minute endpoint without individual explanation from the facilitator. Everyone receives the same exercise time in both approaches.

The observer needs a definition of completion. In this fictional planning workshop, participants draft a handover note naming a next action, its responsible person and its due time. Completion requires all those entries on the supplied worksheet. That narrow definition checks whether someone got through the opening task; it doesn't assess the quality of their planning. An explanation given individually also has an agreed meaning, so normal greetings aren't accidentally counted as help.

That measure has limits. Completing an exercise doesn't establish that someone learned the subject, enjoyed the workshop or can apply it at work. Northbank should keep the conclusion close to the behaviour measured. If the business needs evidence of learning, it will need an appropriate assessment and a different plan, developed with someone qualified to judge it.

Record the eligible group before looking at the result. In this example, participants book and agree to the research before their sessions receive an approach. Later difficulty doesn't make them ineligible. Someone who stops the exercise still belongs in the assigned group, subject to the consent and withdrawal arrangements, with their outcome recorded under the agreed rule.

Keep other checks alongside the main measure. Northbank records requests for help, departures during the exercise and reports that the card was difficult to use. Help requests count requests, so one participant can make several. They aren't another count of unique people. These checks explain delivery and can reveal a problem that the headline completion rate misses.

You also need to distinguish a recorded failure from an unknown outcome. If an observer loses sight of a participant's work, the result is missing. Don't quietly classify it as success or remove it from the total. Show the assigned number, the outcomes observed and the missing records, then assess whether the gap prevents a useful comparison.

Choose these rules before delivery. Otherwise, it's very easy to start with completion, notice that the new version produces more conversation, and declare conversation the real goal. Additional observations can suggest another question. They shouldn't replace the question you promised the team the trial would answer.

Give both approaches a fair comparison

Comparing this year's event with last year's is tempting because the records already exist. It also mixes the change with everything else that changed. The venue, audience, weather, facilitator and programme may all differ. A better result could have several explanations, even when the new approach seems sensible.

Where practical, compare the approaches during the same period with similar participants and delivery conditions. Random assignment means using a chance process to decide which approach a participant or session receives. It reduces the opportunity for staff to give the new approach only to the people or sessions most likely to succeed.

The Cabinet Office's Test, Learn, Adapt guidance describes choosing outcomes in advance and randomly assigning the alternatives. It's public-policy guidance, rather than an event performance benchmark. The underlying comparison principle is useful when you can apply it without disrupting the event or compromising what attendees receive.

You must choose what gets assigned. For a private pre-arrival message, the person or booking might be workable. For a shared workshop instruction, assigning neighbouring people different versions can cause confusion and information sharing. Assigning whole sessions may fit delivery better, although it changes how much independent evidence you have.

Northbank chooses whole workshop sessions. Participants book their workshop time and agree to the research before the team randomly assigns the delivery version, so they aren't choosing a session because they prefer the card. Everyone in a particular session receives the same approach. The consent and operational arrangements still allow people to withdraw from research or request help without losing their workshop place.

The team arranges four pairs of comparable sessions, with twenty research participants in each session. Within each pair, a chance process assigns one session the existing approach and the other the card. The sessions use the same exercise and equipment. Pairing also accounts for known delivery differences as far as the timetable allows.

For example, each pair can use the same facilitator across comparable time slots, with the order of the two approaches varied by the planned assignment. You should also consider whether delivering the first version teaches the facilitator something they carry into the second. Where that is likely, ask your analyst to adjust the design rather than treating the order as irrelevant.

The US National Institute of Standards and Technology explains randomized block designs, which compare alternatives within groups that share relevant conditions. Northbank's paired-session plan applies that idea. Pairing won't make different sessions identical, but it can make a planned comparison more useful than giving all morning sessions one approach and all afternoon sessions the other.

Keep shared experiences and small samples in view

Eight sessions containing twenty people each produce records for 160 participants. They don't give you 160 independently assigned experiences. The session receives the version, and people in that session share a facilitator, timing and room conditions. Treating every participant as an independent trial would make the evidence appear more precise than the design supports.

Ask whoever plans the analysis to work from the unit you assign. If you're assigning sessions, the number of sessions matters. A crowded workshop doesn't compensate for having only one session of each version. You could be comparing two facilitators having different afternoons and mistake that for a reliable effect of the card.

There isn't a universal minimum number of attendees that makes an event experiment conclusive. The required evidence depends on the outcome, how much it normally varies, the improvement you need to distinguish and how the comparison is arranged. Get help with that calculation before recruitment when the decision warrants a formal estimate.

For Northbank, the first run is explicitly exploratory. It checks whether the delivery and measurement work and whether another limited comparison is worth funding. The business does not present eight sessions as proof that the card will improve every workshop. If a supplier needs a large permanent order, this run alone may be inadequate evidence for that commitment.

Consider what happens between sessions too. Participants may compare cards over lunch. A repeat attendee might attend both versions. Staff may share what they think is working. Record those possibilities, limit them where reasonable and avoid collecting intrusive personal information simply to achieve a tidier comparison.

If the change affects the whole venue, participant assignment may be impossible. You can't give half the room a different sound system without affecting the other half. A comparison across several dates may be more practical, with the weaker ability to separate date effects stated openly. A technical rehearsal may answer the immediate delivery question better than an attempted experiment.

If this is your only event of the year, use smaller preparations for the questions they can answer. Invite representative people to try instructions, rehearse the handover and watch where they struggle. Those activities can identify obvious problems. They won't establish a dependable percentage improvement for the full paying audience, and your decision record should say so.

Protect the experience people bought

Both approaches must satisfy the event's promises and requirements. A test is a poor reason to withhold necessary access information, reduce an agreed service or leave a known fault uncorrected. Fix those problems directly. Use the comparison to assess reasonable alternatives that you have already checked are suitable to deliver.

For the workshop, participants in both versions receive the same learning opportunity and can ask for help. An observer never tells a facilitator to leave someone confused until the measurement ends. If help is needed, provide it and record it. A cleaner completion figure isn't worth making the workshop worse for someone who paid to attend.

Check accessibility before selecting a version for live use. The card needs suitable alternatives and a way for participants to request support. Include people with relevant access needs in preparation and don't assume a printed instruction is useful simply because most of the team can read it easily. If an access barrier appears during delivery, address it immediately.

Tell people what research participation involves in language they can use. The UK Government Service Manual's guidance on informed consent for user research covers explaining the purpose, activities, information collected and voluntary participation. Use that as a planning reference, then check the requirements that apply to your event and research method.

Northbank's fictional trial recruits willing adult participants and explains that it is comparing ways to deliver the exercise. Participation in the research is optional and doesn't affect admission or access to support. The team records only the information required for the comparison, using participant codes rather than circulating named worksheets in the review meeting.

Agree how to handle withdrawal, access requests and deletion before collecting records. Keep contact information for arranging the session separate from the working analysis where possible. Someone needs to own the retention date and the process for responding to a participant. Follow those arrangements when data must be removed and report any resulting limits on the comparison. Saying the information is "only for improvement" doesn't resolve those responsibilities.

Name the person who can stop the trial on site. The decision should follow practical conditions, including an inaccessible version, material confusion about the service or a delivery problem that compromises attendee value. Staff need a supported replacement ready. Record why the trial stopped; don't describe the interrupted run as a completed fair comparison.

Rehearse the test as well as the change

Before the event, run through the collection process with someone unfamiliar with it. Can they understand the task? Can the observer identify completion without interrupting the activity? Can the facilitator provide normal support while the records remain usable? Discovering a broken measurement method during your main comparison wastes a scarce opportunity.

Nielsen Norman Group's guidance on pilot testing a research session explains checking tasks, instructions and timing before the main study. That guidance concerns usability research. For an event trial, the useful application is to rehearse how you will collect evidence, rather than assuming a well-rehearsed event automatically produces reliable research.

Northbank asks its observers to practise on the same rehearsal and compare their records. If they disagree about what counts as individual explanation, the definition needs work. This isn't an invitation to create an elaborate scoring system. The aim is to make the few records you need understandable enough that different staff apply the same rule.

Test the mundane details. Confirm where the session code appears, when the timer starts, who collects the worksheets and how records reach the reviewer. Check that neither version gives the observer less visibility. A missing pile of worksheets in the busiest session could matter more than another column in the spreadsheet.

For Northbank, the twelve-minute clock starts after the same spoken opening, when participants begin the exercise. Cards are distributed beforehand but opened at that start. Observers record the worksheet's status at the endpoint, so additions made afterwards don't change the outcome. They also record whether individual explanation occurred during the exercise. People can continue learning and getting help once the measurement window ends.

Keep rehearsal records separate if the process changes before the main run. A participant who receives additional hints while you refine the instructions hasn't had the same experience as someone in the final comparison. Their comments can improve the method without becoming another successful completion in the trial result.

Brief staff on the purpose without asking them to deliver the hoped-for answer. They should know what to do, what to record and when to help. Avoid telling the card facilitator that this version has to win because the design work has already been approved. Expectations can alter delivery even when nobody intends to distort the result.

Save the final materials and the agreed collection rules together. On the day, identify each session's assigned approach before it starts and confirm the correct materials arrived. If the old script and new card are mixed up, record what actually happened. Don't change the assignment record afterwards to make the delivery look compliant.

Write a test brief the producing team can use

The plan should be short enough for the people delivering it to read before they arrive. You can keep technical analysis and research information in supporting documents. The working brief needs the decisions that could otherwise become an argument at the workshop door. Send the final version with the running order and make its owner available for questions.

Northbank's brief could use these entries:

• Decision: whether to fund a limited extension of the task-card method before considering use across the full workshop programme.

• Change: the existing spoken opening plus the agreed task card, compared with the existing opening alone. The exercise and ordinary support stay consistent.

• Participants: willing adult workshop attendees recruited before session assignment, with the agreed consent process and a separate record of attendance and withdrawals.

• Assignment: four comparable session pairs, with one session in each pair randomly allocated to each approach after booking and consent.

• Main outcome: completion of the agreed draft by the twelve-minute endpoint without individual explanation, recorded using the same criteria for both approaches.

• Delivery checks: correct materials, opening time, help requests, departures, accessibility concerns and any change to the script or exercise.

• Authority: the named producer can stop the trial and restore the supported delivery; staff can always provide help without waiting for research approval.

• Review: the named reviewer checks complete session records, costs and limitations before the owner approves any further printing or delivery expense.

Add the actual calendar dates and staff names when you use the brief. A review described as "after the event" is easy to postpone until the next supplier deadline has passed. Work backwards from the next commitment and reserve time to collect worksheets, resolve missing records and check staff hours. If that timetable is impossible, reduce the claim you expect the test to support.

Prepare for attendance differences before they occur. Northbank's later worked result assumes every recruited participant attends. In a real run, record booked, arrived and measured numbers separately for each assigned session. Don't fill empty places with whichever attendees look most enthusiastic about the card. Any replacement process should be agreed before assignment and reviewed for how it changes the comparison.

If the business also wants to compare feedback, use the same question and collection timing for both approaches. Asking card participants immediately while asking the existing-approach group several days later introduces another difference. Show how many people answered in each group. Voluntary feedback can explain a result, but the people answering may differ from those who remain silent.

Keep the question neutral. Ask what helped or hindered the opening exercise rather than whether the new card made it easier. You can learn that someone found the print difficult, used a colleague's explanation or already knew the task. Those answers help you understand delivery. None is a substitute for the planned outcome or a reason to remove an inconvenient participant afterwards.

Finally, confirm what happens if the result isn't ready. The producer should know whether the existing materials remain the default, whether a printing order can wait and who approves a temporary arrangement. That small decision prevents a trial becoming a permanent change simply because someone needs to finish packing the workshop boxes.

Read the results before choosing your explanation

Suppose Northbank completes the fictional exploratory run as planned. All 160 assigned participants attend and have a recorded primary outcome, with no research withdrawals. In the four existing-approach sessions, the numbers completing without individual explanation are twelve, eleven, ten and eleven. In the paired card sessions, they are sixteen, fifteen, fourteen and fifteen. Each session contains twenty participants.

The existing approach therefore records 44 qualifying completions out of eighty participants, or 55%. The card records sixty out of eighty, or 75%. The observed difference is twenty percentage points. Those figures describe this invented run. They aren't a published event benchmark, a prediction or a statistical finding about all Northbank attendees.

Show each pair as well as the totals. Here, every pair points in the same direction, which makes the result more interesting to investigate. You still have only four session pairs. Consistency across them doesn't remove uncertainty about the size of the effect or whether different facilitators, exercises and audiences would produce it again.

Read the supporting records next. Did the card arrive on time? Did participants use it? Did both approaches receive the same opening? Was anyone given an additional demonstration? A favourable completion rate with substantially more facilitator effort answers a different question from a favourable rate with the planned staffing.

Look at who isn't represented. Northbank recruited willing adults for these workshops. The result might not transfer to attendees who decline research, people using a different language or sessions covering a more difficult exercise. You can describe those limits without dismissing the useful observation that the new method worked in the conditions you actually tested.

If results point in different directions, resist choosing the most flattering explanation immediately. A morning improvement and afternoon decline might reflect fatigue, facilitator differences, participant mix or chance. Use the delivery record to identify a plausible next question. Avoid producing a confident story from information you didn't collect.

Missing information needs its own treatment. Suppose an observer had failed to record several outcomes in a busy session. You would show those missing outcomes and examine whether different possible results could change the decision. You wouldn't silently report a completion rate only for the easiest worksheets to collect. Ask the analyst how to handle the gap before drawing an effect conclusion.

Work out what the change will cost to repeat

An experience improvement can be worth paying for without generating immediate additional ticket sales. It may help deliver the event you promised. You still need to know its cost and decide whether that use of the budget is justified. Don't convert a better exercise result into revenue by inventing a future purchase rate.

Northbank plans a possible next edition with twelve workshops and twenty participants per workshop. In this fictional budget, all money is in USD and the stated amounts include applicable supplier charges. Cards cost $0.50 per participant, so supplying 240 participants costs $120. Preparing the cards adds ten facilitator minutes per session, or two hours across twelve sessions.

At an internal planning rate of $30 per hour, that preparation time represents $60. Together, cards and preparation require $180 per edition under these assumptions. The time allowance is a cost of staff capacity. It isn't automatically an extra payroll payment if the team is already paid for that time, but it still has to fit the working schedule.

The first test also had development and research costs. Four design hours at $35 cost $140, and two training hours at $30 cost $60. The eighty cards used in the new-approach sessions cost $40. Those four sessions added forty preparation minutes, valued at $20. Six hours collecting and reviewing evidence at $30 added $180.

That makes $440 for the exploratory test, including development, training, additional delivery and measurement. The six research hours include rehearsing collection, and this example assumes no participant incentives or further incremental expenses. It's separate from the ordinary workshop budget, which both versions needed. The breakdown values the stated staff time without counting any hours twice. Your own quotes, employment arrangements and available staff time may produce a different result.

For the next decision, show both what you have already spent and what you would spend next. If Northbank can reuse the design and the same trained team, the proposed recurring change costs $180 per edition. If staff change or the exercise needs rewriting, add the new training and design work. Calling those costs one-time doesn't make them disappear when circumstances change.

Use the event budget guide to place the proposal inside the full event plan. Ask what the $180 would replace, whether the staff time is available and whether a simpler change could achieve the same practical aim. Another facilitator or a clearer spoken explanation is an alternative to assess, not an equivalent benefit you can assume without trying it.

You can also test a revenue argument without pretending it is evidence. If a future paid admission leaves $45 after the additional costs of serving that attendee, four genuinely additional admissions would cover a $180 recurring expense before other overhead. The workshop result hasn't demonstrated those admissions. Keep that calculation conditional and separate from the completion finding.

Decide whether to expand, revise or stop

Before the trial, agree what sort of evidence could support the next commitment. That doesn't mean choosing an arbitrary completion percentage and treating it as scientific proof. It means deciding how much uncertainty the business can accept for the particular cost, attendee consequence and reversibility of the decision.

For Northbank, a promising exploratory result could support another limited comparison. It wouldn't automatically support ordering cards for a different conference programme or removing staff. The measurement concerned completion without individual explanation. It didn't measure whether the whole facilitator role had become unnecessary or whether participants could work without ordinary support.

The next run could keep the method but broaden the delivery conditions. Include another suitable facilitator or a different comparable time period while preserving a fair comparison. Decide which remaining uncertainty matters most. Changing the audience, exercise, script and staffing simultaneously would make a successful repeat difficult to explain.

Put a limit on that next learning expense. For example, Northbank could propose four further card sessions containing eighty participants, alongside four existing-approach comparison sessions. Under the same invented delivery assumptions, the cards and extra preparation cost $60. A further $120 allowance for evidence collection across the comparison would make the new proposal $180. That is a spending cap to approve, not a forecast of commercial return.

If the card doesn't help under a workable comparison, review whether people used it and whether the suspected problem was real. Perhaps the exercise depends on knowledge the opening never supplies. Rewriting the card again won't solve that. A useful negative result can redirect the next improvement toward the content or the session's suitability for its audience.

Stop if the alternative repeatedly causes a material attendee problem, costs more than the business can justify or cannot be delivered consistently. You don't need another round simply because the first design took time. Keep the materials and reasoning so the next person understands what was tried and why the business chose another approach.

When the evidence is incomplete, state the practical consequence. "We don't yet know" can mean keeping the existing approach, running a smaller preparation exercise or approving a reversible change with explicit monitoring. It shouldn't become a vague status that allows the full rollout to happen without a decision.

Make the next delivery and review easy to run

Write a short decision record while the details are fresh. Start with the problem, the exact alternatives, who was eligible and how assignment worked. Add the planned outcome, actual results, delivery departures, missing information, attendee concerns and complete incremental costs. Finish with the next commitment and the uncertainty it is meant to address.

For Northbank, that record should preserve both the promising result and its limits. Four pairs of sessions produced a higher observed completion rate with the card. The comparison used one exercise and a willing workshop audience. The proposed next step is a capped extension, with the remaining delivery and analysis questions resolved before recruitment. That is more useful than a slide declaring the new format a success.

Assign an owner for the delivery version and another for the review if your team allows it. Someone needs to confirm materials, staffing and participant information. Someone also needs to check the outcome records against the agreed definitions. They can work together without allowing enthusiasm about the change to decide how an awkward result is recorded.

Communicate practical changes to ticket holders through your normal event process. With Loopyah's attendee email tools, you can build the relevant event message and send yourself a test before it reaches attendees. Your team still owns the research explanation, the comparison design and the separate records needed to evaluate it.

Don't confuse sending an email with assigning an experiment. A ticket-based audience segment can help you reach the right holders, but it doesn't establish a random session comparison. Check the actual allocation and booking process independently. Keep research participation voluntary, and make sure people who decline still receive any information they need to attend.

If your real uncertainty concerns the ticket page, use the event ticket-page A/B testing guide for that separate comparison. If it concerns whether people will buy an entirely new event concept, a paid pilot addresses a different decision. Choose the method around the commitment you need to make instead of trying to prove everything through one workshop trial.

Bring the completed record to the person approving the next spend. Ask them to approve the specified delivery, maximum cost and review date, with any unresolved condition written beside it. Then give the producing team that same version. The next event should reflect an explicit decision about what the evidence supports and what you can afford to learn next.