Testing Process & Methodology

Research & AnalysisAdded by Jessica Dunbar - PortlandLabsAugust 16, 2024

BLUF:

This document outlines our A/B testing methodology for optimizing the ArmyMWR.com website. It provides a structured approach to ensure data-driven decisions, efficient project management, and effective communication with all stakeholders.Collect Data

Collect Data:

Our first step is to analyze site analytics using Matomo to identify high-traffic areas and pages with low conversion rates or high drop-off rates. Focusing on these areas allows us to gather data quickly and identify opportunities for improvement.

Continuous_Improvement_Every_Step.png

Write Out a Roadmap:

Creating a detailed roadmap is essential for defining goals, priorities, risks, and timelines. This roadmap will guide our testing process and keep all contributors informed.

Identify Goals:

We define our conversion goals to measure the success of each variation against the original version. Goals may include metrics such as button clicks, link clicks, downloads, and click-throughs.

Generate Hypotheses:

Based on identified goals, we generate hypotheses on why specific variations might outperform the current version. We prioritize these hypotheses by expected impact and implementation difficulty.

Create Variations:

Using Matomo, we make the necessary changes to elements of the website or mobile app. Variations might include changes to button colors, element order, navigation elements, or other custom modifications.

Run an Experiment on a Staging Environment:

We set up the A/B test on a staging site to ensure everything works as expected. The experiment is activated and run internally for quality assurance. Once verified, the A/B testing document is shared with stakeholders to ensure they do not edit the test page/item during the test.

Run Experiment:

The experiment is launched, and visitors are randomly assigned to either the control or variation. Their interactions are measured, counted, and compared to determine performance.

Analyze Results:

After the experiment concludes, we analyze the results using Matomo. The tool presents data showing performance differences between the two versions and indicates whether the differences are statistically significant.

Report on Results:

We prepare a detailed report on the experiment results, including metrics, analysis, and conclusions. This report is shared with stakeholders to inform them of the outcomes and next steps.

Latest News

Metrics_Glossary_3.jpg

Reports Glossary

September 27, 2024

This guide provides an easy-to-understand summary of various Matomo Analytics reports and what they reveal about your website’s performance.

Metrics_Glossary.jpg

Metrics Glossary

September 27, 2024

This page serves as a glossary of key Matomo Analytics terms. It provides clear, concise definitions for a variety of metrics and concepts used in Matomo Analytics

September 26, 2024

Segments

This page provides all the essential labels and definitions related to segments in Matomo.You’ll find explanations of key terms like segment...

September 25, 2024

Visitor Reports

Whether you’re new to Matomo or looking to enhance your analytics skills, this tutorial will help you leverage Matomo’s powerful...

September 25, 2024

Behavior Reports

This tutorial will help you leverage Matomo’s powerful Behavior Reports. This report offers detailed insights into how...

September 25, 2024

Acquisition Reports

Whether you’re new to Matomo or looking to enhance your analytics skills, this tutorial will help you leverage the Acquisition...

September 25, 2024

Dashboard Overview

By leveraging the Main Dashboard and its charting capabilities, you can easily identify performance trends, detect anomalies, and gain actionable...

September 24, 2024

Accessing Matomo

By following this guide, users will be able to: Successfully log in to the Matomo system for the first time using their CAC. Understand...

September 12, 2024

FY24 STRONG B.A.N.D.S.

BLUF: The STRONG B.A.N.D.S.  program pages have experienced mixed trends in site traffic and user engagement. Total users for the...

September 12, 2024

FY24 Summer Reading Program

BLUF: The 2024 Army MWR Summer Reading Program has made significant progress in driving traffic and...