ABacus: fast hypothesis testing and experiment design solution

ABacus is a Python library developed for A/B experimentation and testing. It includes versatile instruments for different experimentation tasks like prepilot, sample size determination, results calculation, visualisations and reporting.

---

Important

Repository moved to the new address: Kolmogorov Lab - ABacus.

---

Important features

• Experiment design: type I and II errors, effect size, sample size simulations.
• Groups splitting with flexible configuration and stratification.
• A/A test and evaluation of splitter accuracy.
• Evaluation of experiment results with various statistical tests and approaches.
• Sensitivity increasing techniques like stratification, CUPED and CUPAC.
• Visualisation of experiment.
• Reporting in a human-readable format.

Installation

You can use pip to install ABacus from Github and use it for your projects:

pip install pip+https://github.com/kolmogorov-lab/abacus

Later the package will be published in PyPI and will be able to be installed with

pip install kolmogorov-abacus

Note the requirement of Python 3.11+.

Quick example

To define an experiment and analyse it is as easy as to describe your experiment and data:

from abacus.auto_ab.abtest import ABTest
from abacus.auto_ab.params import ABTestParams, DataParams, HypothesisParams

data_params = DataParams(...)
hypothesis_params = HypothesisParams(...)
ab_params = ABTestParams(data_params, hypothesis_params)

data = pd.read_csv('abtest_data.csv')

ab_test = ABTest(data, ab_params)

ab_test.report()

The result of code execution is the following:

Documentation and Examples

Detailed documentation and examples are available for your usage.

Communication

Authors and developers:

• Vadim Glukhov
• Egor Shishkovets
• Dmitry Zabavin