A core feature of the
is that the tolerances for tests can be configured externally.
This allows different repositories to share the same tests,
but use different tolerances.
See the “Configuration” section below for details.
To use this fixture, install with
pip install pytest-allclose
import numpy as np def test_close(allclose): x = np.linspace(-1, 1) y = x + 0.001 assert allclose(y, x, atol=0.002) assert not allclose(y, x, atol=0.0005) assert not allclose(y, x, rtol=0.002)
allclose fixture has a number of arguments
that are not part of
One such argument is
which allows arrays that have been shifted along their first axis
by a certain number of steps to be considered close.
import numpy as np def test_close(allclose): x = np.linspace(-1, 1) assert allclose(x[1:], x[:-1], xtol=1) assert allclose(x[3:], x[:-3], xtol=3) assert not allclose(x[3:], x[:-3], xtol=1)
Refer to the
allclose API reference for all additional arguments.
RMSE error reporting¶
allclose fixture stores root-mean-square error values,
which can be reported in the pytest terminal summary.
To do so, put the following in your
from pytest_allclose import report_rmses def pytest_terminal_summary(terminalreporter): report_rmses(terminalreporter)
report_rmses API reference for more information.
allclose_tolerances accepts a list of test name patterns,
followed by values for any of the
These values will override any values provided within the test function itself,
allowing multiple repositories to use the same test suite,
but with different tolerances.
allclose_tolerances = test_file.py:test_function atol=0.3 # set atol for specific test test_file.py:test_func* rtol=0.2 # set rtol for tests matching wildcard test_file.py:* atol=0.1 rtol=0.3 # set both tols for all tests in file test_*tion rtol=0.2 # set rtol for all matching tests in any file test_function[True] atol=0.1 # set atol only for one parametrization
The only special character recognized in these patterns
is the wildcard character
which matches any group of zero or more characters.
If the test is parametrized,
then a pattern like
will match specific parameter settings,
test_name* will match all parameter settings.
Note that the latter will match any test that starts with
If a test has multiple
you can use multiple tolerance lines that match the same test
to set different values for the first, second, third, etc. calls.
If there are more
allclose calls than tolerance lines,
the last tolerance line will be used for all remaining
Example test file:
def test_close(allclose): x = np.linspace(-1, 1) y = x + 0.001 assert allclose(y, x) assert not allclose(y, x)
Example configuration file (
allclose_tolerances = test_close atol=0.002 # affects first allclose call test_close atol=0.0005 # affects second allclose call
Different tolerance lines correspond to calls of the
function, not lines of code. If you have a
loop that calls
allclose 3 times,
each of these calls corresponds to a new tolerance line.
If you have a fourth
you would need three tolerance lines for the three calls
for loop, then a fourth line for the last call.
The patterns for multiple calls of allclose in a function must be exactly the same. This means that if you have specific values for one parametrization and general values for others, you must put the specific values first or they will not have any effect.
Good example, specific takes precedence:
allclose_tolerances = test_close[True-1] atol=0.002 test_close[True-1] atol=0.0005 test_close* atol=0.001 test_close* atol=0.0001
Bad example, general takes precedence:
allclose_tolerances = test_close* atol=0.001 test_close* atol=0.0001 test_close[True-1] atol=0.002 test_close[True-1] atol=0.0005
Returns a function checking if two arrays are close, mimicking
_allclose(a, b, rtol=1e-05, atol=1e-08, xtol=0, equal_nan=False, print_fail=5, record_rmse=True)
First array to be compared.
Second array to be compared.
- rtolfloat, optional
Relative tolerance between a and b (relative to b).
- atolfloat, optional
Absolute tolerance between a and b.
- xtolint, optional
Allow signals to be right or left shifted by up to
xtolindices along the first axis
- equal_nanbool, optional
If True, nans will be considered equal to nans.
- print_failint, optional
If > 0, print out the first
print_failentries failing the allclose check along the first axis.
- record_rmsebool, optional
Whether to record the RMSE value for this comparison. Defaults to True. Set to False whenever
bshould be far apart (when ensuring two signals are sufficiently different, for example).
True if the two arrays are considered close according to the tolerances.
Report RMSEs recorded by the allclose fixture in the Pytest terminal.
This function helps with reporting recorded root mean squared errors (RMSEs). These RMSEs offer a measure of performance for each test by quantifying how close their outputs are to the target values. While this metric has some value on its own, it is most useful as a relative metric, to evaluate if change offers an improvement to tests, and if so, how much.
When using RMSEs, it is important to set
record_rmseto False on any
allclosecall where closer values correspond to a drop in performance (e.g. when using
allcloseto ensure values are different).
The terminal reporter object provided by
- relativebool, optional
Whether to print relative (default) or absolute RMSEs. Relative RMSEs are normalized by the mean RMS of
allclose. Since different tests often compare against values of different magnitudes, relative RMSEs provide a better metric across tests by ensuring all tests contribute proportionally to the average RMSE. One exception is when comparing to a signal that is all zeros, since the relative RMSE will always be 2 no matter how close the values are.
See RMSE error reporting for an example.