Slow tests

[clegaard]

Currently, a small number of tests are taking disproportionatly big chunk of the execution time.
Running: pytest --durations=0 reveals

33.56s call     tests/datasetops_tests/test_caching.py::test_cache
4.22s call     tests/datasetops_tests/test_loaders.py::test_tfds
3.74s call     tests/datasetops_tests/test_datasets.py::test_to_tensorflow
2.19s call     tests/datasetops_tests/test_examples.py::test_readme_example_2
1.13s call     tests/datasetops_tests/test_stream_dataset.py::test_read_from_file
0.72s call     tests/datasetops_tests/test_datasets.py::test_to_pytorch
0.63s call     tests/datasetops_tests/test_examples.py::test_domain_adaptation
0.53s call     tests/datasetops_tests/test_transformation_graph.py::test_serialization_not_same
0.29s call     tests/datasetops_tests/test_datasets.py::TestSubsample::test_subsample
0.26s call     tests/datasetops_tests/test_transformation_graph.py::test_operation_origins
0.22s call     tests/datasetops_tests/test_transformation_graph.py::test_serialization_same
0.13s call     tests/datasetops_tests/test_datasets.py::test_image_to_tensorflow
0.06s call     tests/datasetops_tests/test_caching.py::test_cacheable
0.05s call     tests/datasetops_tests/test_transformation_graph.py::test_common_nodes_equality
0.04s call     tests/datasetops_tests/test_transformation_graph.py::test_roots_kitti
0.03s call     tests/datasetops_tests/test_loaders.py::test_mat_single_with_multi_data
0.02s call     tests/datasetops_tests/test_loaders.py::test_pytorch
0.02s call     tests/datasetops_tests/test_examples.py::test_readme_example_1
0.02s call     tests/datasetops_tests/test_loaders.py::TestLoadCSV::test_names_missing
0.02s call     tests/datasetops_tests/test_transformation_graph.py::test_roots_tfds
0.01s call     tests/datasetops_tests/test_datasets.py::test_image_resize
0.01s call     tests/datasetops_tests/test_scaler.py::test_center
0.01s call     tests/datasetops_tests/test_datasets.py::TestSubsample::test_caching
0.01s call     tests/datasetops_tests/test_scaler.py::test_item_stats
0.01s call     tests/datasetops_tests/test_loaders.py::test_folder_dataset_class_data

I see two options.

1. Make them faster
2. Make it easy only to run the fasts tests

If we go with option 2 we could do

# content of conftest.py

import pytest


def pytest_addoption(parser):
    parser.addoption(
        "--runslow", action="store_true", default=False, help="run slow tests"
    )


def pytest_configure(config):
    config.addinivalue_line("markers", "slow: mark test as slow to run")


def pytest_collection_modifyitems(config, items):
    if config.getoption("--runslow"):
        # --runslow given in cli: do not skip slow tests
        return
    skip_slow = pytest.mark.skip(reason="need --runslow option to run")
    for item in items:
        if "slow" in item.keywords:
            item.add_marker(skip_slow)

[iliiliiliili]

We can't make caching tests faster because we have to test all the operations in different scenarios.
As for me, slow tests start from 5 minutes long duration. Why is it a problem for you to have 30s tests?

[clegaard]

We can't make caching tests faster because we have to test all the operations in different scenarios.
As for me, slow tests start from 5 minutes long duration. Why is it a problem for you to have 30s tests?

That everytime anyone wants to verify a change they have to wait ~40 seconds.
I am not saying the test should necessarily run faster, im saying there should be an easy way to avoid running expensive tests

[LukasHedegaard]

I agree that it would be very nice to shorten tests times a bit, and I think we can gain a bit in the from_tensorflow loader.
Another option may be to investigate whether tests can be run in parallel.