Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enforce pickleability for v2 transforms and wrapped datasets #7860

Merged
merged 10 commits into from
Aug 24, 2023

Conversation

pmeier
Copy link
Collaborator

@pmeier pmeier commented Aug 21, 2023

Fixes #6753 (comment).

  • Fortunately, all our transforms v2 were already pickleable. Thus, this PR just adds tests to enforce this in the future.
  • Due to the dynamic type of the wrapped dataset (see add support for instance checks on dataset wrappers #7239), we need to "help" pickle a little for it to be able to deserialize the object. We only need to overwrite the __reduce__ method. Again, we also add tests to enforce pickleability in the future.

cc @vfdev-5

@pytorch-bot
Copy link

pytorch-bot bot commented Aug 21, 2023

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7860

Note: Links to docs will display an error until the docs builds have been completed.

❌ 29 New Failures, 1 Unrelated Failure

As of commit d256c3f with merge base 92882b6 (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

test/datasets_utils.py Outdated Show resolved Hide resolved
torchvision/datapoints/_dataset_wrapper.py Outdated Show resolved Hide resolved
test/datasets_utils.py Outdated Show resolved Hide resolved
Copy link
Member

@NicolasHug NicolasHug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Philip, some comments below but LGTM if green.

Perhaps we'll want to add more tests w.r.t. multiprocessing_context="spawn"

@pmeier
Copy link
Collaborator Author

pmeier commented Aug 22, 2023

With c68a6de

$ pytest --durations=25 test/test_datasets.py -k v2
[...]
========================================== slowest 25 durations ==========================================
16.05s call     test/test_datasets.py::VOCDetectionTestCase::test_transforms_v2_wrapper
8.05s call     test/test_datasets.py::CityScapesTestCase::test_transforms_v2_wrapper
8.02s call     test/test_datasets.py::CelebATestCase::test_transforms_v2_wrapper
5.38s call     test/test_datasets.py::CocoDetectionTestCase::test_transforms_v2_wrapper
5.23s call     test/test_datasets.py::KittiTestCase::test_transforms_v2_wrapper
3.01s call     test/test_datasets.py::SBDatasetTestCase::test_transforms_v2_wrapper
2.86s call     test/test_datasets.py::HMDB51TestCase::test_transforms_v2_wrapper
2.84s call     test/test_datasets.py::Caltech101TestCase::test_transforms_v2_wrapper
2.82s call     test/test_datasets.py::KineticsTestCase::test_transforms_v2_wrapper
2.82s call     test/test_datasets.py::UCF101TestCase::test_transforms_v2_wrapper
2.74s call     test/test_datasets.py::CIFAR10TestCase::test_transforms_v2_wrapper
2.73s call     test/test_datasets.py::CIFAR100::test_transforms_v2_wrapper
2.70s call     test/test_datasets.py::ImageNetTestCase::test_transforms_v2_wrapper
2.70s call     test/test_datasets.py::VOCSegmentationTestCase::test_transforms_v2_wrapper
2.69s call     test/test_datasets.py::KMNISTTestCase::test_transforms_v2_wrapper
2.68s call     test/test_datasets.py::ImageFolderTestCase::test_transforms_v2_wrapper
2.67s call     test/test_datasets.py::MNISTTestCase::test_transforms_v2_wrapper
2.66s call     test/test_datasets.py::DatasetFolderTestCase::test_transforms_v2_wrapper
2.66s call     test/test_datasets.py::FashionMNISTTestCase::test_transforms_v2_wrapper
2.66s call     test/test_datasets.py::EMNISTTestCase::test_transforms_v2_wrapper
2.64s call     test/test_datasets.py::OxfordIIITPetTestCase::test_transforms_v2_wrapper
2.62s call     test/test_datasets.py::QMNISTTestCase::test_transforms_v2_wrapper
0.07s call     test/test_datasets.py::WIDERFaceTestCase::test_transforms_v2_wrapper
0.07s call     test/test_datasets.py::Caltech256TestCase::test_transforms_v2_wrapper
0.03s setup    test/test_datasets.py::Caltech101TestCase::test_transforms_v2_wrapper
======================================== short test summary info =========================================
FAILED test/test_datasets.py::Caltech256TestCase::test_transforms_v2_wrapper - FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
FAILED test/test_datasets.py::WIDERFaceTestCase::test_transforms_v2_wrapper - ValueError: bad value(s) in fds_to_keep
================== 2 failed, 22 passed, 1 skipped, 479 deselected in 101.67s (0:01:41) ===================

On main and without the DataLoader test

$ pytest test/test_datasets.py -k v2
[...]
====================== 68 passed, 191 skipped, 479 deselected, 5 warnings in 8.36s =======================

Meaning, we are adding roughly 1.5 minutes of test time.

@pmeier
Copy link
Collaborator Author

pmeier commented Aug 22, 2023

After 5358620

$ pytest --durations=25 test/test_datasets.py -k v2
[...]
========================================== slowest 25 durations ==========================================
4.61s call     test/test_datasets.py::CityScapesTestCase::test_transforms_v2_wrapper
4.57s call     test/test_datasets.py::CelebATestCase::test_transforms_v2_wrapper
4.48s call     test/test_datasets.py::VOCDetectionTestCase::test_transforms_v2_wrapper
1.63s call     test/test_datasets.py::HMDB51TestCase::test_transforms_v2_wrapper
1.58s call     test/test_datasets.py::UCF101TestCase::test_transforms_v2_wrapper
1.57s call     test/test_datasets.py::SBDatasetTestCase::test_transforms_v2_wrapper
1.56s call     test/test_datasets.py::Caltech101TestCase::test_transforms_v2_wrapper
1.51s call     test/test_datasets.py::KineticsTestCase::test_transforms_v2_wrapper
1.51s call     test/test_datasets.py::QMNISTTestCase::test_transforms_v2_wrapper
1.49s call     test/test_datasets.py::VOCSegmentationTestCase::test_transforms_v2_wrapper
1.48s call     test/test_datasets.py::CocoDetectionTestCase::test_transforms_v2_wrapper
1.48s call     test/test_datasets.py::ImageNetTestCase::test_transforms_v2_wrapper
1.47s call     test/test_datasets.py::ImageFolderTestCase::test_transforms_v2_wrapper
1.47s call     test/test_datasets.py::DatasetFolderTestCase::test_transforms_v2_wrapper
1.47s call     test/test_datasets.py::CIFAR10TestCase::test_transforms_v2_wrapper
1.47s call     test/test_datasets.py::OxfordIIITPetTestCase::test_transforms_v2_wrapper
1.46s call     test/test_datasets.py::KMNISTTestCase::test_transforms_v2_wrapper
1.45s call     test/test_datasets.py::MNISTTestCase::test_transforms_v2_wrapper
1.43s call     test/test_datasets.py::FashionMNISTTestCase::test_transforms_v2_wrapper
1.43s call     test/test_datasets.py::EMNISTTestCase::test_transforms_v2_wrapper
1.43s call     test/test_datasets.py::CIFAR100::test_transforms_v2_wrapper
1.42s call     test/test_datasets.py::KittiTestCase::test_transforms_v2_wrapper
0.02s setup    test/test_datasets.py::Caltech101TestCase::test_transforms_v2_wrapper
0.01s call     test/test_datasets.py::WIDERFaceTestCase::test_transforms_v2_wrapper

(1 durations < 0.005s hidden.  Use -vv to show these durations.)
======================================== short test summary info =========================================
FAILED test/test_datasets.py::Caltech256TestCase::test_transforms_v2_wrapper - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp731zruyt/caltech256/256_ObjectCatego...
FAILED test/test_datasets.py::WIDERFaceTestCase::test_transforms_v2_wrapper - ValueError: bad value(s) in fds_to_keep
================== 2 failed, 22 passed, 1 skipped, 479 deselected, 3 warnings in 44.24s ==================

Meaning, we are down to roughly 30 seconds of extra time.

@pmeier
Copy link
Collaborator Author

pmeier commented Aug 22, 2023

After f339e6c while pretending I'm on macOS

$ pytest test/test_datasets.py -k v2
[...]
========================================== slowest 25 durations ==========================================
4.43s call     test/test_datasets.py::CityScapesTestCase::test_transforms_v2_wrapper
4.42s call     test/test_datasets.py::CelebATestCase::test_transforms_v2_wrapper
4.39s call     test/test_datasets.py::VOCDetectionTestCase::test_transforms_v2_wrapper
1.71s call     test/test_datasets.py::SBDatasetTestCase::test_transforms_v2_wrapper
1.57s call     test/test_datasets.py::KineticsTestCase::test_transforms_v2_wrapper
1.49s call     test/test_datasets.py::Caltech101TestCase::test_transforms_v2_wrapper
1.48s call     test/test_datasets.py::VOCSegmentationTestCase::test_transforms_v2_wrapper
1.47s call     test/test_datasets.py::CocoDetectionTestCase::test_transforms_v2_wrapper
1.47s call     test/test_datasets.py::KittiTestCase::test_transforms_v2_wrapper
1.43s call     test/test_datasets.py::OxfordIIITPetTestCase::test_transforms_v2_wrapper
1.42s call     test/test_datasets.py::ImageNetTestCase::test_transforms_v2_wrapper
0.03s setup    test/test_datasets.py::Caltech101TestCase::test_transforms_v2_wrapper
0.02s call     test/test_datasets.py::WIDERFaceTestCase::test_transforms_v2_wrapper

(12 durations < 0.005s hidden.  Use -vv to show these durations.)
======================================== short test summary info =========================================
FAILED test/test_datasets.py::WIDERFaceTestCase::test_transforms_v2_wrapper - ValueError: bad value(s) in fds_to_keep
================== 1 failed, 11 passed, 1 skipped, 479 deselected, 1 warning in 27.57s ===================

We are down to ~20 seconds of extra runtime compared to main. Note that we still have a failure that will add a few seconds as well when fixed. Plus, the benchmark was done on my machine, which is more powerful than our macOS CI runners. So the extra time will likely be higher in CI.

@@ -548,7 +549,7 @@ def test_feature_types(self, config):
@test_all_configs
def test_num_examples(self, config):
with self.create_dataset(config) as (dataset, info):
assert len(dataset) == info["num_examples"]
assert len(list(dataset)) == len(dataset) == info["num_examples"]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We never actually consumed the dataset before. Thus, any failures that happen not for the first sample are not detected. Fortunately, only one test was broken that I'll flag below.


class Caltech256TestCase(datasets_utils.ImageDatasetTestCase):
DATASET_CLASS = datasets.Caltech256

def inject_fake_data(self, tmpdir, config):
tmpdir = pathlib.Path(tmpdir) / "caltech256" / "256_ObjectCategories"

categories = ((1, "ak47"), (127, "laptop-101"), (257, "clutter"))
categories = ((1, "ak47"), (2, "american-flag"), (3, "backpack"))
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datasets.Caltech relies on the fact that all categories are present. When actually consuming the dataset (see above), the old fake data setup falls flat. Our options are:

  1. Fix the dataset to account for gaps in the catgories.
  2. Create all categories as fakedata
  3. Create fakedata that starts at the first without any gaps, but not all categories.

Option 3. is by far the least amount of work, so I went for that here.

"occlusion": labels_tensor[:, 7],
"pose": labels_tensor[:, 8],
"invalid": labels_tensor[:, 9],
"bbox": labels_tensor[:, 0:4].clone(), # x, y, width, height
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Views on tensor cannot be pickled correctly. Meaning regardless of the v2 wrapper, datasets.Widerface has never worked in a spawn context.

@NicolasHug
Copy link
Member

Still LGTM, but the test duration is a bit unfortunate.

I think we'd be better-off testing just 2-3 of these datasets (or even just 1 TBH, as one test is still infinitely better than zero). Also ideally we'd let them having transforms, so that what we're testing is closer to a real-world use-case.

@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:01 — with GitHub Actions Inactive
@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:01 — with GitHub Actions Inactive
@pmeier pmeier merged commit 054432d into pytorch:main Aug 24, 2023
32 of 62 checks passed
@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:01 — with GitHub Actions Inactive
@pmeier pmeier deleted the pickle branch August 24, 2023 08:02
@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:02 — with GitHub Actions Inactive
@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:02 — with GitHub Actions Inactive
@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:02 — with GitHub Actions Inactive
@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:02 — with GitHub Actions Inactive
@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:02 — with GitHub Actions Inactive
@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:02 — with GitHub Actions Inactive
@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:03 — with GitHub Actions Inactive
@pmeier pmeier temporarily deployed to pytorchbot-env August 24, 2023 08:03 — with GitHub Actions Inactive
facebook-github-bot pushed a commit that referenced this pull request Sep 6, 2023
…#7860)

Summary: (Note: this ignores all push blocking failures!)

Reviewed By: matteobettini

Differential Revision: D48900370

fbshipit-source-id: be1b23dcab58d2a8b5bca7190f94c0123263d036
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEEDBACK] Transforms V2 API
3 participants