test: fix TTML integration test where ordering was changed by #1364

[cosmin]

No description provided.

[cosmin]

We should turn on integration tests in CI now that they are stable

[joeyparrish]

We should turn on integration tests in CI now that they are stable

It should just be a matter of adding it to CMake, right?

[cosmin]

It's currently skipped in CMake by SKIP_INTEGRATION_TESTS which is ON by default. We could add a separate job just to run the integration tests, either by rebuilding with -DSKIP_INTEGRATION_TESTS=OFF or by directly invoking packager_test.py

[cosmin]

We could also turn them on by default although given the tests take a while to run I think it's convenient to get early feedback from running the unit tests without integration tests.

[joeyparrish]

I bet changing the default won't seriously increase the job time in CI. I'll do a local test to ballpark it.

[joeyparrish]

Clean, configure, build, and test w/o integration tests on my workstation at home:
4:49

Clean, configure, build, and test w/ integration tests on my workstation at home:
5:38

An increase of 49s

Clean, configure, build, and test w/o integration tests in fastest CI configuration:
5:47

Clean, configure, build, and test w/o integration tests in slowest CI configuration:
18:07

I would guess that turning it on by default wouldn't delay the last test result by more than 1-3 minutes.

[cosmin]

I can send a pull request to flip this on by default, although the test times on your workstation seem concerning. On my laptop running

ctest --test-dir build -j 8

runs in 18s

[cosmin]

Ah, this includes a full rebuild of everything including all the 3rd party dependencies.

Something still seems odd though, for me the integration tests take about ~5 minutes to run on their own, because the default unit test runner in Python won't parallelize the tests. I guess the difference could come down to single core performance.

[cosmin]

sent #1381 to turn on the integration tests and see how it looks in CI