Use more extensive test suite

[h-vetinari]

Only testing the import of the python packages is not optimal. We should be using as much of the upstream test suite as possible in our CI (and with a timeout of 6h, we have ample time for testing). After a quick search upstream, these are things that are probably relevant:

• https://github.com/ray-project/ray/blob/ray-1.0.1/.bazelrc#L107-L111
• https://github.com/ray-project/ray/blob/ray-1.0.1/cpp/BUILD.bazel#L60-L85
• https://github.com/ray-project/ray/tree/ray-1.0.1/python/ray/tests
• Some more tests in other places, e.g. https://github.com/ray-project/ray/tree/ray-1.0.1/release

CC @vnlitvinov

[krfricke]

I agree with testing Ray core functionality by running some of the unit tests.

Here is our current Travis pipeline. The following three suites (Streaming, Worker, Tests) should be sufficient and run in ~20 minutes total. We could also opt to run ASAN tests (adding another 30 minutes).

https://github.com/ray-project/ray/blob/master/.buildkite/pipeline.yml#L102-L121

We can also find a (small) subset of sub library tests (for Ray Tune, Serve, RLLib) to run. In my opinion a very narrow set of basic functionality should suffice, as it should generally just serve as a smoke test to decide if the build succeeded and the sub libraries can actually be used at all. For this we will probably add another tag to the bazel rcs in upstream ray to be able to filter these.

[vnlitvinov]

I think we should be good with running some smoke tests, as we take already tested Ray sources for the start.
So we just want to make sure that we didn't screw something up in patching and packaging, thus a bit of everything that pokes most binaries but runs fast is a good thing to get.

This would also benefit the core developers, as it would allow running smoke tests during development...

[h-vetinari]

I don't know how long the full test suite for all things would run, but I prefer to be more comprehensive than just smoke tests, because various corner cases are often only caught by a handful of tests. I'd be fine if running the test suite adds an hour or a bit more to our CI.

[h-vetinari]

@krfricke
Hope you don't mind, I took the liberty of assigning you to this issue. 🙃