-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Testing entry point #6
Comments
What happens in the case where it chooses to not assume this? Or should this read "A test entry point MUST...". Its unclear if this RFC allows for running unit tests outside of a roux-server environment. Does it? I think it should. In other words, I should not have to run a roux-server to run unit tests, I should be able to run these on the command line and view the results in the terminal similar to how rmn's grunt test command works now. |
Yeah, good question @lzilioli, that passage is kinda muddy. What I meant is that the test entry point for an ingredient can always assume that a Roux server will be running, with its ingredient preview, assets, etc. available. This will be the case even when running the tests from the CLI. The entry point doesn't have to make use of the server, of course. This is so that a single test entry point can perform both unit and functional tests. I guess another way to do this would be to only provide that environment variable if the server is running. That might make it easier to run unit tests more quickly. |
I think we should have two separate entry points; one for functional tests and one for unit tests. I don't like the idea of enabling developers of tests to conflate the two since they generally test distinctly different things. The functional tests can assume there MUST be the environment variable while the unit tests MUST work in the absence of said environment variables.
|
Users are still free - and encouraged - to separate their functional and unit tests. There are at least a couple other kinds of tests, however: visual regression and performance. I suspect we could come up with one or two more if we wanted. Rather than have an entry point for each flavor of test, I prefer to have a single entry point that speaks a common API (TAP). As an aside, this is what I plan to propose for linting as well: a single entry point that will run whatever linting tasks the ingredient desires and report the results via a single consistent interface. |
One note, if the spec is going to enforce details of the environment (eg: that a roux server is running):
It should probably also be able to control that environment. I'm thinking specifically about the version of roux server that is running; maybe that means using a |
|
@knksmith57 I would expect the pantry to have a |
So, I think there's a place for unit tests that run in node instead of a browser. That said, I haven't really contemplated non-client-side ingredients. Can you explain a bit further what you're thinking of?
I guess they would put it in whatever config file we choose in #17. The Roux test runner should also pass through command line arguments.
The Roux test runner should probably have some sort of configurable timeout. |
Thinking about this further, I'm coming around to @lzilioli's view that it probably doesn't make sense to assume the server is always running. I'll change this to say that the test entry point MAY receive the ingredient URL via that environment variable. |
I'm going to draft a new version of this RFC and submit it as a PR into a new |
The RFC proposes we specify the behavior of a test entry point. [Fix RetailMeNotSandbox#6]
@lawnsea is this work still underway? Can you close out the RFC issue and update the other issues to reflect an in-progress state where appropriate? |
@lzilioli I am still working on this, yes. It got bumped by other work. I hope to pick it back up next week. |
Summary
Problem
We want ingredients and pantries to be easy to test. Specifically, it should be easy to write both unit and functional tests, easy to run those tests, and easy to obtain coverage data.
This RFC proposes the ingredient interface to support these goals.
Solution Overview
Ingredients may define a test entry point (
test.js
) that runs the ingredient's unit and functional tests and reports TAP results. The test harness will ensure a Roux server is running before executing the test entry point. It will provide the URL for that server as well as any other configuration information via environment variables. The test harness will be able to execute the test entry points of multiple ingredients and report the aggregate results.API
Test entry point
An ingredient MAY define a test entry point by including a file named
test.js
in the ingredient root. If present, the script MUST report its result by printing a Test Anything Protocol (TAP) test stream tostdout
.A test entry point MAY assume that a Roux server is running and WILL receive the root URL for its ingredient as an environment variable named
ROUX_INGREDIENT_URL
.A test entry point MUST NOT assume that it is the only test entry point executing. The test runner MAY execute multiple entry points concurrently, so access to any shared resources MUST be coordinated via other means.
The text was updated successfully, but these errors were encountered: