runtest.py broken on Windows

[Shadlock0133]

This script depends on functionality provided only on unix system. It would be good idea to fix it and make it more portable.

[kanaka]

As long as Windows support is a fairly straightforwards change, I'd be happy to accept a PR. However, runtest.py pretty specific to unix PTY details in order to simulate interactive REPL access so I suspect it might be a bit difficult to add Windows support. I don't have regular access to a Windows system so I can't do it myself. I suspect a better route would be documenting how to use docker on Windows.

All the Travis testing is reliant on docker to pull down build/runtime tools for all the implementations. In order to get a new implementation into the tree you need to support docker testing in Travis. So I think requiring docker on Windows is probably a more direct route to supporting mal testing on Windows (i.e. runtest.py runs within docker so no Windows port of it is necessary).

[dubek]

Here's my experience: On Windows 10 with Docker Desktop I have the Mal git repo cloned in C:\personal\mal; to run the tests I run the following docker command in a PowerShell window:

docker run -it --rm -u 1000 -v c:/personal/mal/:/mal kanaka/mal-test-ocaml make test^ocaml^step2

Here's the full session:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Users\dmurik> cd C:\personal\mal\

PS C:\personal\mal> docker pull kanaka/mal-test-ocaml
Using default tag: latest
latest: Pulling from kanaka/mal-test-ocaml
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the docker.io registry NOW to avoid future disruption.
b8b0bfa9010a: Already exists
cab2395573ae: Already exists
7ebaa68f31db: Already exists
a3ed95caeb02: Already exists
b87e97cab920: Already exists
ef0e23f33732: Already exists
792744784e14: Already exists
e6468e8668e6: Already exists
5b1793a256f8: Already exists
Digest: sha256:1b98a83787f7f1e067b46dd89b7032bf3a759b83b2cb1e3d4595bfffedaf2cae
Status: Image is up to date for kanaka/mal-test-ocaml:latest
docker.io/kanaka/mal-test-ocaml:latest

PS C:\personal\mal> docker run -it --rm -u 1000 -v c:/personal/mal/:/mal kanaka/mal-test-ocaml make test^ocaml^step2
make -C ocaml step2_eval
make[1]: Entering directory '/mal/ocaml'
ocamlopt str.cmxa unix.cmxa mal_lib.cmxa step2_eval.ml -o step2_eval
make[1]: Leaving directory '/mal/ocaml'
----------------------------------------------
Testing test^ocaml^step2; step file: ocaml/step2_eval, test file: tests/step2_eval.mal
Running: env STEP=step2_eval MAL_IMPL=js ../runtest.py  --deferrable --optional   ../tests/step2_eval.mal -- ../ocaml/run
Testing evaluation of arithmetic operations
TEST: '(+ 1 2)' -> ['',3] -> SUCCESS
TEST: '(+ 5 (* 2 3))' -> ['',11] -> SUCCESS
TEST: '(- (+ 5 (* 2 3)) 3)' -> ['',8] -> SUCCESS
TEST: '(/ (- (+ 5 (* 2 3)) 3) 4)' -> ['',2] -> SUCCESS
TEST: '(/ (- (+ 515 (* 87 311)) 302) 27)' -> ['',1010] -> SUCCESS
TEST: '(* -3 6)' -> ['',-18] -> SUCCESS
TEST: '(/ (- (+ 515 (* -87 311)) 296) 27)' -> ['',-994] -> SUCCESS
TEST: '(abc 1 2 3)' -> ['.+',] -> SUCCESS
Testing empty list
TEST: '()' -> ['',()] -> SUCCESS

-------- Deferrable Functionality --------
Testing evaluation within collection literals
TEST: '[1 2 (+ 1 2)]' -> ['',[1 2 3]] -> SUCCESS
TEST: '{"a" (+ 7 8)}' -> ['',{"a" 15}] -> SUCCESS
TEST: '{:a (+ 7 8)}' -> ['',{:a 15}] -> SUCCESS

TEST RESULTS (for ../tests/step2_eval.mal):
    0: soft failing tests
    0: failing tests
   12: passing tests
   12: total tests

But if you plan to develop on Windows you'll need to pay attention to the usual discrepancies like newlines and file permissions. I prefer working in WSL or in a Linux VM so I don't have to deal with these.

[cy20lin]

I made a pull request (#640) which enable the runtest.py script to run mal implementation tests on Windows. You may refer to that PR and see if the script helps.