Running test locally fail

[eamanu]

Hi everybody,

I am working on the packaging of idseq-dag on Debian. But is failling the tests.

Beyond that, I am trying to run the tests locally, but I've this error:

sudo python3 -m unittest tests/test_samples_on_local_steps.py 
E{"time": "2020-04-21T22:09:03.245", "data": {"event": "ctx_exec", "context_name": "command.make_dirs", "uid": "3f3533f1285b", "values": {"path": "/mnt/idseq/results/star_out/257549"}, "duration_ms": 13}, "thread": "MainThread", "pid": 257549, "level": "INFO"}
{"time": "2020-04-21T22:09:03.246", "data": {"event": "ctx_exec", "context_name": "command.make_dirs", "uid": "1441993ea123", "values": {"path": "/mnt/idseq/ref"}, "duration_ms": 0}, "thread": "MainThread", "pid": 257549, "level": "INFO"}
E
======================================================================
ERROR: test_all_local_steps (tests.test_samples_on_local_steps.TestSamplesOnLocalSteps)
----------------------------------------------------------------------
TypeError: test_all_local_steps() missing 3 required positional arguments: 'dag_file', 'test_bundle', and 'output_dir_s3'

======================================================================
ERROR: test_many_samples (tests.test_samples_on_local_steps.TestSamplesOnLocalSteps)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/eamanu/Debian/idseq-dag/github/idseq-dag/tests/test_samples_on_local_steps.py", line 37, in test_many_samples
    self.test_all_local_steps(dag_file, bundle, output_dir_s3)
  File "/home/eamanu/Debian/idseq-dag/github/idseq-dag/tests/test_samples_on_local_steps.py", line 55, in test_all_local_steps
    step_class, step_name, dag_file, test_bundle, output_dir_s3)
  File "/home/eamanu/Debian/idseq-dag/github/idseq-dag/tests/test_utils.py", line 84, in run_step_and_match_outputs
    test_bundle, output_dir_s3)
  File "/home/eamanu/Debian/idseq-dag/github/idseq-dag/tests/idseq_step_setup.py", line 95, in get_test_step_object
    result_dir_local)
  File "/home/eamanu/Debian/idseq-dag/github/idseq-dag/idseq_dag/engine/pipeline_flow.py", line 173, in fetch_input_files_from_s3
    output_file = idseq_dag.util.s3.fetch_from_s3(s3_file, local_dir, allow_s3mi=True)
  File "/home/eamanu/Debian/idseq-dag/github/idseq-dag/idseq_dag/util/s3.py", line 299, in fetch_from_s3
    if is_reference or os.path.abspath(dst).startswith(config["REF_DIR"]):
TypeError: startswith first arg must be str or a tuple of str, not NoneType

----------------------------------------------------------------------
Ran 2 tests in 0.017s

FAILED (errors=2)

Looking on the code seems like PipelineFlow set the config['REF_DIR']

idseq_dag.util.s3.config["REF_DIR"] = self.ref_dir_local 

but that configuration must be saved inside on the PipelineFlow to have persistence
of that dict (or some different way), for that reason whe the test run fetch_from_s3
config is set by default.

[kislyuk]

Hi @eamanu, thank you for your interest in IDseq and apologies for the delay in responding to you. I am migrating your issue to https://github.com/chanzuckerberg/idseq-workflows, where we will respond to your questions in full.