nbgrader autograde does not find submissions

[ykazakov]

We prefer running autograde from the command line rather than from the formgrader so that one can autograde all submissions of a single assignment at the same time. However, with default settings nbgrader autograde does not seem to find the downloaded submissions:

$ nbgrader autograde ps1
[AutogradeApp | ERROR] No notebooks were matched by '/home/instructor/cs101/submitted/*/ps1'

It seems to be that in formgrader the submissions are downloaded to the directory collected rather than submitted.

Looks like the autograder in formgrader is able to find the submissions in the different directory.

Upon the closer inspection of the source code, there appears to be an (undocumented) flag NAAS_FEATURE_MULTI_MARKERS that reverts the submitted directory to the nbgrader default location:

nbexchange/nbexchange/plugin/list.py

Lines 309 to 320 in 985796a

	# This is the code that changes the submitted directory
	# away from default.
	# The feature flag NAAS_FEATURE_MULTI_MARKERS is also used
	# to revert to default nbgrader behaviour
	# Once the feature flag becomes the default, `self.fetched_root`
	# can be ditched
	#####
	if not os.environ.get("NAAS_FEATURE_MULTI_MARKERS"):
	if self.path_includes_course:
	self.coursedir.submitted_directory = os.path.join(self.coursedir.course_id, "collected")
	else:
	self.coursedir.submitted_directory = "collected"

So, a workaround to this problem is to set, e.g.,

NAAS_FEATURE_MULTI_MARKERS=1

Still, it would be nice to make sure that the autograder command would work even without setting this flag.

[perllaghu]

Yes.... this is a complication due to poor design on our part in the early days.

I really do need to correct this

[the whole "multi-marker" thing is something from our in-house service that moves the whole root directory to a different NFS mount, and thus outside a users quota]

[ykazakov]

There is an nbgrader option submitted_directory to set the submitted directory. Why not to use it instead?

## The name of the directory that contains assignments that have been submitted
#  by students for grading. This corresponds to the `nbgrader_step` variable in
#  the `directory_structure` config option.
#  Default: 'submitted'
# c.CourseDirectory.submitted_directory = 'submitted'

It could be easily overridden in nbgrader_config.py using the environment variable NAAS_FEATURE_MULTI_MARKERS like you do.

[perllaghu]

Because I was not clever then.... and it's now biting me [others]

[ykazakov]

@perllaghu That was not meant to be an offence. :-)
I just wanted to point out that the issue could be easily fixed by just removing this part of the code completely and setting the option submitted_directory for your specific use case.

I just verified that adding the next line to nbgrader_config.py:

c.CourseDirectory.submitted_directory = 'collected'

successfully changes the the submissions directory for both formgrader and the command line actions even with NAAS_FEATURE_MULTI_MARKERS=1 (which disables the patch in list.py).

[perllaghu]

Oh.... absolutely not taken as an offence: I made some very poor choices in the early days, and "painted myself into a corner" - and was stuck with what I had written.

In retrospect, changing the config would be the way to go.... however I didn't [and keep getting side-tracked with other work to fix it properly]

Keep these observations/suggestions/comments coming - and feel free to fork & offer feedback :)