Make Dropbox listdir optionally recursive

[smrohrer]

Description

Make the Dropbox listdir function accept an optional keyword argument recursive, True by default to preserve the current behavior.

Motivation and Context

Currently, listdir calls the Dropbox SDK files_list_folder function with recursive=True hard-coded. recursive should be optional, so we can replicate the functionality of LocalFileSystem.listdir which is not recursive.

Have you tested this? If so, how?

• Added a unit test for non-recursive behavior
• Modified the other listdir tests to verify that by default, the list includes the nested file DROPBOX_TEST_FILE_IN_DIR
• Ran tox -e py37-dropbox locally

[dlstadther]

Code change and tests lgtm, with 1 confusing test assertion. I assume it's an underlying sdk implementation, but i would like to doublecheck.

[dlstadther]

I'm a little confused why the input path appears in the results during recursive=True, but not when recursive=False. Is that expected behavior for the underlying api call?

[smrohrer]

Thank you for the quick and diligent review.

I am closing this pull request because of a misunderstanding on my part: I confused the intended behavior of luigi.target.FileSystem.listdir with os.listdir - the former is meant to be recursive, and the latter is not.

However, I think the inclusion of the input path in the Dropbox listdir result does contradict the expected behavior of luigi.target.FileSystem.listdir. This is a behavior of the Dropbox files_list_folder function which is not described in the documentation.

If you think luigi.contrib.dropbox.DropboxClient.listdir should filter out the input path, I'd be happy to open another pull request.