fix(ingest): run sqllineage in process by default

[hsheth2]

I'm not exactly sure what the root cause here was, but it seems like our usage of multiprocessing was causing the main thread to hang on exit. Because we have test order randomization enabled for unit tests, it happened inconsistently and I used --random-order-seed=598371 to reproduce it - it required a specific order of tests to be run. Additionally, this bug would only manifest on Linux, which is why my attempts to repro the issue on mac did not work.

Thread 5918 (idle): "MainThread"
    poll (multiprocessing/popen_fork.py:27)
    wait (multiprocessing/popen_fork.py:43)
    join (multiprocessing/process.py:149)
    _exit_function (multiprocessing/util.py:357)

Main changes:

• Avoids all multiprocessing related stuff by default in both src and tests
• Defaults to in-process SQL parsing for the legacy sqllineage parser. This only impact the redash source
• Refactors redash to use ThreadedIteratorExecutor instead of multiprocessing
• Removes the sql_parser config option from lookml, since it was no longer supported.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[github-actions]

Hello @hsheth2 😄

Thank you so much for opening a pull request!

You can check out your contributor card and see all your past stats here!