[PECO-1803] Splitting the PySql connector into the core and the non core part #417
Conversation
Removed token removed token
Fixed __version__ Fix
jprakash-db
force-pushed
the
jprakash-db/PECO-1803
branch
from
4351410
to
c91d43d
Compare
databricks_sql_connector_core/src/databricks_sql_connector_core/sql/auth/auth.py
Outdated
Show resolved
Hide resolved
databricks_sql_connector_core/src/databricks_sql_connector_core/sqlalchemy/__init__.py
Outdated
Show resolved
Hide resolved
| @@ -0,0 +1,23 @@ | |||
| [tool.poetry] | |||
| name = "databricks-sql-connector" | |||
| version = "3.3.0" | |||
There was a problem hiding this comment.
As we change the package structure in this release. I would suggest to bump the major version to 4.0.0 or at least it should be greater than current release 3.4.0, so it has to start with 3.5.0. @kravets-levko wdyt?
README.md
Outdated
| @@ -60,7 +60,7 @@ In the above example: | |||
| - `server-hostname` is the Databricks instance host name. | |||
| - `http-path` is the HTTP Path either to a Databricks SQL endpoint (e.g. /sql/1.0/endpoints/1234567890abcdef), | |||
| or to a Databricks Runtime interactive cluster (e.g. /sql/protocolv1/o/1234567890123456/1234-123456-slid123) | |||
| - `personal-access-token` is the Databricks Personal Access Token for the account that will execute commands and queries | |||
| `personal-access-token` is the Databricks Personal Access Token for the account that will execute commands and queries | |||
There was a problem hiding this comment.
I think this is some accidental change - please revert it
| @@ -40,6 +44,7 @@ def fetch_rows(self, cursor, row_count, fetchmany_size): | |||
| + "assuming 10K fetch size." | |||
| ) | |||
|
|
|||
| @skipUnless(pysql_supports_arrow(), "Without pyarrow lz4 compression is not supported") | |||
There was a problem hiding this comment.
Let's stick with @pytest.mark.skipif everywhere (please check other test files as well)
poetry.lock
Outdated
There was a problem hiding this comment.
Now, when we have two projects - each of them needs its own lockfile. Keeping the old one at old path is useless, because Poetry won't use it.
Basically, there should be a new lockfile created for databricks_sql_connector. For databricks_sql_connector the old one should be reused, with unnecessary dependencies removed
jprakash-db
merged commit 4099939
into
databricks:PECO-1803/connector-split
PECO-1803
Related Links
databricks_sqlalchemy split is present in this PR - https://github.com/databricks/databricks-sqlalchemy/pull/1
Description
databricks-sql-python library is split so that package size can be reduced for the end user based on their requirements
Particularly pyarrow is the heavy component that is planned to be kept optional
existing library split into
Tasks Completed
How to Test
Testing pipeline remains the same as it is before the split.
pytest can be used to directly run both the integration as well as unit tests, by
pytest [directory_name or file_name]Addition of dist folder in this repo
Github actions have been setup in the databricks_sqlalchemy repo to run tests using the databricks_sql_connector_core. For running those tests currently we need the .whl file in the dist folder and for temporary testing it has been added to the PR.
Once the library gets published to a public repository such as PyPi then databricks_sqlalchemy will automatically download it from that repo and run the tests using Github actions
Performance Comparison - Benchmarking
The pre-split and post-split preformance comparison has been made using the large and small queries to make sure their is no regression of performance
Dashboard has been created so that everytime the benchmarking is run the result are stored in the benchfood, and comparisons can be made easily