-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sql-server Source #18
Conversation
* add destination * add update operation * add tests and coltypes conversion * remove primary key field * add destination readme * add mock
* add destination * add update operation * add tests and coltypes conversion * remove primary key field * add destination readme * add mock
* add configs * Destination (#5) * add destination * add update operation * add tests and coltypes conversion * remove primary key field * add destination readme * add mock * Destination (#7) * add destination * add update operation * add tests and coltypes conversion * remove primary key field * add destination readme * add mock * Source (#8) * add destination * add update operation * add tests and coltypes conversion * remove primary key field * add destination readme * add mock * add combined iterator * add cdc iterator
* change key logic add tests * change key logic, add tests
@maksenius I'm really looking into this. What is missing? |
Hello @AndryHTC However, if you'd like to test it out, you can checkout this branch, build the connector binary and use it in conduit. refer to our docs for more info about that. |
Great @maha-hajja I'm going to play with it next week. I'm trying to track table wide changes, and query subscribed changes. Thank you for your incredible contribution, keep up 🔥 |
I attempted to test out the connector as you suggested, but encountered an error when adding it as a source. The error message is as follows:
It appears to be an issue with multiple identity columns being specified for the 'CONDUIT_TRACKING_STRUCTURE' table. As far as I understand, only a single identity column is allowed per table in MS SQL. Do you have any suggestions or workarounds for this issue? Thank you in advance for your assistance. |
@AndryHTC I'm not the one who developed this connector, but I did take a quick look.
which is trying to add the identity column using this query |
Hello @maha-hajja, Thank you for the detailed explanation and the insights. I've done some more investigation based on your suggestions and here's what I found. You're correct that the The original table has an identity column, and the tracking table is being created with an identical structure, including the identity property on the same column. After that, the code tries to add another identity column to the tracking table, which is not allowed in SQL Server since a table can have only one identity column. To resolve this issue, the creation of the tracking table must be adjusted to not copy the identity property from the original table's identity column. A potential solution could involve a series of SQL operations to "move" the identity property to another column in the table:
Here's an example in SQL:
I understand that this might involve some substantial changes to the code. I hope my findings and suggestions will be helpful for your team when you get to review this connector. Thank you again for your assistance and I'm looking forward to future developments on this. |
@AndryHTC as for the solution, I think the fact that not all tables have an identity column(which is probably the case the developers for this connector tested), makes us need to get to a new solution, because we can't assume that the identity column exists. We could replace the logic for creating the tracking table with a method that copies only the column names and types, so a bit more lengthy but should work. Thanks again for looking into this, and will let you know when any progress is made here. Also, any contributions to this repo or any other repos are welcome ;D |
Description
add source connector
Fixes #38
Quick checks: