Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

additional goodies csv sftp perf sql and more #855

Merged
merged 13 commits into from
Dec 22, 2023

Conversation

keevan
Copy link
Contributor

@keevan keevan commented Dec 21, 2023

  • feat: add connector SQL step
  • fix: allow values to be corrected in bad states
  • feat: add set multiple variables step type
  • fix: warning logs now display and are more legible
  • fix: add copy file error handling for directory creation
  • feat: run name and id are now available as expressable vars
  • fix: SQL trait handling of integer values and improve error message
  • fix: update email flow step to support it as a final step
  • fix: reader csv error to be more correct for header/fields count alignment
  • fix: include row number in csv reader error
  • perf: cache sftp object per identity to speed up subsequent actions
  • feat: add an option to continue processing the csv file on known parse issues

For the SFTP one in particular, it was doing about 200ms initial connection time, 600-700ms initial upload time, but otherwise breezing for subsequent SFTP actions, unlike previously.

Instead of allowing the state to be not recoverable, this will instead fail to render the available variables section only.
@keevan keevan force-pushed the additional-goodies-csv-sftp-perf-sql-and-more branch from a996838 to a9c82ca Compare December 21, 2023 23:55
Similar to set_variable but this allows multiple values to be set on an existing or new path.
This simplifies initialisation of multiple variables per instance run, such as counters used for reporting.
- The background with white text was difficult to read (low contrast)
- The warning regex previously in place was incorrect
Also tweaked comment on string format of run based output logs.
…nment

It is more correct to expect the number of fields to match the number of headers, not the other way around necessarily.
…e issues

The only one is currently a count mismatch on header and the row fields
@keevan keevan force-pushed the additional-goodies-csv-sftp-perf-sql-and-more branch from a9c82ca to 7ffa407 Compare December 22, 2023 03:34
Copy link
Contributor

@Peterburnett Peterburnett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved to merge once CI green

@keevan keevan force-pushed the additional-goodies-csv-sftp-perf-sql-and-more branch from 3e637df to f7485ef Compare December 22, 2023 04:05
@keevan keevan merged commit 66cf26e into MOODLE_35_STABLE Dec 22, 2023
25 checks passed
@keevan keevan deleted the additional-goodies-csv-sftp-perf-sql-and-more branch December 22, 2023 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants