You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This proposal supports the addition of a new configuration model, CsvReaderParams, to the existing DataContract model.
CsvReaderParams collects the existing "csv reader params" within the General and FileDefinition, so that all "csv reader params" are managed within a single document. Collected parameters include:
CsvReaderParams is implemented as an Optional field within the General and FileDefinition configurations. When CsvReaderParams is defined in both General and FileDefinition models, the FileDefinition.CsvReaderParams overrides values set in General.CsvReaderParams.
If CsvReaderParams is not configured within the General model, a default instance is created with appropriate defaults, based on the defaults for the 1.1.0 release.
emptyFieldValues = None
valueDelimiter = ","
convertColumnsToString = True
headers = None
skipRows = None
Finally the existing fields will remain where they are for a "reasonable" amount of time to allow users to migrate to the new configuration model.
The text was updated successfully, but these errors were encountered:
We have a requirement to configure some pandas reader params (chunk size to be specific) per fileDefinition. I think this ticket would be the correct place to implement that functionality.
@dixonwhitmire I will schedule a call with the maintainers to discuss how we want to implement this including support for arbitrary reader params.
In our meeting today with @evbaron , @hammadk373 , @LisaWellman , @klwhaley we decided to opt for a dictionary based configuration. While we aren't locking down specific implementation details we will:
Define our "default" csv reader configuration in a dictionary object.
Load user provided csv reader config within a separate dictionary.
Merge the two dictionaries with update() allowing the user provided config to override default settings.
Provide documentation on default settings and how to override those defaults.
This solution isn't as tightly coupled to the reader, while still allowing us to have documented defaults. We can also use this solution for other Pandas compatible APIs such as Dask.
This proposal supports the addition of a new configuration model,
CsvReaderParams
, to the existing DataContract model.CsvReaderParams
collects the existing "csv reader params" within the General and FileDefinition, so that all "csv reader params" are managed within a single document. Collected parameters include:CsvReaderParams
is implemented as anOptional
field within the General and FileDefinition configurations. WhenCsvReaderParams
is defined in both General and FileDefinition models, the FileDefinition.CsvReaderParams overrides values set in General.CsvReaderParams.If CsvReaderParams is not configured within the General model, a default instance is created with appropriate defaults, based on the defaults for the 1.1.0 release.
Finally the existing fields will remain where they are for a "reasonable" amount of time to allow users to migrate to the new configuration model.
The text was updated successfully, but these errors were encountered: