Decide whether to use Flatterer or Flatten Tool for converting from JSON to CSV format

[duncandewhurst]

We need to decide whether to use Flatterer or Flatten Tool for converting from JSON to CSV format.

Some things to consider:

• Is converting from spreadsheet/CSV format to JSON relevant?
• Suitability for use in publication pipelines.
• (Possibly) ease of adding special casing: Coordinates modelling (add support for WKT to Flatten Tool) #10

[stevesong]

The number one question for me is what is the CSV use case? Perhaps if the records are capturing the kilometre length of links, it would be useful to open in a spreadsheet. Other scenarios?

[duncandewhurst]

Edit: @stevesong, since your question is about the CSV format in general, rather than conversion tooling, I've copied it to the discussion on publication formats and moved my reply there.

[lgs85]

Noting here that this is also dependent on any decision made in #17.

[duncandewhurst]

@kindly if we were to use Flatterer's output as the CSV format for OFDS data, how much work would it be to author a conversion script so that CSV data could be converted to JSON format for validation in CoVE?

If it's a lot of work, could we instead use Flatterer to generate a datapackage.csv that could be used to validate CSV data using frictionless?

[kindly]

@duncandewhurst flatterer is really intended for just flattening as it creates _link fields to express the join conditions and those link fields would be diffucult to create by hand.

Nonetheless recently flatterer has a feature called pushdown which can copy all id fields to the child one-to-many relationships and it prefixes those id fields with the table where they originated. This is similar to the id_field feature in flattentool. Using this feature I think it would be possible to do the unflattening in a script or potentially convert the headings to ones that flatten-tool expects. I have not thought about this deeply though so I am not 100% sure how feesable this.

[duncandewhurst]

Thanks, @kindly

Would using a data package schema be an alternative to unflattering for CSV validation?

If it's a lot of work, could we instead use Flatterer to generate a datapackage.csv that could be used to validate CSV data using frictionless?

[duncandewhurst]

For the Alpha at least, we've decided to use Flatten Tool because it offers support for round-tripping and has better organisational support. I've opened an issue about support for streaming in Flatten Tool: OpenDataServices/flatten-tool#399

[duncandewhurst]

We've not heard any further feedback on this issue so I'm going to close it for now.