Parquet output format

[Thernn88]

Hello!

I was wondering if it would be possible to have Diamond output in the open source Parquet format? This is a more modern output than tsv.

Reading in and processing the TSV is a significant bottleneck when dealing with hundreds of millions of hits.

Using the Parquet format would have several benefits. It would be orders of magnitude faster than reading a tsv/csv. Additionally, it would allow more efficient compression or even compression of individual columns.

https://posit.co/blog/speed-up-data-analytics-with-parquet-files/

https://en.wikipedia.org/wiki/Apache_Parquet

[bbuchfink]

We have considered implementing it before. I'm not sure though if native parquet support would be a lot faster than piping the tsv output to a script that converts it to parquet. Have you tried this?

[Thernn88]

Thanks for the reply. I've done tests involving parquet conversion.

Using an 8gb output tsv from Diamond as a tester.

Read speed using pyarrow. (Not bothering to test standard python read as it is slow.)

kevin@DESKTOP-PQ3G8VJ:~$ time python3 arrow.py

real    0m36.245s
user    0m36.870s
sys     0m17.097s

Above is our baseline. Lets convert it to a parquet.

kevin@DESKTOP-PQ3G8VJ:~$ time python3 convert.py

real    1m37.931s
user    1m27.606s
sys     0m12.685s

Conversion took almost 100s. Oof.

kevin@DESKTOP-PQ3G8VJ:~$ time python3 parquet.py

real    0m10.923s
user    0m6.744s
sys     0m6.653s

Read time, however, is more than twice as fast.

Unfortunately, the conversion is a huge time sink for a file we are only going to read once.

It was our conclusion that unless Diamond natively outputs in parquet it wasn't worth using the format. Thus, why I asked if you had plans to support output in the format in the future.

You believe outputting in this format would not be much faster than a tsv to parquet conversion? (Including the original write time for the tsv itself.)

[bbuchfink]

Thanks, looks like it makes sense to either include native support into diamond or develop a faster way to convert tsv to parquet. I will give this some thought.

[Thernn88]

Thanks! I would also strongly consider JSON format. (Perhaps even more so than Parquet!) There is no option for this correct?

The penalty for outputting in this format should be less than parquet. Tools for parsing it very fast compared to TSV/CSV.

See SIMDJSON.

[bbuchfink]

No json is not supported, but it would not be that difficult. I'll see what I can do.

[Thernn88]

Thanks for looking into this! Json output would be an immense help.

[bbuchfink]

Thanks for looking into this! Json output would be an immense help.

JSON output format has been included in the latest release.

[bbuchfink]

We now provide instructions how to create Parquet output with the help of DuckDB: https://github.com/bbuchfink/diamond/wiki/File-formats