Preserve JSON column order and support list of strings field

[albertvillanova]

Preserve column order when loading from a JSON file with a list of dict (or with a field containing a list of dicts).

Additionally, support JSON file with a list of strings field.

Fix #6913.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005492 / 0.011353 (-0.005861)	0.004087 / 0.011008 (-0.006921)	0.065334 / 0.038508 (0.026826)	0.032282 / 0.023109 (0.009173)	0.246441 / 0.275898 (-0.029457)	0.278807 / 0.323480 (-0.044673)	0.003245 / 0.007986 (-0.004741)	0.003795 / 0.004328 (-0.000534)	0.050082 / 0.004250 (0.045832)	0.050613 / 0.037052 (0.013561)	0.258885 / 0.258489 (0.000396)	0.297257 / 0.293841 (0.003416)	0.028847 / 0.128546 (-0.099699)	0.011377 / 0.075646 (-0.064270)	0.206089 / 0.419271 (-0.213182)	0.037354 / 0.043533 (-0.006178)	0.257319 / 0.255139 (0.002180)	0.275134 / 0.283200 (-0.008066)	0.018064 / 0.141683 (-0.123619)	1.112371 / 1.452155 (-0.339783)	1.160909 / 1.492716 (-0.331807)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.101893 / 0.018006 (0.083887)	0.311084 / 0.000490 (0.310594)	0.000208 / 0.000200 (0.000008)	0.000042 / 0.000054 (-0.000013)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.019548 / 0.037411 (-0.017863)	0.064396 / 0.014526 (0.049870)	0.074900 / 0.176557 (-0.101656)	0.122750 / 0.737135 (-0.614385)	0.076693 / 0.296338 (-0.219646)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.288609 / 0.215209 (0.073400)	2.831354 / 2.077655 (0.753699)	1.453961 / 1.504120 (-0.050159)	1.327702 / 1.541195 (-0.213493)	1.382140 / 1.468490 (-0.086351)	0.568465 / 4.584777 (-4.016312)	2.427199 / 3.745712 (-1.318513)	2.810586 / 5.269862 (-2.459275)	1.839227 / 4.565676 (-2.726449)	0.063219 / 0.424275 (-0.361056)	0.005111 / 0.007607 (-0.002496)	0.341447 / 0.226044 (0.115403)	3.357429 / 2.268929 (1.088501)	1.806501 / 55.444624 (-53.638123)	1.541696 / 6.876477 (-5.334781)	1.755400 / 2.142072 (-0.386673)	0.661442 / 4.805227 (-4.143785)	0.120203 / 6.500664 (-6.380461)	0.044429 / 0.075469 (-0.031040)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.987810 / 1.841788 (-0.853978)	12.765467 / 8.074308 (4.691159)	10.497788 / 10.191392 (0.306396)	0.132723 / 0.680424 (-0.547701)	0.014484 / 0.534201 (-0.519717)	0.285763 / 0.579283 (-0.293520)	0.264377 / 0.434364 (-0.169987)	0.326971 / 0.540337 (-0.213367)	0.429432 / 1.386936 (-0.957504)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005996 / 0.011353 (-0.005357)	0.004092 / 0.011008 (-0.006916)	0.051660 / 0.038508 (0.013152)	0.036661 / 0.023109 (0.013552)	0.271133 / 0.275898 (-0.004765)	0.295728 / 0.323480 (-0.027752)	0.004452 / 0.007986 (-0.003534)	0.002915 / 0.004328 (-0.001413)	0.050669 / 0.004250 (0.046418)	0.044431 / 0.037052 (0.007378)	0.284683 / 0.258489 (0.026194)	0.318799 / 0.293841 (0.024958)	0.031094 / 0.128546 (-0.097452)	0.010810 / 0.075646 (-0.064836)	0.059740 / 0.419271 (-0.359531)	0.034912 / 0.043533 (-0.008621)	0.268779 / 0.255139 (0.013640)	0.291294 / 0.283200 (0.008095)	0.019769 / 0.141683 (-0.121914)	1.124833 / 1.452155 (-0.327322)	1.168301 / 1.492716 (-0.324416)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.097080 / 0.018006 (0.079074)	0.304636 / 0.000490 (0.304146)	0.000232 / 0.000200 (0.000032)	0.000060 / 0.000054 (0.000006)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.023186 / 0.037411 (-0.014225)	0.082232 / 0.014526 (0.067706)	0.089427 / 0.176557 (-0.087130)	0.132715 / 0.737135 (-0.604421)	0.092820 / 0.296338 (-0.203518)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.300672 / 0.215209 (0.085463)	2.969603 / 2.077655 (0.891948)	1.577827 / 1.504120 (0.073707)	1.440768 / 1.541195 (-0.100427)	1.494526 / 1.468490 (0.026035)	0.574599 / 4.584777 (-4.010178)	0.963300 / 3.745712 (-2.782412)	2.847854 / 5.269862 (-2.422008)	1.841248 / 4.565676 (-2.724428)	0.062321 / 0.424275 (-0.361954)	0.005389 / 0.007607 (-0.002218)	0.350853 / 0.226044 (0.124808)	3.463514 / 2.268929 (1.194586)	1.937661 / 55.444624 (-53.506964)	1.665320 / 6.876477 (-5.211157)	1.849028 / 2.142072 (-0.293044)	0.655333 / 4.805227 (-4.149894)	0.119062 / 6.500664 (-6.381602)	0.043387 / 0.075469 (-0.032082)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.004118 / 1.841788 (-0.837670)	13.350894 / 8.074308 (5.276585)	11.179363 / 10.191392 (0.987971)	0.135169 / 0.680424 (-0.545255)	0.016298 / 0.534201 (-0.517903)	0.288467 / 0.579283 (-0.290816)	0.132712 / 0.434364 (-0.301651)	0.325436 / 0.540337 (-0.214901)	0.413406 / 1.386936 (-0.973530)