Timeout on fold/query

[martinsumner]

When running a fold or query, the timeout is monitored in the riak_client wait_for_query_results/2/wait_for_fold_results receive loop. There is also a timeout on the riak_core_coverage_fsm prompted on the init (using gen_fsm:start_timer/2).

If the receive loop within riak_client sees the timeout elapse first, it will return {error, timeout} and a {halt, 503} response is captured at the webmachine layer. However, if the timeout on the init function (i.e. the gen_fsm:start_timer/2) is the first to trigger, it will call Mod:finish, which will return {ReqID, {error, timeout}} to the receive loop - and this prompts a response of {error, {error, timeout}} - which is treated at the webmachine layer as a generic failure and triggers a 500 response.

The response to a timeout on a query should be consistent, and probably should be 503 not 500.

The additional complexity is that some of the processing of the result occurs outside of the timeout (i.e. the json encoding). This may mean that a response is received by the client a significant amount of time after the expected timeout. Is this reasonable? Should the production of the response be included within the timeout period?

[martinsumner]

Timings comparing different json libraries for processing 2i query results (with return_terms and no continuation):

Testing small result sets: 
[0.007 s] ok
  riak_kv_wm_index: fun.encode_tester...
Testing lib mochijson2
Result set of 1K in 0ms Result set of 2K in 1ms Result set of 3K in 1ms Result set of 5K in 7ms Result set of 8K in 4ms Result set of 13K in 6ms Result set of 21K in 10ms Result set of 34K in 19ms Result set of 55K in 31ms Total time 82ms

Testing lib thoas
Result set of 1K in 0ms Result set of 2K in 0ms Result set of 3K in 1ms Result set of 5K in 2ms Result set of 8K in 5ms Result set of 13K in 5ms Result set of 21K in 8ms Result set of 34K in 15ms Result set of 55K in 25ms Total time 65ms

Testing lib thoas_iodata
Result set of 1K in 0ms Result set of 2K in 0ms Result set of 3K in 1ms Result set of 5K in 1ms Result set of 8K in 2ms Result set of 13K in 5ms Result set of 21K in 10ms Result set of 34K in 13ms Result set of 55K in 22ms Total time 58ms

Testing lib jiffy
Result set of 1K in 0ms Result set of 2K in 0ms Result set of 3K in 0ms Result set of 5K in 1ms Result set of 8K in 4ms Result set of 13K in 3ms Result set of 21K in 5ms Result set of 34K in 10ms Result set of 55K in 18ms Total time 45ms


Testing mid result sets: 

Testing lib mochijson2
Result set of 100K in 97ms Result set of 200K in 135ms Result set of 300K in 222ms Result set of 500K in 417ms Total time 873ms

Testing lib thoas
Result set of 100K in 60ms Result set of 200K in 89ms Result set of 300K in 134ms Result set of 500K in 291ms Total time 575ms

Testing lib thoas_iodata
Result set of 100K in 36ms Result set of 200K in 95ms Result set of 300K in 123ms Result set of 500K in 203ms Total time 459ms

Testing lib jiffy
Result set of 100K in 58ms Result set of 200K in 87ms Result set of 300K in 191ms Result set of 500K in 178ms Total time 516ms


Testing large result sets: 

Testing lib mochijson2
Result set of 1M in 943ms Result set of 2M in 4236ms Result set of 3M in 2405ms Result set of 5M in 4293ms Result set of 8M in 6817ms Result set of 13M in 13978ms Total time 32675ms

Testing lib thoas
Result set of 1M in 896ms Result set of 2M in 860ms Result set of 3M in 1390ms Result set of 5M in 2375ms Result set of 8M in 3802ms Result set of 13M in 8799ms Total time 18124ms

Testing lib thoas_iodata
Result set of 1M in 381ms Result set of 2M in 782ms Result set of 3M in 2029ms Result set of 5M in 2105ms Result set of 8M in 3603ms Result set of 13M in 5962ms Total time 14865ms

Testing lib jiffy
Result set of 1M in 1406ms Result set of 2M in 1144ms Result set of 3M in 1838ms Result set of 5M in 4063ms Result set of 8M in 3628ms Result set of 13M in 7684ms Total time 19765ms


Testing huge result sets: 

Testing lib mochijson2
Result set of 21M in 23489ms Result set of 21M in 21110ms Total time 44600ms

Testing lib thoas
Result set of 21M in 10781ms Result set of 21M in 10733ms Total time 21514ms

Testing lib thoas_iodata
Result set of 21M in 9208ms Result set of 21M in 9637ms Total time 18845ms

Testing lib jiffy
Result set of 21M in 12438ms Result set of 21M in 9015ms Total time 21454ms