overflow in utility term calculation

[jpn--]

There is a potential problem in utility calculations for non-sharrow evaluation: When an expression is an evaluation of two int8 values, the pandas.eval engine preserves the datatype in the result, even if it triggers an overflow. For example, in the error @i-am-sijia found below, the problem occurs when evaluating num_escort_tours * start. Both operands have an original dtype of np.int8, and so result of df.eval('num_escort_tours * start') also has a dtype of np.int8. But in this case, the maximum "correct" value for a few rows of the result exceeds 127, causing an overflow.

When using Sharrow, this overflow does not occur (intermediate calculations are promoted to float as sharrow knows they are about to be multiplied against a float coefficient anyhow). BUT this then triggers disagreement between sharrow and non-sharrow, which stops the run in 'test' mode.

... Sharrow is still crashing at the non_mandatory_tour_scheduling.vectorize_tour_scheduling.tour_6.interaction_sample_simulate.eval_interaction_utils, due to the difference in utilities. See attached logs.

activitysim_nonetype_error.log
activitysim.log

...

I did not get this error when running the full-scale benchmarking run with Sharrow. So this might just be happening with the small test example. Should I open a separate issue?

Originally posted by @i-am-sijia in #839 (comment)

[jpn--]

Also, I'm not clear whether this overflow is a new problem created by over-zealous down casting dtypes to save memory, or if it pre-dates the datatype optimization.

I'm not sure what the best fix is here either ... we could circumvent this one specific problem by up-casting the variables in this one expression of this one spec file, but the wider problem would remain lurking, so it's probably advisable not to do that.

[i-am-sijia]

Utility difference is reported in this expression: num_escort_tours * start

The value of num_escort_tours: [6, 6, 6, 6, ...]
The value of start: [22, 22, 23, 22, ...]
The value of num_escort_tours * start in Sharrow: [132, 132, 138, 132, ...]
The value of num_escort_tours * start in non-Sharrow: [-124, -124, -118, -124, ...]
It seems that Sharrow got it right, the non-Sharrow got it wrong.

The num_escort_tours is created here, the dtype is specified as int8 and hence it overflows if used in downstream calculations like this one that generate values greater than 128. Not sure how Sharrow got it right, does that mean Sharrow is upcasting the dtype for the calculation? (@jpn--, sorry, I saw you already answered this in the updated issue description)

This is not a new problem introduced by the Phase 8 data type optimization. Downcasting numeric dtype is turned off by default for all runs due to reasons like this.

Looking at the commit history, the dtype for num_escort_tours has always been coded as int8. It caused a numeric overflow after the changes from School Escorting model that introduced num_escort_tours * start.