testing closet

[adeebshihadeh]

TODOs

• no patch, merge all necessary changes into master
• publish reports publicly
• reproduce statsd crash #32758
• reproduce camerad: persistent crash after restarting openpilot many times #28172
• reproduce encoderd crashed during onroad #26328
• reproduce Acceletometer, Gyroscope timing spikes #30070
• ignition cycling on some devices
• cold boot cycles on some devices

bugs:

• controls mismatch, likely due to the replay
• route format is the old one, see new one
• AGNOS update times out
• get all the devices registered @maxime-desroches
• Log Processing Failure: zero-size array to reduction operation maximum...

PRs to verify

• sensord: improve sensor frequency stability and timestamp accuracy #33257

[Quantizr]

• there's a pull request for changes to master Testing Closet changes #33537, need to also remove the sed statements from app/script/rsync_new_commits.sh in the testing_closet

• reports are exported on jenkins machine to /testing_closet_export/

• statsd crash only seems to happen with no internet connection (best way is prob to disconnect internet on the router side)

• camerad crashes happened a lot while setting up the new testing rack and trying to get everything to engage, not sure what specifically was causing

• the one time I remember encoderd crash happening was immediately after an AGNOS update PR, all the service timings were really bad for the first few segments

• accelerometer, gyroscope timing spikes consistently happen in all circumstances, look at testing closet for any revision

• ignition cycling and cold boot cycles can be done with can_replay (currently either all devices or none, example in testing closet readme), currently set to reset ignition every hour and cold boot every other hour (ignition is turned off first)

• controlsMismatch bug found and is in the Discord

• all the other listed bugs fixed