Sporadic ApertureScatterguard TimeoutErrors reported

[rtuck99]

2 occasions where ApertureScaterguard has reported a TimeoutError on i03

occurrence-2024-10-03.txt
occurrence-2024-10-02.txt

From the stack trace, this line in aperturescatterguard.py is failing
await self._safe_move_within_datacollection_range(value.location)

_safe_move_within_datacollection_range() fails here
ap_z_in_position = await self.aperture.z.motor_done_move.get_value()
However, from the logs it seems that the attenuator settings transmission although it reports a timeout error, did not have any elapsed time. Instead it seems that the concurrent aperturescatterguard movement is cancelled, and this causes a TimeoutError to be raised.

However it's not obvious why the aperture scatterguard set operation would be cancelled.

Steps To Reproduce

This issue occurs sporadically ~1 x per day ?

Acceptance Criteria

• Specific criteria that will be used to judge if the issue is fixed
  Issue does not recur

[DominicOram]

@coretl any idea why we might be seeing this? Seems to be an issue with the read getting cancelled sporadically on BL03I-MO-MAPT-01:Z.DMOV

[DominicOram]

We removed the check on BL03I-MO-MAPT-01:Z.DMOV but this error then occurred with the next PV in the set aperture BL03I-MO-MAPT-01:Z.RBV

[DominicOram]

This error is also suspiciously similar to that in DiamondLightSource/dodal#791, maybe the context branch in aoica will fix this too?

[coretl]

The Cancelled error is a red herring, aioca.caget is being called with timeout=None, then being wrapped with asyncio.wait_for(timeout=10) insider ophyd-async to add a default timeout to it.

It is suspicious that 2 PVs on the same motor are timing out. Has the motor just been moving? Was it using caput-callback? Is the motor ever moved from ophyd as well as ophyd-async? What happens if you check a completely unrelated PV just before BL03I-MO-MAPT-01:Z.{something}?