Single rotation scan with Multipin changes references incorrect h5 data files in VDS

[rtuck99]

During testing of multipin code #556 , the following issue was encountered:

Gridscans completed successfully, but on invoking rotation_scan the generated nexus files contained VDS references to the incorrect data file numbers. This lead to missing/incorrect data shown in ispyb.

See collection
https://ispyb.diamond.ac.uk/dc/visit/cm37235-4/id/15512407

and other subsequent collections

log messages:

[32m[I 241008 15:46:46.263 nexus_callback:82] Nexus writer received start document with experiment parameters {\"sample_id\":5461095,\"sample_puck\":null,\"sample_pin\":null,\"visit\":\"cm37235-4\",\"zocalo_environment\":\"artemis\",\"beamline\":\"BL03I\",\"det_dist_to_beam_converter_path\":\"/dls_sw/i03/software/daq_configuration/lookup/DetDistToBeamXYConverter.txt\",\"insertion_prefix\":\"SR03I\",\"detector_distance_mm\":250.91171812606459,\"demand_energy_ev\":12699.999999999998,\"snapshot_directory\":\"/dls/i03/data/2024/cm37235-4/TestThaumatin/sethau_13/snapshots\",\"snapshot_omegas_deg\":[0.0,90.0,180.0,270.0],\"features\":{\"use_panda_for_gridscan\":false,\"use_gpu_for_gridscan\":true,\"set_stub_offsets\":false},\"parameter_model_version\":\"5.0.0\",\"file_name\":\"sethau_13\",\"exposure_time_s\":0.01,\"comment\":\"Hyperion Rotation Scan - \",\"trigger_mode\":2,\"run_number\":null,\"selected_aperture\":\"LARGE_APERTURE\",\"transmission_frac\":0.03000228852033615,\"ispyb_experiment_type\":\"SAD\",\"storage_directory\":\"/dls/i03/data/2024/cm37235-4/TestThaumatin/sethau_13\",\"shutter_opening_time_s\":0.06,\"rotation_increment_deg\":0.1,\"x_start_um\":null,\"y_start_um\":null,\"z_start_um\":null,\"omega_start_deg\":0.0,\"phi_start_deg\":null,\"chi_start_deg\":0.0,\"kappa_start_deg\":null,\"rotation_axis\":\"omega\",\"scan_width_deg\":144.0,\"rotation_direction\":\"Negative\",\"nexus_vds_start_img\":0}\u001b[0m

[32m[I 241008 15:46:46.263 nexus_callback:86] Writing Nexus file for {"sample_id":5461095,"sample_puck":null,"sample_pin":null,"visit":"cm37235-4","zocalo_environment":"artemis","beamline":"BL03I","det_dist_to_beam_converter_path":"/dls_sw/i03/software/daq_configuration/lookup/DetDistToBeamXYConverter.txt","insertion_prefix":"SR03I","detector_distance_mm":250.91171812606459,"demand_energy_ev":12699.999999999998,"snapshot_directory":"/dls/i03/data/2024/cm37235-4/TestThaumatin/sethau_13/snapshots","snapshot_omegas_deg":[0.0,90.0,180.0,270.0],"features":{"use_panda_for_gridscan":false,"use_gpu_for_gridscan":true,"set_stub_offsets":false},"parameter_model_version":"5.0.0","file_name":"sethau_13","exposure_time_s":0.01,"comment":"Hyperion Rotation Scan - ","trigger_mode":2,"run_number":null,"selected_aperture":"LARGE_APERTURE","transmission_frac":0.03000228852033615,"ispyb_experiment_type":"SAD","storage_directory":"/dls/i03/data/2024/cm37235-4/TestThaumatin/sethau_13","shutter_opening_time_s":0.06,"rotation_increment_deg":0.1,"x_start_um":null,"y_start_um":null,"z_start_um":null,"omega_start_deg":0.0,"phi_start_deg":null,"chi_start_deg":0.0,"kappa_start_deg":null,"rotation_axis":"omega","scan_width_deg":144.0,"rotation_direction":"Negative","nexus_vds_start_img":0}

[32m[I 241008 15:47:11.851 nexus_callback:72] Nexus file created at sethau_13_21�[0m

Notice that the actual nexus data files created for this collection were sethau_13_33_000001.h5 and sethau_13_33_000002.h5 along with associated .nxs and _master.h5 not sethau_13_21 as logged.

Several other subsequent samples all linked to foo_xx_21 files, despite being in different directories.
This points at NexusWriter.data_filename being the incorrect value, given the parameters specified, this logically points to one of two things:

• filesystem gremlins reporting directory listings inconsistent with actual files written during execution of dodal utils.get_run_number(). This seems unlikely given the relatively longstanding existence of several files indexed > 21 in these directories
• Somehow the events were going to a stale external callback handler or the start event was not dispatched