Skip to content

Fix abort when re-running an IB simulation in place - #1760

Closed
YuJenLiu wants to merge 1 commit into
MFlowCode:masterfrom
YuJenLiu:fix/ib-data-status-replace
Closed

Fix abort when re-running an IB simulation in place#1760
YuJenLiu wants to merge 1 commit into
MFlowCode:masterfrom
YuJenLiu:fix/ib-data-status-replace

Conversation

@YuJenLiu

@YuJenLiu YuJenLiu commented Aug 25, 2026

Copy link
Copy Markdown

The bug

Re-running the simulation binary against an existing pre_process output aborts at start-up, before the time loop, for any case with ib = T and parallel_io = F:

FIO-F-208/OPEN/unit=2/'NEW' specified for file which already exists.
 File name = ./p_all/p0/0/ib_data.dat
 In source file src/simulation/m_data_output.fpp, at line number 865

s_write_serial_ib_data writes p_all/p<rank>/0/ib_data.dat with STATUS='new' (src/simulation/m_data_output.fpp:865) and nothing in the simulation binary removes p_all, so the second run trips over the file the first one left behind.

Its twin s_write_serial_ib_state — called on the very next line of s_initialize_modules (m_start_up.fpp:924-925) — already opens with status='replace' (m_data_output.fpp:1046). This PR makes the two consistent.

Reproduction

Any ib = T case with parallel_io = F, e.g. examples/2D_ibm_ellipse/case.py with parallel_io switched to "F":

./mfc.sh run case.py -t pre_process simulation   # ok
./mfc.sh run case.py -t simulation               # exit 127, message above

Fix

-        open (2, FILE=trim(file_path), form='unformatted', STATUS='new')
+        open (2, FILE=trim(file_path), form='unformatted', STATUS='replace')

s_write_serial_ib_data is the sole producer of the file and always rewrites it in full (one unformatted record, :868), so there is nothing to preserve.

Alternatives I rejected: 'unknown' has no defined truncation semantics, which would matter if a future ib_markers were ever smaller than a previous one — the reader (post_process/m_data_input.f90:156-157) does a single read (2) ib_markers%sf(...) and would see a malformed record. 'old' fails on the first run in a fresh directory. An inquire + delete pair is three lines for behaviour identical to one word, and needs no proc_rank == 0 gate here anyway, since each rank writes into its own p_all/p<rank>/ tree (:862).

Scope

:865 is the only unguarded STATUS='new' in src/. Scanning for the pattern turns up 15 sites and 39 MPI_FILE_OPEN calls; every other one is guarded, and none of the MPI_FILE_OPEN calls uses MPI_MODE_EXCL, so MPI-IO cannot produce this error at all:

writer guard
serial restart, p_all/.../*.dat (:319:369) directory wipe at :311-315
MPI-IO restart (:793), file_per_process restart (:721) inquire + MPI_FILE_DELETE
serial ib_state (:1046) status='replace'
file_per_process ib_state (:964-968) inquire + replace/new
time_data.dat, io_time_data.dat (m_start_up.fpp:682,694) inquire at :678,:690
serial ib_data (:865) none

Only the parallel_io = F path is affected. With parallel_io = T the dispatcher at :914-925 takes s_write_parallel_ib_data, which opens with ior(MPI_MODE_WRONLY, MPI_MODE_CREATE). parallel_io defaults to .false. (m_global_parameters_common.fpp:377), but every examples/*ibm* case sets parallel_io: "T", which is why the shipped examples never show it.

Running pre_process is not affected either: it wipes p_all/p<rank>/0 before writing (pre_process/m_data_output.fpp:650-657), so -t pre_process simulation is always safe. It is re-running the simulation alone that hits it — or old_grid = T, which skips that wipe, in which case even -t pre_process simulation aborts on the second run.

rm -rf p_all is not a workaround: with parallel_io = F the simulation reads its IC out of that same directory and aborts if it is missing (m_start_up.fpp:161-171). --clean does not remove it either unless pre_process is also a target (toolchain/mfc/run/input.py:143-149).

Testing

No golden files move. The two dispositions differ only when the file already exists, and in that case the old code aborted — there is no prior output for the new code to differ from. The record written is unchanged: one unformatted record of the integer ib_markers%sf.

Reproduced the abort on unmodified master; the patched build completes the second run. ./mfc.sh precheck and the test suite pass.

Not tested on a GPU, and it should not matter: the statement is host-side Fortran I/O in a routine that runs once from s_initialize_modules, before the time loop. The adjacent $:GPU_UPDATE(host=...) is untouched.

Why the test suite never caught it, and why I did not add a test

toolchain/mfc/test/case.py:265-278 (delete_output) removes p_all before every case, so ib_data.dat never pre-exists and 'new' and 'replace' are byte-identical there. The ib: "T" cases do execute this line — POST_PROCESS_OFF_PARAMS sets parallel_io: "F" (case.py:47-50) — they just never populate the file first. The one test that keeps p_all across a second simulation invocation is the restart round-trip (case.py:196-241), but its phase 2 sets t_step_start = mid_step > 0, so the if (t_step_start == 0 ...) guard at m_start_up.fpp:923 short-circuits and the write never executes — and no restart_check=True case sets ib.

A regression test would require exercising a second simulation invocation with t_step_start = 0 while preserving the existing output. I did not add one, to keep this to a single line — happy to add that coverage in a separate PR if you would like it.

Contribution Policy

We do not accept pull requests generated primarily by AI without genuine understanding or real-world usage context.

All contributions are expected to demonstrate:

  • A clear understanding of the codebase
  • Alignment with product direction
  • Thoughtful reasoning behind changes
  • Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.


Acknowledgement

  • I confirm this PR meets the above expectations and reflects my own understanding and real-world context.

PR template credit: junegunn

s_write_serial_ib_data opens p_all/p<rank>/0/ib_data.dat with
STATUS='new', so a second run of the simulation binary against the same
pre_process output aborts inside s_initialize_modules, before the time
loop:

    FIO-F-208/OPEN/unit=2/'NEW' specified for file which already exists.
     File name = ./p_all/p0/0/ib_data.dat
     In source file src/simulation/m_data_output.fpp, at line number 865

Its twin s_write_serial_ib_state, called on the next line of
s_initialize_modules, already opens with status='replace'. This makes
the two consistent.

s_write_serial_ib_data is the sole producer of the file and always
rewrites it in full, so there is nothing to preserve. Only the
parallel_io = F path is affected; s_write_parallel_ib_data opens with
ior(MPI_MODE_WRONLY, MPI_MODE_CREATE) and cannot raise this error.
@YuJenLiu
YuJenLiu requested a review from sbryngelson as a code owner August 25, 2026 08:38
@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.67%. Comparing base (e2f0e26) to head (ce842a0).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1760   +/-   ##
=======================================
  Coverage   61.67%   61.67%           
=======================================
  Files          84       84           
  Lines       21619    21619           
  Branches     3196     3196           
=======================================
  Hits        13334    13334           
  Misses       6093     6093           
  Partials     2192     2192           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants