Grimoire stops running after a few snapshots in FuzzBench

[renatahodovan]

I'm trying to run a FuzzBench experiment on the mruby target after applying google/fuzzbench#2048 and #13. After applying these patches, the experiment starts successfully, and I get reasonable results from the first few snapshots (the corpus directories contain Ruby-like test cases, and the coverage increases).

However, after several snapshots, it seems that nothing happens anymore. I receive some heartbeat messages in the log, but the corpus directories remain empty, and the coverage stops increasing. This issue occurred in all 5 out of 5 trials after 30 to 60 minutes of fuzzing.

My guess is that there's an issue with the crash, timeout, or restart handling of the SUT (even though it appears to be similar to other grammar-aware fuzzer implementations like Nautilus, Gramatron, and token-level fuzzers, which work fine). Alternatively, it could be a problem with the mutator implementation. Unfortunately, I'm not familiar with Rust or the Grimoire implementation.

Has anyone faced a similar issue or have any idea what might be happening here?

[tokatoka]

I cannot comment on mruby because it was not in the fuzzbench's default target that we usually run.
but it is possible that specific fuzzer, fuzzbench/fuzzres/grimoire is outdated.

I did an experiment with grimoire fuzzer last september using grimoire, and at that time, it did work for the usual 23 targets on fuzzbench (with this result https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2023-09-25-libafl-grimoire/index.html)

If you want you can use it here,
https://github.com/tokatoka/fuzzbench/tree/064ccb8fe137b22a0774f47743a443ba9dda10d9/fuzzers/libafl_fuzzbench_grimoire
by copying all this stuff into your fuzzbench/fuzzers directory.

However, I know there is one issue with grimoire. The fuzzer gets "stuck" for some targets in GeneralizationStage. (like assimp fuzzer).
Maybe your mruby has this problem, too. Can you tell me what setup you used for the mruby?

[renatahodovan]

I tried to evaluate the structure-aware fuzzers + libfuzzer on mruby as a local experiment:

PYTHONPATH=. python3 experiment/run_experiment.py \
  --experiment-config experiment-config.yaml \
  --benchmarks mruby_mruby_fuzzer_8c8bbd \
  --fuzzers libfuzzer nautilus token_level grimoire gramatron \
  -a -mc=5 -rc=25 --no-seeds --concurrent-builds 5 \
  --experiment-name grammars-mruby

I've used the default config with 24 hours of fuzzing with 5 trials.

# The number of trials of a fuzzer-benchmark pair.
trials: 5

# The amount of time in seconds that each trial is run for.
# 1 day = 24 * 60 * 60 = 86400
max_total_time: 86400

# The location of the docker registry.
# FIXME: Support custom docker registry.
# See https://github.com/google/fuzzbench/issues/777
docker_registry: gcr.io/fuzzbench

# The local experiment folder that will store most of the experiment data.
# Please use an absolute path.
experiment_filestore: /tmp/experiment-data

# The local report folder where HTML reports and summary data will be stored.
# Please use an absolute path.
report_filestore: /tmp/report-data

# Flag that indicates this is a local experiment.
local_experiment: true

All the other fuzzers kept working for 24h, but all the 5 trials of Grimoire stopped producing new corpus entries and coverage after ~30-60 minutes.

[tokatoka]

i'll check tomorrow

[tokatoka]

I can reproduce and I have fix in mind, but i'm talking with addison now to check if it's good

[tokatoka]

the issue is that the generalization stage in grimoire takes too long to finish, and this was also mentioned in their paper.
In LibAFL I added the checks so that the fuzzer doesn't run grimoire stage on inputs longer than 8192 bytes. but for mruby, it looks like it still takes a long time to process the input.

If you want you can change this value https://github.com/AFLplusplus/LibAFL/blob/main/libafl/src/stages/generalization.rs#L30 to a lower number, then the fuzzer should go more smooth.

[tokatoka]

let me know if you need further help