Fast-LLM RL: per-token model-version tagging, metrics, and GSPO configs - #155
Open
jlamypoirier wants to merge 5 commits into
Open
Fast-LLM RL: per-token model-version tagging, metrics, and GSPO configs#155jlamypoirier wants to merge 5 commits into
jlamypoirier wants to merge 5 commits into
Conversation
…configs - vLLM per-token model-version tagging: producer patches in vllm1.py plus consumers in async_llm.py, llm.py, and rollouts.py. - Preprocessor/state/actor RL metrics: reward and model-version pass-through, completed-step back-pressure, and the data-pipeline logging flag. - GSPO experiment configs (Qwen 0.5B / 7B, DeepSpeed and Fast-LLM variants), the counting task config, and the conf/base.yaml logging flag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- conf/counting.yaml: the fast_llm GRPO loss entry had no `type:`, so it fails to dispatch on the Fast-LLM backend (base defines no default loss). Add `type: grpo`. - preprocess.py: the debug data-pipeline log file was opened outside the stream `with` block and never closed. Register it on a contextlib.ExitStack entered by that block so the handle closes on exit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- vllm1.py: _create_chat_logprobs captures *args/**kwargs like
update_from_output, so a vLLM signature change disables tagging
instead of crashing at bind time; bounds-guard token_ids[index]
to match top_logprobs[index]; log {error!r} on import failure.
- preprocess.py: hoist `import json` to module top, use the existing
Path import, attribute-access cfg.debug.log_data_pipeline, and use
an identity check for pipeline_log_file.
- math_qwen05_gspo_fllm.yaml: drop inert decay_iterations under a
constant schedule; reword the losses comment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude Opus 4.8 authored this PR description.
Third PR in the Fast-LLM integration stack, extracted from #140:
main ← #151 ← #153 (core integration) ← this PR ← #140 (test suite)What this adds
Runtime code and configs for the Fast-LLM RL path, layered on top of the core integration in #153:
vllm1.py) stamp each generated token with the policy version that produced it; consumers (async_llm.py,llm.py,rollouts.py) parse it back so downstream can associate every token with its producing version.preprocess.py,state.py,actor.py.conf/base.yamllogging flag.Notes
🤖 Generated with Claude Code