You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Today Genkit supports generating multiple "candidates" for a given prompt by passing a number to the generate call:
generate({prompt: "Tell me a story",candidates: 3})
This is honored to the extent that model provider APIs support it - some models/providers do not. While we have built some conveniences around this (if you generate multiple candidates and do .output() we will continue looking through candidates for a valid response if the first is invalid), it also:
Makes streaming responses more confusing (since multiple candidates may be streaming simultaneously)
Makes tool call behavior more confusing (since different candidates may make different tool choices)
Can fairly trivially be equivalently implemented in user code by making the same generate() call multiple times in parallel
I propose that we deprecate the candidates option and remove it entirely in a future version. This would simplify the logic in generate and simplify model implementations as well.
Model Backwards Compatibility
To keep backwards compatibility with existing model implementations, we can continue to honor the existing model API contract and always pass 1 as the candidate count.
Do you rely on multiple candidates in your GenAI workflows?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Today Genkit supports generating multiple "candidates" for a given prompt by passing a number to the generate call:
This is honored to the extent that model provider APIs support it - some models/providers do not. While we have built some conveniences around this (if you generate multiple candidates and do
.output()
we will continue looking through candidates for a valid response if the first is invalid), it also:generate()
call multiple times in parallelI propose that we deprecate the
candidates
option and remove it entirely in a future version. This would simplify the logic ingenerate
and simplify model implementations as well.Model Backwards Compatibility
To keep backwards compatibility with existing model implementations, we can continue to honor the existing model API contract and always pass
1
as the candidate count.1 vote ·
Beta Was this translation helpful? Give feedback.
All reactions