An AI practitioner wants to limit the candidates for the next token to only the top few with the highest probability before selection, suppressing extreme outlier words. Which inference parameter performs this control?
Choosing the inference parameter that limits candidates to the top K by probability.
Temperature
Temperature is a parameter that adjusts the sharpness (randomness) of the overall probability distribution; raising it increases output randomness and lowering it makes output more deterministic (safer). It does not limit the number of candidates to the top K.
It is not a control that fixes the number of candidates, so it is incorrect.
Top-K
Correct. Top-K is a parameter that limits the candidates for the next token to the top K with the highest probability and then selects from among them. Making K smaller makes outlier words less likely to appear and curbs runaway output.
Maximum tokens
Maximum tokens is a parameter that sets the upper limit on the output length.
It is not a control that narrows candidate words to the top K, so it is incorrect.
Context window
The context window is a factor related to the upper limit on the amount of input you can pass at one time.
It is not a control that narrows candidate words to the top K, so it is incorrect.
Remember how the correct answer, 'Top-K,' works.
・Top-K keeps only the top K candidates with the highest probability and selects from them.
・The smaller K is, the more candidates are narrowed and the less likely outlier words appear.
・Top-P narrows candidates 'until a cumulative probability reaches a set proportion' (by total probability, not count).
Temperature controls the sharpness of the whole distribution and maximum tokens controls length, both separate from narrowing candidates.