A research team wants to summarize multiple research papers spanning dozens of pages together in a single request. Without splitting the documents, which is the MOST important model factor that determines whether a long input can be passed at once?
A question about choosing the model factor that determines whether a long input can be passed at once.
The model's vocabulary size
Vocabulary size is the number of unique words (token types) the model can handle.
What determines the length of the input that can be passed at once is the context window, and vocabulary size is a different thing from input capacity, so this is incorrect.
Temperature
Temperature is an inference parameter that adjusts the randomness of output.
This question is about the length of input that can be passed at once, not the diversity of output, so this is incorrect.
Context window
Correct. The context window is the upper limit on the amount of input (plus output) tokens the model can process at once. To summarize multiple papers together in a single request, the input must fit within this limit, and a model with a larger window can handle longer text all at once.
Latency
Latency is the metric for how quickly a response is returned.
This question is about whether a long input can be passed at once, not response speed, so this is incorrect.
Note how the correct answer, the context window, works.
- A model has an upper limit on the amount of input plus output tokens it can process at once.
- If the entire input does not fit within this limit, it cannot be handled all at once.
- So if you want to process long text in one request, choose a model with a large window.
The often-confused vocabulary size is the number of word types it can handle and is a different thing from input capacity.