Generative AI Fundamentals #1
Context window as the number of tokens a model handles at once
Remember the correct answer, the context window.
・The property that represents the amount of tokens a model can handle at one time (the upper limit on input plus output).
・The larger it is, the longer the document or the more conversation history can be passed in at once, which suits tasks such as long-text summarization.
Throughput (amount processed per unit of time), learning rate, and number of epochs are none of them the property that represents the upper limit on tokens handled at one time.
Return on investment as the measure for justifying a project
Remember the correct answer, ROI (return on investment).
・A financial metric that shows how much profit or value was gained relative to the cost invested.
・It serves as a basis for management decisions about the justification and priority of a generative AI project.
DPI (resolution), FPS (frame rate), and bitrate (data volume) are all technical metrics related to image quality or transmission, not a financial metric for return on investment.
How Top-P, Top-K, and temperature decide which word comes next
An LLM is a model that 'predicts the word most likely to come next as a continuation of the text so far'. It computes a probability for each candidate next word, selects one word from them to output, and repeats this process to generate text. The three inference parameters that control this 'way of selecting the next word' are top-p, top-k, and temperature.
・top-p (nucleus sampling): narrows the candidates to the top words until the cumulative probability reaches a set value (because it narrows by the sum of probabilities, the number of candidates changes dynamically).
・top-k: narrows the candidates to the top k items by probability (a fixed count).
・temperature: adjusts the overall sharpness of the probability distribution (lower is more certain, higher is more diverse).
All of these operate at inference time. Number of epochs, learning rate, and batch size are settings for 'training time' and are not parameters that narrow output candidates at inference time.
Distinguishing tokens, parameters, embeddings, and epochs
Distinguish the basic generative AI terms by 'what is being counted'.
・Token: the smallest processing unit of text. The basis for input/output amount, pricing, and the context window.
・Parameter: the number of internal weights of a model (model scale).
・Embedding: a vector representation of meaning.
・Epoch: the number of training iterations.
When the question is about the 'unit for billing or input limit', it is the token.
Splitting the foundation model lifecycle at deployment
The latter half of the FM lifecycle (from deployment onward) is 'deployment (rolling out to production) → feedback (collecting production reactions and improving)'. The early-half 'data selection, pre-training, fine-tuning, and evaluation' are stages before deployment. Remember it split by deployment into the early half (preparation, training, evaluation) and the latter half (operation, improvement).