<p>One of the reasons for the hype around ChatGPT compared to other LLMs (large language models) like GPT-3 and its analogs is that ChatGPT remembers context, meaning that when answering subsequent questions, the LLM takes into account the previous conversation.</p>
<p>But how does ChatGPT do this if it is built on the GPT-3 architecture, which technically lacks support for external memory, and on-the-fly retraining is impossible since it is a lengthy process?</p>
<p>If we look at the OpenAI documentation, we see the following answer to the question “<strong>Does ChatGPT remember what happened earlier in the conversation?</strong>”</p>
<p>“While ChatGPT is able to remember what the user has said earlier in the conversation, there is a limit to how much information it can retain. The model is able to reference up to approximately 3000 words (or 4000 tokens) from the current conversation - any information beyond that is not stored.”</p>
<p>This suggests that what the folks at OpenAI are actually doing is using previous responses as part of the question in the new request. This approach is called “language model cascades”.</p>
<p>Another hypothesis that develops the idea of “language model cascades” is the use of x-prompt, i.e., not just ordinary words, but also “encoded contexts” that convey a block of information in a compressed form. For example, instead of “Criticize the ++ language,” we make the request “Criticize the C++ language in <strong>w_Sheldon_g</strong> style.” In addition to conveying context, x-prompt allows us to formulate more advanced requests, effectively directing the AI in the direction we need.</p>
<p>Other ideas on how ChatGPT retains context?</p>
<p>More details in scientific papers:</p>
<p><a href="https://arxiv.org/pdf/2301.04589.pdf">https://arxiv.org/pdf/2301.04589.pdf</a></p>;
<p><a href="https://arxiv.org/pdf/2212.00616.pdf">https://arxiv.org/pdf/2212.00616.pdf</a></p>;
<p>#ai #gpt #llm</p>