In addition to AI news and my own thoughts, I want to gradually talk about the basic principles in AI and LLM. I recall the meme about "this is the base." So today we will discuss one of the fundamental concepts in the field of AI and LLM - embeddings. An embedding is the process of converting words, phrases, or sentences into vectors of numbers while preserving the semantic proximity between them.
For example: the words "car", "машина", and "автомобиль" have the same meaning (even with very different spellings), and thus will be close in n-dimensional space, while the phrases "critical thinking" and "flat earth" will be far apart.
Numerical example: the vector [1, 0] will be close to the vector [0.98, 0.01], but far from [0, 1].
Embeddings are widely used, for example:
- in translations (LLM understands the semantic proximity of words in different languages);
- in segmentation (questions can be grouped by meaning for support);
- in determining the proximity of a query to information in a knowledge base (services like https://www.chatpdf.com work);
- in sentiment analysis (determining positive and negative reviews);
To use embeddings, you don’t need to understand the details. OpenAI provides an API for calculating embeddings for any text, returning a vector of numbers. By comparing two vectors, you can determine their semantic closeness.
More details
- What are embeddings https://platform.openai.com/docs/guides/embeddings
- Source code on how to use an embed to find relevant info in Wikipedia and use it as context for GPT: https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb
- Course "Deep Learning for NLP" from Stanford University: https://web.stanford.edu/class/cs224n/
- TensorFlow Embedding Projector - interactive visualization of embeddings: http://projector.tensorflow.org/
