Ok, I've been nurturing the idea of a personal AI assistant for a while now, and I've finally decided to take some steps in that direction. I'll be writing small notes about what's happening.
At the moment, the project is open-source and will be available at https://github.com/rvnikita/rv_ai_assistant
The main complaints about the current Voice GPT:
1. Sometimes it glitches on long audio; I can dictate 2-3-5 minutes of thoughts, and then it says, “Oops, something went wrong,” and everything disappears. Technically, if it's voice in Telegram, this shouldn't happen.
2. It can't call functions - for example, I want it to listen and trigger some API for searching or to check the weather. Or, for instance, send a message.
3. GPT works poorly in the subway; it requires a constant connection, unlike Telegram, which can work and process messages even when there's no internet in the subway.
So, at a high level, the current architecture is:
- High-Level Architecture
- Dispatcher Agent:
- The main agent that handles incoming messages and decides which module/agent to use based on the request.
- Uses GPT-4o for decision making.
- Interacts with a RAG (Retrieval-Augmented Generation) system to leverage past interactions and module usage data.
- Modules/Agents:
- Individual modules that handle specific tasks, such as downloading Twitter videos or future functionalities.
- Each module has a standardized interface for inputs, outputs, and descriptions.
- Database:
- PostgreSQL for storing messages, configuration, user interactions, and feedback.
- Alembic for database migrations.
- Transcription Service:
- Uses OpenAI's Whisper model for transcribing voice messages.
- Environment Management:
- Uses .env files to manage environment variables and API keys.
For now, everything works on my local computer, but soon I'll try to deploy it on a server so that I can play with it.
In the current zero version, it simply copies the text that was sent to it as text, and also recognizes voice and returns text for the voice message.
The first module I want to implement is downloading videos from x.com, as this is a common task, and I constantly have to use third-party sites for this.