SW-Architecture around LLMs (Introduction)

Components

  • Frontend / UI
  • Orchestrator 1
  • Tools (e.g. code interpreter, image generator, web search)
  • Backend / LLM

Examples

Company Frontend Backend
Google Bard 2 LaMDA / PaLM 2 2
Meta Meta AI LLaMa
OpenAI ChatGPT GPT

1 Routes the context to the backend and uses, if necessary, the right tool on its result.
2 Those names are historic; these days they are combined under the name Gemini.

Workflow

Frontend ---> Orchestrator ---> Backend ---> Orchestrator ---> Frontend
                                                  |
                                                Tools

Example (for GitHub Copilot)

  1. Frontend: Gets the request “Change the output message to ‘Hello World!’.”
  2. Orchestrator: Provides the prompt plus additional context to a selected LLM. 3
  3. Backend: Generates an answer containing a tool call to edit a file. 4
  4. Orchestrator: Executes the tool call and modifies the answer accordingly.
  5. Frontend: Shows the modified answer as an output like “Modified main.cpp”.

3 That contains system instructions, chat history, retrieved knowledge, and tool definitions.
4 The LLM learns to handle general tool calling in post-training on specific training data.