Components
Three application patterns for building LLM apps:
- Direct conversation: take user input and generate a reply
- Knowledge processing: semantically process, store, and retrieve text documents
- Tool calling: decide from context and call the matching tools
Eino abstracts commonly used capabilities into reusable Components.
How the component abstractions map onto those patterns:
Conversation components:
- Modular handling of parameters for talking to the model:
ChatTemplate - Talking to the model directly:
ChatModel
Text-semantics components:
- Fetching and processing text documents:
Document.Loader,Document.Transformer - Turning documents into semantics:
Embedding - Storing the index after embedding:
Indexer - Indexing and recalling semantically related documents:
Retriever
Decision-and-execution components:
The model can decide and call tools: ToolsNode
Custom components:
User-defined code logic: Lambda
Eino's component abstractions follow these design principles:
- Modular and standardized: capabilities that do the same job get one unified module; components have clear duties and boundaries, and you can compose them freely.
- Extensible: interfaces constrain as little as possible, so component authors can easily write custom ones.
- Reusable: wrap the most common capabilities and implementations, and give developers tools that work out of the box.
Chain & Graph orchestration
Orchestration: compose and chain the atomic capabilities of Components.
- Don't let business logic leak into the orchestration.
- The core of an LLM app is composing and chaining "components that provide atomic capabilities". Components are first-class citizens of orchestration.
- From an abstract view, orchestration is building a network, and data flows through that network. Every node has format/content requirements on the flowing data. For the network to flow smoothly, the key is: are the data formats of upstream and downstream nodes aligned?
- The complexity of a business scenario shows up in the complexity of the orchestration result. Only horizontal governance keeps complex scenarios from going off the rails.
- LLMs will keep developing fast, and so will LLM apps. Only apps that can extend stay alive.
Eino provides an orchestration solution based on the Graph model (edge + node), with components as atomic nodes, built on upstream/downstream type alignment.
- Components at the center, which standardizes how business features get packaged.
- Business-logic complexity is encapsulated inside components; the orchestration layer gets a more global view, so the layers stay clear.
- Aspect-style capabilities: the callback mechanism supports unified node-level governance (what is an aspect, anyway)
- A call-option mechanism: extensibility is the most basic need of a system that's iterating fast
- Stronger "type alignment" as a way of developing, which lowers the mental load and actually uses Go's type safety
- Automatic stream conversion, so "streaming" drops off the list of things that make an orchestration system complicated (Eino stream programming)
Graph's downside: a Graph based on the "node" / "edge" model makes you use both graph.AddXXXNode() and graph.AddEdge() to open a data channel. Powerful, a bit complicated.
Eino wraps a more convenient interface, Chain. Chain is a wrapper around Graph; except for cycles, Chain exposes almost all of Graph's capabilities.