Months ago Manus posted a blog talking about Context Engineering.
https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus
You don't need all context to live in the messages history of your agent, so we need context offloading.
Langchain experience
Offload context to a file system
So one of the most popular ideas here is just using a file system.
Take the output of a tool message as an example, dump it to the file system, send back to your agent just some minimal piece of information necessary so it can reference the full context if it needs to, but that full payload, for example, a web search result that's very token-heavy, isn't spammed into your context window for perpetuity.
Offloading context takes some piece of information, like a tool message that's token-heavy, and not sending it all back to your messages list, but dumping it to a file system where it can be retrieved only as needed.
Reduce context
Summarize or compress information to reduce context. Summarizing tool call outputs is one intuitive way to do this. So this idea of pruning old tool calls with tool outputs or tool messages is something that Claude is now kind of built into their SDK.
Cognition (an agent application) also talks about the idea of summarizing / pruning at agent-to-agent handoffs.
Retrieve Context
Claude Code Force only uses the file system and simple search tools, notably glob and grep. So there's different ways to retrieve context on demand for your agent.
Indexing and something like semantic search, file system and simple file search tools, both can be highly effective.
Context isolation
Context isolation is major, in particular splitting context across multi-agents.
Each sub-agent has its own context window and sub-agents allow for separation of concerns.
Caching Context
langchain open deep research
https://github.com/langchain-ai/open_deep_research
It has three phases: scoping of the research, the research phase itself using a multi-agent, basically architecture, and then a final one-shot writing phase. We use offloading, so we basically create a brief to scope our research plan.
We offload that so we don't just save that in the context window because that context window is going to get peppered with other things.
We offload it so it's saved independently, it can be accessed in our case from the LangGraph state, but it could also be from file system, it's the same idea.
So you create a research plan, you offload it, it's always accessible. You go do a bunch of work, you can pull that back in on demand so you can put it kind of at the end of your message list so it's accessible and readily available to your agent to perform, for example, the writing phase.
We use offloading, as you can see, to help steer the research and writing phases. We use reduction to summarize observation from token-heavy surf tool calls, that's done inside research itself.
And we use context isolation across sub-agents within research itself. And this is kind of a summary of a bunch of different of these various ideas across a bunch of different projects.
Manus experience
Instead of building specialized models too early, startups really should lean on general models and context engineering for as long as possible.
Context Reduction: Compaction vs. Summarization
For compaction, in Manus, every tool call and tool result actually has two different formats: a full format and a compact one.
The compact version strips out any information that can be reconstructed from the file system or external state. For example, here, let's say you have a tool that writes to a file and it probably has two fields, a path and a content field.
And once the tool returns, you can ensure that the file already exists in the environment. So in the compact format, we can safely drop the super long content field and just keep the path.
And if your agent is smart enough, whenever it needs to read that file again, it can simply retrieve it via the path. So no information is truly lost. It's just externalized.
We think this kind of reversibility is crucial because agents do chain predictions based on previous actions and observations and you never know which past action will suddenly become super important like 10 steps later.
You cannot predict it. So this is a reversible reduction by using compaction.
Of course, compaction only takes you so far. Eventually your context will still grow and will hit the ceiling, and that's when we combine compaction with the more traditional summarization, but we do it very carefully.
For example, here, before summarizing, we might offload key parts of the context into files. And sometimes we even do more aggressively, we can dump the entire pre-summary context as a text file or simply a log file into the file system so that we can always recover it later.
And like Lance just mentioned, some people just use glob and grep. You know, glob also works for log files. So if the model is smart enough, it even knows how to retrieve those summarized, those pre-summarized context.
The difference here is that compaction is reversible, but summarization isn't. Both reduce context lengths, but they behave very differently.
To make both methods coexist, we have to track some context length thresholds. At the top you'll have your model's hard context limit, say like 1 million tokens, pretty common today.
But in reality most models start degrading much earlier, typically maybe around 200k, and you'll begin to see what we call a context rot, like repetitions, slower inferences, degraded quality.
So by doing a lot of evaluation, it's very important for you to identify that pre-rot threshold, it's typically 128K to 200K, and use it as the trigger for context reduction.
And whenever your context size approaches it, you have to trigger context reduction, but starting from compaction, not summarization.
And compaction doesn't mean compressing the entire history. You know, we might compact the oldest 50% of tool calls while keeping the newer ones in full detail so the model still has fresh few-shot examples of how to use tools properly.
Otherwise in the worst case, the model will imitate the behavior and output those compact format with missing fields and that's totally wrong.
And after compaction, we have to check how much free context that we actually gain from this compaction operation. Sometimes like in this graph, after multiple rounds of compaction, the gain is tiny because even it's compact, it still uses context.
And that's when we go for summarization, but also keep in mind that when summarizing, we always use the full version of the data, not the compact one.
And we still keep the last few tool calls and tool results in full detail, not summary, because it can allow the model to know where it left off and will continue more smoothly.
Otherwise, you'll see after summarization sometimes the model will change its style, change its tone, and we find out like keeping a few tool call / tool result examples really help.
Context Isolation: Communicating vs. Sharing Memory
Cognition's blog shows that they warn against using multi-agent setups because when you have multiple agents, syncing information between them becomes a nightmare.
Multi-process or multi-thread coordination has been a classic challenge in the early days of computer programming, and I think we could borrow some wisdoms here.
In the Go programming language community there's a famous quote from this gopher, "Do not communicate by sharing memory, instead share memory by communicating."
https://chatgpt.com/share/68f4f8c3-baac-8004-9cf7-421375260909
Of course, this isn't directly about agent and it's sometimes even wrong for agents, but I think the important thing is it highlights two distinct patterns here which is by communicating or by sharing memory.
Like if we translate the term memory here into context, we can see that parallel pretty clear. "By communicating" is like the easier one to understand because it is the classic sub-agent setup here.
For example, the main agent writes a prompt and the prompt is sent to a sub-agent, and the sub-agent's entire context only consists of that instruction.
We think if a task has a short, clear instruction and only the final output matters, say like searching a codebase for a specific snippet, then just use the communication pattern and keep it simple.
Because the main agent doesn't care how the sub-agent find the code, it only needs the result.
And this is what Claude Code does, typically using its task tool to delegate a separated clear task to some sub-agents.
But for more complex scenarios, in contrast, "by sharing memory" means that the sub-agent can see the entire previous context. It means all the tool use, tool use history, tool usage history, but the sub-agent has its own system prompt and its own action space.
For example, imagine a deep research scenario, the final report depends on a lot of intermediate searches and notes. And in that case, you should consider using the share memory pattern or in our language "by sharing context," because even you can save all that notes and searches into file and making the sub-agent to read everything again, but you're just wasting latency and context.
And if you count the amount of token, maybe you're using even more token to do this. So we think for those scenario that requires a full history, just use a share memory pattern.
But be aware that sharing context is kind of expensive because each sub-agent has a larger input to prefill, which is like you'll spend more on input tokens, and since the system prompt and the action space differs, you cannot reuse the KV cache, so you have to pay the full price.
Context Offloading: Layered Action Space
When people say offload, they usually mean moving parts of the working context into external files.
But as system grows, especially if you decide to integrate MCP one day, you realize that the tools themselves can also take up a lot of context, and having too many tools in context leads to confusion.
We call it context confusion, and the model might call the wrong ones or even non-existing ones.
So we have to find a way to also offload the tools. A common approach right now is like doing dynamic RAG on tool descriptions, for example like loading tools on demand based on the current task or the current status.
But that also causes two issues. First of all, since tool definitions sit at the front of the context, your KV resets every time.
And most importantly, the model's past calls to removed tools are still in the context, so it might fool the model into calling invalid tools or using invalid parameters.
So to address this, Manus is experimenting with a new layered action space. Essentially, we can let Manus choose from three different levels of abstractions: number one, function calling, number two, sandbox utilities, and number three, packages and API.
We go deeper into these three layers of layered action space. Let's start from level one, function calling, and this is a classic, everyone knows it. It is schema safe thanks to constraint decoding, but we all know the downsides.
For example, we mentioned breaking the cache and maybe too many tool calls will cause some confusion, too many tools may cause confusion.
So Manus uses a fixed number of atomic functions, for example like reading and writing files, executing shell commands, searching files in internet, and maybe some browser operations.
These atomic functions have super clear boundaries, and they can work together to compose much more complex workflows.
Then we offload everything else to the next layer, which is the sandbox utilities. As you know, each Manus session runs inside a full virtual machine sandbox. It's running on our own customized Linux system, and that means Manus can use the shell commands to run pre-install utility that we develop for Manus.
For example, we have some format converters, we have speech recognition utilities, and even a very special, we call it MCP CLI which is how we call MCP.
We do not inject MCP tools to the function calling space. Instead, we do everything inside that sandbox through the command line interface.
And utilities are great because you can add new capabilities without touching the model's calling space, and you know it's just some commands pre-installed in your computer.
And if you're familiar with Linux, you always know how to find those new commands and you can even run --help to figure out how to use a new tool.
And another good thing is for larger outputs, they can just write to files or return the result in pages.
And you can use all those Linux tools like grep, cat, less, more, to process that results on the fly. So the trade-off here is it's super good for large outputs, but it's also not that good for low latency back and forth interactions with the front end.
Because you always have to visualize the interactions of your agent and show it to the user.
And then we have another layer, the final layer, we call it packages and APIs. You know, here Manus can write Python scripts to call pre-authorized API or custom packages.
For example, Manus might use a 3D designing library for modeling or call a financial API to fetch market data. And here actually we've purchased all these API on behalf of a user and pay the money for them.
It's included in the subscription. So we basically we have a lot of API keys pre-installed in Manus and Manus can access these APIs using the keys.
I think these are perfect for task that requires lots of computation in memory, but do not need to push all that data into the model context.
For example, imagine if you're analyzing a stock's entire year of price data, you don't feed the model all the numbers. Instead, you should let the script to compute it and only put the summary back into the context.
And you know, since code and APIs are super composable, you can actually chain a lot of things in one step.
For example, in a typical API, you can get city names, get city ID, get weather, all in one Python script.
There's also a paper from one of my friend called Code Act. A lot of people were discussing about it. I think it's the same idea because code is composable and it can do a lot of things in one step.
But also it's not schema safe. It's very very hard to do constrained decoding on code.
So we think you should find the right scenario for these features. For us, we use all as we mentioned everything that's that can handle inside a compiler or interpreter runtime, we do that using code.
Otherwise, we use sandbox utilities or function calls.
And the good thing is if you have these three layers from model's point, all three levels still go through the standard function calls, so the interface stays simple, cache friendly, and orthogonal across functions.
Because you know, we mentioned sandbox utilities, you're still accessing these tools using the shell tool, accessing these tools using the shell function.
And also if you're using APIs in third-party applications, you're just using the file function to write or read file and then execute it using the shell function.
So you think it does not add overhead to the model. It's still all the things that models are trained and they're already familiar with.
Connecting the Five Dimensions and Avoiding Over-engineering
Let's zoom out and connect the five dimensions: offload, reduce, retrieve, isolate, and cache. You can find out that they are not independent.
We can see that offload and retrieve enables more efficient reduction and stable retrieve makes isolation safe, but isolation also slows down context and reduces the frequency of reduction.
However, more isolation and reduction also affects cache efficiency and the quality of output. So at the end of the day, I think context engineering is the science in art that requires a perfect balance between multiple potentially conflicting objectives.
I want to leave with maybe one final thought, and it's kind of the opposite of everything I just said, which is please avoid context over-engineering.
Looking back at the past six or seven months since Manus launch, actually the biggest leap we've ever seen didn't come from adding more fancy context management layers or clever retrieval hacks.
They all came from simplifying or from removing unnecessary tricks and trusting the model a little more.
Every time we simplify the architecture, the system got faster, more stable, and smarter, because we think context engineering should, the goal of context engineering is to make the model's job simpler, but not harder.
So if you take one thing from today, I think it should be build less and understand more.
Q&A
Q&A - Shell Tools and Sandboxing
Q: How does the LLM call the various shell tools? How does it know which tools exist and how to invoke them?
Maybe you can explain a little bit about kind of the multi-tier kind of sandboxing setup that you use with Manus.
A: First of all, we have a hint in the system prompt telling Manus that, hey, there's a lot of pre-installed command line utilities located in some specific folder.
And also for the most frequently used ones, we already injected in the system prompt, but it's super compact. We do not tell the agent how to use the tools.
We only list them and we can tell the agent that you can use the --help flag safely because all the utilities are developed by our team and they have the same format.
Q&A - Indexing vs. File System for Context Retrieval
Q: I know you talked a lot about using file system. What's your take on using indexing, and do you utilize like do you spin up vector stores on the fly if the context you're working with gets sufficiently large?
A: There's no right and wrong in this space, like you've mentioned, but at Manus we do not use index databases because right now, you know, every sandbox in Manus session is a new one and user want to interact with things fast, so actually we don't have the time to build the index on the fly.
So we're more like Claude Code, we rely on grep and glob. But I think if you consider to build something like more long-term memory or if you want to integrate some enterprise knowledge base, you still have to rely on that external vector index because it's only about the amount of information that you can access.
But for Manus it operates in a sandbox and for coding agent, you operate in the codebase, so it depends on the scale.
Q: So let's say I'm a user, I have my Manus account, I interact with Manus across many sessions. Do you have the notion of memory?
So Claude has CLAUDE.md files, they persist across all the different sessions of Claude Code. How about you guys? How do you handle kind of long-term memory?
A: Actually in Manus we have a concept called knowledge, which is kind of like explicit memory.
For example, every time you can tell Manus, hey, remember every time I ask for something, deliver it maybe in Excel, and it's not automatically inserted into some memory.
It will pop up a dialogue and say, here's what I learned from our previous conversation and would you like accept it or reject it? So this is the explicit one, it requires user confirmation.
But also we are discovering new ways to do it more automatically. For example, a pretty interesting thing in agents is that compared to chatbots, user often correct the agent more often.
For example, a common mistake that Manus make is when doing data visualization, you know, if you're using Chinese, Japanese, or Korean, a lot of time there will be some font issues and there will be errors in those render visualizations.
So the user will often say, hey, you should use a CJK font. And for these kind of things, the user will, a different user will have the same correction and we need to maybe they'll find out a way to leverage these kind of a collective feedback and use it.
That's kind of like we call it self-improving agent with online learning, but in a parameter free way.
Q&A - Adapting to Evolving Models
Q: You mentioned towards the end of your talk that you gained a lot from removing things, and a lot of that is probably because of the fact that also the models are getting better.
So model capabilities increasing and so you can kind of remove scaffolding over time. How do you think about this?
Because this is one of the biggest challenges that I've faced is like over time the model gets better, and I can remove things like certain parts of my scaffolding, so you're building on top of this, the foundation that's like the water's rising.
Do you revisit your architecture every some number of months with new releases and just delete as the models get better, and how do you approach that problem?
A: This is a super good question here because you know, actually we have already refactored Manus for five times, and we've launched Manus in March and now it's October already, five times.
So we think you cannot stop because models are not only improving, but they are changing. Models' behaviors are changing over time.
One way is you can work closely with those model providers, but we also have another internal theory for how we evaluate or how we design our agent architecture.
I cover a little bit on Twitter before. It's basically like we do not care about the static performance of a static benchmark.
Instead, we fix the agent architecture and we switch between models.
If your architecture can gain a lot from switching from a weaker model to a stronger model, then somehow your architecture is more future-proof because the weaker model tomorrow might be as good as a stronger model today.
So we think switching between weaker and strong models can give you some early signals of what will happen next year and give you some time to prepare your architecture.
For Manus, we often do these kind of review every one or two months, and we often do some research internally using open source models and maybe early access to proprietary models to prepare the next release even before the launch of the next model.
Q&A - Data Storage Formats
What about best practices or considerations for format for storing data?
So like markdown files, plain text, log, anything you prefer in particular?
I think obviously it's, yeah, how do you think about that kind of file formats for?
A: I think it's not about plain text or markdown, but we always prioritize line based formats because it allows the models to use grep or read from a range of lines.
And also markdown can sometime cause some troubles. You know, models are trained to use markdown really well, and sometimes it will maybe for some model, I don't want to say that name, but they often output too many bullet points if you use markdown too often.
Yeah, so actually we want to use more plain text.
Q&A - Prompting for Summarization
How about on the topic of compaction versus summarization?
Let's hit on summarization. This is an interesting one that I've been asked a lot before, how do you prompt to produce good summaries?
So for example, summarization like you said, it's irreversible, so if you don't prompt it properly, you can actually lose information.
The best answer I came up with is just tuning your prompt for high recall, but how do you approach this?
How do you think about prompting for summarization?
A: We tried a lot of optimizing the prompt for summarization, but it turns out a simple approach works really well is that you do not use a free form prompt to let the AI generate everything.
Instead, you could define a kind of a schema. It's just a form, there's a lot of fields and let the AI to fill them.
For example, here are the files that I've modified and here's the goal of the user, here's what I left off.
And if you use this kind of a more structured schema, at least the output is kind of stable and you can iterate on this, so just do not use free form summarizations.
Q&A - Compaction of Search Results
How about with context, how about with compaction then?
And actually, I want to make sure I understood that. So with compaction, let's say it's a search tool, you have the raw search tool output and would it be that would be your raw message and then the compaction would just be like a file name or something, is that right?
A: Yeah, it is. It's not only about the tool call, it's also applied to the result of the tool.
We interestingly we find out that almost every action in Manus is just kind of like reversible if you can offload it to the file system or an external state.
For most of these tasks, you already have a unique identifier for it. For example, for file operations, of course, you have the file path.
For browser operations, you have the URL, and even for search actions, you have the query.
So it's naturally it's already there.
Lance: And just want to hit that again because I've had this problem a lot. So for example, I'm an agent that uses search, I perform a, it returns a token-heavy tool call.
I don't want to return that whole tool message to the agent.
I've done things like some kind of summarization or compaction and send the summary back, but how do you approach that because you might want all that information to be accessible for the agent for his next decision, but you don't want that huge context block to live inside your message history?
So how do you approach that? You could send the whole message back, but then remove it later, that's what Claude does now.
You could do a summarization first and send the summary over. You could send everything and then do compaction so that later on you don't have the whole context in your message history.
You only have like a link to the file. How do you think about that specifically if you see what I'm saying?
A: I know actually it depends on the scenario. For example, for complex search, I mean for complex search, I mean it's not just one query.
For example, you have multiple queries and you want to gather some important things and drop everything else.
In this case, I think we should use sub-agents or internally we call it agent as tool. So from the model's perspective, it's still a kind of function, maybe called advanced search.
It's a function called event search, but what it triggers is actually another sub-agent, but that sub-agent is more like a workflow or agentic workflow that has a fixed output schema, and that is the result that returns to the agent.
But for other kinds of more simpler search, for example, just like searching Google, we just use the full detail format and append it into the context and rely on the compaction thing.
But also we always instruct the model to write down the intermediate insights or key findings into files in case that the compaction happens earlier than the model expected.
And if you do this really well, actually you don't lose a lot of information by compaction because sometimes those old tool calls are irrelevant after time.
Q&A - Agent-to-Agent Communication & MapReduce
Q: I like the idea of agent as tool, we do that quite a bit and that is highly effective, but that brings up another interesting point about, and you referenced this a little bit, agent agent communication.
How do you address that?
So Walden Yen from Cognition had a very nice blog post talking about this is like a major problem that they have with Devin.
So like kind of communication between agents, how do you think about that problem and yeah, ensuring sufficient information is transferred but not overloading like you said the prefill of the sub-agent with too much context?
A: We've launched a feature called Wide Research a month ago, like it's basically like we call, yeah, internally we call it agentic map reduce because we got inspired from the design of MapReduce.
And it's kind of special for Manus because, there's a full virtual machine behind the session, so one way we pass information or pass context from the main agent to sub-agent is by sharing the same sandbox.
So the file system is there and you can only pass like the different path here.
And I think sending information to sub-agent is not that hard. The more complex thing is about how to have the correct output from different agents.
And what we did here is like we have a trick for every time if the main agent want to spawn up a new sub-agent or maybe 10 sub-agents, you have to design, you have to let the main agent to define the output schema.
And in the sub-agent perspective, you have a special tool called submit_result, and we use constraint decoding to ensure that what the sub-agent submits back to the main agent is the schema that is defined by the main agent.
Yeah, so you can imagine that this kind of MapReduce operation, it will generate a kind of like spreadsheet and the spreadsheet is constrained by the schema.
Lance: That's an interesting theme that seems to come up a lot with how you design Manus, you use schemas and structured outputs both for summarization and for this agent agent communication.
So it's kind of like use schemas as contracts between agent sub-agent or between like a tool and your agent to ensure that sufficient information is passed in a structured way in a complete way, like when you're doing summarization, you use a schema as well.
Q&A - Model Choice and Open Models
I'm poking around some other interesting questions here. Any thoughts on models like I think you guys use Anthropic, but do you work with open models?
Do you do fine-tuning? You talked a lot about kind of working with KV cache, so for that, maybe using open models.
How do you think about like model choice?
A: Actually right now we don't use any open source model right now because I think it's not about quality, it's interestingly it's about cost.
We often think that open source model can lower the cost, but if you're at the scale of Manus and if you're building a real agent, which the input is way longer than the output, then KV cache is super important.
And distributed KV cache is very hard to implement if you use open source solutions.
If you use those frontier pro LLM providers, they have more solid infrastructure for distributed cache globally.
So sometimes if you do the math, at least for Manus, we find out that using these flagship models can sometimes they can be even more cheaper than using open source models.
And right now, we're not only using Anthropic.
Like Anthropic's model is the best choice for agentic task, but we're also seeing the progress in Gemini and in OpenAI's new model.
I think right now these frontier labs are not converging in directions. For example, if you're doing coding, of course, you should use Claude.
And if you want to do more multimodal things, you should use Gemini.
And OpenAI's model is super good at complex math and reasoning. So I think for application companies like us, one of our advantage is that we do not have to build on top of only one model.
You can do some task level routing or maybe even subtask or step level routing if you can calculate if you can pull in that kind of KV cache validation.
So I think it's advantage for us and we do a lot of evaluations internally to know which models to use for which subtask.
Lance: With KV cache, so what specific features from the providers are you using for cache management?
I know like Anthropic has input caching as an example.
Q&A - Tool Selection and Layered Action Space (Revisited)
Q: Tool selection is a good one. Right, so you were talking about this, you don't use indexing of tool descriptions and fetching tools on the fly based on semantic similarity.
How do you handle that? Like what's the threshold for too many tools?
Tool choice is a classic. How do you think about that?
A: First of all, it depends on the model. Different model has different capacity for tools, but I think a rule of thumb is try not to include more than like 30 tools.
It's just a random number in my mind, but actually I think if you're building a we call it a general AI agent like Manus, you want to make sure those native functions are super atomic.
So actually there are not that much atomic function that we need to put inside the action space.
So like for Manus, we right now we only have like 10 or 20 atomic function, and everything else is in the sandbox.
We don't have to pull things dynamically.
Lance: Let's explain that a little bit more, so you have, let's say, 10 tools that can be called directly by the agent, but then I guess it's like you said the agent can also choose to for example write a script and then execute a script.
So that expands its action space hugely without giving it like you don't have an independent tool for each possible script, of course, that's insane.
So our very general tool to write a script and then run it does a lot.
A: Why we are super confident to call Manus a general agent?
Because it runs on a computer, and computers are Turing complete. The computer is the best invention of human.
Like theoretically, an agent can do anything that maybe a junior intern can do using a computer.
So with the shell tool and the text editor, we think it's already complete, so you can offload a lot of things right to sandbox.
Lance: You mentioned code with code agents. My understanding is the model will actually always produce a script and that'll then be run inside a code sandbox so every tool call is effectively like a script is generated and run.
It sounds like you do some hybrid where sometimes Manus can just call tools directly, but other times it can actually choose to do something in the sandbox, is that right?
A: I think this is super important because actually we try to use entirely CodeAct for Manus, but the problem is if you're using code, you cannot leverage constraint decoding and things can go wrong.
CodeAct has some special use cases as I mentioned earlier in slides, for example, processing a large amount of data.
You don't have to port everything in the tool result.
It's that you put it inside maybe the runtime memory of Python and you only get the result back to the model.
So we think you should do it like in a hybrid way.
Q&A - Planning and To-Do Lists
Q: Tell me about planning and I know Manus has this to-do tool or it generates a to-do list and start of tasks.
A: At the beginning Manus uses that to-do.md paradigm.
It's kind of, I don't want to use the word stupid, but actually it wastes a lot of turn.
You know, like back in maybe March or April, if you check the log of some Manus task, maybe like one-third of the action is about updating the to-do list.
It wastes a lot of tokens.
So right now we're using a more structuralized planning. For example, if you use Manus, there's a planner at the bottom of the system.
Internally, it's also kind of a tool called it's, we implemented using the agent as tool paradigm so that there's a separate agent that is managing the plan.
So actually right now the latest version of Manus, we are no longer using that to-do.md thing.
Like todo.md still works and it can generate like good results, but if you want to save tokens, you can find another way.
Q&A - Multi-Agent Design and Roles
So you might have like a planning agent with its own context window, makes a plan, produces like some kind of plan object, maybe it's a file or maybe it just calls sub-agents directly.
How do you think about that like and how many different sub-agents do you typically recommend using?
A: I think this is also depends on your design, but here at Manus actually Manus is not kind of like the typical multi-agent system.
For example, we've seen a lot of different agent that divides by role.
For example, you have a designer agent or programming agent, manager agent, we don't do that because we think why we have this is because this is how human company works and this is due to the limitation of human context.
So in Manus, Manus is a multi-agent system, but we do not divide by role.
We only have very few agents. For example, we have a huge general executor agent and a planner agent and a knowledge management agent and maybe some data API registration agent.
So we are very very cautious about adding more sub-agents because of the reason that we've mentioned before, communication is very hard.
And we implement more kinds of sub-agents as agent as tools as we mentioned before.
Lance: I see this mistake a lot, or I don't know if it's a mistake, but you see anthropomorphizing agents a lot like it's my designer agent, and I think it's kind of a forced analogy to think about like a human org chart in your sub-agents.
It's like a planner and knowledge manager. A knowledge manager might do what like what will be the task of knowledge manager like?
A: We mentioned we have a knowledge system in Manus.
What the knowledge agent does is that it reviews the conversation between the user and the agent and figure out what should be saved in the long-term memory.
Q&A - Safety and Guardrailing in Sandboxed Environments
How about guardrailing? Someone asked a question about kind of safety and guardrailing.
A: If you have a sandbox that's connected to the internet, everything is dangerous. So we have put a lot of effort in guard railing, like at least we do not let the information to get out of the sandbox.
For example, if you got prompt injected, we have some checks on outgoing traffic.
For example, we'll ensure that no token things will go out of the sandbox.
And if the user wants to print something out of the sandbox, we have those kind of removing things and to ensure that no information go out of the sandbox.
But you know, for another kind of thing is that we have a browser inside of Manus, and the browser is very complicated.
For example, if you log into some your websites, you can choose to let Manus to persist your login state, and this turns out to be very tricky because sometime the content of the web page can also be malicious, maybe they're doing prompt injection.
And this, I think, is somehow out of scope for application company. So we're working very closely with those computer use model provider.
For example, Anthropic and Google. Yeah, they're adding a lot of guardrails here.
So right now in Manus, every time you do some sensitive operations whether inside the browser or in the sandbox, Manus will require a manual confirmation and you must accept it or otherwise you have to take over it to finish it yourself.
So I think it's pretty hard for us to design a very well-designed solution, but it's a progressive approach.
So right now we're letting the user to take over more frequently, but if the guard rail itself in the model gets better, we can do less.
Q&A - Evaluation Strategies
How about the topic of evals? This has been discussed a lot quite a bit online if you probably seen, you know, Claude Code, they talked a lot about just doing less formal evals at least for code because code evals are more or less saturated, lots of internal dog fooding.
How do you think about evals? Are they useful? What evals are actually useful?
What's your approach?
A: Yes, yeah, you know, at the beginning at the launch of Manus, we're using public academic benchmarks like Gaia, but then after launching to the public, we find out that it's super misaligned.
Models that get high scores on Gaia, the user don't like it.
So right now we use like three, we have three different kinds of evaluations.
First of all, most importantly is that for every completed session in Manus, we'll request the user to give a feedback to give one to five stars.
This is the gold standard. Like we always care about the average user rating. This is number one.
And number two, we're still using some internal automated tests with verifiable results.
For example, we have created our own data set with clear answers. But also we still use a lot of public academic benchmarks, but we also created some data sets that's more focused on execution because most benchmark out there are more about read-only tasks.
So we designed some executing tasks or transactional task because we have the sandbox, we can frequently reset the test environment.
So these are the automated parts. And most importantly, number three, we have a lot of interns, you know, you have to use a lot of real human interns to do evaluations on things like website generation or data visualization because it's very hard to design a good reward model that knows whether the output is visually appealing.
It's about the taste.
Q&A - RL with Verifiable Rewards vs. Tool Calling Agents
I do want to ask you about this emerging trend of reinforcement learning with verifiable rewards versus just building tool calling agents.
So like Claude Code, extremely good, and they have the benefit because they built the harness and they can perform RL on their harness and it can get really really good with the tools they provide in the harness.
Do you guys do RL, or how do you think about that?
Because of course, in that case, you would have you using open models.
I've been playing with this quite a bit lately. How do you think about that, just like using tool calling out of the box with model providers versus doing RL yourself inside your environment with your harness?
A: I've been doing pre-training, post-training, RL for a lot of years, but I have to say that right now if you have sufficient resource, you can try.
But actually like we, as I mentioned earlier, MCP is a big changer here because if you want to support MCP, you're not using a fixed action space.
And if it's not a fixed action space, it's very very hard to design a good reward, and you cannot generate a lot of the rollouts and feedbacks will be unbalanced.
So if you want to build a model using that supports MCP, you are literally building a foundation model by yourself.
So I think everyone in the community like model companies they're doing the same thing.
They're doing the same thing for you. So right now I don't think we should spend that much time on doing RL right now, but like as I mentioned earlier, we are just discovering exploring new ways to do like maybe call it like personalization or some sort of online learning, but using like parameter free way.
For example, like collective feedbacks.
Lance: One little one along those lines is is it the case that for example Anthropic's done reinforcement learning at verified rewards on some set of tools using Claude Code.
Have you found that you can kind of mock your harness to use similar tool names to kind of unlock the same capability if that makes sense?
Like for example, I believe they've just, you know, they've obviously performed, you know, they utilized glob, uses grep, uses some other set of tools for manipulating the file system.
Can you effectively reproduce that same functionality by having the exact same tools with the same tool name, same descriptions in your harness or kind of how do you think about that like unlocking, yeah, right, you see what I'm saying?
A: I know the clear answer here, but for us we actually try not to use the same name because if you design your own function, you maybe have different requirements for that function, and the parameters, the input arguments might be different.
So you don't want to confuse the model like if the model is trained on a lot of post training data that has some internal tools, you don't want to let the models to be confused.