Introduced in v4 — The agent system is a major v4 feature that enables Honeydipper to interact with AI models through persistent, stateful conversations. It is fully integrated with the Honeydipper workflow engine, allowing agents to call systems, workflows, sub-agents, and remote MCP tools.
The agent system enables Honeydipper to run stateful, multi-turn conversations with AI models. Unlike traditional stateless API calls to LLMs, the agent system:
AgentSession (internal/agent/agent_session.go) holds the runtime state of a single agent inference or chat-turn session. It is the central object in the agent system.
| Field | Type | Description |
|---|---|---|
ID |
string |
Unique session identifier (UUID) |
ConvoID |
string |
Conversation identifier grouping multiple sessions |
UnifiedConvoID |
string |
Cross-conversation identifier for sub-agent chains |
Agent |
*config.Agent |
Resolved agent configuration |
history |
[]AgentMessage |
Conversation history (chronological message list) |
Type |
string |
SessionTypeInference or SessionTypeChatTurn |
TTL |
string |
Session cache TTL (default: 72h) |
ToolCalls |
[]AgentToolCall |
Pending tool calls from the model |
ToolResults |
[]map[string]interface{} |
Collected tool call results |
ParentSessionID |
string |
Parent session ID (set for sub-agent sessions) |
Sessions are stored in the distributed cache under the key prefix agent_session:<ID>.
ConvoState (internal/agent/convo_state.go) is the persisted, queryable view of a conversation. While an AgentSession tracks a single inference turn, the ConvoState spans the entire conversation.
| Field | Type | Description |
|---|---|---|
ConvoID |
string |
Canonical conversation ID |
UnifiedConvoID |
string |
Shared ID across sub-agent conversations |
FirstSession |
*ConvoSessionRef |
First session in this conversation |
LastSession |
*ConvoSessionRef |
Most recent session (UI-visible) |
ActiveSession |
*ConvoSessionRef |
Currently active session (for cancel targeting) |
Cancelled |
bool |
Whether the conversation has been cancelled |
TotalTokens |
int |
Cumulative input+output tokens |
Generation |
int |
Number of times compaction has occurred |
ArchivedConvos |
[]string |
Keys of archived pre-compaction histories |
Agent |
*config.Agent |
Resolved agent configuration (shared across sessions) |
Skills |
map[string]string |
Loaded skill name → path mapping |
Sessions in a conversation go through status transitions:
active → complete
→ failed
→ cancelled
ConvoState is stored under key prefix convo_state:<ConvoID> and is also published to a stream (convo_stream_) for UI consumption.
A ConvoSessionRef records a compact summary of each session:
type ConvoSessionRef struct {
SessionID string
AgentName string
Type string // "inference" or "chat_turn"
Status string // "active", "complete", "failed", "cancelled"
ErrorReason string
InputTokens int
OutputTokens int
TotalTokens int
CreatedAt time.Time
UpdatedAt time.Time
}
PersistentAgentStore (internal/agent/agent_store.go) is the eventbus-driven interface that bridges Honeydipper’s workflow engine and the agent system. It implements the AgentStore interface and handles lifecycle management of agent sessions.
Key methods:
| Method | Trigger | Description |
|---|---|---|
StartInference |
eventbus:agent_start |
Creates a new agent session and begins inference |
ContinueInference |
eventbus:agent_continue |
Continues a session with tool call results |
ReceiveInference |
agentbus:receive |
Processes model responses from AI drivers |
PollInference |
eventbus:agent_poll |
Polls session for streaming output |
StartAgentCall |
eventbus:agent_call |
Starts a sub-agent session (agent-as-tool) |
StartMCPCall |
eventbus:mcp_call |
Calls a remote MCP server tool |
CancelConvo |
eventbus:convo_cancel |
Marks a conversation as cancelled |
StartTurn |
HTTP API | Starts a new chat turn from the UI |
StartNewConvo |
HTTP API | Creates a brand-new conversation from the UI |
The store uses a sync.WaitGroup to track in-flight sessions and supports graceful draining via Stop() / Wait().
Agents are defined in the Honeydipper configuration under the agents section. Each agent maps to an AI model driver and a set of tools.
agents:
my_assistant:
driver: openai # AI model driver name
engine: gpt-4o # Model/engine name
system_prompt: |
You are a helpful DevOps assistant. You can help with deployments,
monitoring, and infrastructure management.
should_stream: true
max_history_len: 100
pre_context:
- /path/to/CONTEXT.md
skills:
- /path/to/skills/
file_tool: sys__files__read
compaction_policy:
strategy: summarize
threshold_type: history_len
threshold: 50
preserve_recent: 10
summarization_agent: fast_summarizer
summarization_prompt: "Summarize preserving key decisions."
continue_policy:
mode: auto # auto = supported; never = RESERVED (not honored in this phase)
# prompt: "Resume the task exactly where it stopped." # optional override; omit to use the built-in marker
tools:
- type: system
name: kubernetes
- type: workflow
name: deploy_app
- type: agent
name: code_reviewer
- type: mcp
name: github_server
type CompactionPolicy struct {
Strategy string // "summarize"
Threshold int // when to trigger compaction
ThresholdType string // "history_len" or "total_tokens"
PreserveRecent int // number of recent messages to keep verbatim
SummarizationAgent string // agent name to produce the summary
SummarizationPrompt string // optional custom prompt for summarization
}
When ThresholdType is "history_len", compaction triggers when len(history) >= Threshold.
When ThresholdType is "total_tokens", compaction triggers when the context size of the latest
model call >= Threshold. That value is InputTokens + OutputTokens of the most recent complete agent
message in the conversation history. In the default mode (token_counter unset) these are
driver-reported counts — the exact token unit the user observes, populated by hd-driver-openai from
msg.Usage.PromptTokens / msg.Usage.CompletionTokens. Under token_counter: simple the baseline is
instead heuristic: appendConvoHistory overwrites each message’s InputTokens/OutputTokens with
simple character-based counts before persistence, so the baseline reflects those heuristic values rather
than driver-reported usage. The baseline is derived from persisted history, refreshed once per turn under
the turn lock, and maintained incrementally as new complete agent messages arrive, so it stays accurate
even after interrupted, cancelled, or errored turns. This is a comparison against that context size; it
does not mutate the conversation’s ConvoState.total_tokens accounting field.
The default PreserveRecent is 10 messages. The default summarization prompt is:
“Summarize the above conversation history, preserving key decisions, context, and any critical information that will be needed to continue the conversation. Be concise but thorough. Explain what is currently happening at the end.”
Compaction degrades gracefully on failure: a failed, errored, or empty summarization result is validated
before anything destructive happens, so the live conversation history is never truncated or lost. The
pre-compaction history (including the triggering user message) was archived to a <ConvoID>_g<N> key before
the summarizer ran, so on failure it is rolled back from that archive when complete (or kept as-is when the
archive is partial), a RoleSystem placeholder noting the failure and archive location is injected, and the
conversation resumes with the full history (automatic path) or posts a clear failure reply (/compact path).
No compaction boundary or baseline is recorded when no compaction actually completed, and compaction cannot
re-fire within the same failed turn.
type ContinuePolicy struct {
Mode string // "auto" (default) or "never" (RESERVED; not honored in this phase)
Prompt string // optional override for the resume marker; empty -> built-in default
}
continue_policy controls the resume marker the engine may append to the history it sends an AI
driver when that history would otherwise not end on a user-role message. It follows the same config
shape and validation style as compaction_policy.
mode: auto (default, and the fallback for a nil/unknown mode) synthesizes a marker only for an
interrupted agent turn.mode: never is RESERVED for a later phase in which drivers advertise their own
require_trailing_user handling. It is not honored yet: the daemon emits an assembly-time warning
(mode: never requires driver-side handling; using auto) and degrades to auto at runtime. Config
validation never fails assembly.prompt overrides the built-in marker text. It is intentionally not interpolated — interpolating
session data into a user-authority turn is the anti-goal, and the marker must stay byte-stable across
turns. An empty or whitespace-only prompt falls back to the built-in default and is never transmitted
as empty content (Anthropic and Gemini reject empty content; OpenAI treats it as inert). Omit the key
entirely to use the default silently; setting it to an empty string is treated as a likely
misconfiguration and logged at assembly time (agent %q continue_policy: empty prompt; default text
will be used).type AgentToolDef struct {
Type string // "system", "workflow", "agent", or "mcp"
Name string // System/workflow/agent/MCP server name
Only []string // (MCP only) Whitelist of tool names to expose
Excludes []string // (MCP only) Blacklist of tool names to exclude
}
At the start of each model call, the agent session assembles a tool map from its configured tools. Tools are exposed to the AI model using a naming convention based on their type.
sys_ prefix)Each function of a named system is exposed as a callable tool:
sys_<system_name>__<function_name>
Example: If a system named kubernetes has functions deploy, get_pods, and scale, the following tools are generated:
| Tool Name | Maps To |
|---|---|
sys__kubernetes__deploy |
kubernetes.deploy() |
sys__kubernetes__get_pods |
kubernetes.get_pods() |
sys__kubernetes__scale |
kubernetes.scale() |
System functions with skip_agent: true in their metadata are excluded.
wf__ prefix)Named workflows are exposed directly:
wf__<workflow_name>
Example: A workflow named deploy_app becomes wf__deploy_app.
ag__ prefix)Other agents can be called as sub-agents:
ag__<agent_name>
Example: An agent named code_reviewer becomes ag__code_reviewer.
Agent tools accept these parameters:
| Parameter | Type | Description |
|---|---|---|
input |
string |
Input text to send to the sub-agent |
one_shot |
boolean |
If true, runs without persisting conversation history |
forget_history |
boolean |
If true, clears the agent’s previous conversation history |
compaction_id |
string |
(Internal) Used during compaction for loading archived history |
mcp__ prefix)Remote MCP (Model Context Protocol) server tools are exposed with a two-underscore separator:
mcp__<server_name>__<tool_name>
Example: If an MCP server named github_server exposes a tool create_issue, the resulting tool is mcp__github_server__create_issue.
MCP tool lists are cached (default TTL: 6 hours) to avoid slow chat turns on repeated calls. The cache TTL is configurable via daemon.services.agent.tools_cache_ttl.
Using Only and Excludes in AgentToolDef, you can filter which MCP tools are exposed:
tools:
- type: mcp
name: github_server
only:
- create_issue
- list_repos
hd_load_skill)When the agent’s systemPrompt contains the skills header marker and SkillsPaths are configured, a special hd_load_skill tool is automatically registered. This allows the model to dynamically load skill content during a conversation:
hd_load_skill(skill_name: "file_ops/read_file")
The agent system uses long-polling for streaming model output. This allows real-time delivery of model responses without requiring WebSocket or SSE support from the HTTP layer.
Flow:
agent_poll message.The polling mechanism includes:
PendingContent, PendingThoughts) are accumulated and flushed on poll.Completion-only polls (wait_for_completion: true) finish on a complete assistant
response without tool calls, including an empty response after a successful tool
call. Each active wait cycle reloads persisted history. A completed session retains
its unread response independently of conversation trimming, compaction, or reset,
so a subsequent turn cannot replace its answer or produce a stale cancellation
warning. A Slack delivery tool result alone is not turn completion.
With cancel_on_timeout: true, cleanup cancels only the polled session and retains
the original timeout reason if cleanup fails. Conversation turn locks use the
session ID as the Redis lock’s optional owner; release atomically compares that
owner before deleting. Missing, expired, already-released, or successor-owned
locks are safe no-ops. Calls without owner retain the legacy locker behavior.
Deploy the core and bundled redislock driver together and drain existing turns
before upgrading: locks acquired by older code lack session ownership and are
left to expire rather than being deleted by owner-checked cleanup.
When conversation history grows too long, the agent system automatically compacts it to reduce token usage. Compaction is triggered lazily, before the next model call, when the configured threshold is exceeded.
Compaction process:
shouldCompact() checks if the threshold is reached.compactHistory() archives the current history under a generation-suffixed key (convo_history:<ConvoID>_g<N>).compaction_id, preserve, and a summarize_upto boundary.summarize_upto boundary), produces a summary, and returns it via eventbus:agent_continue.handleCompactionResult() replaces the history with the summary (as a system message) plus the most recent PreserveRecent messages.Summarizer instruction adherence: the summarizer sub-agent is generic and would otherwise see the triggering user’s question as its newest message and frequently answer it instead of summarizing (losing history). Two mechanisms guarantee it summarizes instead:
summarize_upto boundary: compactHistory() passes the index of the last non-slash user message (the
triggering user message) as a summarize_upto param. The summarizer honors it when loading its archived
history (loadConvoHistory), so the triggering user message is excluded from its input. The boundary is
gated on the param being present, so non-compaction ag__ sub-agent calls are completely unaffected.SummarizationPrompt), supplementing (never replacing) the summarizer agent’s own system prompt.The triggering user message stays in the preserved tail and the full _gN archive (intact for the UI);
only the summarizer’s loaded input excludes it, so the model still answers the user’s question post-compaction
while the compaction-boundary bookkeeping (CompactionHistoryIdx / LastCompactionHistoryLen /
PrevContextSize / refreshContextSize()) is completely unchanged by the exclusion.
Key constraints:
threshold_type values (history_len and
total_tokens); it is effectively a no-op for history_len in normal operation because compaction
shrinks history back below threshold, but it guarantees total_tokens never loops within a single turn.threshold_type: total_tokens, the metric compared against Threshold is the context size of the
latest model call: InputTokens + OutputTokens of the most recent complete, non-slash agent message in
the conversation history. In the default mode (token_counter unset) these are driver-reported
counts; under token_counter: simple the baseline is heuristic (character-based counts written by
appendConvoHistory). This baseline is derived from persisted history (so existing long conversations
get a correct baseline automatically on upgrade), refreshed under the turn lock after a fresh history
load, and kept up to date as complete agent messages arrive — even when a prior turn was interrupted,
cancelled, or errored. The compaction boundary marker is persisted in ConvoState and re-seeded on each
new user turn so a preserved-tail agent message carrying pre-compaction tokens cannot re-trigger
compaction when the post-compaction resume produced no new agent reply. After compaction the baseline
self-heals from the next agent reply.Threshold is set below roughly (summary + preserve-window)
tokens, compaction will re-trigger on every subsequent user turn. Set Threshold comfortably above the
expected post-compaction baseline to avoid re-firing on each turn.convo_history:<ConvoID>_g1, _g2, etc.Some driver protocols can require the history to end on a user-role message. The engine therefore
synthesizes a trailing user turn only when the serialized history ends on an interrupted agent
turn — a trailing agent message with no pending ToolCalls. The marker is unambiguous to the model
(an automatic continuation, not new user input), provenance-marked (Message.IsSynthetic), and
configurable per agent (agents.<name>.continue_policy).
Decision matrix (decided on the slash-filtered serialized history):
| Trailing real message | Synthesis |
|---|---|
user |
no — already a user turn |
tool_result (a tool round-trip) |
no — every supported driver terminates correctly on tool results |
agent with pending ToolCalls |
no — this means resolveUnresolvedToolCalls() did not run; a Warning is logged rather than hiding the bug behind a user turn |
agent without ToolCalls (interrupted turn) |
yes — marked IsSynthetic |
tool (legacy) / unknown role |
no — logged at Debug |
| empty / system-only serialized history | no — logged at Debug (continue_policy: no actionable turn, sending as-is) |
Every sendToDriver() entry point (run, processToolResult, handleCompactionResult, slash
/retry, and post-restart recover) yields a non-empty, non-system tail, so the “no actionable turn”
case is a defensive no-op, never an injection.
Invariants: the synthesized turn is ephemeral. It is appended only to the local driver request
and is never persisted to the conversation history (s.history / convo_history:<id>), never counted
in ContextTokens, and never subject to max_history_len trimming. Message.IsSynthetic is the
opposite of IsSlash: write-only in this phase — no daemon code reads it back; it exists so tests
and logs can assert provenance now and drivers can act on it later.
/retry keeps working: after resolveUnresolvedToolCalls() clears a hanging tool call the tail is a
tool_result, which is a valid continuation for every driver, so no marker is synthesized. A /retry
on an interrupted agent turn resumes with the marked marker.
Driver version compatibility: the Phase-1 behavior changes require no minimum hd-driver-openai
version. The OpenAI driver ignores is_synthetic and needs no trailing user turn (a tool message is a
valid terminal message), so removing the old injection after tool round-trips is safe.
hd-driver-claude— do not deploy/roll out yet.hd-driver-claudemust not be deployed against this daemon release until itstool_resultblock-shape fix lands (the deferred follow-up ticket;tool_resultblocks need thetool_use_idcorrelation). The old synthetic trailing user turn was masking that bug on thetool_resultpath: with the injection removed, the driver’s malformedtool_resultblock is now the trailing message and the defect becomes reachable. This is a pre-existing driver bug that the removed workaround happened to hide, not a regression introduced here — the daemon Phase-1 change touches no driver code. Onlyhd-driver-openaiis safe to run today.
Drivers that consume is_synthetic (or honor mode: never via a per-engine require_trailing_user) are
a later phase.
┌──────────────┐ eventbus:agent_start ┌──────────────────┐
│ Workflow │ ──────────────────────► │ Agent Service │
│ / UI │ │ (agent_svc.go) │
└──────┬───────┘ └────────┬─────────┘
│ │
│ eventbus:agent_poll │ agentbus:send_to_model
│ ◄──────── agent_response ─────── │ ──────────────► AI Driver
│ (full_messages, live_message, │
│ state: {history_len, total_tokens}) │ agentbus:receive
│ │ ◄────────────── AI Driver
│ │
│ eventbus:agent_continue (tool result)
│ ◄─────────────────────┤
│ │
│ Tool calls dispatched via: │
│ eventbus:agent_command (→ sys_*) │
│ eventbus:agent_workflow (→ wf__*) │
│ eventbus:agent_call (→ ag__*) │
│ eventbus:mcp_call (→ mcp__*) │
eventbus:agent_start with resume_key. The agent service creates a session and begins inference.agentbus:send_to_model.eventbus:agent_continue.eventbus:agent_poll to receive streaming output.agent_continue back to the workflow, which resumes.Agents can be configured with pre-context files and skill directories that are loaded at the start of a new conversation using the configured file_tool.
Pre-Context (pre_context): Files whose content is appended to the system prompt before the first turn. Useful for providing project-specific context, coding standards, or environment details.
pre_context:
- /workspace/CONTEXT.md
- /workspace/architecture.md
Skills (skills): Directories containing SKILL.md files. At conversation start, all skills are parsed and listed in the system prompt. The model can then use hd_load_skill to load a specific skill’s full content during the conversation.
skills:
- /workspace/skills/
Skills are parsed as YAML frontmatter. Each SKILL.md must have:
---
name: read_file
description: Read the contents of a file from the workspace.
---
# Skill content here...
| Stage | Description |
|---|---|
| Creation | setup() initializes a new session or restores from cache. Acquires distributed lock. Acquiring the lock uses a 600-second expiry. |
| Pre-Context | loadPreContextAndSkills() loads configured files and skills before the first run. |
| Run | run() appends the user message, then sends history + tools to the AI driver via sendToDriver(). |
| Streaming | The driver returns partial chunks. processAgentMessage() accumulates content and thoughts. |
| Tool Call | When the model requests tools, nextToolCall() dispatches to the appropriate handler. |
| Tool Result | processToolResult() collects results. If more tools are pending, dispatches the next. Otherwise, feeds all results back to the model. |
| Polling | processAgentPoll() blocks until new content is available or timeout (default 9s). Workflows can opt into terminal-response polling with a separate completion timeout and cancellation on timeout. |
| Compaction | Triggered when history exceeds threshold. Archives old history, summarizes, and resumes. |
| Completion | Final agent message is emitted. Session state is synced to ConvoState. Lock is released. |
| Caching | persist() serializes the session to the distributed cache. Cache TTL defaults to 72 hours. |
| Cancellation | checkCancelled() is called before processing. Cancelled sessions abort with error. |
If the daemon restarts mid-inference, interrupted sessions are recovered via eventbus:agent_recover:
agentbus:rpc/interrupted with the agent_session_id label.agent_recover message through Redis for durability.recover(), which re-sends the history to the driver.agents:
chat_helper:
driver: openai
engine: gpt-4o-mini
system_prompt: "You are a helpful assistant for the DevOps team."
should_stream: true
tools:
- type: system
name: pagerduty
- type: workflow
name: manage_incident
agents:
long_running_agent:
driver: openai
engine: gpt-4o
system_prompt: "You are an expert code reviewer."
max_history_len: 200
compaction_policy:
strategy: summarize
threshold_type: history_len
threshold: 80
preserve_recent: 10
summarization_agent: fast_summarizer
agents:
fast_summarizer:
driver: openai
engine: gpt-4o-mini
system_prompt: "You are a summarization expert. Create concise summaries."
agents:
github_assistant:
driver: openai
engine: gpt-4o
system_prompt: "You help manage GitHub repositories and issues."
tools:
- type: mcp
name: github_mcp
only:
- create_issue
- list_issues
- create_pull_request
- get_file_contents
agents:
codebase_assistant:
driver: openai
engine: gpt-4o
file_tool: sys__workspace__read
system_prompt: |
You are a codebase assistant. You have access to the project files
and skills to help with development tasks.
pre_context:
- /repo/docs/ARCHITECTURE.md
- /repo/docs/CONTRIBUTING.md
skills:
- /repo/skills/
- /shared/skills/
tools:
- type: system
name: git
- type: system
name: github
- type: workflow
name: run_tests
agents:
orchestrator:
driver: openai
engine: gpt-4o
system_prompt: "You coordinate between specialist agents."
tools:
- type: agent
name: code_reviewer
- type: agent
name: security_scanner
code_reviewer:
driver: openai
engine: gpt-4o-mini
system_prompt: "You are a code reviewer. Review code for quality and best practices."
security_scanner:
driver: openai
engine: gpt-4o-mini
system_prompt: "You are a security expert. Scan code for vulnerabilities."
workflows:
ask_assistant:
steps:
- call_agent:
agent: chat_helper
prompt: "Check the current incident status for team Alpha."
- notify:
channel: slack
message: "Agent response: $result"
rules:
- when:
source:
system: webhook
trigger: alert
do:
call_workflow: ask_assistant