This commit is contained in:
@@ -0,0 +1,193 @@
|
||||
# Agent loop follow-up issues
|
||||
|
||||
## Issue 1: Fix dangling control tool calls across movement transitions
|
||||
|
||||
### Problem
|
||||
|
||||
`transition` and `complete` tool calls can be appended to the live conversation
|
||||
without a matching tool result message. When the same `Conversation` is reused by
|
||||
the next movement, strict OpenAI-compatible providers may reject the next request
|
||||
because every assistant `tool_call` must have a corresponding `tool` message.
|
||||
|
||||
### Evidence
|
||||
|
||||
- `src/engine/agent-loop.ts`: assistant messages are recorded with all pending
|
||||
tool calls, including control calls.
|
||||
- `src/engine/agent-loop/terminal-control.ts`: valid `transition` returns a
|
||||
`MovementResult` without a tool result message.
|
||||
- `src/engine/agent-loop.ts`: valid `complete` returns immediately without a
|
||||
tool result message.
|
||||
- `src/engine/context/conversation.ts`: `replayableTurns()` sanitizes persisted
|
||||
transcript replay, but live `conversation.messages` shared across movements is
|
||||
not sanitized.
|
||||
|
||||
### Expected behavior
|
||||
|
||||
After a movement exits via `transition` or `complete`, the next model request
|
||||
must not contain unresolved control `tool_call`s.
|
||||
|
||||
### Suggested fix
|
||||
|
||||
Normalize live conversation state before entering the next movement, or avoid
|
||||
recording terminal/control tool calls without corresponding tool messages.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- Add a regression test that uses a shared `Conversation` across a real
|
||||
`transition`.
|
||||
- The second movement's model input contains no dangling `transition` or
|
||||
`complete` tool calls.
|
||||
- Existing agent-loop tests continue to pass.
|
||||
|
||||
---
|
||||
|
||||
## Issue 2: Make context overflow terminal defaults consistently abort
|
||||
|
||||
### Problem
|
||||
|
||||
Context overflow handling has two different policies:
|
||||
|
||||
- `buildContextOverflowResult()` converts terminal defaults (`COMPLETE`, `ASK`)
|
||||
to `ABORT`.
|
||||
- `applyContextManagerUpdate()` with `force_transition` uses
|
||||
`movement.defaultNext ?? 'ABORT'` directly.
|
||||
|
||||
This means context pressure can falsely complete or ask from a movement whose
|
||||
default next step is terminal, even though the safer policy is to abort on
|
||||
context loss.
|
||||
|
||||
### Evidence
|
||||
|
||||
- `src/engine/agent-loop/context-control.ts`: `buildContextOverflowResult()`
|
||||
normalizes `COMPLETE` and `ASK` to `ABORT`.
|
||||
- `src/engine/agent-loop/context-control.ts`: `applyContextManagerUpdate()`
|
||||
force transition uses `movement.defaultNext` directly.
|
||||
|
||||
### Expected behavior
|
||||
|
||||
All context-overflow forced exits should use the same terminal-default policy.
|
||||
When context is compromised, terminal defaults should not be treated as
|
||||
successful completion.
|
||||
|
||||
### Suggested fix
|
||||
|
||||
Share one helper for context-overflow movement results, or apply the same
|
||||
terminal-default normalization in `applyContextManagerUpdate()`.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- Add a regression test for `force_transition` with `defaultNext: "COMPLETE"`.
|
||||
- The result is an abort-style movement result, not a successful completion.
|
||||
- Existing context overflow tests continue to pass.
|
||||
|
||||
---
|
||||
|
||||
## Issue 3: Enforce `why_no_default` for `needs_user_input`
|
||||
|
||||
### Problem
|
||||
|
||||
The prompt and tool description require `why_no_default` when calling
|
||||
`complete({ status: "needs_user_input" })`, but runtime validation only requires
|
||||
`missing_info`.
|
||||
|
||||
This lets the agent ask the user without documenting why it could not choose a
|
||||
reasonable default, which weakens the intended "avoid unnecessary user
|
||||
questions" behavior.
|
||||
|
||||
### Evidence
|
||||
|
||||
- `src/engine/agent-loop/prompt.ts`: instructs the model to provide
|
||||
`why_no_default`.
|
||||
- `src/engine/agent-loop/terminal-control.ts`: tool description says
|
||||
`why_no_default` is required.
|
||||
- `src/engine/agent-loop/terminal-control.ts`: `validateCompleteArgs()` only
|
||||
checks `missing_info`.
|
||||
- `src/engine/agent-loop/terminal-control.ts`: movement output ignores
|
||||
`why_no_default`.
|
||||
|
||||
### Expected behavior
|
||||
|
||||
`needs_user_input` should be rejected unless both `missing_info` and
|
||||
`why_no_default` are non-empty strings.
|
||||
|
||||
### Suggested fix
|
||||
|
||||
Validate `why_no_default` in `validateCompleteArgs()` and include it in the
|
||||
result/debug output if useful.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- Add a negative test for missing `why_no_default`.
|
||||
- Add a positive test for `needs_user_input` with both fields.
|
||||
- Existing `complete` behavior remains unchanged for other statuses.
|
||||
|
||||
---
|
||||
|
||||
## Issue 4: Validate interactive browse waiting-human session id
|
||||
|
||||
### Problem
|
||||
|
||||
`parseInteractiveBrowseWaitingHuman()` casts `sessionId` to `string` without
|
||||
checking that it is present and actually a string.
|
||||
|
||||
If the tool returns malformed `waiting_human` output, the movement result can
|
||||
carry `browserSessionId: undefined` despite the type expecting a string.
|
||||
|
||||
### Evidence
|
||||
|
||||
- `src/engine/agent-loop/tool-dispatcher.ts`: `sessionId` is read via
|
||||
`parsed["sessionId"] as string`.
|
||||
- The function validates `action` and `waitReason`, but not `sessionId`.
|
||||
|
||||
### Expected behavior
|
||||
|
||||
Malformed waiting-human output should not produce a typed waiting-human movement
|
||||
result with an undefined session id.
|
||||
|
||||
### Suggested fix
|
||||
|
||||
Require `typeof parsed["sessionId"] === "string"` and a non-empty value before
|
||||
returning a waiting-human result.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- Add a test for malformed `waiting_human` output without `sessionId`.
|
||||
- Add a test for valid `waiting_human` output.
|
||||
- Existing interactive browse behavior remains unchanged for valid tool output.
|
||||
|
||||
---
|
||||
|
||||
## Issue 5: Split `tool-dispatcher.ts` before it becomes the next agent-loop
|
||||
|
||||
### Problem
|
||||
|
||||
`tool-dispatcher.ts` now owns several distinct responsibilities:
|
||||
|
||||
- cache routing
|
||||
- cache hit/miss event logging
|
||||
- tool execution
|
||||
- memory checkpoint behavior
|
||||
- waiting-human parsing
|
||||
- batched result recording
|
||||
|
||||
This file is becoming the next high-coupling module after the agent-loop split.
|
||||
|
||||
### Expected behavior
|
||||
|
||||
Tool dispatch should be decomposed into focused modules so bugs in cache
|
||||
routing, execution, and special tool outputs are easier to test independently.
|
||||
|
||||
### Suggested fix
|
||||
|
||||
Split along behavioral boundaries, for example:
|
||||
|
||||
- `tool-cache-routing.ts`
|
||||
- `tool-execution.ts`
|
||||
- `tool-result-recorder.ts`
|
||||
- `interactive-browse-result.ts`
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- No behavior change.
|
||||
- Existing agent-loop and tool-loop tests pass.
|
||||
- New modules expose narrow, testable functions.
|
||||
@@ -0,0 +1,324 @@
|
||||
# Improve task memory and conversation recall
|
||||
|
||||
## Summary
|
||||
|
||||
Long-running local tasks can lose early user instructions, constraints, and
|
||||
decisions after several exchanges. The codebase already has partial mechanisms:
|
||||
|
||||
- `MissionUpdate` / Mission Brief pins `goal`, `done`, `open`, and
|
||||
`clarifications`.
|
||||
- `Conversation` persists `logs/transcript.jsonl` and can replay prior turns on
|
||||
continuation.
|
||||
- `buildLocalConversationContext()` injects recent task comments.
|
||||
- `handoffContext` carries the previous piece result.
|
||||
|
||||
However, these do not give the agent a reliable way to preserve task state or
|
||||
actively search older conversation history.
|
||||
|
||||
This issue proposes three related improvements:
|
||||
|
||||
1. Make Mission Brief a stronger task-state ledger.
|
||||
2. Add agent-facing conversation search/read tools.
|
||||
3. Update prompts so the agent proactively revisits prior conversation when
|
||||
task context is ambiguous or long-running.
|
||||
|
||||
## Problem
|
||||
|
||||
In multi-turn tasks, the agent often appears to forget early conversation
|
||||
details, especially:
|
||||
|
||||
- original user constraints
|
||||
- decisions made after clarification
|
||||
- "do not change X" instructions
|
||||
- previous failed attempts
|
||||
- which files were already inspected or changed
|
||||
- why a plan was chosen
|
||||
|
||||
Current behavior is understandable from the implementation:
|
||||
|
||||
- `MissionUpdate` is optional and model-driven; it is not enforced at movement
|
||||
boundaries.
|
||||
- Mission Brief fields are too coarse for long implementation tasks.
|
||||
- local task context only injects the recent conversation window.
|
||||
- transcript replay is automatic context carry-over, not searchable retrieval.
|
||||
- prompt compaction/summarization can remove or compress important details.
|
||||
|
||||
## Existing Mechanisms To Reuse
|
||||
|
||||
### Mission Brief
|
||||
|
||||
Source: `src/engine/tools/mission.ts`
|
||||
|
||||
Mission Brief is the best existing place for pinned task state because it is:
|
||||
|
||||
- per local task
|
||||
- visible at the top of every movement prompt
|
||||
- editable by the agent through `MissionUpdate`
|
||||
- editable by the user in the UI
|
||||
- intended to prevent long-conversation drift
|
||||
|
||||
This should be extended rather than replaced.
|
||||
|
||||
### Conversation transcript
|
||||
|
||||
Source: `src/engine/context/conversation.ts`
|
||||
|
||||
`logs/transcript.jsonl` is already written during execution and replayed on
|
||||
continuation. It should remain the raw audit source. New search tools should
|
||||
read from it instead of stuffing the whole transcript into prompt context.
|
||||
|
||||
### Local task comments
|
||||
|
||||
Source: `src/engine/local-context.ts`
|
||||
|
||||
Task comments are the user-facing conversation layer. They should be searchable
|
||||
alongside transcript entries because user instructions often live in comments,
|
||||
requests, and interjections.
|
||||
|
||||
## Proposed Design
|
||||
|
||||
### 1. Extend Mission Brief schema
|
||||
|
||||
Add fields that separate task facts from progress:
|
||||
|
||||
```ts
|
||||
interface MissionBrief {
|
||||
goal?: string;
|
||||
user_constraints?: string;
|
||||
decisions?: string;
|
||||
done?: string;
|
||||
open?: string;
|
||||
current_focus?: string;
|
||||
touched_files?: string;
|
||||
risks?: string;
|
||||
clarifications?: string;
|
||||
last_updated_by_movement?: string;
|
||||
}
|
||||
```
|
||||
|
||||
Keep backwards compatibility with existing `goal`, `done`, `open`, and
|
||||
`clarifications`.
|
||||
|
||||
### 2. Make movement-boundary updates explicit
|
||||
|
||||
At the end of each movement, encourage or enforce a Mission Brief update when
|
||||
the movement made meaningful progress.
|
||||
|
||||
Suggested behavior:
|
||||
|
||||
- On movement start, render the current Mission Brief as today.
|
||||
- During movement, the model can still call `MissionUpdate`.
|
||||
- Before `transition` or `complete`, prompt guidance should require the agent to
|
||||
update `done`, `open`, `current_focus`, and relevant `decisions` if stale.
|
||||
- Optionally add a lightweight stale-check:
|
||||
- if a movement used tools or edited files
|
||||
- and no `MissionUpdate` occurred
|
||||
- inject a reminder before accepting terminal control calls
|
||||
|
||||
Do not make this a hard blocker initially; use a reminder first to avoid
|
||||
creating loops.
|
||||
|
||||
### 3. Add `SearchTaskConversation`
|
||||
|
||||
Add a META tool available to all local-task pieces.
|
||||
|
||||
Suggested signature:
|
||||
|
||||
```ts
|
||||
SearchTaskConversation({
|
||||
query: string;
|
||||
source?: "comments" | "transcript" | "both";
|
||||
author?: "user" | "agent" | "system";
|
||||
kind?: "request" | "comment" | "interjection" | "result" | "ask" | "progress" | "handoff";
|
||||
limit?: number;
|
||||
})
|
||||
```
|
||||
|
||||
Search sources:
|
||||
|
||||
- DB local task comments
|
||||
- `runtimeDir/transcript.jsonl` when available
|
||||
|
||||
Return compact excerpts only:
|
||||
|
||||
```md
|
||||
## Conversation Search Results
|
||||
|
||||
- comment:123 user/comment 2026-06-30T...
|
||||
excerpt: ...
|
||||
|
||||
- transcript:42 user 2026-06-30T...
|
||||
excerpt: ...
|
||||
```
|
||||
|
||||
Constraints:
|
||||
|
||||
- cap result count
|
||||
- cap excerpt length
|
||||
- never return full transcript by default
|
||||
- search only the current task's conversation
|
||||
- respect existing task/space authorization boundaries
|
||||
|
||||
### 4. Add `ReadTaskConversation`
|
||||
|
||||
Add a companion tool to read context around a known hit.
|
||||
|
||||
Suggested signature:
|
||||
|
||||
```ts
|
||||
ReadTaskConversation({
|
||||
ref: "comment:123" | "transcript:42";
|
||||
before?: number;
|
||||
after?: number;
|
||||
})
|
||||
```
|
||||
|
||||
Return nearby entries with strict caps. This keeps search cheap and lets the
|
||||
agent inspect the exact conversation around an old decision.
|
||||
|
||||
### 5. Wire conversation retrieval into Mission Brief refresh
|
||||
|
||||
Update prompt guidance for long tasks:
|
||||
|
||||
- If user constraints or decisions are unclear, use `SearchTaskConversation`
|
||||
before asking the user.
|
||||
- When updating `user_constraints` or `decisions`, prefer citing old comments or
|
||||
transcript refs in the brief.
|
||||
- Before asking the user a clarification, search the task conversation when the
|
||||
missing information may already have been stated earlier.
|
||||
- Before changing previously touched files or revising a prior decision, search
|
||||
for older constraints/decisions related to the file or topic.
|
||||
|
||||
Example Mission Brief snippet:
|
||||
|
||||
```md
|
||||
### User constraints
|
||||
- Keep existing auth flow unchanged. Source: comment:17
|
||||
- Do not rewrite unrelated UI. Source: transcript:42
|
||||
|
||||
### Decisions
|
||||
- Extend Mission Brief instead of creating a second TaskState store. Source: comment:23
|
||||
```
|
||||
|
||||
### 6. Add proactive recall guidance to system/movement prompts
|
||||
|
||||
The new tools should not be passive. Update prompt guidance so the agent
|
||||
actively uses them when forgetting early conversation is likely.
|
||||
|
||||
Suggested rules:
|
||||
|
||||
- At the start of a follow-up task, review Mission Brief first.
|
||||
- If Mission Brief is missing, stale, or too vague, call
|
||||
`SearchTaskConversation` before proceeding with assumptions.
|
||||
- If the task has multiple user turns or interjections, search conversation
|
||||
history for constraints before making broad edits.
|
||||
- If the agent is about to ask the user something, first search prior comments
|
||||
and transcript for the answer unless the question is truly new.
|
||||
- If a movement resumes after context compaction, search for relevant older
|
||||
decisions before modifying files.
|
||||
- When search results reveal durable constraints or decisions, update
|
||||
Mission Brief immediately.
|
||||
|
||||
Candidate prompt copy:
|
||||
|
||||
```md
|
||||
## Conversation recall
|
||||
For long-running or follow-up tasks, do not rely only on the visible recent
|
||||
context. If an earlier user constraint, decision, or clarification may affect
|
||||
your next action, use SearchTaskConversation / ReadTaskConversation before
|
||||
proceeding. Prefer retrieving prior context over asking the user to repeat it.
|
||||
When you find durable task facts, update MissionUpdate so they remain pinned.
|
||||
```
|
||||
|
||||
Keep this guidance concise so it does not crowd out movement-specific
|
||||
instructions.
|
||||
|
||||
## Files Likely Involved
|
||||
|
||||
- `src/db/repository.ts`
|
||||
- extend `MissionBrief`
|
||||
- update parse/update tests
|
||||
- add query helpers for task comments if needed
|
||||
|
||||
- `src/engine/tools/mission.ts`
|
||||
- extend `MissionUpdate` schema
|
||||
- clamp/render new fields
|
||||
- tests for partial updates
|
||||
|
||||
- `src/engine/agent-loop/prompt.ts`
|
||||
- render new Mission Brief fields
|
||||
- update movement guidance around state refresh
|
||||
- add concise proactive recall guidance for long/follow-up tasks
|
||||
|
||||
- `src/engine/agent-loop/watchdogs.ts`
|
||||
- optionally add stale Mission Brief reminder logic
|
||||
- optionally remind the agent to search conversation history when Mission Brief
|
||||
is empty/stale during later iterations
|
||||
|
||||
- `src/engine/tools/index.ts`
|
||||
- add new conversation tools to META_TOOLS
|
||||
|
||||
- `src/engine/tools/conversation.ts` or `src/engine/tools/task-conversation.ts`
|
||||
- implement `SearchTaskConversation`
|
||||
- implement `ReadTaskConversation`
|
||||
|
||||
- `src/engine/tools/core.ts`
|
||||
- expose safe context fields needed by conversation tools, e.g. taskId,
|
||||
runtimeDir, workspacePath, repo access if needed
|
||||
|
||||
- `src/worker.ts`
|
||||
- ensure ToolContext has enough current-task context for the new tools
|
||||
|
||||
- `ui/src/components/detail/tabs/OverviewTab.tsx`
|
||||
- expose new Mission Brief fields for user editing
|
||||
|
||||
- `ui/src/content/help/*.md`
|
||||
- update help docs if behavior/UI changes
|
||||
|
||||
- `ui/src/content/help/00-changelog.md`
|
||||
- add a user-facing changelog entry
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Mission Brief supports new fields while preserving existing data.
|
||||
- Existing tasks with old Mission Brief JSON still load correctly.
|
||||
- `MissionUpdate` can update new fields independently.
|
||||
- Mission Brief rendering keeps a bounded prompt size.
|
||||
- Movement guidance encourages updating Mission Brief before transitions.
|
||||
- Prompt guidance tells the agent to search prior conversation before asking the
|
||||
user to repeat information or making assumptions in long/follow-up tasks.
|
||||
- Prompt guidance tells the agent to search prior conversation before asking the
|
||||
user to repeat information or making assumptions in long/follow-up tasks.
|
||||
- `SearchTaskConversation` can find old user comments by keyword.
|
||||
- `SearchTaskConversation` can find transcript entries when `transcript.jsonl`
|
||||
exists.
|
||||
- `ReadTaskConversation` can return bounded context around a search result.
|
||||
- Tools are scoped to the current task and cannot read other task logs.
|
||||
- Tests cover:
|
||||
- old Mission Brief compatibility
|
||||
- new Mission Brief fields
|
||||
- conversation search over comments
|
||||
- conversation search over transcript
|
||||
- prompt text includes proactive recall guidance
|
||||
- bounded output and excerpt truncation
|
||||
- authorization/task scoping
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Do not put task-specific state into `UpdateUserMemory`; that is user-wide
|
||||
memory and would pollute future unrelated tasks.
|
||||
- Do not inject the full transcript into every prompt.
|
||||
- Do not replace `Conversation` replay; retrieval should complement it.
|
||||
- Do not make Mission Brief update a hard terminal blocker until reminder-only
|
||||
behavior has been observed.
|
||||
|
||||
## Suggested Implementation Order
|
||||
|
||||
1. Extend Mission Brief schema and UI rendering.
|
||||
2. Extend `MissionUpdate` and prompt rendering.
|
||||
3. Add reminder-only movement-boundary guidance.
|
||||
4. Implement `SearchTaskConversation` for DB comments.
|
||||
5. Add transcript search.
|
||||
6. Add `ReadTaskConversation`.
|
||||
7. Add proactive recall prompt guidance and tests.
|
||||
8. Update help docs and changelog.
|
||||
@@ -0,0 +1,229 @@
|
||||
# Add workspace file provenance so agents can identify which task created or owns files
|
||||
|
||||
## Summary
|
||||
|
||||
Persistent workspaces make it harder to tell which task created, uploaded, or
|
||||
modified a file. This is especially confusing when multiple tasks share the same
|
||||
workspace tree. The agent may see a file and assume it belongs to the current
|
||||
task, even though it was an input or output from a different task.
|
||||
|
||||
Add a file provenance ledger so both users and agents can answer:
|
||||
|
||||
- Which task created this file?
|
||||
- Was this uploaded by the user or generated by an agent?
|
||||
- Which task last modified it?
|
||||
- Is it safe for the current task to edit, or should it be treated as read-only
|
||||
context?
|
||||
|
||||
## Problem
|
||||
|
||||
In persistent workspace mode, files can outlive a single task. That is useful,
|
||||
but it creates ambiguity:
|
||||
|
||||
- old input files remain visible to later tasks
|
||||
- previous task outputs look like current task artifacts
|
||||
- generated files and uploaded files are not clearly distinguished
|
||||
- agents may edit or rely on files that belong to unrelated tasks
|
||||
- users cannot easily audit why a file exists
|
||||
|
||||
Current workspace/runtime separation helps logs:
|
||||
|
||||
- `workspace_path` points to the shared files tree
|
||||
- `runtime_dir` separates per-task logs/checklists/raw outputs
|
||||
|
||||
But shared workspace files themselves do not have provenance metadata.
|
||||
|
||||
## Proposed Design
|
||||
|
||||
Add a sidecar provenance ledger for files in persistent workspaces.
|
||||
|
||||
Prefer a DB-backed table for queryability, with optional JSONL export for
|
||||
debugging. Do not write task tags into file contents.
|
||||
|
||||
### Schema
|
||||
|
||||
Suggested table:
|
||||
|
||||
```sql
|
||||
CREATE TABLE workspace_file_provenance (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
space_id TEXT,
|
||||
workspace_path TEXT NOT NULL,
|
||||
rel_path TEXT NOT NULL,
|
||||
created_by_task_id INTEGER,
|
||||
created_by_job_id TEXT,
|
||||
created_by_piece TEXT,
|
||||
created_by_movement TEXT,
|
||||
source_kind TEXT NOT NULL,
|
||||
first_seen_at TEXT NOT NULL,
|
||||
last_modified_by_task_id INTEGER,
|
||||
last_modified_by_job_id TEXT,
|
||||
last_modified_at TEXT,
|
||||
checksum TEXT,
|
||||
note TEXT,
|
||||
UNIQUE(workspace_path, rel_path)
|
||||
);
|
||||
```
|
||||
|
||||
Suggested `source_kind` values:
|
||||
|
||||
- `user_input`
|
||||
- `agent_output`
|
||||
- `agent_edit`
|
||||
- `bash_generated`
|
||||
- `subtask_output`
|
||||
- `imported_existing`
|
||||
- `unknown`
|
||||
|
||||
### Recording Rules
|
||||
|
||||
Record provenance at these boundaries:
|
||||
|
||||
- UI upload to `input/`
|
||||
- `source_kind=user_input`
|
||||
- `created_by_task_id=current task`
|
||||
|
||||
- `Write`
|
||||
- new file: `source_kind=agent_output`
|
||||
- existing file: update `last_modified_by_task_id`
|
||||
|
||||
- `Edit`
|
||||
- update `last_modified_by_task_id`
|
||||
- keep original `created_by_task_id`
|
||||
|
||||
- `Bash`
|
||||
- compare workspace file snapshot before/after command
|
||||
- new files: `source_kind=bash_generated`
|
||||
- changed files: update `last_modified_by_task_id`
|
||||
|
||||
- subtask outputs copied/visible to parent
|
||||
- `source_kind=subtask_output`
|
||||
- include parent task and subtask job id
|
||||
|
||||
- existing files in a persistent space before this feature ships
|
||||
- backfill as `source_kind=imported_existing` or `unknown`
|
||||
|
||||
### Agent-Facing Tools
|
||||
|
||||
Add META tools:
|
||||
|
||||
```ts
|
||||
GetFileProvenance({ path: string })
|
||||
```
|
||||
|
||||
Returns a compact provenance record for one file.
|
||||
|
||||
```ts
|
||||
ListWorkspaceFiles({
|
||||
path?: string;
|
||||
sourceKind?: string;
|
||||
createdByTaskId?: number;
|
||||
lastModifiedByTaskId?: number;
|
||||
includeUnknown?: boolean;
|
||||
})
|
||||
```
|
||||
|
||||
Returns bounded file listings with provenance summaries.
|
||||
|
||||
### Prompt Guidance
|
||||
|
||||
Update the system prompt / workspace guidance so the agent uses provenance
|
||||
before editing ambiguous files.
|
||||
|
||||
Candidate prompt copy:
|
||||
|
||||
```md
|
||||
## Workspace file provenance
|
||||
Persistent workspaces may contain files from older tasks. Before editing a file
|
||||
whose provenance shows `source_kind=user_input` or `created_by_task_id` differs
|
||||
from the current task, verify that it is relevant. Prefer creating a new output
|
||||
file when unsure. Use GetFileProvenance / ListWorkspaceFiles to inspect file
|
||||
origin.
|
||||
```
|
||||
|
||||
### UI
|
||||
|
||||
Show provenance in the file browser / preview:
|
||||
|
||||
- created by task id/title
|
||||
- source kind
|
||||
- last modified by task id/title
|
||||
- timestamp
|
||||
|
||||
This can be a small metadata row or tooltip; avoid cluttering the file list.
|
||||
|
||||
## Files Likely Involved
|
||||
|
||||
- `src/db/schema.sql`
|
||||
- add provenance table
|
||||
|
||||
- `src/db/migrate.ts`
|
||||
- migration for existing installations
|
||||
|
||||
- `src/db/repository.ts`
|
||||
- read/write/query provenance records
|
||||
- optional backfill helpers
|
||||
|
||||
- `src/engine/tools/core.ts`
|
||||
- hook `Write`, `Edit`, and `Bash` file changes
|
||||
- expose task/job/piece/movement context needed for provenance
|
||||
|
||||
- `src/engine/piece-runner.ts`
|
||||
- thread task id, job id, piece, movement into `ToolContext`
|
||||
|
||||
- `src/engine/tools/index.ts`
|
||||
- register provenance tools as META tools
|
||||
|
||||
- `src/engine/tools/file-provenance.ts`
|
||||
- implement `GetFileProvenance`
|
||||
- implement `ListWorkspaceFiles`
|
||||
|
||||
- `src/bridge/local-tasks-api.ts` / file APIs
|
||||
- record UI uploads
|
||||
- return provenance metadata with file listing/detail responses
|
||||
|
||||
- `ui/src/components/files/*`
|
||||
- display provenance metadata
|
||||
|
||||
- `src/engine/agent-loop/prompt.ts`
|
||||
- add concise workspace provenance guidance
|
||||
|
||||
- `ui/src/content/help/*.md`
|
||||
- document how provenance labels work
|
||||
|
||||
- `ui/src/content/help/00-changelog.md`
|
||||
- add user-facing changelog entry
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- New files created by `Write` record current task provenance.
|
||||
- Files changed by `Edit` preserve original creator and update last modifier.
|
||||
- Files created or changed by `Bash` are detected and recorded.
|
||||
- User-uploaded input files are recorded as `user_input`.
|
||||
- File listing or preview can show provenance metadata.
|
||||
- Agent can call `GetFileProvenance` for a file.
|
||||
- Agent can list files filtered by provenance.
|
||||
- Prompt guidance tells the agent to avoid editing unrelated/user-input files
|
||||
without checking relevance.
|
||||
- Existing persistent workspace files remain usable after migration.
|
||||
- Provenance output is bounded and does not dump large file contents.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Do not embed task tags into file contents.
|
||||
- Do not make files from other tasks globally read-only; the agent may still
|
||||
need to use shared workspace artifacts.
|
||||
- Do not block all edits to `user_input` files immediately. Start with warnings
|
||||
and guidance to avoid breaking existing workflows.
|
||||
- Do not use `UpdateUserMemory` for per-file task metadata.
|
||||
|
||||
## Suggested Implementation Order
|
||||
|
||||
1. Add DB schema and repository helpers.
|
||||
2. Record UI uploads and `Write`/`Edit` provenance.
|
||||
3. Add `Bash` before/after file change detection.
|
||||
4. Implement `GetFileProvenance`.
|
||||
5. Implement `ListWorkspaceFiles` with filters and caps.
|
||||
6. Add prompt guidance.
|
||||
7. Surface metadata in the file browser/preview.
|
||||
8. Update help docs and changelog.
|
||||
@@ -0,0 +1,114 @@
|
||||
# Add segmented screenshots to BrowseWeb
|
||||
|
||||
> **Status: Implemented (2026-07-03).** 設計との主な差分は下記「Implementation notes」を参照。
|
||||
|
||||
## Implementation notes (設計との差分)
|
||||
|
||||
- **既定を分割に変更**: 当初案は `screenshotSegments: true` のオプトインだったが、実際のユーザー意図は「スクショは既定で 1 画面ぶんずつ区切る」だったため、**BrowseWeb では分割を既定挙動**にした。フルページ 1 枚が欲しい場合は `screenshotSegments: false`(基本モード)/ アクションの `segments: false` でオプトアウトする。
|
||||
- **短ページは連番なし**: 1 画面に収まるページは連番を付けず `output/<name>.png` の 1 枚のまま(後方互換)。2 画面ぶん以上のときだけ `-001` / `-002` の連番になる。
|
||||
- **分割単位・撮影方式**: ビューポート高さ(1 画面ぶん)を単位に、`fullPage: true` + `clip` でページを縦に切り出す。スクロールしないため、(1) `position:fixed`/`sticky` なヘッダーが各セグメント先頭に重複して本文を隠さない、(2) 撮影後にページのスクロール位置を変えないので後続アクション(ref/selector クリック等)に副作用が出ない。最後のセグメントは残り高さぶんだけ切り出し、下端に余白を作らない。
|
||||
- **枚数の絶対上限**: `maxSegments` はユーザー(LLM)指定でも `ABSOLUTE_MAX_SEGMENTS`(50)を超えない。巨大な `scrollHeight` を返すページや極端な指定値による撮影ループの暴走(worker 時間・ディスク枯渇)を防ぐ最終防波堤。非有限値(NaN/Infinity)は既定値にフォールバック。
|
||||
- **TestWorkspaceApp は現状維持**: 共通ハンドラ(`runPageActions`)の既定を分割にしたため、`TestWorkspaceApp` の screenshot アクションには `segments: false` を明示して単一フルページ撮影を維持(本 issue の scope 指示どおり)。
|
||||
- **打ち切り通知**: `maxSegments`(既定 10)で打ち切った場合は戻り値にその旨を出力(無音打ち切りにしない)。
|
||||
- 実装: `src/engine/tools/browser.ts`(`planScreenshotSegments` / `segmentFilename` / `captureScreenshots`)、テスト: `browser.screenshot-segments.test.ts`(純関数)・`browser.screenshot-capture.test.ts`(実ブラウザ、`CONTAINER=1` ゲート)。
|
||||
|
||||
## Summary
|
||||
|
||||
Long HTML pages are hard for agents to verify from a single screenshot. A full-page
|
||||
capture can become extremely tall, causing image understanding to miss details or
|
||||
compress important UI states. Viewport-only screenshots avoid giant images, but they
|
||||
only show the first visible area.
|
||||
|
||||
Add a segmented screenshot mode to `BrowseWeb` so agents can capture a page as a
|
||||
sequence of viewport-sized images.
|
||||
|
||||
## Problem
|
||||
|
||||
For generated HTML verification and UI review, agents often need to inspect the
|
||||
whole rendered page. Current screenshot modes are not ideal:
|
||||
|
||||
- viewport screenshot: readable, but only captures one screen
|
||||
- full-page screenshot: complete, but can be too tall and information-dense
|
||||
|
||||
This makes visual verification unreliable for long reports, dashboards, and other
|
||||
HTML outputs.
|
||||
|
||||
## Proposed Design
|
||||
|
||||
Keep existing screenshot behavior and add an explicit segmented mode.
|
||||
|
||||
### Basic BrowseWeb mode
|
||||
|
||||
```js
|
||||
BrowseWeb({
|
||||
url: "output/report.html",
|
||||
screenshot: "report.png",
|
||||
screenshotSegments: true
|
||||
})
|
||||
```
|
||||
|
||||
Expected output files:
|
||||
|
||||
```text
|
||||
output/report-001.png
|
||||
output/report-002.png
|
||||
output/report-003.png
|
||||
```
|
||||
|
||||
### Action mode
|
||||
|
||||
```js
|
||||
BrowseWeb({
|
||||
actions: [
|
||||
{ type: "goto", url: "output/report.html" },
|
||||
{ type: "screenshot", value: "report.png", segments: true, maxSegments: 8 }
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
### Behavior
|
||||
|
||||
- Scroll the page from top to bottom and capture one viewport-sized screenshot
|
||||
per segment.
|
||||
- Use stable generated names based on the requested screenshot filename:
|
||||
`name-001.ext`, `name-002.ext`, etc.
|
||||
- Return the saved file list in the tool output.
|
||||
- Default to a bounded number of segments, for example `maxSegments: 10`, to
|
||||
avoid runaway captures on infinite-scroll pages.
|
||||
- Allow callers to override the cap with `maxSegments`.
|
||||
- Preserve existing modes:
|
||||
- default screenshot remains a single viewport image
|
||||
- full-page screenshot remains available via the explicit full-page option
|
||||
- Do not change `TestWorkspaceApp` iframe screenshots unless a separate need is
|
||||
identified.
|
||||
|
||||
## Files Likely Involved
|
||||
|
||||
- `src/engine/tools/browser.ts`
|
||||
- extend `BrowseWebAction` with `segments?: boolean` and `maxSegments?: number`
|
||||
- add basic-mode inputs `screenshotSegments` and `screenshotMaxSegments`
|
||||
- implement shared segmented capture helper
|
||||
|
||||
- `src/engine/tools/browser.runpageactions.test.ts`
|
||||
- add regression coverage for segmented screenshots on a tall local HTML page
|
||||
- assert filenames and image heights/counts
|
||||
|
||||
- `docs/tools/browseweb.md`
|
||||
- document segmented screenshot usage
|
||||
|
||||
- `ui/src/content/help/00-changelog.md`
|
||||
- add a user-facing changelog entry when implemented
|
||||
|
||||
- `ui/src/content/help/16-tools.md`
|
||||
- mention that browser screenshots can be captured as viewport, full-page, or
|
||||
segmented images
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- `BrowseWeb({ url, screenshot, screenshotSegments: true })` saves multiple
|
||||
viewport-sized screenshots for a tall page.
|
||||
- Action-mode `screenshot` supports `segments: true`.
|
||||
- The tool output lists all saved segment paths.
|
||||
- Segment count is capped by default and configurable.
|
||||
- Existing viewport and full-page screenshot behavior continues to work.
|
||||
- Tests cover viewport, full-page, and segmented capture behavior.
|
||||
Reference in New Issue
Block a user