Hierarchical Task Decomposition#
Overview#
Hierarchical Task Decomposition is Marcus’s system for intelligently breaking down large, complex tasks into smaller, manageable subtasks that can be worked on autonomously. This system enables better parallelization, clearer interfaces, and improved coordination while maintaining Marcus’s core values of guided autonomy and context-rich task management.
The Problem#
In real-world software development, tasks vary dramatically in size and complexity. A task like “Build user management system” might be appropriate for a project plan, but it’s too large and ambiguous for a single agent to work on effectively. Breaking it down manually is time-consuming and requires deep technical understanding.
Challenges with Large Tasks#
Ambiguity: Large tasks contain multiple components that need clear interfaces
Task: "Build authentication system"
Questions: What about password reset? 2FA? Social login? Session management?
Integration Risk: Multiple components need to work together seamlessly
Problem: Agent 1 builds API with JWT, Agent 2 expects sessions
Result: Integration failure, rework required
Coordination Overhead: Agents need to understand how their work fits together
Without decomposition: "Build auth" → unclear scope, duplicate work, integration issues
With decomposition: "Create User model" → "Build login API" → "Add session management"
Progress Tracking: Hard to measure completion on monolithic tasks
Task: "Build authentication" - 50% complete
What does 50% mean? Which parts are done? What's left?
The Solution#
Marcus uses AI-powered hierarchical task decomposition to automatically break large tasks into well-defined subtasks with clear interfaces, dependencies, and shared conventions.
Key Principles#
1. Intelligent Decomposition
Marcus uses heuristics and AI analysis to decide which tasks benefit from decomposition:
✅ Decompose when:
Task estimated > 4 hours
Multiple distinct components (API + DB + UI)
Clear sequential phases
Epic-level features
❌ Keep intact when:
Bug fixes (focused scope)
Small enhancements (< 3 hours)
Refactoring tasks (atomic changes)
Deployment tasks (coordinated execution)
2. Clear Interfaces
Every subtask defines what it provides and what it requires:
Subtask("Create User model")
provides: "User model with email validation, password hashing"
requires: None # Foundation task
file_artifacts: ["src/models/user.py"]
Subtask("Build login endpoint")
provides: "POST /api/login returning {token, user}"
requires: "User model from subtask 1"
file_artifacts: ["src/api/auth/login.py"]
Subtask("Add session management")
provides: "Session tracking and refresh tokens"
requires: "Login endpoint from subtask 2"
file_artifacts: ["src/api/auth/sessions.py"]
3. Shared Conventions
All subtasks in a decomposition share common patterns to avoid integration issues:
shared_conventions = {
"base_path": "src/api/auth/",
"response_format": {
"success": {"status": "ok", "data": {...}},
"error": {"status": "error", "message": "..."}
},
"authentication": "JWT tokens in Authorization header",
"naming_convention": "snake_case for endpoints, camelCase for JSON"
}
4. Automatic Integration Subtask
Every decomposition automatically includes a final integration subtask that:
Verifies all components work together
Runs end-to-end tests
Creates consolidated documentation
Validates all promised interfaces
5. Parent Auto-Completion
When all subtasks complete, the parent task automatically:
Moves to “Done” status
Updates progress to 100%
Adds completion comment with subtask summary
Triggers dependent tasks
Architecture#
System Components#
┌─────────────────────────────────────────────────────────────┐
│ Hierarchical Task Decomposition System │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Decomposer │ │ SubtaskManager │ │
│ │ │ │ │ │
│ │ • should_decomp │ │ • Track subtasks │ │
│ │ • AI breakdown │ │ • Manage state │ │
│ │ • Conventions │ │ • Persistence │ │
│ │ • Integration │ │ • Completion │ │
│ └──────────────────┘ └──────────────────┘ │
│ │ │ │
│ └──────────┬───────────┘ │
│ │ │
│ ┌───────────────────┼─────────────────────────────────┐ │
│ │ │ │ │
│ │ Subtask Assignment & Coordination │ │
│ │ │ │ │
│ │ • Dependency resolution • Context enrichment │ │
│ │ • Task conversion • Progress tracking │ │
│ │ • Parent updates • Integration checks │ │
│ │ │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Workflow Integration#
┌─────────────────────────────────────────────────┐
│ Task Assignment with Subtasks │
├─────────────────────────────────────────────────┤
│ │
│ Agent: request_next_task() │
│ │ │
│ ▼ │
│ Check SubtaskManager for available subtasks │
│ │ │
│ ┌────┴────┐ │
│ │ Found? │ │
│ ▼ ▼ │
│ Yes No │
│ │ │ │
│ │ ▼ │
│ │ Regular task assignment │
│ │ │ │
│ ▼ │ │
│ Check dependencies satisfied │
│ │ │ │
│ ▼ │ │
│ Convert Subtask → Task │
│ │ │ │
│ └────┬─────┘ │
│ │ │
│ ▼ │
│ Assign with enhanced context │
│ │ │
│ ▼ │
│ Agent works on subtask │
│ │ │
│ ▼ │
│ Reports completion │
│ │ │
│ ▼ │
│ Update subtask status │
│ │ │
│ ▼ │
│ Update parent progress │
│ │ │
│ ▼ │
│ Check if all subtasks done │
│ │ │
│ ┌───┴────┐ │
│ │ Done? │ │
│ ▼ ▼ │
│ Yes No │
│ │ │ │
│ ▼ └─→ Continue │
│ Auto-complete parent │
│ │ │
│ ▼ │
│ Trigger dependent tasks │
│ │
└─────────────────────────────────────────────────┘
How It Works#
1. Decomposition Decision#
When a task is created, Marcus evaluates whether it should be decomposed:
def should_decompose(task: Task) -> bool:
# Size heuristic
if task.estimated_hours >= 4.0:
return True
# Complexity heuristic
indicators = ["api", "database", "model", "ui", "frontend", "backend"]
indicator_count = sum(1 for i in indicators if i in task.description.lower())
if indicator_count >= 3:
return True
# Type heuristic - skip certain types
skip_types = ["bugfix", "hotfix", "refactor", "deploy"]
if any(t in task.labels for t in skip_types):
return False
return False
2. AI-Powered Breakdown#
If decomposition is warranted, Marcus uses AI to create subtasks:
decomposition = await decompose_task(
task,
ai_engine,
project_context={
"labels": task.labels,
"existing_tasks": [t.name for t in project_tasks],
"tech_stack": ["Python", "FastAPI", "PostgreSQL", "React"]
}
)
AI Analysis Includes:
Component identification (API, DB, UI, etc.)
Sequential vs parallel opportunities
Interface design between components
File artifact organization
Time estimation per subtask
Dependency relationships
3. Subtask Generation#
AI generates structured subtasks with complete metadata:
{
"success": True,
"subtasks": [
{
"id": "task_123_sub_1",
"name": "Create User database model",
"description": "Implement User model with fields: id, email, password_hash, created_at...",
"estimated_hours": 2.0,
"dependencies": [],
"file_artifacts": ["src/models/user.py"],
"provides": "User model with email validation and password hashing",
"requires": None,
"order": 1
},
{
"id": "task_123_sub_2",
"name": "Build authentication endpoints",
"description": "Implement POST /api/login and POST /api/register endpoints...",
"estimated_hours": 2.5,
"dependencies": ["task_123_sub_1"],
"file_artifacts": ["src/api/auth/login.py", "src/api/auth/register.py"],
"provides": "POST /api/login, POST /api/register with JWT token response",
"requires": "User model from subtask 1",
"order": 2
},
{
"id": "task_123_sub_integration",
"name": "Integration testing and validation",
"description": "Verify all auth components work together, run e2e tests...",
"estimated_hours": 1.0,
"dependencies": ["task_123_sub_1", "task_123_sub_2"],
"file_artifacts": ["docs/integration_report.md", "tests/integration/test_auth.py"],
"provides": "Validated, integrated authentication system",
"requires": "All auth components",
"order": 99 # Integration always last
}
],
"shared_conventions": {
"base_path": "src/api/auth/",
"response_format": {"success": {...}, "error": {...}},
"authentication": "JWT tokens",
"error_handling": "Custom AuthException with HTTP status codes"
}
}
4. Subtask Assignment#
When agents request work, subtasks are prioritized:
async def find_optimal_task_for_agent(agent_id: str, state: Any) -> Optional[Task]:
# Check for available subtasks first
if state.subtask_manager:
available_subtask = await find_available_subtask(
state.subtask_manager,
agent_id,
state.project_tasks
)
if available_subtask:
# Convert subtask to Task for assignment
task = convert_subtask_to_task(available_subtask, state.subtask_manager)
return task
# Fall back to regular task assignment
return await find_optimal_task_original(agent_id, state)
5. Enhanced Context#
Agents receive rich context about their subtask’s role:
{
"is_subtask": True,
"subtask_info": {
"name": "Build authentication endpoints",
"provides": "POST /api/login, POST /api/register with JWT tokens",
"requires": "User model from subtask 1",
"file_artifacts": ["src/api/auth/login.py", "src/api/auth/register.py"]
},
"parent_task": {
"id": "task_123",
"name": "Build user authentication system",
"description": "Complete auth system with login, registration, and sessions"
},
"shared_conventions": {
"base_path": "src/api/auth/",
"response_format": {...},
"authentication": "JWT tokens"
},
"dependency_artifacts": {
"task_123_sub_1": {
"name": "Create User database model",
"provides": "User model with email validation",
"file_artifacts": ["src/models/user.py"],
"status": "done"
}
},
"sibling_subtasks": [
{"name": "Add session management", "status": "todo"},
{"name": "Integration testing", "status": "todo"}
]
}
6. Progress Tracking#
As subtasks complete, parent progress updates automatically:
# When subtask completes
await report_task_progress(agent_id, "task_123_sub_2", status="completed", ...)
# System automatically:
# 1. Updates subtask status
state.subtask_manager.update_subtask_status("task_123_sub_2", TaskStatus.DONE, agent_id)
# 2. Calculates parent completion
completed = sum(1 for s in subtasks if s.status == TaskStatus.DONE)
progress = int((completed / len(subtasks)) * 100)
# 3. Updates parent task
await kanban_client.update_task_progress("task_123", progress)
# 4. Checks for auto-completion
if all(s.status == TaskStatus.DONE for s in subtasks):
await kanban_client.move_task_to_done("task_123")
Benefits#
For Agents#
Clearer Scope: Work on well-defined components instead of vague epics
Before: "Build authentication system" (unclear what's included)
After: "Build login endpoint returning JWT tokens" (specific deliverable)
Better Context: Understand how your work fits into the larger system
Context shows:
- What you can use from completed subtasks
- What future subtasks will depend on your work
- Shared patterns to follow for consistency
Reduced Integration Risk: Shared conventions prevent mismatch
All subtasks follow same:
- File structure
- Response formats
- Error handling
- Naming conventions
For Project Management#
Accurate Progress: Real-time visibility into feature completion
Parent: "Build authentication" - 66% complete
✓ Create User model (done)
✓ Build login endpoint (done)
⧗ Add session management (in progress)
○ Integration testing (todo)
Better Parallelization: Independent subtasks can run simultaneously
Sequential: User model → Login → Register → Sessions (8 hours serial)
Parallel: User model → [Login + Register + Sessions] (5 hours with 3 agents)
Easier Recovery: Failed subtasks can be reassigned without losing context
If agent fails during "Add session management":
- Other subtasks remain complete
- New agent gets same context and conventions
- No need to restart entire feature
For System Intelligence#
Learning Patterns: Marcus learns effective decomposition strategies
Memory System records:
- Which decompositions worked well
- Common subtask patterns for feature types
- Optimal subtask sizes and dependencies
- Integration task effectiveness
Improved Estimates: Better time predictions from subtask history
"Authentication system" historically breaks down to:
- User model: 1.5-2.5 hours
- Login endpoints: 2-3 hours
- Session management: 1.5-2 hours
- Integration: 0.5-1 hour
Total: 5.5-8.5 hours
Comparison with Other Approaches#
vs. Manual Task Breakdown#
Manual:
Time-consuming upfront planning
Requires deep technical knowledge
Hard to maintain consistency
Doesn’t adapt to project changes
Marcus Hierarchical:
Automated AI-powered breakdown
Learns from project patterns
Enforces shared conventions
Adapts to new information
vs. Flat Task Lists#
Flat:
All tasks at same level
Unclear relationships
Hard to track feature completion
Integration happens implicitly
Marcus Hierarchical:
Parent-child relationships clear
Explicit interfaces defined
Feature progress tracked automatically
Integration tested explicitly
vs. Epic/Story/Task Hierarchy#
Traditional Agile:
Manual story pointing
Subjective estimates
Human-defined dependencies
Stories often too large for one sprint
Marcus Hierarchical:
AI-powered decomposition
Time-based estimates
Inferred + explicit dependencies
Subtasks right-sized for agents
Key Takeaways#
Hierarchical Task Decomposition enables:
Intelligent Breakdown - AI identifies components and interfaces
Clear Contracts - Subtasks define provides/requires explicitly
Shared Conventions - Consistent patterns across components
Automatic Integration - Final validation subtask ensures cohesion
Better Parallelization - Independent subtasks run simultaneously
Accurate Tracking - Parent progress reflects subtask completion
Easier Recovery - Failed subtasks don’t invalidate entire features
This system embodies Marcus’s core values:
Context Compounds - Rich subtask context enables autonomy
Guided Autonomy - Clear interfaces with implementation freedom
Relentless Focus - One subtask, complete → request next
Fail Forward - Subtask failures don’t cascade to entire features
Next Steps#
Guide: Using Hierarchical Task Decomposition - Practical usage
System: Task Decomposition System - Technical details
Integration Guide - Developer integration
Hierarchical Task Decomposition transforms large, ambiguous features into manageable, well-defined subtasks with clear interfaces—enabling autonomous agents to collaborate effectively on complex software projects.