Memory-Augmented Policy Harness for Frozen VLA Policies

RoboHarness 🏇: A Memory-Augmented Policy Harness for Vision-Language-Action Model Robustness via In-Context Adaptation

RoboHarness is a memory-augmented policy harness that upgrades frozen Vision-Language-Action (VLA) policies for robust in-context adaptation without parameter fine-tuning, via contrastive Dual-Memory RAG, an attribution-driven vision-language orchestrator implemented with a multimodal large language model, Dynamic MCP Interventions, and Offline Memory Consolidation.

56.6%

Avg. Absolute Gain
(LIBERO-PRO + LIBERO-RoboHarness)

89.1%

Absolute Improvement
(Long-Horizon Task Chaining)

Core Highlights

Dual-Memory RAG

Contrastive retrieval from both success and failure memory banks for stable, low-cost intervention planning.

Vision-Language Orchestrator

LLM-based strategic planner that generates precise tool-chain plans and task refinements.

Dynamic MCP Interventions

Extensible visual tools (overlay, distractor removal) and control flow (rollback, decomposition).

RoboHarness Framework

Here, we demonstrate how RoboHarness perceives visual input and retrieves historical experience to handle image/task prompts without requiring VLA fine-tuning, leading to a higher success rate.

Figure 1: Overview of the RoboHarness framework architecture.

Generalization Performance Demos

Our experiments are based on the LIBERO Benchmark. We modified LIBERO to create LIBERO-RoboHarness and also utilized LIBERO-PRO to evaluate robustness.

Visual Intervention Tools

MCP Tool	Task Prompt	Origin Image	Modified Image	Success Output
Visual Overlay	"Pick the red bowl from center of the cross formation and place it on the plate"
Distractor Remove	"Pick the black bowl from left of the cross formation and place it on the plate"

Text & Task Refinement Tools

MCP Tool	Original Prompt	Refined Prompt	Observation	Success Output
Prompt Simplify	"Hey, umm... look down there. Can you grab that bottle? You know, the one for fries? Yeah, put it in the basket."	"Pick the red sauce bottle and place it in the basket"
Task Decompose	"Sort the items: milk and cream cheese into the basket"	[SubTask-1] "Pick the milk and place it in the basket." [SubTask-2] "Pick the cream cheese and place it in the basket."

More Experiment Results & Ablation Studies

LIBERO-RoboHarness + LIBERO-PRO Backbones: π₀, π₀.₅, SmolVLA Zero-shot In-Context Adaptation OOD-Generalization

Quick Glance KPI

Metric	Baseline	RoboHarness	Gain
LIBERO-PRO Avg (Pos)	2.33%	57.2%	+54.87%
LIBERO-PRO Avg (Task)	0.86%	55.2%	+54.34%
Dual-Memory Avg Turns	7.40 (No Memory)	1.07	-85.5%

Figure: Main performance comparison across visual, linguistic, and long-horizon OOD challenges.

RoboHarness consistently reduces sensitivity to sensory noise, linguistic ambiguity, execution instability, and long-horizon error accumulation.

Ablation Study

Dual-Memory Ablation

Memory Setting	Avg. Turns	First-Turn Score
No Memory	7.40	60.0
Failure Only	7.33	73.75
Success Only	1.47	91.07
RoboHarness (Dual)	1.07	96.3

RAG Richness Ablation

RAG Config	Success Rate
No-RAG	19.2%
Limited-RAG	48.8%
Rich-RAG (RoboHarness)	60.1%