The best visual AI assistant finished 53% of guided tasks in Salesforce's test. Speed was part of the failure

A technician points a phone camera at a router and asks an AI assistant which cable goes where. The assistant sees the scene, gives the right instruction, and the technician has already plugged the cable into the wrong port two seconds earlier.

On a static benchmark, that answer scores as correct. In the field, it failed. Salesforce AI Research built a test for exactly this gap, and the results are sobering for anyone planning to put visual assistants in front of frontline workers.

A test where the user reacts

Interactive Visual Gym is a simulation framework for assistants that watch a user’s video feed and guide them by voice. It recreates physical workflows, such as preparing food or configuring a device, and pairs the assistant with a simulated user. When the assistant speaks, the simulated user changes what it does, and the state of the simulated environment changes with it.

The harness also creates trouble on purpose: missteps, a change of plan halfway through, clarification questions, and checks on whether the assistant remembers earlier steps. Simulated users vary along four traits: domain expertise, trust in AI, working speed and tolerance for interruptions.

Salesforce evaluated 14 leading visual assistants across 315 scenarios in three environments. The strongest, Gemini 4 Argon, completed 53.4% of tasks within the time limit, averaged across the three environments. Salesforce reports weaknesses in factual grounding and in giving instructions a person can act on, and says leading models were often too slow to prevent errors once latency was counted. Open-weight models generally trailed, and some became less accurate as a task went on.

Automated scores agreed with human participants’ preferences nearly 80% of the time, which is reasonable for a simulator but not a substitute for field trials.

Why closed-loop testing matters

Most multimodal benchmarks show a model an image and grade the answer. They separate the answer from its effect, so they cannot tell whether the guidance would have prevented the mistake. Salesforce’s argument is that for an assistant guiding physical work, correctness includes timing. A right instruction after the error is a wrong instruction.

That is the same lesson enterprise teams learned with voice agents, which we described as real-time systems rather than chatbots with a microphone. Visual guidance adds a camera to the loop and makes the timing harder, because the assistant must notice the moment to speak, not only answer when asked.

The research also shows a path forward. Training a small 4-billion-parameter open model in the simulator raised its success on unseen tasks from 33.3% to 42.2%. That suggests task-specific simulation can close part of the gap, which matters for companies that would rather run a small model on a device than stream video to the cloud.

For field service and frontline AI

If your roadmap includes AI that watches and guides technicians, nurses, warehouse staff or installers:

• Measure task completion, not answer quality. The question is whether the person finished correctly, in time.
• Measure latency from event to instruction. An assistant that is right but late should fail the test.
• Test with difficult users. Fast workers, skeptical workers and workers who change plans expose failures that a cooperative tester never will.
• Keep the procedure authoritative. Where the stakes are physical safety, the assistant should point to the approved procedure, not improvise one.

A 53% success rate for the best model is not a reason to wait indefinitely. It is a reason to start with tasks where a missed or late instruction is cheap, and to evaluate the way Salesforce did before expanding.