MIT researchers have developed an AI system called F3RM that enables robots to interpret natural language commands for manipulating unfamiliar objects. Inspired by human adaptability, the technology combs visual, verbal and spatial clues to fulfill less-specific instructions.

Robots normally perform pre-set motions on objects they've been repeatedly trained on. Breaking this mold, F3RM allows leading-edge models to construct 3D scenes from images then associate semantic details.

Using a camera on a stick, F3RM creates a rich digital replica of the surroundings by capturing photos from diverse angles. This point cloud forms the environmental context linked to object characteristics and names via advanced deep learning algorithms.

Combined, the capabilities give F3RM intuitive spatial understanding and ability to handle opaque requests like "grab the tall thermos" or "raise that action figure." Without specific object data, the system determines the most fitting target object and how to appropriately grasp it.

Lead researcher Ge Yang says such ad-hoc flexibility will be critical as robots enter unpredictable real-world environments like warehouses. To aid overloaded human staff, they must capably handle constantly varying items.

"We want robots as adaptable as ourselves, grasping new objects without prior exposure," says Yang. "F3RM takes an aggressive step toward generalization needed in messy industrial settings."

By merging 3D mapping, language and object recognition, the team brought sci-fi-like responsiveness closer to reality. They see great promise automating pick-and-place using natural communication as training data multiplies.

The research establishes a framework to imbue robots with more contextual awareness necessary for seamless human collaboration. This intelligence could allow futuristic factories and supply chains to harness both unique strengths of man and machine.