​From Words To Actions: How LLMs Are Reshaping Robotics

Nishanth Prakash, Principal Member of Technical Staff, AI/ML Engineer, Oracle.

​AI is rapidly moving beyond software that generates text, images and code. A new generation of AI systems is beginning to interact with the physical world through robots capable of interpreting instructions, understanding their surroundings and translating high-level goals into actions.

For decades, robots have been extremely capable within carefully controlled environments. Industrial robots can repeatedly weld, assemble or move components with extraordinary precision. But these systems are typically programmed around predefined tasks, known objects and structured environments.

Large language models (LLMs) and, increasingly, multimodal models are beginning to change this paradigm. Instead of programming every possible action individually, developers can use language as a higher-level interface between humans and machines.

This is where an important architectural shift is beginning to emerge: Robots are moving from executing predefined instructions toward interpreting human intent.

Why Traditional Robotics Alone Is Not Enough

Traditional robotic systems are built around components such as perception, motion planning, localization and control. These technologies remain essential because robots ultimately need precise and predictable mechanisms for interacting with the physical world.

The challenge emerges at a higher level. Consider a robot asked to perform a task such as “Clear the table and put the dishes in the sink.” A traditional system may require the task to be decomposed beforehand into several explicitly programmed steps: identify dishes, determine grasp positions, navigate toward the sink, avoid obstacles and place each object in the appropriate location. A language-enabled system can potentially interpret the broader goal and help determine the sequence of actions necessary to complete it.

From Language Models To Vision-Language-Action Models

1. Understanding human instructions