In a world-first study, Purdue University researchers have integrated ChatGPT into autonomous vehicles, significantly improving their ability to understand and respond to passengers’ commands.
Imagine telling your car, “I’m in a hurry,” and it intuitively navigates the fastest route. Researchers at Purdue University have demonstrated that this could soon be a reality, thanks to integrating ChatGPT and other advanced chatbot technologies into autonomous vehicles (AVs).
The groundbreaking study, set to be presented at the 27th IEEE International Conference on Intelligent Transportation Systems on Sept. 25, reveals that AVs can now interpret and act upon more nuanced passenger commands using large language models (LLMs).
Ziran Wang, assistant professor in Purdue’s Lyles School of Civil and Construction Engineering and lead researcher for the study, emphasized the potential of this innovation.
For vehicles to be fully autonomous one day, they’ll need to understand everything that their passengers command, even when the command is implied, Wang explained in a news release. This mirrors how a human taxi driver might understand and respond to a passenger’s urgency without detailed instructions.
While current AV systems require passengers to issue explicit commands, LLMs offer a more human-like interaction. Trained on vast datasets, these models can understand and generate responses more naturally.
“][The power of large language models is that they can more naturally understand all kinds of things you say. I don’t think any other existing system can do that,” Wang said in the news release.
In the study, Wang and his team integrated ChatGPT with a level-four autonomous vehicle, as defined by SAE International, one step below full autonomy. The LLMs processed passenger commands and provided real-time driving instructions to the AV’s control systems, taking into account traffic conditions, weather and sensor data.
The results were promising. The participants reported feeling more comfortable and satisfied with the AV’s driving decisions compared to traditional AV systems. The vehicle’s performance also surpassed baseline safety and comfort metrics. The memory module installed in the system allowed the AV to learn and adapt to individual passenger preferences over time.
Even though the models processed commands in an average of 1.6 seconds, considered adequate for non-critical scenarios, Wang noted that further optimization is required for more urgent situations. Additionally, addressing the issue of “hallucinations,” where LLMs can misunderstand commands, remains crucial.
The breakthrough has garnered attention for its potential to revolutionize AVs, but much more testing and regulatory approval will be necessary before such systems become mainstream. Wang’s lab is continuing their research by evaluating other chatbot technologies like Google’s Gemini and Meta’s Llama AI assistants and exploring LLMs’ capability to enable AV-to-AV communication at intersections.
Moreover, the team is investigating large vision models, which can aid AVs in navigating extreme weather, a significant concern in the Midwest.