With Tesla's release of the V12 version of the FSD intelligent driving system as a symbol, intelligent driving has entered an end-to-end era overnight.
At the 2024 Hangzhou Yunqi Conference, He Xiaopeng, Chairman of Xiaopeng Motors, said that after adopting the end-to-end large model, Tesla's FSD is completely different from before, and it may be stronger than human old drivers next year.
Xiaopeng Motors is one of the first domestic car companies to follow Tesla. At the end of July this year, Xiaopeng Motors began pushing the XNGP intelligent driving system based on an end-to-end big model to users. As of September this year, car companies such as Huawei and Ideal have also started to push corresponding end-to-end large-scale model-based intelligent driving systems to users; NIO has applied end-to-end large-scale models to the AEB system and released its self-developed world model.
With the launch of end-to-end large-scale models, car companies have become increasingly aggressive in promoting intelligent driving. The once lively and noisy smart driving city opening and high-precision maps are no longer in high demand. The launch of a driving assistance system with door-to-door and point-to-point capabilities has been officially elevated in the schedule. Xiaopeng Motors claims that it can achieve L3+level autonomous driving user experience with the hardware cost of L2 level intelligent driving.
For a while, intelligent driving systems without end-to-end capabilities seemed to have hooked up with backwardness. Intelligent driving without using large models will be eliminated, "He Xiaopeng also said, stating that all L4 autonomous driving companies should switch to large models as soon as possible.
Chentao Capital, in collaboration with three parties, has released the "End to End Autonomous Driving Industry Research Report" (hereinafter referred to as the "Report"). The Report shows that among the more than 30 frontline experts in the autonomous driving industry interviewed, 90% of them stated that their companies have invested in developing end-to-end technology, and most technology companies believe that they cannot afford to miss the consequences of this technological revolution.
But not all "players" agree that end-to-end big models are the disruptors of the current intelligent driving system landscape.
Hou Cong, CTO of Qingzhou Zhihang, told First Financial reporters that he experienced Tesla's FSD V12.3 system in the United States. Although it has made great progress compared to Tesla's previous FSD, there is still a significant gap compared to Waymo Robotaxi, which focuses on regulation. Former founder of Tucson Future, Hou Xiaodi, called on the industry to take a rational approach and avoid myths from end to end.
In this technological controversy, car company leaders such as Musk and He Xiaopeng have provided end-to-end support; However, executives from L4 intelligent driving companies such as Hou Cong, Hou Xiaodi, and Lou Tiancheng (CTO of Xiaoma Zhixing) believe that the end-to-end big model cannot directly upgrade L2 intelligent driving assistance technology to L4 autonomous driving.
The report also shows that due to the fact that technology is still in the early stages of development, there are still many application challenges and pain points that urgently need to be addressed for end-to-end large-scale models, such as significant differences in technical routes, high demands for data and computing power, immature testing and verification methods, and huge resource investment.
On the road to the ultimate goal of autonomous driving, end-to-end large models have become another technological controversy after pure visual perception, radar fusion perception, and so on.
Tesla leads technological change again?
Starting from technologies such as integrated die-casting and battery body integration, Tesla has become the industry benchmark for new energy vehicle technology. Many Chinese car companies are considered to be "crossing the river by feeling Tesla", with end-to-end large models getting on board. Tesla has once again led the transformation of new energy vehicles.
Before the end-to-end large model is loaded onto the vehicle, intelligent driving assistance systems are mostly divided into multiple modules such as perception, planning, decision-making, and control. Among them, artificial intelligence and machine learning are mostly applied in perception, planning, and other aspects, but the modules are mainly defined by manually handwritten rules, known as "rule-based".
However, in actual system operation, vehicles often encounter endless cone cases (long tail problems). To solve such problems, engineers need to write code and set rules based on specific scenarios. In this mode, intelligent driving assistance or auto drive system often requires manual input of a large number of rules.
Wu Xinzhou, Global Vice President and Head of the Automotive Division at Nvidia, believes that most of the existing algorithms for autonomous driving are rule-based, which is easy to say, from what you see to how you do it. However, it is difficult to set rules for it well, requiring many human engineers to think of all possibilities as much as possible, and this method has limits.
Unlike traditional rule-based intelligent driving assistance systems, end-to-end autonomous driving solutions mean that the entire process from perception to regulation is processed through advanced algorithms and deep learning techniques.
The application of end-to-end technology in autonomous driving has transformed the architecture of combining multiple models such as perception, prediction, and planning into a single model architecture that integrates perception and decision-making.
A research report released by Xinda Securities shows that "end-to-end" refers to one end inputting environmental data information such as images, going through a multi-layer neural network model similar to a "black box", and the other end directly outputting driving instructions such as steering, braking, and acceleration.
Compared with the traditional rule-based modular architecture, end-to-end implementation will bring a series of advantages: fully data-driven global task optimization, with better and faster error correction capabilities; Can further reduce the lossy transmission, delay, and redundancy of information between modules, avoid error accumulation, and improve computational efficiency; Stronger generalization ability, shifting from rule-based to learning based, with zero sample learning ability, and stronger decision-making ability when facing unknown scenarios.
With the support of end-to-end large models, intelligent driving systems can achieve faster iteration and progress. Taking Xiaopeng's XNGP as an example, after applying end-to-end big models, its three in one neural network XNet+regulatory big model XPlanner+AI big language model XBrain can achieve iteration every 2 days, and its intelligent driving ability can be improved by 30 times in 18 months; The data system capability and neural network architecture can achieve rapid diagnosis and solve long tail problems on an hourly basis.
With Tesla's end-to-end large model getting on board, the intelligent driving technology roadmap of Chinese car companies will also begin to undergo significant changes in 2024.
In the past few years, the controversy over the technology route of intelligent driving assistance systems among Chinese car companies has mostly focused on visual perception and integrated perception, with more competition in terminal areas such as opening speed and opening quantity. At the beginning of 2024, companies such as Huawei and Xiaopeng are still competing for high-precision graphics and truly 'nationwide accessibility'.
After the end-to-end large model is loaded onto the vehicle, the generalization ability of the intelligent driving assistance system is greatly improved, and the importance of verification and city opening for a single region decreases. At the same time, the previous differentiation of perception, planning, decision-making, control and other modules has been weakened end-to-end, and multiple car companies have also begun to readjust the organizational structure of their autonomous driving teams based on the needs of end-to-end big models.
At the end of 2023, Ideal conducted an organizational restructuring of its intelligent driving team. In this restructuring, Ideal reorganized the large model into a team and placed it under the front-end algorithm research and development team, with overall responsibility for end-to-end architecture development and deployment; In 2024, NIO established the Large Model Department, Deployment Architecture and Solutions Department, and Spatiotemporal Information Department, and abolished the original Perception Department, Planning and Control Department, Environmental Information Department, and Solution Delivery Department.
Despite the booming trend of end-to-end driving, most Chinese car companies have not yet achieved the theoretical "One Mode" end-to-end intelligent driving.
The CTO of a certain autonomous driving company told reporters that the intelligent driving application of end-to-end models can be divided into two stages: the first stage is a two model solution, consisting of end-to-end perception and end-to-end regulation, which is currently a mainstream direction used in the industry; The second stage is the one model solution, which involves a large model that solves the problem of information input to decision output, closer to the direction of AGI. However, this direction is relatively difficult and is estimated to take 3-5 years to achieve some large-scale applications.
At present, the industry generally believes that the R&D progress gap between domestic car companies and Tesla is about 1.5-2 years. Gu Junli, Deputy General Manager of Chery Automobile Co., Ltd., believes that in order to catch up with Tesla in terms of business model, it is necessary to form a product scale. When the data reaches the Tesla level of millions or more, through reinforcement training of the model, intelligent driving can learn video streams and directly tell drivers the direction of driving, just like the popular ChatGPT, "said Gu Junli.
Is there a route disagreement between the vehicle manufacturer and the supplier?
As numerous car companies launch end-to-end models and promote the potential arrival of the autonomous driving era, many suppliers specializing in autonomous driving have issued different voices.
After Tesla launched its end-to-end FSD, there were some issues with the car. The car is always prone to scratches on the road shoulder, especially at night, and sometimes it hits the road shoulder directly, causing the tire to deflate. "Hou Cong told reporters that in the United States, Waymo did not adopt an end-to-end large model, but it has been able to achieve unmanned Robotaxi operation in multiple cities, and the user response has been quite good.
The end-to-end big model itself is not a new technology that has only achieved breakthroughs in recent years.
Before the emergence of deep learning around 2010, it was called model analysis algorithm. At that time, we did pedestrian detection at Tsinghua University and had to extract some feature information from images, such as the curvature of human shoulders, the color of eyes, and so on. These features were artificially induced by us, which is rule based. After deep learning was developed, we input images and let deep learning learn autonomously. Finally, each person's different features were learned by deep learning, not defined by humans. This is based on learning, just like today's end-to-end intelligent driving assistance. "Hou Cong told reporters that this system, like current end-to-end intelligent driving assistance, requires massive data support.
This is also considered one of the important factors for car companies to compete in choosing end-to-end large models.
Compared to L4 autonomous driving suppliers that only operate test fleets of over a hundred vehicles, car companies typically have hundreds of thousands or even millions of products on the road, generating massive amounts of data during user driving. This helps car companies train their end-to-end intelligent driving systems and enables rapid iteration of the system.
In addition, Dong Jun, an engineer from a certain L2+intelligent driving assistance system supplier, told reporters that for suppliers, end-to-end intelligent driving is difficult to become a standardized product; The changes in vehicle body form and sensor installation positions require the entire system to retrain the model, which incurs significant costs and time, resulting in poor efficiency.
The significance of end-to-end big models for L2 driving assistance lies in their ability to accelerate the driving speed and achieve what car companies call "nationwide driving". But for L4 level autonomous driving companies, end-to-end large models can also reduce the system's dependence on high-precision maps in the initial stage of operation, allowing the company to expand its operational scope faster; However, in the middle and later stages of operation, high-precision maps still have an important impact, which can further improve the reliability, safety and smoothness of the auto drive system.
On the other hand, compared to profitable car companies such as Tesla and Ideal, the vast majority of autonomous driving companies currently rely mainly on financing and blood transfusion. However, getting an end-to-end big model on board not only requires massive amounts of data, but also a significant investment of funds.
In the future, intelligent driving has entered the L4 stage, with exponential growth in data and computing power every year. This means that at least $1 billion is needed every year, and continuous iteration is needed after 5 years. At this level, it is difficult for a company's profit and profitability to support investment. Therefore, we do not need to focus on how many billion we invest in autonomous driving now, but rather start from the essence, whether we have sufficient computing power and data support, and then see how much money we need to invest. "Lang Xianpeng, Vice President of Intelligent Driving Research and Development at Ideal Automobile, told reporters.
Xia Yiping, CEO of Jiyue Automobile, believes that 20 billion yuan was once recognized as the capital threshold for car manufacturing, but now companies cannot do well in intelligent driving without 50 billion yuan.
More importantly, for autonomous driving companies like Waymo and Xiaoma Zhixing that aim to achieve L4 Robotaxi, their considerations for system weight, cost, and other aspects differ greatly from those of vehicle manufacturers.
Different from L2 driving assistance, automatic driving above L3 level will transfer the responsibility subject of the accident to the vehicle, which puts forward high requirements for the stability and safety of the auto drive system. The inexplicability of the black box of the end-to-end large model brings certain risks to the auto drive system.
Car companies have successively launched end-to-end large-scale intelligent driving models and heavily promoted them, with the core goal of creating differentiation and selling cars, "said Dong Jun.
Hou Xiaodi said in a media interview that if Tesla's FSD has an accident, the responsibility lies with the driver. Tesla requires the driver to keep their hands on the steering wheel throughout the entire process, and the accident has nothing to do with Tesla; In addition, Tesla's business is selling cars, and FSD is the added value of selling cars. If you want to consider how to sell more cars, you cannot delve deep into a limited area like L4 and solve all corner cases (extreme cases) in this area.
Interviewees from autonomous driving companies such as Hou Cong pointed out that L4 autonomous driving requires 100% safety and cannot accept the unexplainable and uncertain nature of end-to-end "black boxes". In addition, there are significant differences in business logic between L2 and L4.
For automobile manufacturers, selling cars is their main business, and cost determines profit and market competitiveness. Therefore, it is inevitable that too much safety redundancy cannot be arranged in products; L4 Robotaxi, on the other hand, places more emphasis on operations and will primarily focus on B2B services for a considerable period of time, rather than directly serving consumers. Therefore, relevant companies need to consider not only vehicles, but also various situations in vehicle operations.
For example, what to do if a car gets stuck, hardware breaks down, or an accident occurs, which requires more redundancy. Tesla cannot reserve a lot of redundancy like Waymo because their business logic is different, "said Hou Cong.
Is the world model achieving autonomous driving?
Although there are differences, several technicians from autonomous driving companies also agree in interviews that end-to-end large models can enhance the upper limit of current intelligent driving assistance systems in cars. Several practitioners have expressed that the end-to-end big model presents a "seesaw" state, and getting on the car with the end-to-end big model can improve the upper limit of intelligent driving assistance system capabilities, but it will also lower the lower limit of system performance.
The end-to-end big model is based on a probabilistic model training, and it has a problem that for relatively simple and easy to describe scenarios, its output is often not so accurate, and the bottom line is relatively low. Tesla has done quite well in this area, but it has not completely solved this problem. We believe that in the current lack of sufficient data, it is still necessary to gradually implement end-to-end, replacing each module one by one, while ensuring safety at the same time. With this relatively solid engineering infrastructure and fast iteration method, the performance upper limit of the system can be gradually improved, while also ensuring the lower limit of system performance, "said Chen Liming, President of Horizon Robotics.
The end-to-end big model is data-driven, with sensor data as input and driving decisions as output. However, it has strong interpretability in the middle, and people cannot know the process of the system making the final decision. It is often compared to a black box.
Hou Cong believes that the current end-to-end big model intelligent driving and the previous rule-based intelligent driving have some similarities with the production process of automobiles. "In the past, when making cars, car companies bought parts from different companies to assemble them together. On the one hand, it was convenient for procurement, dispersing suppliers and not easily getting stuck. On the other hand, it was easy to repair, where it was broken. The same goes for multi module autonomous driving, which has the advantage of better defining and solving problems
Taking traditional multi module autonomous driving as an example, if the system encounters problems during testing, R&D personnel can discover bugs in the corresponding sections based on the situation and fix them. But for end-to-end big models like black boxes, developers can only train strategies, retrain, or modify models, but modify the parameters in the "black box". And with the upgrading and iteration of the system, the more difficult the problems it solves, the more cost investment is required, which sets a high threshold for end-to-end large models.
On the other hand, end-to-end big models are data-driven, but massive amounts of data may not necessarily have a positive impact on the system.
Xiao Bo, the head of Xiaoma Zhixing AI team, believes that even if the algorithm is good and the system training is done well, the ability learned from massive human driving data is almost at the level of an average human driver, which is sufficient to cope with L2 level intelligent driving assistance; However, the automatic driving capability of L4 or above needs to be 10 times or more than that of human drivers, which is not enough to support.
At a time when the end-to-end trend is rapidly becoming popular, domestic car companies and suppliers have once again proposed the concept of a new "world model". Lou Tiancheng believes that the world model is currently the best and most important thing, and understands it as the only solution to autonomous driving.
The world model can be understood as the simulation and modeling of the real world, which can accurately reproduce changes in scenes such as intersections. For example, pedestrian trajectories obstructed by ghost probes; The reaction between pedestrians and other vehicles at the moment of vehicle collision; Even reflecting details such as the acceleration due to gravity when a person slows down while running. At the same time, the world model is also a scoring system. By evaluating the performance of the auto drive system, we can know who is better than system A and system B.
Previously, car companies such as NIO and Ideal have successively released their "World Models".
NIO's Vice President of Autonomous Driving, Ren Shaoqing, said, "Compared to conventional end-to-end models, the new world model has three main advantages that we consider. The first is in terms of spatial understanding, it extracts information more broadly through generative models by reconstructing sensors. Through autoregressive models, it automatically models long-term environments. The third is that the vast world requires more data, and through self supervision, there is no need for manual labeling. It is a multiple autoregressive generative model structure that allows us to learn better
Lou Tiancheng believes that the world model can be understood as a "coach" simulated by humans, and for the L2 system, its driving ability is equivalent to that of an experienced driver; For the L4 system, its driving level is much higher than that of human drivers. If it were to train the intelligent driving system, the result would definitely be better than that of human drivers.
Although there is still controversy, most respondents believe that in the L2 intelligent driving assistance stage, end-to-end large models can indeed improve the performance ceiling of related systems. What most practitioners of L4 autonomous driving companies do not agree with is that Tesla, Xiaopeng, and other car companies vigorously promote end-to-end technology, and their products are based on L2 intelligent driving, even achieving L4 autonomous driving capabilities at the hardware level of L2.
At present, car companies are vigorously promoting end-to-end technology, shaping it into a cutting-edge technology leading to autonomous driving, with the ultimate goal of selling more cars, "said Dong Jun.