What are Neural Networks?

Neural networks are a set of algorithms, that are designed to mimic the human brain, that is designed to recognize patterns. They interpret data through a form of machine perception by labeling or clustering raw input data.

Let’s take a moment to consider the human brain. Made up of a network of neurons, the brain is a very complex structure. It’s capable of quickly assessing and understanding the context of numerous different situations. Computers struggle to react to situations in a similar way. Artificial Neural Networks are a way of overcoming this limitation.

First developed in the 1940s Artificial Neural Networks attempt to simulate the way the brain operates. Sometimes called perceptrons, an Artificial Neural Network is a hardware or software system. Consisting of a network of layers this system is patterned to replicate the way the neurons in the brain operate. The network comprises an input layer, where data is entered, and an output layer. The output layer is where processed information is presented. Connecting the two is a hidden layer or layers. The hidden layers consist of units that transform input data into useful information for the output layer to present.

In addition to replicating the human decision-making progress, Artificial Neural Networks allow computers to learn. Their structure also allows ANNs to reliably and quickly identify patterns that are too complex for humans to identify. Artificial Neural Networks also allow us to classify and cluster large amounts of data quickly.

How does the Biological Model of Neural Networks Function?

What are neural networks emulating in human brain structure, and how does training work?

All mammalian brains consist of interconnected neurons that transmit electrochemical signals. Neurons have several components: the body, which includes a nucleus and dendrites; axons, which connect to other cells; and axon terminals or synapses, which transmit information or stimuli from one neuron to another. Combined, this unit carries out communication and integration functions in the nervous system. The human brain has a massive number of processing units (86 billion neurons) that enable the performance of highly complex functions.

How do Artificial Neural Networks Work?

As we have seen Artificial Neural Networks are made up of a number of different layers. Each layer houses artificial neurons called units. These artificial neurons allow the layers to process, categorize, and sort information. Alongside the layers are processing nodes.

Each node has its own specific piece of knowledge. This knowledge includes the rules that the system was originally programmed with. It also includes any rules the system has learned for itself. This makeup allows the network to learn and react to both structured and unstructured information and data sets. Almost all artificial neural networks are fully connected throughout these layers.

Each connection is weighted. The heavier the weight, or the higher the number, the greater the influence that the unit has on another unit. The first layer is the input layer. This takes on the information in various forms. This information then progresses through the hidden layers where it is analyzed and processed.

By processing data in this way, the network learns more and more about the information. Eventually, the data reaches the end of the network, the output layer. Here the network works out how to respond to the input data. This response is based on the information it has learned throughout the process. Here the processing nodes allow the information to be presented in a useful way.

ANNs are statistical models designed to adapt and self-program by using learning algorithms in order to understand and sort out concepts, images, and photographs. For processors to do their work, developers arrange them in layers that operate in parallel. The input layer is analogous to the dendrites in the human brain’s neural network. The hidden layer is comparable to the cell body and sits between the input layer and output layer (which is akin to the synaptic outputs in the brain). The hidden layer is where artificial neurons take in a set of inputs based on synaptic weight, which is the amplitude or strength of a connection between nodes. These weighted inputs generate output through a transfer function to the output layer.

Different Types of Neural Networks

The most commonly used type of Artificial Neural Network is the recurrent neural network. In this system, data can flow in multiple directions. As a result, these networks have greater learning ability. Consequently, they are used to carry out complex tasks such as language recognition. Other types of Artificial Neural Networks include convolutional neural networks, Hopfield networks, and Boltzmann machine networks.

Each network is capable of carrying out a specific task. The data you want to enter, and the application you have in mind, affect which system you use. Complex tasks such as voice recognition may require more than one type of ANN. Now that we’ve established what Artificial Neural Networks are here are 10 examples of how they are currently being applied.

Tesla Bets Farm On Neural Network-Based Autonomy With Impressive Presentation

Tesla today held an “Autonomy Investor Day” at their HQ in Palo Alto, CA. There, Tesla outlined some of their plans for advanced driver assist and eventual autonomy in Tesla cars. The presentation was more technical than Tesla has revealed in the past, and significantly improved my impression of Tesla’s methods and chances. This was certainly the most important press conference that Tesla has given.

Tesla has taken a very different approach from the bulk of companies trying to build a truly autonomous car that can drive empty or let the passenger sleep. Tesla plans to use only radar and an array of video cameras around the vehicle to do the job. Almost all other teams use this but add LIDAR (laser) sensors which give the vehicle superhuman 3-D vision regardless of the lighting. At the meeting, they went into much more depth as to why they have taken that approach, and why the others are wrong.

Well, not just wrong. Elon Musk said LIDAR was a “fool’s errand” and those who depend on it are “doomed.” He predicted all other players “will dump LIDAR, that’s my prediction. Mark my words.” He said similar things about the use of detailed “HD” maps when storing understanding of the road based on past trips over it.

In short, Tesla is placing a significant bet that they will be able to solve all self-driving problems using neural networks. They believe, in particular, that the problem can’t be solved without neural networks (which almost all would agree on) but go further and say that the neural network approaches needed to make all other approaches (including additional sensors like LIDAR) a distraction and needless cost.

If the bet pays off, it is a big win, and they could have the lead in what is perhaps the biggest opportunity in modern industry.

There is a lot to unpack from this presentation, and there will be further articles to come.

New Chip

Tesla has created its own chip, custom-designed to do processing based only on what they feel a car needs, and it is now putting the chip in all new vehicles. They believe it is all the computing needed for full self-driving. The chip was designed to focus its silicon only on hardware useful for driving and to keep the power usage below 100 watts to stop it from eating into range. Most of the chip is devoted to doing dot products for neural network convolutions. Musk claims that this chip is the “best by a huge margin” at neural networks in the world, a claim that many other companies developing neural network chips might dispute. Tesla mostly compared its performance to NVIDIA general-purpose GPU chips.

The hardware has impressive specs and probably is sufficient for the computation needed. While I believe similar chips will be available from other providers, Tesla feels that by designing their own chip and putting it in millions of cars, they will save money in the long run, even with the cost of development. On top of the neural network hardware, the chip contains a mid-level GPU and 12 64-bit ARM cores for general-purpose computing. The hardware is redundant to survive the failure of any component.

Network training

With their new network hardware, Tesla has put most of its focus on training better neural networks to classify everything they will see on the roads. They believe, as has long been said, that their advantage will come from the large fleet of cars they can use to help train their networks — around half a million cars today, and growing.

Andrej Karpathy outlined some of their approaches. They began training their networks in the way everybody does, by creating human-labeled images. When they identify something of interest that they want to train their network better on, they dispatch a request to the cars in their fleet that says, “If you see something that looks like this, upload it to us.” Thus, if they notice they don’t handle a bicycle mounted on a car well (because you see both things) they ask the fleet to send them thousands of images of bikes on cars, and they tag and add those to their training data, eventually creating a network very good at understanding this.

They have done this with all sorts of static and moving objects, and also can look for things based on patterns of movement, such as asking for examples of cars that have cut in front of Tesla cars. When they get an example of a car that does that, they can ask for the video going back in time well before the cut-in, to train the network on what it hints there are before a car actually does a cut-in. This helps them predict the future activity of cars on the road.

They have also done this for path planning, watching the paths taken by human drivers in various road situations, to learn what typical human actions are when a given situation is seen. If they see a car making an error in planning a path or recognizing things, they prioritize getting better data to train the networks.

They have also had impressive success in training their networks to estimate the distance to objects in the view. One approach has been to make use of the radars in the car, which have an objective measurement of the distance to all radar targets. Once they can match a radar target with a visual target, they can train the network to learn how to estimate distances to purely visual targets.

Tesla’s fleet of drivers gives them immediate access to new data about any item of interest to their team. It should be noted that anybody with a large video network of recordings from dashcams could also do this (though they would not normally be able to get the radar data used above.) Such data is available to many players if they elect to record it. Tesla has more flexibility in controlling its fleet because it regularly updates the software in all its cars.

This approach gives Tesla an excellent system for training neural networks for perception and driving. The central question will be whether that is enough to attain the “final 9s” of reliability needed to remove the steering wheel from the car. Tesla feels this extremely high level can only be attained with immense amounts of training data, which they have an edge at gaining with their fleet. Almost all agree that more such data is better, but there is debate on whether it’s enough, or whether other techniques are needed to get to that extreme level of reliability.

Software management

Tesla has been applying this with their recent “Navigate on Autopilot” update which now makes lane changes on its own. This product began by requiring drivers to confirm any lane changes. Tesla looked at what drivers did with suggested changes and when they confirmed them, trained the system to do a better job. Now that changes are automatic, it is receiving feedback on 100,000 automated lane changes each day. It reports zero accidents involving these lane changes.

They also plan to use this method to make their automatic emergency braking (AEB) more predictive. It should learn, by the end of this year, to brake for obstacles that are about to enter your path (including pedestrians, cyclists, and cut-ins) and not just obstacles that are already in your path.

Tesla vs. the Industry

The central issue of the whole presentation was Tesla’s decision to avoid both LIDAR and maps, a decision different from almost all major teams out there. (Tesla does use maps, but not detailed high-definition maps as used by other companies.)

The non-use of LIDAR by Tesla has been controversial. Musk’s view that LIDAR is a crutch is a minority view, but they made a better case for it than has been made in the past. I have a more detailed article on this central question of cameras vs. LIDAR which goes into these issues. In essence:

1. LIDAR sees the same regardless of lighting conditions, camera views change greatly based on night/day, weather, and the position of the sun.
2. LIDAR sees in true 3D, cameras require software to understand the scene to figure out where things are in 3D.
3. LIDAR sees the world at a much lower resolution and shorter range
4. LIDAR is much more expensive, but dropping quickly in price. It is not available in quantity and quality levels today, except Waymo. The cameras are very cheap.
5. The computer vision needed to make cameras work is not yet reliable enough for self-driving. People hope the breakthroughs to make that happen are around the corner.
6. LIDAR is not enough for certain situations, such as proper identification of road debris, signals, and more distant situations, and as such extensive computer vision is definitely needed.

Tesla Network

Elon Musk presented the future Tesla network, which I will write about in more detail tomorrow. You will be able to specify times and rules about how your car can be used by others. Some initial points:

Tesla projects current cars will last one million miles. 2018 battery packs last 300K to 500K miles. By 2020 they will also last 1M miles by getting to 4,000 cycles.

1. Tesla predicts the operating cost of a Model 3 robotaxi at just 18 cents.
2. “Snake” charger will allow recharge with no human.
3. 25%-30% cut for Tesla, 65 cents/mile profit per mile (including 50% empty miles, which seems high.)
4. 90,000 miles of taxi service per year (NY Taxis do about 62K miles per year.) 11 Year life, $30K profit per year. A total present value of $200,000 was estimated by Musk.
5. Tesla’s contract forbids the use of Tesla cars in any other ride-hail network.
6. Eventually, after the steering wheel is removed and capped, car cost drops to $25K per year.

Tesla has promised that eventually, they will create a ride-hail service (similar to Uber in appearance) where the private cars of Tesla owners provide the rides in autonomous modes, making money for that owner. Ie. you would declare that you don’t need your car for the next 5 hours, and it would join the network, and scoot off to give rides, then return to you. They have predicted this may be available in just 3 years, and it will cause each Tesla to become much more valuable because of its ability to make money.

It is uncertain how many people would want to do this, or how many will keep their car in the state where it could be dispatched on short notice to serve somebody. (Many people keep things in their cars and don’t want the battery suddenly depleted.) For those who do, the car will, of course, incur costs and depreciation which estimates and calculations suggest is around 37 cents/mile but which Tesla predicts could be 18 cents/mile with their car. They are predicting a net cost of $1/mile (half of Uber) but have not come to final conclusions.

Tesla is extremely dedicated to this idea. In fact, Musk declared that they will now be pushing customers to buy the lower-end “Standard Plus” Model 3 rather than the long-range Model 3 because they are limited in how many cars they can sell by how many batteries they can make. If they sell smaller batteries, they sell more cars, and that means more cars in the future to go into their robotaxi service. Musk was asked how much Tesla was spending on Autonomy and he replied “It’s basically our entire expense structure.” This suggests they are really betting the farm on this plan.

“The advantage that Tesla will have is that we’ll have millions of cars in the field with full autonomy capability and no one else will have that.”

— Elon Musk

Throughout its journey, AI and Big Data have remained steady partners of the firm. Tesla has taken excellent use of AI and Big Data for expanding its customer base. The firm has made use of existing customer databases for its data analytics using it to comprehend customer requirements and regularly update its systems accordingly.

Conclusion

In the near future scenario where autonomous cars are widespread, these networks will most likely also interface with cars from some other manufacturers as well as other systems such as road-based sensors, traffic cameras, purge light-up masks, or smartphones.

So, sure these self-driving cars already exist, but are they ready for prime time? Perhaps not yet, since the vehicles are currently required to have a driver present for safety. So despite exciting developments in this new field of automated transportation, the technology isn’t perfect yet. But give it a few months or years, and you’ll probably want to have one of these cars yourself.

Thank you for your time!