In a recent YouTube interview with Lex Fridman, Andrej Karpathy, the former AI lead at Tesla, spent some time talking about software 2.0 and why it has been so important to Tesla. But the effect the effects of Software 2.0 can and likely will be felt far outside of Tesla itself. In fact Software 2.0 could mark a sea change in the very nature of software development!
Writing software the Software 1.0 way has massive upside, as is obvious from the immensely computer rich world in which we live today
My video take on Karpathy's chat with Lex Fridman
First, in order to understand Software 2.0, let’s examine Software 1.0. Software 1.0—or simply software to most of us—is the traditional type of coding that programmers do today. You choose a language—like Python, or C++, or Swift—you optionally choose an IDE (Integrated Development Environment) in which to write your code, and then you use your brainpower to actually write the code within the confines of the syntactic needs of the given language. You can think of an IDE as something like Microsoft Word for coding: it isn’t necessary (for simple programs I’ve used a basic text editor in the past) but it makes the process of coding and debugging much easier, so people generally use them. Writing software the Software 1.0 way has massive upside, as is obvious from the immensely computer rich world in which we live today.
Dr. Andrej Karpathy, former Head of AI at Tesla
There are however some significant problems with Software 1.0. First, it requires highly trained humans to be able to do the programming task. One caNeural Networkot simply say “hey computer, create a search engine that scours all the websites on the internet.” Rather many many (many) humans have to figure out each exact step in the sequence needed to do this task, and then imagine how one would convert that to a computer language’s syntax, and then implement that in the code itself… and then debug and error correct when inevitable problems occur. And in the end, when the code is successfully created, computers can still only do exactly what we humans tell them to do. This process is not only tedious but limited by our imagination and extremely deterministic as well. To be a good coder, one has to spend years training (either at school or on one’s own), then one has to have the ability to imagine how to translate a desire into exact computer language, and then in the end if one is successful the program is an unchanging artifact: it will produce the same results over and over again.
Dall-e 2 is as creative as many humans now
Software 2.0, according to Dr. Karpathy, is based on Neural Networks rather than traditional coding
What, then, is Software 2.0, and how does it change this time tested and highly successful coding paradigm? Software 2.0, according to Dr. Karpathy, is based on Neural Networks rather than traditional coding. To be clear, Neural Networks have been created using traditional coding techniques, but now that we have multiple architectures on which to build them (PyTorch and Tensorflow being two very popular ones) we can begin to consider how to move to Neural Networks as a basis for “coding” rather than the way it’s been done since the dawn of the computer age. As Dr. Karpathy noted, many people believe Neural Networks are just another Machine Learning or AI technique, but he believes they provide a fundamentally different way of building software itself.
Tesla’s Full Self Driving is one great example, and where the whole Software 2.0 concept arose
Why make this change? Hasn’t Software 1.0 done fine by us and isn’t it still useful? Certainly yes to both of these, and Software 1.0 will be with us for a long time to come. But the twin issues of limited human ability to translate problem solutions into code, plus deterministic output of that code, mean that there is a massive opportunity for a different paradigm to have a huge impact on problems that are extremely difficult to solve using older methods. Tesla’s Full Self Driving is one great example, and where the whole Software 2.0 concept arose: at the start, Elon Musk, Andrej Karpathy, and the team at Tesla AI were using traditional hard coding methods to solve the autonomous driving problem. The problems with this approach are that the world is immensely complex, and thus human imagination and ability to solve every problem in code fails, and when code is created, each bit of code can only do what it was programmed to do—there is no flexibility.
As Neural Networks can theoretically learn any function, they can (at least in theory) learn to operate in the real world, with all the crazy diversity that entails. In addition, Neural Networks can deal with “fuzzy” data: changing lighting conditions, grainy images, variations in signage, etc. This non-determinism is very very important to solving real world problems. The challenges for humans can then move from coding EVERY possible scenario by hand (which is basically intractable) to creating a robust Neural Network architecture and then collecting, cleaning, and utilizing massive data sets to train the Neural Networks.
We are shifting from creating each painting (or algorithm) by hand—being the artists—to collecting the amazing paintings already existent (data from the real world) and putting them in a complex building—being curators
As an analogy, consider this: we wish to create an art museum. We can either move into a simple building (a programming language), then paint each painting by hand (hand coding solutions) and hang them around the gallery (Software 1.0) or we can build an architecturally complex building (a Neural Network architecture) and go out into the world and find as many amazing paintings (real world data) as we can to fill the building (Software 2.0). In other words we are shifting from creating each painting (or algorithm) by hand—being the artists—to collecting the amazing paintings already existent (data from the real world) and putting them in a complex building—being curators. Even better if we do it right, the museum building puts all the paintings in just the right place to show them off all by itself. This shift in priorities from artist to curator has many repercussions, including emotional ones: humans who are excellent coders might find the shift disturbing as their hard won skills might be less necessary in the future. In fact, during their conversation you could tell that both Fridman and Karpathy were nervous with the potential for coding skills to be less necessary in the future. This reticence to a potential future gets to the heart of why disruption usually occurs from the outside rather than the inside, but that’s the topic for another post!
The democratization of programming... will fundamentally change the nature of our interactions with computers and with the world itself
This potential reduction in need for traditional coders actually follows suit in the way technology has displaced highly specialized skills—for example, type layout artists—with simple to use software that unskilled people utilize every day. Could we see coding follow this pattern in the future? I say yes, and it will have a similar effect to the way previous technology shifts have worked. The type layout person, for example, was a skilled artisan and few had the skills or equipment to do that job. Today we have many page layout programs on our computers, and even online on the internet, and most people can use these tools to create a more beautifully laid out world. In other words, these programs have democratized the page layout process. What could happen to programming? The same thing: we could go from a relative few people who can code well to excellently to a world in which many of us with little skill can create programs that do useful things.
Check out this blog post about machine based creativity
Imagine in the future you say “hey computer, create a search engine that scours all the websites on the internet,” and the computer generates and trains a neural network that does just this—no coding required on your part! Suddenly the domain of the few (search engine companies in this case) becomes the domain of the many (those interested in creating a search engine). The democratization of programming beyond the limited subset of those with years of experience and highly developed skills to the more general masses will fundamentally change the nature of our interactions with computers and with the world itself.
The limitation then will be not what we can code, but what we can imagine can be done, and as Han Solo famously said, “I can imagine a lot!”
This vision of Andrej Karpathy is something to get excited about, isn’t it?!