C Programming Language | Brian Kernighan and Lex Fridman

So what's to you, so you wrote a book, C Programming Language, and C is probably one of the most important languages in the history of programming languages, if you kind of look at impact. What do you think is the most elegant or powerful part of C? Why did it survive? Why did it have such a long-lasting impact? I think it found a sweet spot of expressiveness, that you could really write things in a pretty natural way, and efficiency, which was particularly important when computers were not nearly as powerful as they are today. You've got to put yourself back 50 years, almost, in terms of what computers could do, and that's roughly four or five generations, decades, or more is law, right? So expressiveness and efficiency, and I don't know, perhaps the environment that it came with as well, which was Unix. So it meant if you wrote a program, it could be used on all those computers that ran Unix, and that was all of those computers, because they were all written in C, and that way, Unix, the operating system itself, was portable, as were all the tools. So it all worked together, again, in one of these things where things fed on each other in a positive cycle. What did it take to write sort of a definitive book, probably definitive book on all of programs, like it's more definitive to a particular language than any other book on any other language, and did two really powerful things, which is popularized the language, at least from my perspective, maybe you can correct me, and second is created a standard of how this language is supposed to be used and applied. So what did it take, did you have those kinds of ambitions in mind when working on that? Is this some kind of joke? No, of course not. So it's an accident of timing, skill, and just luck. A lot of it is, clearly, timing was good. Now, Dennis and I wrote the book in 1977. Dennis Ritchie. Yeah, right, and at that point, Unix was starting to spread. I don't know how many there were, but it would be dozens to hundreds of Unix systems, and C was also available on other kinds of computers that had nothing to do with Unix, and so the language had some potential. And there were no other books on C, and Bell Labs was really the only source for it, and Dennis, of course, was authoritative because it was his language, and he had written the reference manual, which is a marvelous example of how to write a reference manual. Really, really, very, very well done. So I twisted his arm until he agreed to write a book, and then we wrote a book. And the virtue, or advantage, at least, I guess, of going first is that then other people have to follow you if they're gonna do anything. And I think it worked well because Dennis was a superb writer. I mean, he really, really did, and the reference manual in that book is his, period. I had nothing to do with that at all. So just crystal clear prose, very, very well expressed. And then he and I, I wrote most of the expository material, and then he and I sort of did the usual ping-ponging back and forth, refining it. But I spent a lot of time trying to find examples that would sort of hang together and that would tell people what they might need to know at about the right time that they should be thinking about needing it. And I'm not sure it completely succeeded, but it mostly worked out. Mostly worked out fairly well. What do you think is the power of example? I mean, you're the creator, at least one of the first people to do the Hello World program, which is like the example. If aliens discover our civilization hundreds of years from now, it'll probably be Hello World programs. Just like a half-broken robot communicating with them with a Hello World. So what, and that's a representative example, so what do you find powerful about examples? I think a good example will tell you how to do something and it will be representative of, you might not want to do exactly that, but you will want to do something that's at least in that same general vein. And so a lot of the examples in the C book were picked for these very, very simple, straightforward text processing problems that were typical of Unix. I want to read input and write it out again. There's a copy command. I want to read input and do something to it and write it out again. There's a grab. And so that kind of find things that are representative of what people want to do and spell those out so that they can then take those and see the core parts and modify them to their taste. And I think that a lot of programming books that I don't look at programming books a tremendous amount these days, but when I do, a lot of them don't do that. They don't give you examples that are both realistic and something you might want to do. Some of them are pure syntax. Here's how you add three numbers. Well, come on, I could figure that out. Tell me how I would get those three numbers into the computer and how we would do something useful with them and then how I put them back out again, neatly formatted. And especially if you follow that example, there's something magical of doing something that feels useful. Yeah, right. And I think it's the attempt, and it's absolutely not perfect, but the attempt in all cases was to get something that was going to be either directly useful or would be very representative of useful things that a programmer might want to do. But within that vein of fundamentally text processing, reading text, doing something, writing text.


GPT-3 vs Human Brain

The human brain is at least 100 trillion synapses, and it could be as high as 1,000 trillion. And a synapse is a channel connected to neurons through which an electrical or chemical signal is transferred, and is the loose inspiration for the synapses, weights, parameters of an artificial neural network. GPT-3, the recently released language model from OpenAI that has been captivating people's imagination with zero-shot or few-shot learning, has 175 billion synapses or parameters. As mentioned in the OpenAI paper, the amount of compute that was used to train the final version of this network was 3.14 times 10 to the 23rd flops. And if we use reasonable cost estimates based on Lambda's test of U100 cloud instance, the cost of training this neural network is 4.6 million dollars. Now the natural question I had is, if the model with 175 billion parameters does very well, how well will a model do that has the same number of parameters as our human brain? Setting aside the fact that both our estimate of the number of synapses and the intricate structure of the brain might require a much, much larger neural network to approximate the brain, but it's very possible that even just this 100 trillion synapse number will allow us to see some magical performance from these systems. And one way of asking the question of how far away are we, is how much does it approximately cost to train a model with 100 trillion parameters? So GPT-3 is 175 billion parameters and 4.6 million dollars in 2020. Let's call it GPT-4 Hb with 100 trillion parameters. Assuming linear scaling of compute requirements with respect to number of parameters, the cost in 2020 for training this neural network is 2.6 billion dollars. Now another interesting OpenAI paper that I've talked about in the past, titled Measuring the Algorithmic Efficiency of Neural Networks, indicates that for the past seven years the neural network training efficiency has been doubling every 16 months. So if this trend continues, then in 2024 the cost of training this GPT-Hb network would be 325 million, decreasing to 40 million in 2028, and in 2032 coming down to approximately the same price as the GPT-3 network today at 5 million. Now it's important to note as the paper indicates that as the size of the network and the compute increases, the improvement of the performance of the network follows a power law. Still, given some of the impressive Turing test passing performances of GPT-3, it's fascinating to think what a language model with 100 trillion parameters might be able to accomplish. I might make a few short videos like these focusing on a single simple idea on the basics of GPT-3 including technical, even philosophical implications, along with highlighting how others are using it. So if you enjoy this kind of thing, subscribe, and remember, try to learn something new every day.