AI Researcher, #101
Joscha Bach is, per Wikipedia, an AI researcher and cognitive scientist. A German guy, he spends the first few minutes of this interview talking about his childhood as a nerd, and then segues into philosophy. Lex says he really enjoys talking to Joscha and will probably interview him again, and indeed he does, over a year later, in podcast #229, I think.
<iframe width="560" height="315" src="
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
Reverse engineering Joscha Bach
Nature of truth
Original thinking
Sentience vs Intelligence
L: To stay for one more time in the early days, when, for you, was the dream to understand or create human-level intelligence born for you?
J: I think you can see AI today as largely advanced info. processing. if you changed the acronym AI into that, most would be happy. We’re automating statistics; statistical models are more advanced. It’s pretty good work, and productive. The other aspect is philosophical project. It’s very risky, very few work on it, and it’s unclear if it succeeds.
L: You used the term “info. processing,” just info. processing, as if it’s muck of existence. The entirety of the universe might be info. processing. Maybe you can comment on, if the advanced info. processing is a limitation. … Second, what do you mean by the philosophical project.
J: I suspect general intelligence is the result of trying to solve general problems. Second, intelligence is the ability to model. To be presented with a number of patterns and see a structure in those patterns, and bea ble to predict. … it’s the ability to make models … So as a child, for instance, you notice you do certain things despite perceiving yourself as wanting different things. You become aware of your own psychology. You need to model yourself, reverse-engineer yourself, in order to predict how you’ll react to certain situations. If you reverse-engineer yourself, if you go all the way—this is basically the project of AI. .. T/he Touring test; you ask a system, ‘What is intelligence?’ If the system can answer, you should assign it a property of being intelligent in this sense. … touring didn’t express it in the original 1950 paper … In order to understand this test, you’ve got to be intelligent enough yourself. .
L: ‘so hidden inside the Touring test is a recursive test
J: Yes, it’s a test on us. Whether people are intelligent enough to understand themselves.
L; The project of AI; do you think emergent self-awareeness is a fundamental aspect of intelligence … Is coming to grips with the idea you’re an agent in the world.
J: Many intelligent people are not self-aware. Inteligence and self-awareness are not the same thing. … You don’t need
L: You said self-awareness seemed important to children
J: … I called it sentience. I’d distinguish it from intelligence; sentience is possessing certain classes of models. Intelligence is a way to get to these models if you don’t already have them.
L: Can you try to answer the maybe-unanswerable question of “what is intelligence?”
J: I think it is the ability to make models.
L: NNs, very popular now, form representations of a large-scale data set. When you say models, and look at today’s NNs, can you compare? In saying intelligence is the process of making models?
J: One is, the representation—is the representation adequate for the domain we want to represent? Second, is the type … you want to arrive at adequate? In both of these cases, I think modern AI is still lacking. Not criticism. Most of the people who designed our paradigms are aware of that. One aspect we’re missing is “unified learning.” … We learn it all into one model, and we call this model the universe. …. Physical reality is aa quantum graph we can never experience or get acces to …
L: I want to disengangle … Can you describe the basics of … dualism, idealism, functionalism, materialism, and what conectis with you. YOu mentioned there’s a reality we don’t have access to? Why not?
J; The trajectory that exists in the West is the result of our indoctrination by a cult for 2000 years; the Catholic cult, mostly. It’s defined modes of interaction we have; it has also, in some sense, scarred our rationality. The intuition that exists if you’d translate the mythology of Cath. church into reality—the world in which you and me interact is a role-playing adventure … the money and objects we have in this world is all not year; Eastern philosophers would call it maya. … The identification with the means of secular, everyday existence. Teh Catholics introdduced the notion of higher meaning, the sacred … the platonic form of the civilization you’re part of .. formed by individuals as a … agent … They implemented software on the minds of people; got the software synchronized to make people walk lockstep; to get this God online and to make it efficient and effeective. God is juts a self that spans multiple brains … So in some sense you can construct a self … a self implemented by brains that exists across brains. This is god with a small “g”
L: This is Yval Havari, talking about … a nice feature of our brains, we can all download the same piece of software.
J: You give everybody a spec, the mathematical constraints … you come up with a compatible structure
L: So ideas we all share, that’s “the mind” … that’s separate from Christianity—there’s a separate thing …
J: There’s a real world … the world in which god exists; god is the coder of the multiplayer adventure; we are all players in this game
L: That’s dualism
J: That’s because the mental realm exists in a different implementation than the physical. The mental realm is real. A lot of people have inutiion that there’s this real room in which we talk and speak right now. Then comes layer of physics … then another realm where are souls are … This of course is a very confused notion. It’s the result of connecting materialism and idealism in the wrong way
L: I apologize; I think it’s helpful if we try to define terms. What is dualism? Idealims? Materialism?
J: Dualism—there’s two substances; a mental, and a physical. Tehy interact by different rules. The physical world is causally closed; built on low-level causal structure. That’s the mechanical level; it’s computational. There’s a physical world where info flows around; physics describes laws by which info flows around. A computer is a generalization of info. flowing around. Touring discovered universal principal; you can describe universal machine that can do all the computations … So all these machines have the same power; you can always define translation between them as long as they have unlimited memory; they can preform each other’s computations
L: Materialism—this whole world is just the hardware; idealism—the whole world is just the software?
J: Not necessarily; Most idealists don’t have a notino of software. Softrware comes down to info processing. What you notice is the only thing that’s real to you and me is the experiential world of taste and color
L: You are bringing up consciousness
J: This is distinct from physical world, where things have value only in an abstract sense, where you only have cold patterns moving around … no feelings … idealims starts out with mind primary; materialism thinks matter is primary. For Idealists, the material patterns we see playing out are part of dream the mind is dreaming; we exsit in mind on higher plane of existence. For materialists, there’s only this material thing; that generates some models, and we’re the result of these models. In somes sense, I think irf we understand it porpertly .. they’re two different aspects of the same thing. Weird thing—we don’t existt in the physical world; we exist inside a story the brain tells itself.
L: (mutters) …
J: Basically your brain can’t feel anything; neurons can’t feel; physcial systems can’t experience anything. But it would be useful for the brain to know what it would be like to be a person, to feel something. So the brain cereates a simulacrum of such a person; it uses it to omodel the interactions of the person; it’st the best model of whtaa that brain thinks it is in relation to its environment. It’s a story, a novel, the brain is continuously writing and updating
L: You said we exist in that story. What is real in any of this? These terms … you said there’s a “quantum graph” … Is the story, impossible to get access to, fundamentally? Isn’t the brain in something? Existing in some kind of context?
J: As CS-ists, we can engineer systems and test our theeoires this way; that might have the necesssary and sufficient properties to produce pheenomena we’re observing; there’s a self in a virtual world generaied in somebody’s neurocortex …
Mind vs Reality
J: … that is contained in the skull of this primate. When I point at this .. I do create something that’s likely to give rise to patterns on your retina that allow you to interpret what I’m saying. But what we are saying is not the real, physical world. What we’re seeing is a virtual reality generated in your brain to explain the patterns on your retina?
L: How close is it to the real world? Donald Hoffman says you’re really far away; that interface we have is very far away from anything—we don’t have anythign close. Or is it a very surface piece of architeutre?
J: imagine look at the Mandebrot fractal? If you see an overall shape in there, but if you truly understand it, you know it’s two lines of code. It’s a serious that’s being tested for complex numbers … testing in complex numbers for every point; if the series is divering, you paint it black; if converging, you don’t. You get intermediate colors by taking how far it diverges. That tgives you shape of this fractal. Imagine you live inside the fracgtal and have not disccovered the generator function … what you see is—all I can see is the spiral … Is this an accurate model of reality? Yes. You know there’s no spiral in the fractal; it only appears that way to an observer (from a certain perspective). So .. at this level, you have the spiral … at osme point it disappears. At this point your model is no longer valid. You observe again, you hit another spiral … If you make 30 layers of these laws; then you come close to what we have when we describe the reality around us. … It’s reasonably rpedictive; it doesn’t cut to the core of it; nor eexplain how it’s being generated. But it’s relatively good to explain the world …
L: But you don’t think the tools could step outside, explain the whole driving, explain the basic mechanism
J: Imagine you would find yourself embeddded into a Mandlebrot fractal … With a touring machine with the ability to think. You come up with the theory that it’s an autonmaton … you enumerate all the possible autnomaton until you get to the one that produces your reality … So you can identify necessary and sufficient conditions; For example we discover math itself is domain of all languages. Most of domain of math we’ve disocovered are describing the same fractals. That’s What category theory is obsessed about … that you can map these domains to each ohter … So they’re not that many fractals. Some have interesting .. symmetry breaks. You can discover what region of this fractal you’re embedded in … using first principles. The only way is from first principles! Your understanding of universe has to start with autonoma, number theory … and so on
L: Wolfram still dreams; he’ll be able to come up with … what’s behind our universe. You’ve said, on this topic, in a recent conversation—”Some people think a simulation can’t be conscious, and only a physical system can; they have it backward. Only a simulation can be conscious. Consciousness is a simulated property of a simulated self.” Like you said, the mind is a story-narrative. So our mind is essentially a simulation.
J: Usually I try to use the terminology so the mind is basically principles that produce the simulation; the software implemetned by your brain, the mind is creating both the universe you’re in and the self. The idea of a person on the other side of attention and embedded in this world.
L: Why is that important, the idea of a self? Why is that a feature in the simulation?
J: It’s basically a result of the purpose the mind has; its’ a tool for modeling; we’re not actually monkeys; we’re side effects of the regulations needs of monkeys. What we need to regulate is the relationship of an organism to an outside world that is in large part consisting of other organisms. As a result, it has regulation targets it tries to get to; these targets start with priors; unconditional reflexes we’re more-or-less born with …. then we can reverse-engineer them to make them more consistent. Then we get more detailed models of how the world works and how to interact with it. These priors you commit to are largely target values that our needs should approach. The deviation to the set point creates some urge, tension. We live inside of feedback loops; consciousness emerges over dimensions of disagreements with the universe; things where you care; things are not the way they should be, you need to regulate. The self is all the identifications you have; the id is the regulation target you’re commititng to; a dimension you decide is important. This locks you in; if you let go of these identifications; you get free, there’s nothing you have to do anymore. If you let go of all of htem, you’re completely free, and can achieve nirvana
L: By the way, thanks to Gustav Soderstrom who turned me on to you; I’m glad the Internet exists and i could find your talks. … You’re describing this emergent phenonmenon of consciousness from the simulation. Can you linger on the hard problme of consciousness? I understand the self is nan important part of the simulation. But why does the simulation feel like something.
J: If you look at a book by George R. Martin where the characters have plausible psycchology … They stand on a hill and want to conquer the city below the hill; they look at color of the sky and are apprehensive. Why do they. have these emotions; because they’re written into the story? Wh written into the story …it’s an adequate model of the person that predicts what htey’ll do next .Same is true for us; it’s a story the brain is writing; it’s written not in words but basically in multimedia content. It’s a bottle of what the person would feel if it existed. It’s a virtual person. You and me happen to be these virtual persons … If this v.p. gets access to language center and talks about the sky being blue—this is us.
L: Do I exist? In your simulation.
J: Yes, in a similar way as me. Tehre are internal states that are less accessible … to me … that you have. And so on. And my model might not be adequate; there are alos things I might perceive about you that you don’t perceive. But in some sense, you and I are two puppets that enact a play in my mind; I identify with one of them because I can control one of the pupets directly. With the other, I can create things in between; we can go on an interaction that leads to a coupling, a feedback loop. We can think things together or feel things together …. in a certain way. But this coupling is not physical; it’s entierely a software phenomenon—two different implementations interacting with each other
L: Are y ousuggesting—the entire existence is a simulation, and each mind is a sub-simulation … Why doesn’t your mind have access to my mind’s full state?
J: For the same reason that it doesn’t have access to its own full state. There is no trick invovled. When I know something about myself, it’s because I made a model; one part of your brain is tasked with modeling what other parts of your brain are doing.
L: But there seems to be an incredible consistency about this world; in the physical sense. It does repeatible experiments and so on. How does that fit into our simulation? Why is so much so repeatable, fundamental physics experiments, for a long time, all over the place—the laws of the physics
J: It seems parts of the world that aren’t deterministic aren’t long-lived. If you build a system, any kind of automaton … if you build simulations of something, you’ll notice, the phenomenon that endure, are those that give rise to stable dynamics. IF you see anything that is complex in the world, it’s the result of some control of feedback that keeps it stable around certain attractors. And the things that are not stable that don’t give rise to certain harmonic patterns; they tend to get weeded out over time. If we’re in a region of the universe that sustains complexity, required to implement minds like ours, this is going to be a region of the universe that is tightly controlled and controllable. It’s going to have lots of symmetries and symmetry breaks, that allow the creation of structure.
Hard problem of consciousness
Connection between the mind and the universe
L: … My question is, to try to understand how that fits with the entirety of the universed; you’re saying there’s a region that allows enough complexity to create creatures like us. What connection between the brain, the mind, and the broader universe? Which comes first ;which is more fundamental? Is the mind the starting point and the universe emergent; or is the universe the starting point and the mind emergent?
J: I think quite clearly the latter; it’s an easier explanation; it allows us to make causal models. And I don’t see any way to construct an inverse causality
L: So what happens when you die, to your mind’s simulation?
J: My implementation ceases; the thing that implements myself will no longer be present. If I am no longer implemented upon the minds of other people, the thing that I identify with … good thing is, I don’t actually have an identity beyond the one that I construct. If I was the Dali Lama, he identifies as a form of government. So the DL gets reborn, not because he gets confused, but because he is not identifying as a human; he runs on a human; he’s basically governmental software that’s instantiated in every generation anew; his advisors will pick someone who does this in the next generaiton. If you identify with this, then you are no longer human; you don’t die in a sense; what dies is only the body of the human that you run on. to kill the DL, you’d have to kill his tradition. If you look at ourselves … if you have childrene, you realize something lives on in them … or if you spark an idea in the world, or if you identify with a society around you. Because you are in part that, you are not just this human being.
L: In a sense, you are kind of like a DL, in the sense that you, J Bach, is just a collection of ideas. You’re an OS on which these ideas live and interact. Once you die, some of them .. jump off the ship
J: Put it the other way; identity is a software state; it’s a construction .IT’s not physically real; it’s not a physical concept. It’s basically representation of different objects on the same world line
L: But identity lives and dies? What’s the fundamental thing. Is it the ideas that ocme together to form identity; or is each individual identity actually fundamental
J: It’s a representation you can get agency over if you care; you can hcoose what you identify with if you want to.
L: It seems if the mind is not real, then that birth and death is not a crucial part of it. Maybe I’m silly; maybe I’m attached to this biological organism; but it seems that the physical being, being a physical object in this world, is an important aspect of birth and death. Feels like it has to be physical to die. Feels like simulations don’t have to die.
J: The physics that we experience is not the real physics. There experiences no color and sound in the real world. C & S are types of representations that you get if you want to model reality with ___ data. Colrs and sounds ahve octaves; because they are represented by osdcillators. That’s why colors form a circle of hues. And colors have harmonics, sounds have harmonkcs, as a result of synchronizing oscillators in the brain. the world we subjectively interact with is fundamnetally the result of the represtnation mechanism in our brian. They’re mathematically to some degree universal; they are certain regularities you can discover in the patterns and not otehrs. But the patterns that we get; this is not the real world. The world that we interact with is always made of too many parts to count. When you look at this table; it has so many atoms and molecules; you cannot count them. So you only look at aggregate, limit dynamics. IF you had almost infinitely many particles, what would be the dynamics of the table; this is roughly what you get. The geometry you interact with is the reuslt of discovering those operators that work on the limit, that you get by building an infintie series that converges. For those parts where it converges that’s geometry; for where it diverges, that’s chaos.
L: So all of that is filtered through consciousness that’s emergent in our narrative. That gives it color, feeling, flavor.
J: I think feeling, flavor and so on is given by the relationship that a feature has to other features. A giatn relational graph that is our subjective universe. Color is given by those aspects of representation … the expreiential color that you tcare about. Wjhere you have identifications; where osmething means something; your’e on the inside of a feedback loop. Dimensions of caring are basically dimensions of this motivational system that we emerge over
L: The meaning of the relationships—can you elaborate? Maybe even step back and a..
What is consciousness
J: I think that consciousness is largely a model of the contents of your attention. IT’s a mechanism that has evolved for a certain type of learning. At the moment, learning systems largely work by building chains of weighted sums of real numbers, with some nonlinearity. … You learn by piping in —- signals, of these different chained layers, and adjusting the weights of weighted sums. You can approximate most polynomes with this, if you have enough training data. But the price is, you need to change a lot of these weights. The error is piped backward into the system until it accumulates at certain junctures in the network. Everything else evens out statistically; only at these junctures, where you have actual error in the network, you make the change there. This is a very slow process; our brains don’t have time for this;… we don’t get old enoughto play go the way machines learn to play go. What we have is n attention-based learning. We pinpoint the probable region in the network wehre we can make an imporvement … then we store this binding state together with expected outcome in the protocol. This ability to make index memories for the purpose fo learning to revisit these commitments later, this requires … memory of the contents of our attention. Another aspect is when I construct my reality I make misteakes. I see things that utern out to be reflections or shadows … which means I have to be able to point out which features of my perception gave rise to the present construction of reality. The system needs to be able to pay attention to the features that are currently in its focus. It also needs to pay attention to whether it pays attention itself. In partt because the atention mechanism gets trained … with the same mechanism, so it’s reflexive. But also in part because the attention lapses if you don’t pay attention to the attention itself. So is this thing I’m currently seeing just a dream my brain has spun off into some sort of daydream? Or, am I still paying attention to my precepts. You have to periodically go back and see if you’re still paying attention. If you have this loop and you make it tight enough, between the system becoming aware of the contents of its attention and the fact that it’s paying attention itself, makes attention the object of its attention—I think this is the loop in whiich wwe wake up.
L: So there’s this attentional mechanism that’s somehow self-referential that’s fundamental to what consciousness is.
Language and concepts
L: NLP, they use transformers, to learn patterns and sentences by allowing the network to focus attention to particular parts of the sentence … parameterize and make it learnable, the dynamics of a sentence, by having a little window into the sentence. Do you think that’s a little step … that will take us to … the intentional mechanism from which consciousnes scan emerge?
J: Not quite. I think it models only one aspect of attention. In the early days of automated language translation, there was an example … funny … Someone tried to translate a text from English into German … there was a passage, “a bat broke the windwo.” The translation in German was lksdfjlksdjf … to translate back into English was, “the flying mammel broke the window with a baseball bat.” It seemed to be the most simlar to this program; itt somehow maximized the possibility of trnaslating the concept “bat” into German in the same senteence. IT’s a mistake that the transformer model is not doing; it’s tracking identity. The attentional mechanism in the transformer model is putting its finger on individual concpets, and make sure that these concepts pop up later in the text. And tracks basically the individuals through the text; it’s why the system could learn things that other systems couldn’t before it. Which makes it possible to write a text where it talks about the scientist, and the scietnist has a name, and gives it a pronoun, and tells a consistent story about that thing. What it does not do; it doesn’t fully integrate this; its meaning falls apart at some point… It does not yet understand that everything that it says has to return to the same universe. This is where it falls apart. What the atention in the transfomrer model does not go beyond tracking identity. That’s an important part of attention. But it’s a specific attentional mechanism; it’s not the one that gives rise to the type of consciousness that we have.
L: What do you mean by identity in the context of language?
J: So, when you talk about language, we have differnet words that can refer to the same concept. … IT can also be, in teh nominal sense, lexical sense, that you say this world, doesn’t only refer to this class of objects, but to a definite object, sone kind of agent that weaves its way through the story; and only is referred by different ways in the language. The language is basically a projection from a conceptual representation from a scene that is evolving into a discrete stream of symbols. The transsforemr learns aspects of this projection mechanism that other models couldn’t learn.
L: Have you ever seen an AI or any kind of construction idea … unlike NNs, or perhaps within NNs, that’s able to form something where the space of concepts continues to be integrated. Building a knowledge base, building this consistent larger and larger sets of iddeas that would then allow for deeper understanding.
J: Wittgenstein thought we could build everything from langauage …. grammatical construct. … That’s why _ focused so much on common-snese reasoning … Project that was inspired by him was Psyche; … of course, ideas don’t die; only people die. .
L: That’s true but …
J: And Alt-psyche is a productive project. Just probably not one that’s going to converge to general intelligence. The thing Wittgeinstein couldn’t solve, he looked at this at the end of his life, was the nottion of images. Images play an important role in Tractados … Tractados an attempt to turn philosophy into logicla programming language. To design a logical language in which you can do actual philosophy. The difficult ywas to deal with perceptual content. Eventually, I think he decided he was not able to sovle it. This preempted the failure of the Logitus program in AI. The solution as we see it today; we need more general function approximation. There are functions, geometric ones, that we learn to approximate, that cannot be efficiently expressed and comptued in a gramamtical language. We can of course build atomata … that go by a number theory … to learn linear algebra, and then compute an approximateion of this geometry. But to equate langauge and geometry is not an efficient way to think about it.
L: The approach NNs takes, is actually more general than what can be expressed through language
J: What can be efficiently expressed through language at the data rates through which swe proces gramattic language.
L: So you disagree with Witgenstein
j: I agree with him; with the late Wittgeinstein. I also agree with the beauty of the early Wittgenstein. The Trattados itslef is probably the most beautiful philosphical text that was written in the 20th century.
L: But language is not fundamental to cognition and intelligence nad consciousness
J: I think langbuage is a particular way; or the natural language that we’re using is a particulra level of abstraciton that we use to communicate with each other. But the languages in which we express geometry are not gramatical language in the same sense. They’re more general expressions of functions. I think the general nature of a model is, you have a bunch of parameters. These have a range; variances of the world. You have relationships between them which are constraints. If certain parameters have these values, then other parameters have to have the following values. And tis is a very early insight in CS; I think some of the earliest formulations is the Boltzmann machine. The problem with the B machine is that it has a measure of whether it;s good; this is basically the measure of energy in the system; the amount of tension that you have left in the constraints when the constraints don’t quite match. It’s difficult to, despite having this global measure, to train it. Because as soon as you add more than trivially few elements, parameters into the system, it’s very difficult to get it settled into the right architecture. The solution that Hinton and Senovsky found was to use a restricted B machine that which uses the hidden links in the B machine, with only an input and output layer. This limtis the expressivity of the B machine; so now he builds this network of smaller, primiteve B machines. In some sense you can see almost continuous development from this to the DL models that w’ere using todayu. Even tho we don’t use B machines at this point. The idea is you use take this model ; you clamp some of the values to perception. This forces the entire machine to go into state that is compatible to the states that you currenetly proceed. And this state is your model of the world. I think it’s very general way of thinking about models. We have to use a different approach to make it work. We have to find diferent networks that train the B machines. The mechainsm that trains the B mahcine and the mechanism that makes the. B machine settle into its state … …
The direction I think our research is going to go …what you notice in perception is, our models of the world are not probabilistic but possibilistic … which means, you should be aable to perceive things which are improbable but possible. A perceptual state is valid … if it’s possible. If you see a tiger coming after you, you hsould be able to see it, even if it’s unlikely. And, the probaibliy is necessary for convergence of the model. Given the state of possiblities is very large, and the set of perceptual features, how should you hcange the states of the model to get it to converge with your perception?
L; The space of ideas which are coherent with the context that you’re sensing, is perhaps not as large. That’s perhaps pretty small.
J: The degree of coherence you need to achieve depends how deep your models go. For instance, politics is very simple when you know very little about game theory and human nature … You get a coherent statiscs from relatively few inputs. The more layers you model reality, the harder it gets to satisfy all the constraints.
Meta-learning
L: The current NNs are primarily supervised learning systems with feedforward NN, that use backpropagation to learn. What’s your intuiton about what kind of mechanism might we move towards to imporve the learning procedure
J: I think one big aspect is going to be meta-learning. In some sense the first wave of AI .. identified possible problem, identified a solution, and implemented the solution. The second wave, instead of writing the algorithm that implements the solution; we write an alagorithm that automatically searches for the algoreithm that implements the solution. The learning system in some sense is an alg. that itself just covers the alg. that solves the problem like Go. Go is too hard to implement the solution by hand, but we can implenet a n alg. that finds the solution. NOw let’s move to at teh third stage; thee third stage would be meta-learning. Find an alg. that just covers the learning alg. for the given domain. AOUr brain is problably not a learning system but a meta-learning system. This is one way of looking at what we’re doing. Another way, if you look at the way our brain is implemented; there is no cetnral control that tells the neurons how to wire up. Instead every neuron is an individual, reinforement learning agent. .. quite motivated to get fed; it gets fed if it fires on average at the right time. The right time depends on the context that the neuron exists in, which is the eletrical and chemical environment in which it lives … It has to learn a function over its environemtn. .that tells it when to fire to get fed. If you see it as an RL learning agent, every neuron is in a sense making a hyupothesis when it sends a signal … And tries to pipe a signal through the universe. ..and get positive feedback through it. And teh entire system is set up .. so that it’s robustly self-organizing into a brain. Which means you start out with different neuro types … you put them in different concentraitons … in different alignment; sas a result you develop a nice brain.
L: Okay a brain is a nice meta-learning system with a bunch of RL agents. Just to clarify, there’s no centralized govenrment that tells you, here’s a loss function, here’s a los function, here’s a los function
J: Also govenrments which imposee loss functions on different parts of the brain. {We have difrferential attention; Some parts of your brain get specially awarded if you can tell faces. IF you don’t get that, you get ____ … it’s na extraordinary attention that we have for faces … People with _____ don’t look at faces more than they look at cups. … People that don’t have (this disease) look obsessively at faces. For you and me, it’s imposisble to look through a crowd without scanning faces. And as a reuslt, we make insanely detailed models of faces that allow us to discern mental states of people.
L: Still, we took a leap from soemthing much dumber, to that, through the evolutionary process. Can you maybe say, how big of a leap is that, from our brain, our ape ancestors, to multicell organisms; is there something we can think about … as we start to think about how to engineer intelligence, is there something we can learn from evolution?
J: In some sense, life exists because of the market opportunity of controlled chemical reactions. We compete with dumb chemical reacitons; we win in some areas against dumb combustion; we can harness those entropic gradients where you need to add energy in a specific way to harvest more energy
L: So we outcompete combustion
J: Yes, we try very hard. When we aren’t in competition we lose. The combustion is going to close the entropy gradients much faster than we can run. We do this because every cell has a Trouing machine built into it; it’s lieke read/write head on a tape. Everything that is more complicated than a molecule … a vortex around attractors that has a Toruuing machine in ti for its regulation. Then you bind cells together and you get next-level roganization; where the cells together implement some kind of sofwtware. For me, an interesting discovery in the last year is the word “spirti.” What “spirit” actually means is an operating system for an autnonoous robot. And when the word was invented, people needed this word; tbut they didnt’ have robots that they built themselves yet. The only robots that were known were people, animals, plants, ecosystem,s cities, and so on—and they all had spirits. and it made sense to say that a plant is an OS; if you pinch a plant in one area, it’s going to have repercussions throughout the plant. Everything in the plant in some sense is connected into some global aesthetics, like in other organisms. An organism is not a collection of cells; it’s a function that tells cells haw to behave. The function is not implementned like some supernatural thing. It’s an emergent result of interactions of each cell with each other cell.
L: Ohjmygod… you’re syaing the organism is a function that tells cells what to do; and the function emerges from the interaction of the cells.
J: Yes. It’s basically a descirption of what the plant is doing in terms of micro-states. The micro-states, the physical implementation, therre are too many of them to describe them, so the software that we use to descrigbe what the plant is doing, the spirit of the plant is the software, the OS of the plant. … This is aw way we the observers make sense of the plant. … Same for people; people have spirits; we have OS in a way; aspects of that which relate to how the body functions, social interactions … how you interact with yourselrf and so on. We make models of that spirit; we think it’s a loaded term, because it’s from a pre-scientific age. But it took the scietnifi c age a long time to re-disoocver a term, that is prety m uch thesame thing. I suspect the differences we still see are translation errors …
L: Can you linger on that. The word “spirti” … why did you say in the last year or so you discovered this. You mean the same old traditional idea of a spirti
J: I try to understand what people mean by spirit. When poeple say psiriutality in the US, it usually refers to the phantom lymph that they develop in the absence of culture. culture is in some sense the spirit of a society that is long-gain … this thing htat becomes self-aware at a level above the individuals. Where you say, if you don’t do the following things, the grandchildren of our children will have nothing to eat. So, if you take this long scope, you try to maximize the length of the game your’e playing as a species; you realize you’re part of a larger thing you can’t control, you probably need to submit to the ecosphere instead of trying to completely control it. There needs to be a certain level at which we can exist as a specise if we want to endure. Our culture is not sustaingin this anymore; we basically made bet with Indust. Rev. that we can control everything. Modern societ8ies, and unfettered growth, we depend on the ability to control the entirep lanet. Bedcause we are not able to do that, as it seems, this culture will die. We realize that it desn’t have a future; we call our future Generation Dead; it’s not an optimsitic thing to do.
L: YOu have this ntotion that our civilization, its entirety, may not exist for long. Can you entangele that? What’s your intuition behind that. You hoffhand mentioned to me, that the IR .. was the moment we agreed … we doomed ourseves.
J: It’s a suspicion; I don’t know how it plays out. It seems to me that a society that …you leverage yourself very far across an entropic abyss without land on the other side; it’s relatively clear that your cantilever at some point its goint ot break down into the entropic abyss ..
Spirit
Our civilization may not exist for long
L: So many of the things you say are poetic
J: and mispronounced (laughs)
L: which makes it more poetic. Let’s rewind; the IR, how does that get us into the entropic abyss
J: In some senes we burned 100 million years of trees to get everybody plumbing. .. The society we had before that had very limited number of people. Basically since 0 b.c., we hovered between .3 and .4 billion people; this only changed with Enlgihtenment and IR … the Enlightenemtn freed our rationality it also freed our norms from the preexisting order, gradualy. It was a process that happened in feedback loops; it wasn’t just, one caused the other. The dyanmic worked by increasing productivity to such a degree that we could feed all our children. I thin the definition of poverty .. is that you have as many children as you can feed before they die. Which is the state all animals on earth are in. In our societyies, you can basically have as mnay children as you want, they don’t die. The reason why we don’t have as many chilren as we want … is because we also have to pay a price; we have to insert ourselves … basically anyone in the lower, middle, and upper class has limited number of hcildren .. because it means big economic hit to the family; children are super-expensive to have. You are taken out of this if you are super-rich or super-poor … because in those areas it doesn’t matter …
L: HOw does this lead to self-destruction
J: We try to let our children survive even if they have diseases; I would have died in my mid20s without modern medicine, and most of my friends would have as well. We get our protein alrgely by … ___ the entirety of nature. Imaagine a very clever microbe living in our organisms, that would harvest them and change them … Would discover that brain cells are kind of edible .. and not quite nice l… would turn them into fat cells … Very alive, brittle … when the environment changes
L: some par tof that organism; there would still be somebody thrinving
J: I suspect we’re not the smartest thing on this fplanet. I suspect every complex system has to have complex regulation if it relies on feedback lops. .. We should ascribe some intelligence to plants. Problem is, plants don’t have a nervous system. They rely on chemicals from adjacent cells … Their signal processign speed … if it’s intelligent; it won’t be intelligent at simlar time scales.
L: So you ssupect … we might not be the most intelligent; we’re the most intelligent at our time scale.
J: If you would zoom out very far; you might discover there were intelligent ecosystems on the plantet; possibly they actively related the environment ; changed the course of evolution within … … and made the ecosystem more efficient and less brittle …
L: So plants …as a set of living organisms … could be operating at a different time scale … and human beings will die out, and plants will still be there.
J: Evolutionary adaptation plays role at all these levels … If mice don’t get food; get stressed; the next generation will be more sparse, or more scrawny … If they overgraze, all the things that sustain them might go extinct; so .. to make sure there will be mice in five years … the mice scale back. Same thing with predators of mice; if they’re smart enoguh .. they’re tasked with shepherding their food supply. Maybe lions have larger brains than antelopes because lions need to make complex models of their environment.
L: Just describing that, makes me feel a little betetr about the extinction of human species …
J: Maybe … God’s ploy to put the carbon back into the atompshere. The big stain on evolution was not us, it was trees. Earth evolved trees before they could be digested again; there were no insects that could devour all of them apart … many of these trees fell into swamps; carbon became inert; couldn’t be recycled into organismsl. We’re the species that is desitned to take care of that. .. Take it out of the ground; put it back in the atmosphere, and the earth is already greening. Within a million years ago, when ecosystmes have recovered from the rapid changes, the Earth is going to be awesome again. … I think there will be memories of us. I suspect we’re the first generally intelligent species in this sense: we are the first species . with industrial society. We will leave more phones than bones in the stratsophere.
L: Let me push back. You’ve suggested we have a very narrow definition of intelligence. Why aren’t trees a higher level of general intelligencde
J: Well, if they are, it’s at a different time scalel
L: Maybe the trees are the ones that made the phones
J: The entirety of life—you could say, the first cell never died; it just split. Maybe every cell on our body is an instant of this very first cell. So the cell is more than a building block of life; it’s a hypo organism.
L: Nevertheless .. humans, because of IR< and the exponential growth of technology; we might destroy ourselves. What’s the most likely way we might do that? Some worry about genetic manipulation. Some worry about AI, dumb AI destroying us. Some worry about nuclear weapons. What if you were a betting man?
J: (Laughing) .. It’s very likely nothing we bet on, after we win our bets; so I don’t think bets are the right way to go about it. It’s also not lcear if we as as pecies go exitnct, but our present civilization isn’t sustainable. There will probably be fewre poeple on the planet than there are todya … Even if not, still most of people in our life today will not have offspring years from now, because of geographic changes and changes in food supply. It’s likley many areas of the planet will only be livable with a close-cooling chain … many areas around the equator, and subtropical climates that are now quite pleasant … will become uninhabitable
L: Wow, close-knit cooling chain communities. You have a strong worry about the effects of global warming
J: It’s not a big issue by itself; if you life in Arizoan right now, you have three months in summer whenn you can’t be outside; so you have a close-cooling chain … uou have AC in your house and your car and its fine. If you don’t have AC for three days, you won’t be able to survive.
L: What is that term: close cooling chain?
J: I imagine people use it when you describe how to get meat into a supermarket …
l: That’s a beautiful way to put it. It’s like calling a city, a close social chain
L: It means you live your life in a climatized places. In between you have very short distances hwen you run from your car to the uspermarket; you want to make sure your planet doens’t approach the temp. of the environment. The curcial thing is the wetbot—temperature. Tkae wet clothes, hold it around your thermometer. .. As soon as you cannot cool your body temperature bleow 35 degrees, you die. If the outside world is dry, then you can cool your body by sweating; but if the atmoshere is humid, then sweating will not safe you … Without climatizing equipment .. you will die. This itselkf, as long as you maintain civilization, and you have energy supply, and food trucks coming to your own, it’s fine. But wyat if you lose large-scale open agriculture at the same time—you run into food insecurity .. you have a lot of exreme weather events. You need to grow most of your food indoor; import your food from certain regions … Get the infrastructure to get the food to your home
L: There could be wars over resources. .. but ultimately .. what do you make of the capacity of tech. innovation to help us prevent some of the worst damages this condition could create. For example, the work of SpaceX, trying to also consider our propagation throughout the universe in deep space, to colonize other plantes.
J: Of course, what he’s trying on Mars is not to save us from global warming, since Mars looks much worse .. it’s essentially not habitable.
L: What he is doing .. is a lot of different tech. innovation with some kind of target; and unexpected new things come up. Trying to terraform Mars .. might give us totally new ideas of how to exapnd, or increase the power of this closed cooling circuit that empowers the community. It seems like there’s a little bit of a race between our open-ended tech. innovation … of this communal operating system that we have, and our general tendency to overuse resources, and thereby destroy ourselves. You don’t think tech. can win that race?
J: I think the probability is low; our technology in the US is stagnating since the 19790s;
L: What about intel? What about Moore’s Law?
J: The invention of the microprocessor was a major thing. Miniaturization of transistors. But the things we did after that weren’t that innovative … Scaling CPUs into GPUs .. I don’t htink there are .. many things .if you take a person that diedi nt he 70s they wouldn’t need to read muc hto bee current again.
L: Who cares about books, and papers, and knowledge—that’s a concept of a time, when you were individual consumers of knowledge. What about the impact of social media; the reason you and I are sitting here today, is because of Twiter and YouTube; the ripple effect … two minds, two dumb apes, are coming up with new clean insights. And there’s 200K other apes listneing. That effect .. might be bigger than any advances of microprocessor .. the ability to speread knowledge; that knowledge … it allows good ideas to reach millions, much faster. The effet of that; that might be the 21st cenutyr; the multiplying of ideas … of good ideas. If you say one good thing today; that iwll multiply across huge numbers of people. ..
J: We should have billiions of phenomenons in Tourings; and we don’t, for some reason. I suspect the reason is we destroy our attention span. The reason we are doing this, if you and me don’t have the attention to write a book together right now, and you listening don’t have the attention span to readi t. So let me tell you … in short burst—take care of your atteniton!
L: 80% of the people are still listening. Who said the book is the optimal way to transfer information?
J: That’s something social media could be oding. I think the end game of social media is a global brain. I think that Twitter i s i some sense a global brain that is completely hooked on dopamine; doesn’t have any sense of inhibition, and is caught in a permaennt seizure. It’s also a multilplayer role-playing game. People use it to play an avatar that is not like them; they look through world ithrough lens of their phones; think it’s the real world. But it’s the Twitter world, thwarted through the popularity incentives of Twiter
L: The incentives; and the biological .. dopamine rush, of a like. I try to be Zen-like and minimalist, and not be motivated by likes and so on. Speaking of Twitter;
Twitter and social media
L: How can Twitter be done better … it has a huge impact on society; doing exactly what you describe … it’s some knid of game; we’re individual RL agents in this game; it’s not controllable; there’s not really a centralized control … engineers at Twtiter don’t seem able to control this game. Any hadvice you would give?
J: Our brain has solved this problem to some dgerre; our brain has individual agents that manage to play together. Other organims have … solved problems of cooperation we don’t solve on Twitter. Mahybe a solution is an evolutionary approach. Imagine you have Reedit, Twiter, and Facebook. What they have in comon is they are companies that in some sense own a protocol. A protocol that is imposed on a community … And the protocol has different componetns for … monetization, user management, user display, rating, import of other content, etc. imagine taking components of the protocol apart; in some sense as communities … these communities are allowed to mix and match their prootocols and define new ones … Rules for sharing content can be redefined; monetization can be redefined; the ways users cna represent themselves can be redefined
l: Who could be the redefiner
J: This itself oculd be part of the protocol. It could be a single person who comes up with these things. Some might implement a voting scheme. Who knows what would be the best principles? … It can be automated .. Let’s not make an assumption about this thing, if we don’t know. Those areas we don’t know if the best will be people adding things ad hoc, or computers implemetning. ..
What system of government might work well?
The way out of self-destruction with AI
AI simulating humans to understand its own nature
Reinforcement learning
Commonsense reasoning
Would AGI need to have a body?
Neuralink
Reasoning at the scale of neurons and societies
Role of emotion
Happiness is a cookie that your brain bakes for itself
Writing on Writing: author of The War of Art, #102
Steven Pressfield, author of several powerful .. books, including The War of Art … whose passion is to create in art, science, business, sports, and everywhere else. I highly recommend it, and others of his books on this topic, including Turning Pro, Do The Work, Nobody Wants to Read Your Shit, and The Warrior Ethos. Also his other books are some of the best historical fiction novels ever writeen. Some of you know, I don’t shy away from taking a big difficult challenge. One of the hardest for me is looking at an empty page every day. In his work, Steven has articulated the struggle better than anyone I have ever read. … This is the AI podcast. I recently considered renaming the podcast. AI is my passion, and in some sense, thi spodcast is more about the journey of an AI researcher struggling to ecplore the human mind (and a variety of other topics). I will continue to return home to the technical; but also venture out to talk to people who had a big impact on my life outsisde the technical fields. Possible future podcasts: Steven King, Tom Waits, and … political leaders.
<iframe width="560" height="315" src="
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
L: Modern society in many ways dreams of creating universal peacde. And yet war has ___ at civilization as we know it throughout its history. IF you could imagine a world without war, how different would it be? Waht purpose has war served—why do we fight?
S: I think we’re basically the same creatures internally that we were in the cave, in tribla society hundreds of thousands of eyars ago. Which means, the dynamic in our mind is an us-versus-them dynamic, in which our tribe is the people, and evertybody else is whatever. I don’t think that’s changed one iota, over the centuries. It’s a question of how one might sublimate that urge to compette. You’re a martial artist; a great part of your day, I’m sure, is dedicated to reaching that place of total commitment, in the face of adversity, etc. on a n individual basis. The hope I have, if there is any—personally I don’t think the human race is goingto be around very long … would be in sports, or in other sublimated activities, where people can act out their need for conquest or aggressison or so forth, but at the same time relate to their opponents as human beings, and so on.
L: So you think war is ineviatble, a part of human nature; as opposed to a creative force in society
S: I”m sure it has benefitted, psreading and mixing cultures. but I think the urge to conquest, if you think about Alexander the Great … Napoleon … or if you even think of one of the plants we’re looking at outside. if you let a particular plant have its way, it’ll take over the hillside. If you look at the days of Alex. the Great … there were hundreds of little kingdoms, and every prince that grew up dereamt of conquering his neighbor and another neighbor after that. That seems to be a universal human imperative, at least in the male of the species. …
L: You’ve written about Spartans, the battle of Thermopoly …. about the y-dayh war in Isreal in ‘67 … What wra, not just out of those, has been most transformative for the world?
S: Great questions! I wish I knew more about the Mongols. What little I know, I think their conquest was very transformative .. .bringing cultures, in horrible bloody way, together. Gosh, the most transofrmative? Maybe the Roman Conquest, establishing the Roman empire? Maybe Alex. the great’s wars that united East and West, at least for a minute?
L; so building of empire? There’s wars, the 6-day war isn’t about building empires; it’s about deeply held religious cultural conflict, and holding the line, holding the border. Then the're’s conquest, like the Mongols—some large percent of the population is a descendant of Ghenghis Khan. Then World War II—think of that one
S: Let me ask you, what are you trying to get at? What is the theme?
L: I talked to ERic Weinstein—he said everythign is great about war except the killing. There’s a romantic notion of war; certaintly a romantic notion fo being a warrior, that somehow, there’s a creative force to tit. Because we fight, out of that fighting comes culture, music, and art—more and more desire to create, with the societies that win. To me, war is not just, “Hey, I have a stick—and want your land.” IT’s some kind of; it has echos of the creative force that makes humans unique to other animals. IT can’t just be four or 10 or 100 people; you have to have thousands of people, or more, agreeing for something so deeply that yuou’d be willing to risk your own life. There’s ar omantic notion to that. Youv’e writen so well about some of these; I wanted to untnagle these. … to see, is there a reason we fight more than just anger, hate, and wanting to conquer
S: Let me take it from a different side. I don’t think that I, in writing about war, am interested in war, per se. I’m more interested in a metaphor. I’m really writing about my own internal war. The war against my self, against my own .. resistance, my own negativity, all those things that are … that spirituality would be the opoiste of. I’m not really an expert on war. It’s not like talking to Jim Mattis, or Victor Davis Hansen, or whatever. To me, the human being … we are spiritual beings in a physical envelope. There’s a automatic terrible tension within that. Which creates a war inside ourselves. The outer war, when I think about the Isreali army, standing up to whatever, 10-to-1 odds, or whatever it was, that is a metahpor to me of the fihgt we’re fighting inside ourselves. For me, the 6-day war was, about a return from exile. It was sort of the culmination fo the re-establishment of the stateo f Isreael, which hadn’t really been complete,d because holiest places of the state were in the hands of their enemies. The conquest s of Alex. the great was a different scenario. Alex’s father Philip created the first Nation—srort of a pathway for these guys, who were mountain men, and basically barbarians, Macedonians … by creating this army, and this dream, of conquering the world, which Alexander which he enacted … he gave them a way of transcending themselves,e individually, and as a people. That would go along with what you’re saying, Lex, with creativity to itt. But again … I’m just realizing this as I’m talking, that’s not what’s interesting to me about these stories. The Spartans, at Thermopoly, that was a whole other metaphor of war. That was a sort of willingly going to one’s own death, for a greater cause. For me, the Spartans at Thermopoly enacted as a group what Jesus enacted as an individual—sacrifice of their lives for the greater good. .I do feel like … I get invited to speak to Marine Core groups all the time. I declinee, because i don’t think I’m a spokesman for the warrior class; that’s not what’s interesting about it, to me.
L: But didn’t you just say, war is a metaphor that we’re all, in various ways, warriors.
S: If we think of it in terms of Jungian arche-types; if we think of our lives, as males. The earliest ones are the Student, the Wanderer … At some point between age 15-twentyosmething, the Warriorr archetype kicks in. We want to play football, join special forces, hang out with buddies, that’s our great bond. At some point, we move beyhond that arche-type; we become fathers and teachers, andd so on. There are many arche-type beyond that, toward the end. I’m interested in the warrior arche-type, but not over everything else. In my book The Virtues of WAr, there’s a character Telemon, a long stoyr, when he’s with Alexander’s army, and they arrive in India, he becomes fascinated by the yogis, the naked wise men. HE says to Al;exander that these guys are warriors beyond what we are. Even though they do nothing; they are insdie their own selves, all day long.
Love and hate in a time of war
L: In Lions gate, you write about the 6-day war in Israel; the war we’re still in many ways in the midst of today. What is at the core of that conflict?
S: The Isreali-Palestinean conflict?
L: today yeah, but it’s echoes of the same conflict in that part of the world, with Isreael. What is, in your sense, the nature of that conflict; hwat can we learn about society and human nature from that?
S: When I was working on (that book), I wrote in the Intro that this was not gonna be a multi-sided story; I’m a Jew; I identify with the Isreali people; I was gonna see it entirely from their side. That’s proably not what your’e asking, but to me, the 6-day war and that whole … it’s a peace of land that’s holy to at least 3 religions and probably more …. From the Jewish point of view, it’s where the state of Isreal .. was founded by David … where the 12 tries were; where Moses came and brought the people … To me, the 6-day war was about, as I said, a return from Exile, from Diaspora, after 2000 years. Obviously from the (other) POV, it’s a whole other scenario
L: Religion is at the core
S: REligion and racial/ethnic tribal iddentity. Again, what is a Jew. Is it somebody that believes in the reliigon., or somebody of a certain race that arose in a certian place. Same thing with a Muslim; what isa. Muslim? D/o they believe in Mohammad? If we landed from Mars, we couldn’t tell a Jew from a Palestiean just looking at them
L: The specifics of the faith is not necessariliy the thing that defines … Many are secular Jews living in Isreal and still have a strong bond
S: Defininteely. In fac,t almost all of teh Jewis, fighters that I spoke too, were secular; it wasn’t a religious thing as much as a national thing.
L: Having spent time in Isreal … how’s the world where military conflict is directly. felt, as opposed to the U.S.
S: IT’s very different, as you know. … … You should definitely go. Here in the U.S., where … an incident like Charlottesville comes up; people are chnating “Jews won’t replace us,” the impulse in the Jewish community is, “how can we reach out to the other side.” How can we show them we’re human beings … that we care them. That’s sort of distant from war. If your’e in Isreal … if you and I were Isreali citizens right now, you’d be a fighter pilot or a tank comander or whatever; you wouldn’t just be in MIT. I would be in the army, too. From their POV, they say, all those people who hate us—Fuck them, we’ll kill ‘em! IF they dare to cross the line. That’s a whole different point of view; to me, it’s actually healtheir
L: You think so? Hard question—how do we resolve that conflict? In Isreal—or anywhere the instinct is to reach out and say, “F You,”
S: The only way the two warring sides, or two sides so opposed, can come together is if there’s mutual respect, and they can see each other as equals, and when there’s mutual fear. Where one side says, we don’t dare cross a line with this other side, and the other side says the same thing … Then we can say, “Okay, you stay here, you stay here.” We’ll mingle in cultural ways, have intermarriaage. But as soon as one side has no power, as the Jewish people had no power throughout the diaspora, then it’s human nature … you can see it in Trump and what he odes to any vulnerable minority. And he’s nto alone, I’m not blaming him alone … tha’ts human nature. That idea, fuck you, if you cross the line we’ll kill you—is really a good place to start from. then you can sit at opposite sides of the table …. … how can we do this in a way that we’re not hurting each other.
L: You kind of siad we need to arrive at an ew balance of power … you haven’t spoken to the fact that there’s deeply rooted hatred … of the other. Is there no way to alleviate that hatred? What role does love ..
S: I think it can go away, I really do. Even now, I haven’t seen this in person, but they say the Saudis and Isrealis are collaborating on certain things. .. by mutual fear and antagonism to Iran. Even long, long, long-standing anamiosities can go away under the right circumstances
L; On what time scale? Do generations have to die and apss away and new generations come up with less ahte? Or can a single individual learn to not hate?
S: For instance …. I think a single individual … We’re in a real spiritual realm when we’re talking about that. We’re in a realm of Buddha, Jesus, whatevder. Something where a true change of soul happens.
L: What do you think is the future of warfare?
Future of warfare
L: especiall ywith what many see about the expansion of the military industrial conflict. I’m asking as a metaphor. Do you see us as people continuing to fight?
S: Now, with social media, TV, movies … all these things that create empathy across cultures, it becomes harder and harder—I think—to totally demonize the other the way it was in previous wars. I also think, I don’t really see an appetite for people wanting to go to war these days. I don’t know if that’s good or bad; everyone’s so fat and lazy and concerned with (social media) clicks … The younger generations in WWI were eager to go to war. It was insane, but it was that warrior archetype we were talking about; that generational testosterone, Eros thing. Nowadays, I don’t know … it’s hard. to say there’s not gonna be another war, because there always are; but it’s hard to imagine people getting off their ass these days to do anything.
L: You mention social media as a place for empathy; but it’s a place for war ( and hatred0 as well! Perhaps the positive aspect of hatred on social media is it’s somewhat less harmful than mudder; it kind of idssipates .. you get the hate out, at a … less … on a daily basis, and thereby it never boils up to a place where you want to kill
S: Like with video games; where kids are acting out these incredible horror things; you know if they cut their finger; they would freak out! I don’t think many of the people tha tare hateful on social media … wouldn’t (act the same way if face-to-face). There’s two mental spheres happening at the same time.
L: If the U.S. had a draft. .. the population would respond differently.
Technology in war
L: I work on building AI systems; in our community many are worried about AI being used in war; automating the killing process; it’s being used more and more
S: I should recuse myself on that one; I haven’t thought about it. I should ask you
L: Interesting, fundamentally different, if you lok at Battle of Thermopoly. If we look at the differnce between a gun and a sword
S: Anecdote. Spartan King; they showed him a new invention that launched a bolt and could kill a person at a range of 200 yard.s The king wept; he said, alas, valor is no more. Their view of war was highly ritualized … you werne’t supposed to be able to kill another person unless you yourself were at equal risk of being killed. Even bow and arrow were considered less than manly, less than honorable. At least that makes the stakes real and true. We should go back to that maybe? Not that we could.
What it takes to kill a person
L: You were in the Marine Core … Have you … thought about what it takes to kill ap erson? If you yourself could do it?
S: I have thought about it … (L: and how that would make you feel). .. I haven’t been in combat; I haven’t done it. I would imagine, in the ral world, it would change you, utterly, forever. Because … you can’t help but identify with the person that you’ve just killed. Its another human being, and I would have a hard time killing a spider. I would imagine it’s somethiung that warriors understand and nobody else understands.
L: You’ve spoken with many (people who have seene combat). Have they been able to artiuclate teh experience of killing?
S: It’s sort of what I said. I’m thinking of one pilot I interviewed … He was strafing a tank in his Mustang, at low altitude, and saw what his bullets did to the guy, could see his face … and that’s even one remove from what an infaantryman does. He said that same thing—it changes you … you can never … look at the world or anything the same way again.
L: Whe nthat happens at scale, thousands … that changes entire societies.
LS: The problem is, it doesn’t change the politicians back home!
Mortality
L: HHow important is mortality, the fact that this thing ends—to the creative process? In general, the fact that this thing ends …
S: IT does? (laughs) Shit! (L: Adn on a serious note, do you think about your own mortality?) Actually, I’m 75 … I had breakfast in NY a few motnhs ago; a friend of mine who’s my exact same age. I asked him the same qeustion—he said every fucking minute … I was relived to hear that, because I do too. I always ahve, I think. The fact of mortality is … kind of gives meaning to lfie; I think htat’s why we want to create; why we want to make a mark of some kind … The other aspect of it is, whats’ on the other side of that mortality? I’m a bleiever in previous lives … the question I’ve never been able to answer, among many others, is why are we even here? Why are we in the flesh? I’d like to believe God, or some force—that we’re on some kind of journey; but I’m not sure why we were put in this world, where the ground rules are, if you think about animal life, you can’t live from one day to the next without killing and eating some other form of life. Why couldn’t we just have a solar panel on our head, and be friends with everybody? I don’t get what that was all about, but
L: Have you read Ernest Becker’s Denial of Death? He said that the fear of death is really the primary driver of everything we do. Freud had …
S: I would agree with that ….
L: Can you elaboraet on the reincarnation aspect of what you were talking about? What’s your sense that we had previous lives? Have you thought concretely—
S: Yeah … It’s very clear .. when you see children, young kids or even dogs and cats .. that they come into the world with personalities. Three kids and a family are going to be completely different, their own person. That person that they are doesn’t change over life. ONe of the things I did in my book, The Artist’s Journey … I tracked or listed all of Bruce Springsteen’s albums, or Philip Roth’s books … over 30, 40, 50 years. You can see .. that there’s a theme running through all of those things; completely unique to that person .. nobody else could have written them. You can even see a destiny there. I asked myself, where did that come from? It seems to be a continuation of something that happened before, and it will lead to something else. It’s not starting from scratch; there’s a calling, a destiny, in there already. This gets back to the muse and that sort of thing
The muse
L: Let’s call it a god. Almost smapling parts of a previous human that has lived, and putting those into the new one. … You can’t take all the good parts; because the bad parts is what makes the … is this humans only, or does it involve animals, in your view? … Oka, you talk about the muse as the source of ideas, maybe. Sinc eyou’ve gotten a few glimpses of her in your writing, what is it possible for you to tell me about her—where does she reside? What does she look like?
S: You can look at it in many different ways. The Greeks did it in a anthropomorphic way. If you look at it from a Kabalistic way, Jewish mysticisim, you could say that it’s the soul … soul is above us on a higher plane … and it’s trying to reach down to us and communicate with us. We’re trying simulatneously to reach up to ti. thrrough prayer, or as a writer/aritst, when you sit down at the keyboard, you’re entering into a kind of prayer; an altered consciousnes;s you’re opening the pipeline, tuning into the cosmic radio station. Another way of looking at it—did you ever see City of Angels; the visual of it, it was Meg Ryan and Nicholas Cage … and the visual of it was … Meg Ryan is a heart surgeon, and as she’s operating on somebody, suddenly Nic Cae in this long duster coat appears right next to her in the OR, he’s an angel; and he’s waiting to take out the soul of the patient on the oeprating table. They’re all unawarwe of him, except the guy who’s about to die. I kind of believe, that there are being, like that—or, a force, a consciousness, something—that is right here, right now, and they’re trying to communicate to us. Through a membrane; like tapping on that window over there; they’re like, right out there. They carry the future. They are everything that is in potential. All the works that you will do, Lex, your start-up, whatever else you’re doing—they know that. It’s not really you, that’s coming up with those idea, IMO. Somebody knocks on the door and puts it in there. In the Iliad, when gods and goddesses appear with humans on the battlefeild all the time … Homer flashes to Olympus and back to the real wordl. … Aphrodite let’s say wants to help Paris, she says, I’ll appear to him in a deream, and I’ll take the form of his brother … That’s createrues, beings, on one dimension, communicating … and I believe that’s exactly what’s going on .. whatever analogy you want to use.
L: To which degree do you play the role in that communication; as opposed to sitting at the computer, as a writer, and staring at the blank page, and putting in the time, and waiting. In your ivew, are these creatures waiting to tell you about your future? Or is there choice? How many posible futures and ideas are there?
S:Yes, there are alternatives, degeres within it. But if you look at Bruce Springstseen’s albums, how much could he have done really differently? You can see there’s an impetus going through the whole thing. Nothing was going to shake him off that. He was dealing with certain issues; his conscious self was, and they were really out of his control. He was drawn, he was called to it, nothing could stoph im. It is sort of a partnerhship … with creative impulses coming from some other place. Or it’s coming from deep within us. If we’re acorns, growing into oaks; the conscious artist, who’s sitting there at the keyboard or whatever, is applying his or her consciousness to that. But is also opening themselves to the unconscious, to this other realm, whatever that is. Certainly songwriters have for a million years said songs just ame into their head. … Keats’ notes for A Thing of Beauty … they go on forever; his conscious mind is working on it. So I do think it’s a partnership. When I was first starting out as a writer, trying to do novels I couldn’t do, I was really unskilled at tuning into that station. I beat my brains out and was unable to do it. I was trying too hard; it was sort of like a monk of some kind trying to meditate, and constantly thoughts driving you crazy. Over time … I’ve sort of got beter at it; I can sort of let go of that part of me that’s trying so hard, so these angels can speak a little more easily through the membrane. . … …. .It took probably 30 years; I would liken it tom editation, although I’m not a meditator. It seems to me one of the hardest things in the world to sit still and stop thinking. That’s why these teachers of meditation use tricks and koans, stuff like that. For me, it was just a process of years and years of trying, and beating my head on the wall. Little by little giving up that beating of the head. But there doesn’t seem to be any trick. Everyone wants a hack these days; I don’t think there is one. Look at it in terms of the goddess … Like a marine going through an obstacle course … Uma Thurmna (in Kill Bill) l… trying to punch through that piece of wood. Finally she’ll say, alrigh the’s paid his dues—I’m gonna give it to him.
Editing
L: I was also getting at, certainly there’s no shortcut; but is there a process? The process of practice … You had two, one you had an example of meditation; it’s essentially the practice of meditation
S: Drill, is a good way to look at it too. … You’re writing. Then you’re looking at what you wrote. You’re hitting moments when it flows. And then yo’re hitting moments where you can’t do anything … from the moments where it flowed … you go back and say, what did I do; how did it happen? I think it’s a process, over and over and over until it gets a little bit easier.
L: When you read something you write, did you always have a good radar for what’s good?
S: No. (laughs0 I think I do now, but … no. It was always really hard for me to know what was good.
Resistance
L: The invisible force that … tries to prevent you from doing the work. Where do you think it comes from? Why is it … trying to jeopardize our efforts? With laziness, excuses, and so on
S: It’s called the Yetzir hurrah … If this up here is your soul … the yetzir hurah is this negative force in the midel. Here’s my answer: I think that there are two places where we as human beings can seat our identity. One is the conscious ego, and the other is the greater self. The self, in the Jungian sense, includes the unconscious, and butts up against the Divine Ground, which I would call the muse, the goddess or whatever. The ego is just this ltitle dot insdie this bigger self. The ego has this completely different view of life from the self. The ego believes … (apolgoies for long answer) .. The ego believes death is real; that time and space are real; that each one of us is separte from the other; I could punch you in the face and it would only hurt you. In the ego’s world, the dominant emotion is fear. We are all made of flesh; we can all die … we are protecting ourselves and our desire to create comes out of that fear of death. Self, the greater self, OTOH, believes that death is not real, that time and space aren’t real, that the gods travel swift as thought; and the Self also believes that there’s no difference between you and me, that we’re all One—karma! In the world of the greater Self, the domiannat emotion is love, not fear. … I’ll go farther back … to answer the question: when Jesus died on the cross, or when the 300 spartans willingly sacrificed their lives at Thermpoly .. .they were acting according to the rules of the SElf. … The predominant emotion is love. In my opinion, we as conscious human vessels, are in a struggle between these two things, the ego and the self. To me, resistance is the voice of the ego. It’s a fearful voice. When we … identify with the Self, we move our consciousness over to it, opening ourselves up to the cosmic dimension, the ego is tremendously threatened. If we’re in that space, that head space, we don’t need the ego anymore. I think resistance is a voice of the ego trying to keep control of us. A bad example—Trump, is the ego.
L: That’s probably a very good example (laughs)
S: It’s a zero-sum world for him, and probably for anybody … And the opposite example is probably MLk, or Gandhi. .. and that’s of courswe why they end up getting assassination—because the ego is hanging on to itself and feels so threatened …
L: That’s fascinating; it’s interesting why fear is attached to theego. I really like this dichotomy. Ego … the self-obsession of it … why fear is such a predominant thing; why is resistance trying to undermine everything …
S: Let’s think about it in terms of stories. In a story, the villain is always resistance; si always the ego. The hero … is always (pretty much), the Self. If you think about the Alien on the spaceship—it’s the ultimate kind of villain; it keeps changing form. It always has that one monoaniacal thing —to destroy. Just like the ego, ust like resitsance. Maybe Alien is a bad example, because S.Weaver has to fight on the same terms as the alien. Maybe a better one might be Casablanca, where in the end H. Bogart has to, operating out of Self, give up his … selfish dream, of bgoing off with Ingrid Bergman .. and instead, puts her on the plane to Lisbon, while he goes off to fight the Nazis in the desert. Don’t know if that’s clear, but in almost every story, the villain is the ego, resistance, fear, htat zero-sum theing. …
L: Do you think there’s evolutionary advantage to resistance? What would life look like without it?
S: I also believe that resistance, like death, gives meaning to life. If we didn’t have it, what would we be? In the garden of Eden, picking fruit, happy and stupid, you know? I think that myth of Adam & Eve is about this—they decide to take matters into their own hands and acquire knowledge. Until then, God had said, I’m the only one that’s got that knowledge. Once they acquire it, they’re cast out iinto the world we live in now, where they do have to deal with that fear …
L: And the meaning and the purpose come fro mthe resistance being there and the struggle to overrcome it.
S: The other saspect of it is, it’s not real at all. It’s not an actual force. It’s all here (gestures toward the head). In a way you sort of surrender to it, you know? Or ..
L: Surrender toward reality
S: It’s like turning on the light in the dark, because it’s gone.
L: But not quite because it keeps returning ….
S: It’s always the same, it’s about writing for me; evolving within my own body of work; it never goes away, it never gets any less.
L: Do you have particular excuses, justifications?
S: IT’s alwayss the same. It’s really not, because resistance is so Protean; it keeps changing form. The resistance gets more nuanced and subtle, trying to fake you out. I think you learn that it’s always there, you’re always going to have to face it.
L: Your battle is sitting down and writing some number of words to a blank page. Do you have a process there, with this battle? A number of hours that you put in?
S: I’m definitely a believer that, even though this battle is fought on the highest spiritual level, the way you fight it is on the most mundane … martial arts must be the same way. I go to the gym first thing in the morning; I sort of am rehearsing myself … I go to the gym; it’s resistance trainnig … i’m fortifying myself to be ready for the day. Like I said, over the years, I’ve learned to get inot the right kind of mindset; it’s not as hard for me as it used to be. … The question of sort of, what’s the next idea. What’s the next book, the next project … When I ask; I’m asking it of the muse, or of my unconscious. If we’re looking at Bruce S’s albums … what is the next idea. Now he’s on Broadway. That was a great idea! Where did that come from? And for him, what’s after that? That body of work, is already alive; it already exists. It’s kind of like a woman’s biological clock—we have to serve it. Otherwise, it’ll give us cancer, you know? You knwo what i mean; it’ll take its revenge on us. A big aspect of it is what’s next?
L: At the same time, you have a kind of … sense that there is Bruce S. single line of albums, so like, it’s already known somewhere in the universe what you’re going to do next, is the sne you have?
S: Yes; I don’t know if it’s predetermined, but there’s something like that
L: Yeah, I’d like to believe … it’s kind of like quantum mechanics. Once you obsere it, once you talk to the muse; it’s one thing for sure, it was always going to be that one thing. GBut really, in reality, it could be any distribution of things.
S: … Yeah … but they’re not that far apart. Bruce S. is not going to wrtite a Joni Mitchell thing. …
L: Do you visualize yourself completing the work? Olympic athletes visualize themselves getting the gold medal. l.. Every. day, thy visualize how the day of the champinship will go. … Do you do anything like that, in how you approach writing?
S: No. Because it’s such a mystery; I think it’s different from sportts. …
L: There’s no gold medal.
S: No. I would like to think, as soon as you finish one, the next day you’re on the other. Hopefully you’ve already started the other. You’re alreayd 100 pages into the other, when you finish the first one. It is a … it’s a journey, it’s a process. … I don’t think it is … I think it’s very dangeorus to think that awya. “This, I’m going to win the Oscar!'“
L: For the creative process it might be dangerous. Why is that dangerous? Because …
S: Ego … IYou’re giving yourself over to the ego. I keep saying this myself—my job, I’m a servant of the muse. I’m there to do what she tells me to do. IF I suddenly think, I just won whatever. The muse doesn’t like that. She’s on another dimension from me.
L: I’m trying to square that … I think there’s a meditation to visualizing success in the athletic realm, to where it removes everything else away, to where you focus on this particular battle. I think you could do this in many kinds fo ways. In sports, the ego serves a more important role than it does in writing. ..
S: When you say that, I know what you mean Lex. I do think there’s sort of — it’s interesting to watch inteviews with Steph Curry, who’s obviously such a jnice guy, but he’s got such tremendous self-confidence … IT doesn’t border on ego, so much, because he’s worked so hard for it. He knows, he has visuzlied … maybe not so much winning, as just him being the best he can be, him being in the flow … doing his thing, that he knows he can do. I do think, in the creative world, yeah, there is sort of a thing like that. A choreographer or filmmaker or whatever … might do an internal thing where they’re saying, “I can make an Oscar-winnign movie, I can direct this movie. I’m banishing these thoughts that I’m not good enough”. I don’t think that’s really ego; that’s part of the process in a good way; like an athlete does
L: Extreme confidence iis what some of the best athletes come with. You think it’s possible, as a write,r to have extreme conifdence
S: I do. I’m sure when John Lennon sat down to write a song; he though, “I can do this.”
L: I’m not sure; the great artists I’ve seen, you’re haunted by self-doubt
S: Yes. But beyond the self-doubt is self-belief. Yeah I’m freaking out, but I know I could do this.
Loneliness
L: Is the process (of writing) lonely
S: NO, becausue your’e with your characters … … I’m in here, I’m seeing the Spartans, I’m seeing whatever. The characters that are on the page, or that you create, are not accideents. They’re coming out of some deep issues that you have, whether you realize it or not. Your characters are sort of fascinating to you. Their dilemmas are fasicnating to you. You’re also trying to come to grips with them; you see them through a glass darkly, and you want to see them more clearly. In fact, it’s not lonely at all. I’m more lonely going out ot dinner with people, or talking to people.
L: Do you miss them after it’s over?
S: I have affection for them, like children who have gone off to college. Even the bad guys.
L: Maybe especially the bad guys? … Yo
Is a warrior born or trained?
L: You’ve said that writers are often not tought-minded enough; that you have to be professional in the way you handle your emotions; that you have to be a bit of a warrior …
S: I think they’re born to some etent—you have the gift … But training is the big thing. 90% training, 10% genetics. I use another analogy, otehr than warrior … to be a mother! If you’re a writer, you’re giving birth to something. In terms of bravery, if your two-year-old child is underneath a car; a mother is going to stop a Buick with her bare hands. That’s another way to think about how a writer has to think about … what they’re doing. This child, this new creation that they’re bringing forth.
L: Yeah, so the hard work that’s underlying it. A couple weeks ago I talked to … Ariana Huffington; I did this conversation with her. I didn’t know much about her before then, but she has recently … wrote a couple books; has been proomitng a lifestyle; she created the Huffintgnon Post … gave herself 20 hours a day obsessed with her work; she fainted, passed out, had helath issues. Her book says … you want to estaablish a lifestyle that doesn’t sacrifice health. She thinks you can have both—lifestyle and health. Criticizing Elon Musk for working too hard and thereby sacrificing … effectiveness. I’m trying to get at this balance between health and excessive work. To me, I’m torn. Passion and reason don’t interlap much. Maybe I’m being too Russian. I feel madness and obsession doesn’t care for health or sleep or diet or antyhing like that. Hard work is hard work and everything else can go to hell. If you’re erally focused … Where do you stand?
Hard work and health
S: I’m falling on the health side. There was a period of my life where I was just .. had no obligations; was just living in a little house and working nonstop. But even then, I would get up in the morning and have liver and eggs for breakfast … and do my exercisees … but I was still doing 18 hours a day. I got to think about it as an athlete does. I’m sure that Steph Curry is totally committed. But he has his family; I’m sure he eats perfect , grreat stuff; gets his sleep … gets the … training … for knees an dankles …. Or Kobe Bryant, or anybody operating at a high level. I’m in the health school. The good thing about being a writer is you can’t work very many hours a day. Four hours is the max. I can work. I’ve heard of peple doing 10-12 hours; I don’t know how they do it. That gives you other time to … optimize … other stuff.
Daily ritual
L: What is a perfect day look like for you, if you’re talking about writing—an hour by hour schedule of a perfect day?
S: I get up early (3:15 a.m.), go to the gym; have breakfast with friends of mine. That’s what I’m on now. So I ‘m in bed .. when I’m with my nephews that are 3-4 years old, I’m in bed before them. … Exercise; I have a trainer, I have a couple guys that I work out with. It’s maybe an hour, maybe a little more … Warmup before, stretching afterward. It’s an intesnese kind ofa. thing, that I definitely don’t want to do.
L: So you feel like you’ve accomplished something first thing?
S: Yes, but it’s not so hard that I’m completely exhausted. I come home, handle correspondence; tand then I work for maybe three hours. And then I sort of crash … the office is closed; I don’t think about anything, don’t htink about work at all.
L: You listen to music?
S: No. That’s just me.. Sompebody could do it a million different ways.
L: Fascinating. Of many writers, you’ve got … optimzed this conversation with the muse you’re having .You’ve at least thought about it. Can you say more about hte trivialiities of that process; like you said, facing the wall. Do you have little rituals?
S: The granular aspect of it? I do little rituals; I have all kinds that I’m not even gonna tell you about. The one thing, that I don’t want to talk about too much ,because it jinxes things … when I sit down, I immediately get into it—ifrst second. I don’t sit and fuck aorund with anything. I try to get into it as quickly as I canl. The other thing is … writing a book o r screenplay is a porcess of mutliple drafts. The first draft is where you’re most with the muse. Right now I’m on the 5tgh, 6th draft of something I’m working on. I’ve got pages I’ve already written; and I’m reading them afresh … Where I am now is not a deep muse scenario. Partly it is. But it’s also bouncing back and forth between the right and left brains. Looking at it, trying to evaluate it, go into it and try to change it a little bit.
L: Do you know the night before what the starting point is?
S: I always try to stop—when you know what’s coming next. So you’re not facing a chasm. You jknow?
L: And afterwards, when you’re done, the office is closed
S: I let the muse take care of it. I think it’s very unehalthy to worry or think about any creative process.
L: Whta about on a long walk later?
S: I’ll keep my mind open to it, but I won’t be obsessing about it. … That’s not your ego doing it, that’s the deeper level
L: Okay, how does the day end?
S: Maybe go out to dinner; my gf is not here now … she’s in NY working. Go out to dinner, somethiung like that. Maybe read something, nothing heavy. Go to bed pretty early … The gym is a big thing for me … probably for you with martial arts. I’ll be visualizaingg what I ahve to do the next day, and getting myself psyched up for htat. I’ll conk out like a light and wake up the crack of dawn
L: So looking out into the future … hwat do you think the muse has in store for you?
S: I don’t think you can ever know …
Popularizer of the term AGI
This interview includes a brief Covid discussion in June of 2020
Ben Goertzel, one of the most interesting minds in the AI comunity. He’s the founder of singularity-net; designer of OpenCog AI frameowrk. Formerly a director of resarch … and chief scientist of Hansen robotics, the company that created the Sophia robot. He has been a key figure in the AGI community for many years; including in his ortganizign and contributing to the conference on AGI, the 2020 version of which is happening this week, Wed-Fri … it’s virtual and free. I encourage you to check out th etalks, including from Jos. Bach … episode 101 of this podcast.
<iframe width="560" height="315" src="
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
L: What books, authors, ideas had a lot of impact on you … in the early days?
B: What got me into sci-fi in the first place wasn’t a book but the original Star Ttrek TV show … in its first run, 1968, 1969 or something. That was incredible. Every show visited a different alien civilization with differnet culture and weird mechanism.s That got me into sci-fi; there wasn’t much of it on TV; it got me into the literature of sic-fi .. from the beginning of the previous century until that time. So many sci-fi writers were inspirational. If I had to pick two, it’d be Stanislav Lem … Solaris and he had a bunch of more obsure writings on superhuman AIs that were engineered … Then PHilip K. Dick, who … utlimtately .. my fandom for him brought me together with David Hansen, my colllaborator on robotics projects. Lem was very much an intellectula; he had a broad view of intelligence going beyond the human … to open-ended superintelligence … In a complex and confusing way, so that human beings could never quite connect to it. … Goldman 4 supercomputer, in one of Lem’s books .. was engineered by peoplee, but it became intelligent in a different direction than human; it decided humans were trivial and not that interesting. It put some impenetrable shield around itself … isused some philosophical screed about the pathic nature of humanity and all human thought, and hten dissappeared. Dick was different; he was human focused. His main thing was … human compassion, the heart and soul, will keep us going, through whatever aliens we discover, or telepathy machines, or whatever it might be. He didn’t believe in reality; the reality we see may be a simulation; but he believed in love and compassion, something persistent through the various simulated realities. A little older than that, I got into Dostoevsky, Niezsche, Rimbaud, and more literary type writings.
Are there intelligent beings all around us?
Dostoevsky
Russian roots
When did you fall in love with AI?
B: … … When my dad was at Antioch College, before I was born, he led a protest movement called SLAM (STudent League AGainst Mortality); they were protesting against death, across the campus. Whether AI could confront logical paradoxes, he didn’t know … Later I discovered Goedel-Escher-Bach. … Can an AI model itself reflexively, or does that lead to some sort of paradox? Same question for the human mind. I read it when I was 12 years old; 16-hour-dya, I read it cover-to-cover; I re-read it after that. That was the first book I read that gave me a feeling for AI as a practical, academic, engineering discipline that people were working in. Before I read that, I was into AI from the POV of a sci-fi fan. I had the idea, it may be a long time before we achieve immortality; I should figure out how to drive a space craft … Reading G-E-B, though it didn’t all ring true to me, a lot of it did; I could see there were a lot of people at universities working on building AGI; then I started to think, well this is something that I could practically work on.
L: Instead of flying away and waiting it out, you could actually be one of the people that builds something
B: … what we now call nanotechnology … but AI seemed like, if Hofstadter was right, you just figure out the right program, go and type it. You don’t need to align stars into the right configuratiions or do crazy experiments on humans. It’s just programming. There was another book by Gerald Finebaum, The Prometheus Project; I encountered it in the mid-70s. It said humanity’s going to create super-thinking machines. The challenge we’ll have is what to do with it. Do we use it to expand human consciousness in a positive direction. Or to further vapid consumerism? He proposed the UN should do a survey on this; go to the remotest villages … and let everyone on the whole planet vote on whether we should develop super-AI nanotech. and immortality for expanded consicousness or rampant consumerism. Needless to say, this didn’t happen; and this guy died in the mid-80s. . But many of the themes I’m engaged in now, including trying to democratize tech., many of these things were there in finebaums’ book in the late 60s. Of course Valentin Churchin, a great Russian physicist, who I got to know when we both livedi n NY; he had a obok in the late 60s in Russia, The PHenomenon of Sicence; Val died in 2004 or 2005 of Parkinson’s.
Are humans good or evil?
Colonizing Mars
Origin of the term AGI
AGI community
How to build AGI
OpenCog
L: So maybe this is a good chance to step back and mention OpenCog 2.0 …
B: Go back to Open Cog 0.0 …
:: Maybe talk to the history fo OpenCog
B: Absolutely. 2.0 is a term worth throwing around tongue-in-cheek, because existing openCog system .. is not remotely cllose to what we consider a 1.0 … It’s been around, 13 years or something but it’s still an early-stages resaerrch systsem. we are going back to the beginning in terms of … theory and implementation; we feel that’s the right thing to do. I’m sure what we end up with will have a huge amount in cmomon with the current system
L: What is it?
B: OC is an open source software project that I launched with several others in 2008; probably the first code written toward it was in 2001 or ‘02, or something. Developed as propietary codebase within my company; we decideed to open source it; clean up the code, threw out some things. It’s written primarily in C++.
L: And it’se separate from SingularityNET; it was born as a non-network thing.
B: Correct. There are many levels of networks involved here
l: N connectivity to the Internet at birth
B: SingularityNET is a separate project, a separate body of code … you can use singularityNet as part of an OpenCog system; they are different layers. OpenCog as a software framework could be used to implement a variety of different AI architectures and algorithm.s Group of developers I’ve been co-leading which have been using OC platform and infrastructure to implement certain ideas about how to make an AGI. There’s been some ambiguity about OC software platform versus OC AI design ….
L: What does the osftware platform provide?
B: Let me talk, first, about it as a software platform …
… it’s way easier than it was when I got my PhD
SingularityNET
L: Great we talked about OpenCog … an actual attempt to build an AGI system … Can you say what is SingularityNET;
B: Sure; It’s a platform for realizing a decentralized network of AI’s. So Marvin Minsky, the AI pioneer, who I know a bit, had an idea for a society of minds; you should achieve AI not through one program, but by putting a bunch of AI’s out ther; they’ll interact, and the totality of the society of AI’s will be what displays the human-level intelligence. I had many debates with Marvin about this idea. I think he really thought the mind was more like a society than I dod. You can have a mind as disorganized as human society … But we have this thalamus, medula, limbic system, a sort of top-down control system that guides much of what we do, more so than a society does; so I think he stretched the meetaphor a little far. But there’s something interesting there. In the 90s, when I started WebMind, an AI startup in NY, what I was aiming to do was make a distributed society of AIs, that would live on different computers all around the world; each would think about data local to it, but they would hsare and outsource, and the intelligecne woudl be in the whole collective. Conference in Brussels at 2001, the Global Brain Zero conf., I organized it. We’re doing a sequel in 2021 …
L: The timing’s right, yeah
B: The idea with the global brain was, maybe the AI won’t just be in a computer; it’ll be in the Internet as a whole, with cooperation of various AI modules living in different places. One of the issues you face in building a system like that—how is the whole thing controlled? Do you have a central control unit? Or do you have a fundamentally decentralized network? Where th society of AIs are in a democratic and self-organized .. system. Frances and I had a differnt view of many things, but we both wanted this global society of AI minds with a decentralized organization. He wanted the … AIs to each be incredibly simple; I thought that was cool, but that a more practical way to do it might be … You could have a bunch of OpenCogs out there; these are all cooperating, coordinating together. The brain as a whole is a general intelligence; some parts of the cortex have a fair bit of intelligence on theeir own. Not so the limbic system. … They’re contributing by their connectivity to other modules.
L: … instantiations …
B: Yeah Yeah … Sophia and other robots …. collective intelligence infusing them. … The thing is, at that time, as well as WebMind being implemented in Java 1.1, blockChain wasn’t there yet. So … We sort of knew about distributed systems, about encryption, the key principles of what underlies BlockChain now. So when Butterin came out with theory on BlockChain in 2013, then I was like, well this is interesting. … Scripting language; it’s kind of dorky in a way; I don’t know why you need a cmoplete language for this purpose. But this is the first time I can sit down and script infrasturcture for decentralized control in a tractable way. … Ethereum scripting language is nice and easier to use; I’m annoyed with it at this point. Like Java, these languages are amazing when they first come out. … So Singularity—a decentralized agent system … each of them has their own identity on the BlockChain … this community of AI has no central control, no dictator. The coordination of the society of minds is done entirely by decentralized network. The motto of BitCoin is “IN Math we trust.” That’s what you need; the society to trust only in math. …
L: so ..
B: I would have loved to put AI’s operations on chain in some sense; but in Ehtereum, it’s just too slow.
L: … it’s the basic communication …
B: AI’s just a software process living in a container … there’s a proxy .. connection with the rest of singularityNET … when one AI wants sto communicate with another .. they set up channels .. once set up, the data flows on those channles. All that goes on the blockChain is the fact that some data went along that channel.
L: So there’s not a shared knowledge
B: Well the identity of each agenet is on the blockChain; if the agents rate each other, that goes on the BC; if they publish … what they wil fulfill, that will go on the BC … but ____ is not on the BC
L: Do you think it shoudl be?
B: Eh sometimes. Now there’s more modern and faster blockChains where you could start to do that in some cases; two years ago it was less so.
L: One example; I worked a lot on autonomous vehicles; you could see each individual vehicle as an AI system. Vehicles from Tesla, Ford, GM; they’re all running the same kind of system on each sets of vehicles; the instantiation is the same for each within the same company. You could invision a situation where all those AI systems are put on singularityNET. How do you see that happneing? What would be the benefit? Would they share data? The power is in the decentralized control. But a benefit, what’s really nice, is if they. can share knowledge in an open way if they choose to.
B: I think the benefit from being on the decentralized network, as we envision it, is that we want Ai’s and the network to be outsourcing work to each toher. … the real gbenefit would be if that AI wanted to outsource some cognitive processing … or . to other AI’s in the network which specialize in something different. This really requires a different way of thinking about AI software development. Just as object-oriented programming is different … Shifting to agent-based programming, where AI agent is asking otehr real-time evolving agents for feedback about what they’re doing …
B: … the brain has regions … tha tightly interconect; then there’s looser connection with the different lobes of the brain; and the brain connects with the endocrine systems. Then your body connects even more loosely with other people that you talk to. You often have networks within networks within networks. You have that in biology; you have that in the Internet. I think that’s what we’ll have in the network of software processes leading to AGI.
L: Beautiful. Similar question as with OpenCog … ‘
Sophia
L: Can you tell me who Sophia is?
B: I’d rather tell you who David Henson is; the guy behind the creation of Sophia. He’s currently still more interesting than Sophia. We met at some futures conference in 2007 whwere we were both speaking. We had a lot in common, both crazy, both had a passion for AGI, and the singularity, and we’re both big fans of Philip K. Dick; I wanted to create benevolent AGI, that would create massively better life for all humans and sentient beings, and David, he wanted exactly the same thing; he had a different idea how to do it. He wanted to get computational compassion; he wanted machines that would love people and empathize with people; get a machine that could look people eye-to-eye and face-to-face; make people love the machine, and the machine loves the people back. Different way of looking at it, I’m a math person … I was looking at it as an algorithm … We hit it off quite well, and we talked to each other off and on. I moved to Hong Kong in 2011; I’ve been living all over the place; Australia, New Zealand .. Las Vegas for a while, New York in the late 90s, D.C. for nine years doing government stuff. But HK in 2011 mostly because I met a Chinese girl I fell in love with; she’s from mainland China; we converged together in Hong Kong; we’re married now, have a 2-year-old baby. I started doing cool research there. Got involved with a project called IDEA … Got to know something about consumer electronics ecosystem in XianZhen across the border; the only place where it makes sense to make electronics at large scale at low cost. The ecosystem in South China; people here cannto imagine what it’s like. David was starting to explore that also. I introduced him in HK to investors that were interested in his robots. Then he had robot Philip K.Dick and robot Einstein and robot Xeno (the name of his son). He managed to get some funding to move Hansen Robotics to Hong Kong. At the time I was doing AGI research, and didn’t get involved. But as we hung out more, I started to think more about how to make robots smarter … For a few years I was chief scientist and head of software at Hansen; then as we got closer to blockChain, I stepped back and co-founded singularityNET with David and others. … Part of our goal was to make a cloud- … platform for Sophia.
L: Sophia would be one of the AI’s in SingularityNET
B: Exactly; many copies of it … …. David and I talked about that for quite a while before co-founding singularityNET
L: In your vision, was Sophia tightly couple to a particular system? OR that you could plug in any system …
B: David’s view was always that Sophia would be a platform, much like the Pepper robot is a platform … with a set of nicely designed APIs that anyone could use to experiment with different AI algorithms … SingularityNET is part of that … OpenCog is a little bit different. That’s more my thing. David has more passion for biologic-based approaches. He’s a character sculptor, right? He also worked a lot with rule based and logic-based systems, too. He’s interested in all the Henson Robots; it’s a powerful social and emtotional roboticsp latofrm. What I saw in Sophia was a way to get AI algorithms out there in front of a lot of differernt people in an emotionally compelling way. Part of my thought was connected to AGI ethics. ..
… so you. keep fooling them when they want to be fooled. The whole media industry is based on fooling people when they want to be fooled. We are fooling people toward a good end; playing on people’s sense of empathy and compassion so we can give them a good experience with helpful robots, and fill the AI’s mind with love and compassion. I’ve talked a lot with Henson Robotics lately about …
… elder care … In the Covid era, having a robot that could be a nursing assistnat could be quite valuable; robots can’t spread infection. If you have a robot helping a patient with Covid; if that patient attributes more empathy and understanding to that patient because it’s made to look like a human … is that bad? … It’s about how you use it. If you have door-to-door sales robot that used its human-looking appearance to scam people out of their moneyh, then they’re using that human connection in a bad way.
Coronavirus
L: Like you said, we’re living in the era of the Covid; this is 2020, one of the craziest years in recent history … If we zoom out and look at this pandemic … Maybe let me ask you this kind of thing, in viruses in general … When you look at viruses, do you see them as a kind of intelligence system?
B: I think intelligence is not that natural of a concept. … Human minds and bodies are a self-organizing adapative system. viruses certainly are that. If you look at intelligence as Marcus Sutter defines it, optimizing computable reward functions over computable environments—for sure, viruses aree doing that! In doing so, they’re causing some harm to us. The human immune system is a very complex self-organizing adaptive system with a lot of intelligence to it. Viruses are adapting, develoipng into new mutant strains. Ultimately the solution is going to be nanotechnology; making nano-bots … we can use them to detect, combat, and kill the viruses. Now we’re stuck with the biological mechanimss to combat these viruses. AGI is not yet mature enough to use against Covid. But we’ve been using ML and some reasoning in OpenCog to help doctors do personalized medicine … Given a person’s genomics … how do you figure out the best combination of antivirals that’s most effective against Covid for that person. … Getting OpenCog with machine reasoning; it can help with transfer learning with not that many cases to study …
L: So there’s a lot of different disparate data to work with …
B: IT’s very hard to get that data—that’s a shameful thing. We’re doing clinical trials with another group; they’re sharing data with us. But what should be the case; ever Covid clinical trial shoudl be putting data online somewhere, so someone with the right algorithms shoudl be able to analyze it. Instead that data is silo-ed in whatever hospital is running the clinical trial. Completely asinine and ridiculous. Why does the world run that way. Look at hydroclorequine … trials done by Sergisphere; a compnay no one had ever heard of … Why isn’t that data just out there so everyone can analyze it and see what’s going on?
L: Do you hvae hope that our society will move in a direction where that data will be more out there?
B: Not in the immediate future because US/China friction is very high. There’s some sharing of data; different groups are … Then it’s a very cool playground for putting deep NNs and OpenCog together. … Human longevity and biology knowledge bases on line … We were doing this in the context of a project called Rejuve, spun off from singularityNET, to do longevity analytics; undrestand why some live to 105 years. We had to spin off singularity studio, working with some health care companies on data analytics. ..
…
Decentralized mechanisms of power
Life and death
Would you live forever?
Meaning of life
Hat
L: You were on the Rogan podcast wearing that same amazing hat. Does the hat have its own story?
B: The hat story has not been told yet; we’ll come back and you can interview the hat. It’s too much to pack in. …
Question for AGI
L: If you’re successful at building the AGI system and you get to talk to her and ask her one question, what would it be?
B: Wer’e not allowed to ask—what is the question I should be asking, huh? I’m .. I wrote a story with Stefan Bugai once, where these AI developers create a supersmart AI, aimed at answering all the philosophical questions that had been worrying them … They got this super AGI built, and it … it turned a while, it said, “those are really stupid questions,” and then put itself on a spaceship and left the earth.
L: You’d be afraid of scaring it off?
B: Honestly, there is no one question that rises among … all the others. REally. What interest me more is upgrading my own intelligence so that I can absorb the whole worldview of the super-AGI. Of course, if the answer could be, what’s the chemical formula for the immortality pill, then I would do that. Or, emit a bitstring which will be the code for a super-AGI on the I7 processor.
L: So if your mind was expanded to be super-intelligent, there’s kind of a notion that ..l intelligenc eis a burden; that with greater and greater intelligence, then that other metric of joy that you mentioend becomes more and more difficult.
B: Pretty stupid idea … I think getting root access to your own brain will enable new forms of joy that we don’t have now. .. What IA met was … keep multiple versions of myself. I’d like to keep one version like I am now; but keep the dial to turn pain up nad down and get rid of death. Make another version that fuses its mind with superhuman AGI, and then will become massively trans-human; whether it will send messages back to the human me or not .. will be interesting to find out … It will be on a whole different basis (esp. with respect to time). Could be interesting to put your mind into a dolphin or a space amoeba …. You could imagine one version that doubled its intelligence ever year. Right now, w’ere constrained to think: one mind, one self, one body. But I think we don’t need to be that constrained … when we think about future technology. After we’ve masteredd … this and that.
Berkeley prof, inventor, and hardware guru, #104
David Patterson, Touring award winner and CS professor at Berkeley. He is known for his contributions to RISC (Reduced Instruction Set Computer-design) processor architecture, used by 99% of new chips today, and for co-creating RAID storage. He is also one of the great educators of computer science in the world. His book with John Hennesy, Computer Architecture … was seminal to me (Lex) when I was studying.
<iframe width="560" height="315" src="
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
L: How have computers changed in the last 50 years, at fundamental architectural level, and in general?
D: Biggest thing—invention of the microprocessor. So computers that took up several rooms, can fit inside your cell phone. They got smaller, and a lot faster. They’re much cheaper, and they’re ubiquitous. Probably half the people on the planet have cell phones right now.
L: They’re probably more microprocessors than theree are people …. What is a microprocessor?
D: Well, here’s what’s inside a computer. Computers forever had five pieces: Input and output. Input is speech or typing; output is displays. There’s a memory, like the name sounds, it remembers things. Integrated circuits, you put information in … The third part is the processor—that’s where microprocessor comes from. It has two pieces as well: the control, the brain of the processor. And the arithmetic unit, the braun of the computer. … it does the number-crunching. Those five piecees: input, output, memory, arithmetic unit, and control. The last two are considered the processor. A microprocessor simply means a processor that can fit on a micro-chip.
L: Interesting—the arithmetic unit being compared to the body, the braun. A microprocessor does computation; it processes information. Most of what it does is arithmetic operations. What are the operations, by the way?
D: It’s like a calculator. There are add instructions, subtract instructions, mutliply, and divide … The brilliance of the invention of the comptuer, or the processor, is that it performs very trivial operations, but it performs billions of them per secodn. We’re capable of writing software which takes these trivial instructions and allows them to create tasks better than human beings do.
L: Did you anticipate how good we would be able to get at doing these small operations? How many surprises along the way-?
D: Fundamental driving force is Moore’s Law, named after Gordon Moore, a berkeley alum. He made this observation about semiconductors; where you could build these simple switches … he said, “I think hwat’s going to happen, is the number of switches … is going to double every year for the next decade.” Then he said, maybe it’ll double every two years. That guided the industry. When Moore made the prediction, he wrote a paper back in the 70s … he said, “What will be the implications of this.” He showed ideas of computers being in cars, in things we buy in the grocery store … He not only called his shot; he called the implications of it. If you were in the ocmputing field, and you believed his prediction. .. (you wouldn’t have been surprised). … OTOH, there were these shocking events in your life. I remember driving in Marin, across the SF Bay, and seeing a bulletin board at a local Civic Center, and it had a URL on it. For people at the time … people thought it looked like alien writing. They’d see these advertisements and commercials with alien writing on it. For lay people, it was, “WTH!?” And for the rest of us; it was like, it’s leaking out of our nerdy world into the real world. Then we start seeing advertisements for personal computers; so popular, iti made the newspaper. It was shocking.
L: Taking a step back, what layers of abstraction do you see in a computer?
D: CS Fundamentals; these things are complicated in the ways we … the layers … That means, we suspend disbelief and pretend the only thing you know is that layer; that’s the way we make very complicated things. Probably it started with hardware; it’s proven extremely useful. I’d say in a modern computer today, there might be 10-20 layers … The contract is, all you know is the interface; there’s a set of commands you’re allowed to use. You stick with those commands and we’ll faithfully execute that. You peel down the layers, and you get news sets of instruction. For people who want to study CS … you keep finding those layers. You can take a follow-on course, and get to a lower-level language like C. You get down to the transistor I talked about … you can understand all those layers all the way up to the highest-level application software. … In particular what’s happening right now… there’s getting to be open source version of all of those things. That means, what the engineer'/programmer designs isn’t secret to a company. It’s on the web. You can look, for lots of pieces of software that you use … You can start going layer by layer and see what’s inside—a remarkable time for the interested individual.
RISC vs CISC Computer Architecture
L: Are you thinking, when you say open source at the hardware level—is this going to the design architecture instruction set level? OR to the manufacturer of the actual hardware, whether Asix, Specialized …
D: When you get to the bottom layer of software, the way software talks to hardware is in a vocabulary; we call the words of that vocab “instructions.” Or instruction set. So add, subtract, multiply, divide. There’s instructions to put data into memory, “Store,” and to get data back, “Load.” Those simple instructions go back tto the dawn of computing in 1950. That’s the instruction set … Up until 10 years, these instruction sets were all proprietary. A popular one is owned by Intel; it’s in the Cloud … It’s referred to as the X-86. The first number was 8086; there’s isnce been a lot of numbers; they all end in 86. (“That’s proprietary,” notes Lex.) Then we have Arm; it powers all the cell phones in the world … That’s also proprietary. The new idea that got started at Berkeley, by accident, 10 years ago … Earlier in my career, we pioneered a way to do these instruction sets that was controversial at the time. It used to be that in attempts to have powerful instructions; polysollabic words. “Polynomial-vide” or “sort-a-list.” The hope was that’d make it easier for software. We thought that didn’t make sense for microrpocessors. We called our opposite philosophy RISC. We said for microprocessors, with which Moore’s Law are changing really fast, we think it’s better to have a really simple set of instructions; a reduced set. We’ll just use standard software to cover, to generate more of those simple instructions. One of the pieces of software in that software stack is a compiler; it translates between levels. The technical question was: Since tehre's these reduced instructions, you have to execute more of them. But maybe you execute them faster, since they’re simpler … It ended up, we executed 30-50% more instructions, but they ran four times faster. So these RISC systems were 3-to-4 times better
L: I love that this was controversial and rebellious at first.
D: It was violently debated at sevderal conferences. It was thought it was de-evolution; we’re goingt to make software worse …
L: It’s not intuitive to me, why RISC has won. Maybe I can say a bunch of dumb things … to me, this is an interesting thing, if you look at C++ versus C; with modern compilers, you can write faster code with C++; so relying on the compiler to reduce your complicated code into something simple and fast. To me, comparing RISC, why is it that focusing the definition, the design of the instruction set on very few simple instructions in the long run provide faster execution; versus coming up witih, like you said, complicated instructions—over time, over decades, you come up with compilers that change those into simple instructions for you …
D: If the compiler can do that for you, if the compiler can take a complicated program and reduce to simpler instructions, then the programmer doesn’t care. How fast is the computer I’m using? How costly is it? What happened in the software industry, in the 1980s, critical pieces of software were written not in C or C++, but in assembly language, where you have humans writing at the level that a computer can understand. They were writing add, subtract, multiply instructions—very tedious. To write this lowest-level language called operating systems, that people use, it was believed it had to be in assembly language, since higher-level languages were too inefficient. That changed with a famous operating system called UNIX; the grandfather of all operating systems today. UNIX demonstrated you could write something as complicated as an operating system in a language like C. … Once that was true, that meant we could hide the instruction set from the prorammer; the programmer didn’t have to write lots of these simple instructions—that was up to the compiler … That was part of our argument for RISC; if you had to write your own assembly languages, that was a good argument for CISC, but if the compiler could do that, it’s gonna be, … the computer tranlsates it onces, and then every time you run the program, it runs potentially simpler instructions. That was the debate. People would acknowledge that the simpler instructions could lead to a faster computer. Think of monosyllabic instructions—you could read them faster and say them faster; that analogy works pretty well for hardware. … That’s the basic idea
Measures of performance
RISC instruction set
RISC-V open standard instruction set architecture
Why do ARM implementations vary?
Simple is beautiful in instruction set design
D: … I was always atracted to the idea, “small is beautiful.” It’s easy to make thingsm ore copmlicated. It’s more difficult to come up with a simple, elegant solution. There’s small features of RISC in general, where you can see this example of making it more simple making it more legant. Specificaly in RISC-V … I was the mentor of the program; it’s really led by two graduates (names them). It’s driven by a subset of instructions; the software in RISC-V can run just on those 40 instructions. … Optional features build on that; you don’t need to hvae them. So it has maybe five optional subsets you can pull in, but the software runs without them. This is fantastic educationally; you can explain computers with only 40 instructions; not thousands of them. Also if you invent … new tech., like biological computing, you’d like a nice and easy instruction set. You can run really interesting programs on top of it with optional features; the compilers free to use them. I think it’s a powerful combination. … When they add new instructions, it becomes a required piece, and all microprocessors in the future have to use those instructions … that’s the old way of doing it. As they get bigger and older together … it’s tough. RISC-V means you only have to add extra features if you’r really going to use thme; so you stay as thin as a teenager.
L: Nice analogy. “Here’s the simple core,” that’s the essential …
D: If we brough tback the pioneers from the 1950s and showed them the instruction set architectures—they’d understand them! Talk about philosophical things; there may be something powerful about those 40-50 instructions. All you need is those commands that we talked about, and that’s sufficient to bring about AI. It’s remarkable; surprising to me that it’s as complicated as it is to build these things. A microprocessors … the line widths are narrower than the wavelengths of light. The comands a software executes are straight-forward and haven’t changed much in decades—how surprising!
L: So underlying all Touring machines, all AI systems … might be a very simple instruction set, like a RISC-V
D: Yeah. I had a colleague who’d published an article about 25 years in the future … “I was interested to see how it would turn out.” (??). There must be something fundamental about those instructions that were capable of creating intelligence from primitive operations and just doing them really fast.
L: You mentioned … a lot of ideas. ASICs, domain specific, quantum computers … all those different mediums and types of conversations. Swapping out different hardware systems versus swapping out instruction sets—are they disjoint or fundamentally coupled?
D: When Moore’s laws in full effect, computer designers think, how can we take advantage of that? Turn those transistors into better computers. In the 80s and 90s, computers were doubling performance every 18 months. If you weren’t around them—what would happen was, you’d have your computer, and your friend’s computer, a year or two newer, was much faster than yours. He or she could get their work done much quicker than you. People took their perfectly good computers and threw them away because the up-to-date ones were so much faster. With the slowing down of Moore’s Law, that’s no longer true. I only get a new laptop now when it breaks. The display broke, I gotta get a computer. You’d thrlow them away because they were just so sluggish! That’s a huge change of what’s gone on. Since this lasted for decades, programmers, and maybe all of society, were used to computers getting faster regularly. Now, those of us in computer design … the path forward is to add accelerators … We don’t think general purpose computers are going to get a lot faster; the Intel processors have been barely improving, a few percent a year—it’s doubling every 20 years, now. So to be able to deliver on what Moore’s Law used to do … what’s happening right now is, people adding accelerators that only work well for some domains. By sheer coincidence, at the same time this is happening, has been this revolution in AI called Machine Learning. As I’m sure your other guests have said, AI had two competing schools of thought. Figure out AI by writing the rules top-down, or look at data and figure what the rules are through machine learning. In the last decade or two, ML has won. The hardware you build for ML has multiplied. Matrix-multiply is key for how ML is done. That’s a god send for computer designers; we know how to make … matrix-multiply really fast.. .. Much faster than general-purpose computers have done. We need a way to accelerate things. The danger of accelerating just one application is, how important is that application? Well, ML is important in all sorts of ways—the revolution of Machnine Learning!
How machine learning changed computers
D: As long as people are moving their programs to be embracing more ML … we know how to give them more performance … even as Moore’s Law is slowing down
L: You can say it’s domain-specific, but because it’s leveraging data, it could be very broad in terms of the domains it could be aplied in. (“Yeah, exactly right.”) Software 2.0, we’re almost taking another step up in the abstraction layer in designing ML systems; you’re programming in the space of data, hyperparameters. The specialized devices that accelerate … NN-based ML systems … might become the new general.
D: Interesting to point out: these are not tied together. The enthusiasm about … ML … figuring out programs through data … That’s a movement that’s going on at the same time. Co-incidentally … the first word in ML is “machines”—that’s going to increase the demand for computing. Instead of programmers being smart, writing those things down … That’s the idea. Remarkably, this gets used for image recognition, language translation, game-playing. IT gets into pieces of the software stack like databases, and htingsl ike that. What’s happening on the hardware sisde—Moore’s Law is failing us right when we need it! This idea we’re going to do domain-specific. Here’s a domain, where your greatest fear is you’ll. make this onee thing work, that’ll help 5% of the people in the world. The timing is fortuitious; perhaps if we can keep building hardware that can accelerate ML and NNs; the timing will be right; that NN revolution … The software of the future will be very different than the software of the path. We’ll still have the same basic RISC instrutions; we can accelerate the small piece … That’s what makes this from computer designers perspective a really interesting decade. … … …
Machine learning benchmarks
D: We talked about benchmarks earlier; ML didn’t have a set of benchmarks. We created something called ML-PERF; Machine Learning benchmark suite. Only companies that invested in the software suite could run it. … We’re seeing companies go out of business, and then companies like … Habana, an Isreali company; they came up with ML accelerators; the yhad good pERF scores. … There was another company called Nirvana, owned by Intel … they canceled that, and they bought Habana.
L: Let’s linger on ML-PERF; I love standards and metrics. What are aspectso f thaat portfolio of metrics.
D: One of them; I was involved in the start, with Peter Madson from Google who got it off the ground; we had to reach out to competiotrs. We siad, there’s no bnechmarks here; it’s bad for the field. In RISC days; competitors got together to build RISC microprocessors to create a seet of benchmarks called “spec.” “You can believe microsoft on this but those guys are liars” was what people said before we had agreed-upon benchmarks. So we made the same argument with ML-PERF. Companies worried originally it was a kind of trap; but we got together and did the right thing. From an engineer’s perspectrive, as long as the results are fair, you can live with it. What you hate is if it’s false. From an engineeer’s perspective, if we don’t come in first place, that’s too bad, but it’s fair. Now there’s 10 universities and 50 companies involved, so it’s pretty much now the way you measure ML performance, and it didn’t exist even two years ago.
L: One thing I enjoy about the INternet is, people can see through b.s. a little better. It’s the cool thing to do. to put your engineers forward and show off how well you do on thes emetrics; there’s less emphasis on marketing.
D: Well … I think, because of things like social networking and twitter and stuff; if you put up bullshit stuff that’s purposely misleading, you can get a violent reaction in social media, pointing out the flaws in your arguments. From a marketing perspective you have to be careful today! You can get the word out .. much more easily today than in the past. The other thing that’s been happening for starting-off engineers, in the software sisde, people have largely embraced open source software. Even Microsoft is a proponent now; it’s the standard way software gets built. If you look at the source code, you can see who’s making the permits, who’s making the improvements. You can see who are the great engineers at all of these companies.
L: But that’s of course, not everywhere; in the autonomous vehicles space, the machinery of hype and marketing is still very strong, and there’s less willingness to be open … The ML world is much more willing to hold themselves open.
D: Historically, it wasn’t alwasy that way. I worked with a grad student David Martin—In some fields, benchamrking has been around forever. Computer architecture, databases, … David was working with me and Prof. Malek … David told me, “they don’t have benchmarks” Everyone has their own vision algorithm; everybody had their own image. David figured out a way to do benchmarks when he did his dissertations; to see which algorithms run well. That was, AFAIK, the first time people did benchamrks in computer vision—that led to ImageNet … as the vision commnity got reliigon. Then the guy from Toronto was able to win the ImageNet competition
L: To enter tthe world of benchamrks, you actually have to be good to participate.
D: Yeah-, as opposed to—you just believe you’re the best in the world. I don’t know if people purposely mislead; they just don’t know! …I don’t know, fields that don’t have benchmarks—how do they figure out if they’re making progress
Quantum computing
L: These are vacuum-tube days of quantum computing
D: Idea’s been around for a while. I thought, “I sure hope I retire before I have to start teaching htis.” I talk about the slowing of Moore’s Law. .. and domain-specific accelerators. Quantum computing is in the news all the time. But it’s not right around the corner. Two national reports, one by the Computing Consortium .. their frank assessment of quantum computing said, as far as we can tell, it’s a dcade away. I think about it like I do about nuclear fusion. If we ever get it, it’s going to be fantastic for the world. …. we’re not going to have a quantum cell phone; it’s a 2030 kind of thing. It’s hard with all the news about it not to think that it’s right around the corner; we need to do something, as Moore’s Law is slowing down, to keep computers getting better for the next decade. We shouldn’t be betting on quantum computing delivering for the next few years. I’m glad if it becomes … viable. The old-fashioned computers are going to keep getting better for quite a while.
L: There’s going to be a teenager a few years ago saying, “Look how silly David Patterson was, saying we’re not going to have quantum cell phones.”
D: Well, look—I’m busy trying to handle things that are going to happen in the next decade. I’m not good enough to see what’s going to happen in 30 years …
Moore’s law
L: I spoke with Jim Keller; he’s a little bit rebellious
D: Yeah, he quotes me as being wrong
l: For the record, Jim talks about, his intuition that Moore’s Law is not in fact dead yet, and that it may continue for some time to come. Your thoughts?
D: This is just marketing. What Gordon MOore said is a quantitative prediciton—check the facts. Doubling the number of transistors ever two years. Let’s look at Intel for the last 10 years. Look at D-RaM chips six years ago. Do they have eight times as many chips? Eight times as many transistors? That’s not the case. The problem has been, because Moore’s Law was genuinely embraced by the semiconductor industyr; they made investments into their equipment to make it true. In people’s minds, semiconductor improvement and Moore’s Law are the same thing. Some people thik, when I say it’s no longer happenign, tech. is no longer improving. Jim is trying to … counteract the impression that semiconductors are frozen in 2019, are never going to get better. I never said that! I said, Moore’s Law is no more; that’s what Moore’s Law is … There’s been this aura associated with Moore’s Law that they’ve enjoyed for 50 years; look at the field we’re in, we’re doubling …. But even as Gordon Moore said, no exponential can last forever. It’s amazing how long it lasted; it’s had huge impact on the indsutry. He claims, “Pattern says—it’s no more, but look, it’s still going.” But there’s quite a bit of evidence for what I’m saying. So now I acknowledge there’s a perception problem; I’ll moderate and say “Moore’s Law is slowing down.” I think Jim wouldn’t disagree that it’s slowing down. … Things are still getting better, just not as fast.
L: You don’t like expanding the definition of Moore’s Law
D: As an educator … This is like modern politics. Does everybody get their own facts? Moore’s Law … Carver Meade looked at Moore’s conversations; drawing on a log scale of a straight line … That’s what it is. What Intel did for a while, before Jim joined them, they re-branded Moore’s Law as the cost of the individual transistor going down. That’s not what Moore said, but they re-interpreted it. They believed the cost was continuing to drop, even though they couldn’t get as many chips … The cost isn’t going down anymore, I’ve heard—so even that corollary might not be true.
… the added accelerators to ___ the experience of the phone.
RAID data storage
L: Another exciting space … RAID. In 1988, you co-authored a paper: a case for Redundant Arrays of Inexpensive Discs. That’s where you introduced the idea; that paper had this ripple effect, a revolutionary effect. What is RAID?
D: Work done with Randy Katz, and Darth Gibson (grad student). We’d just done 4th-generation RISC project, and Katz, an early Apple Mac computer guy … everything had floppy discs … To get any work done you were always sticking your floppy disc in and out. The ystarted building hard disk drives; magnetic material … for the Mac. Randy asked the quesiton, when he saw it—”Gee, these are brand new small things.” Befrore that, disks would be the size of washing machines, and here’s something the size of a book or so. “I wonder what we could do with that.” Randy had been involved with RISC 4; we figured out how to make the computation part; what about the storagee part? We hit upon the idea—taking a few of these disks, many of them together; 40 of these little PC disks, instead of one of these washkng-machine sized htings. .. We wrote a paper, and sent it to one of our former Berkeley students at IBM. He said, “What about the reliability of these things.” There are 40 of them; that’s a lot of room for error; the reliability’s going to be awful! It' would break every two weeks! We thought, we’d better address this. So the name, “RAID” … we have extra copies, so if one breaks, we won’t lose all the info; the redundancy is insurance against the breaking that’s going to ahppen; with the redundancy, they become reliable—in fact, more reliable than the washing-machine thing-sized things.
L: Did you hvae a sense, as with RISC, that RAID would take over as a method of storage?
D: I’m naturally an optimist; I thought our ideas were right. If you looked at the history of disk drives, they were getting smaller and smaller. … We thought that was a tech. trend … volume … 8 inches diameter, then 5, then 3 … it made sense to figure out how to deal with things using an array of disks. Logically, we think the tech. forces were on our side; we expected it to catch on. IBM was the big pusher of these disk drives … in the real world, would the tech. advantage turn into a business advantage or not? …
L: If you look up those 30 years—any interesting developments in the space of storage?
D: Yeah. As people built bigger and bigger storage systems, they added more redundancy to tolerate more fialures. Also, in storage, you had things physically spinning—hard disk drives; the noise at a computer’s startup was the disk drives spinning … IF you remember vinyl records, that’s what it looked like … with a needle. Teh big change has been switching over to FLASH; an increasing fraction of all computers are using semiconductor for storage—flash drive. Instead of being magnetic, they’re writing information very densely … all cell phones use flash … The only place you see magnetic disks is in the cloud. Both disks and flash are used in the Cloud. So … the shift has been from primarily disks to primarily semiconductors. … People still use RAID ideas … Psychologically, if you think about it, people have always worried about the reliability of computing. IF you talk about computation, if your computer makes a mistake … The program that was running, you have to re-do it. For storage, if you’ve sent important info away, and it gets lost, you go nuts! It’s the worst. If you have a laptop and you’re not backing it up, you lose all that info! You go crazy! … So yes, RAID ideas are still very popular. Although, flash drives are more reliable. If you’re not backing it up to get some redundancy, you’re taking great risk
Teaching
L: You’ve said teaching and research don’t conflict; they kind of helpe each other. How has teaching helped you?
D: I think .. what happens is, when you’re a college student, you know there’s this tenure system in doing research. This thing that’s popular in America; we c an attract great faculty to research universities … People are up to date and they’re teaching the latest things. If you run into a really bad teacher; students assume they must be a good researcher, or else why are they here? Overall—that is not my experience! I saw a phooto of five of us in the department who won the Distinguished Teaching Award. Manuel Blum, Richard Carp, Randy Katz, ___ Osterhaugh … and myself. All of us are in the National Academy of Engineering. Blum and I have Touring Awards. So that’s the opposite, right?! They’re highly correlated. If you’re very successful people, you’re successful at everything you do.
L: Interesting question, though, specifically for teaching. The Richdard Feynaman idea … is there something for teaching that brings you outside the box.
D: I was going to bring up Feynman. He criticized IAS; it was supposed to be heaven, a university without any teaching. He thought it was a mistake! Having to ecxplain things to students, having them ask questions—that makes you think. That’s synergistic … a university without teaching wouldn’t be as vital, and exciting. I think it helps stimulate research.
L: Favorite concept or idea to teach? Or inspires you, or the students?
D: In general, people are surprised—people who don’t think they like teaching, come give guest lectures, teach a course—and they see the lights come on. They get something that’s important or difficult. Just seeing the lights turn on is a real satisfaction. I don’t think there’s a specific exampel of that. Just the general joy of seeing them understand.
Wrestling
L: I have to talk about this. I wrestle. I do martial arts. I talked to Dan Gabel … You wrestled at UCLA, among many other things you’ve done in your life. AGain, continuing with romanticized questions, what have you learned about life and science from wrestling
D: I wrestled also at El Camino Community College. Just now, we were state champions at El Camino. Talking to my mom; I got into UCLA, BUT Decided to go to the com. college. I did that because of my gf, but also because the wrestling team was really good. We beat UCLA! Part of the reason I brought this up; they’ve invited me back to give a lecture there next month. My friend who was on the wrestling team … we’re reaching out to other members of the team to see if we can have a reunion. In terms of me, it was a huge difference. I was always the youngest person in my class, and I matured later, so I was always the smallest guy. I took nerdy courses, but I was wrestling—that was huge for my self-confidence in high school, and then I got bigger in college. .. I had this physical self-confidence, and it’s translated into research self-confidence … and also, I’ve had this feeling, even today in my 70s—if something’s going on in the street that’s bad physically, I’m not going to ignore it ….
L: That confidence carries through in the entirety of your life
D: The same is true intellectually; if someone’s saying something that’s not true … it’s my job to stand up and say something. …….. I had great coaches, and they believed, even though it’s an individual sport, we’ll do better as a team if we bond together. Made us more successful. So I picked up skills of how to build successful teams. Most would say, a strength I have is to get teams of faculty to pull together toward a common goal. Also, I think, I heard this line about, if people are in sports with physical contact, people are a little bit more assertive, or something? I think that also comes through. I didn’t shy away from the RISC-CISC debates … I’m really glad I did wrestling … Sports done well, there’s really lots of positives you can take about it. Leadership, how to form teams, how to be successful.
L: Metric you’ve develpoed in terms of weight lifting. Let’s talk about metrics more broadly.
Meaning of life
L: Let’s look at the meaning of life. What mettrics would you put on a life well lived?
D: Randy Katz said this—when it’s time to sign off, the measure isn’t the number of zerioes in your bank account; it’s the number of inches in your obituary in the NYT. But then, people don’t die wishing they’d spent more time in the fofice. A lot of what I’m proud of is not papers. My family; my wife, married to me longer than 50 years. Kids and grandkids. Education things I’ve done. I did some help with underrepresented groups that was effective. I’ve had hundreds of papers. Some of the (things I was proud of) were the papers. .. But going after the dollars, or going after all the papers in the world, those probably aren’t the things you’re gonna care about. I read a book, just before I got to Berkeley: what I got out of that book was, the people who felt good about what they did was the people who affected people. … That was a correct assessmetn. It’s the people you work with, the people you influence, the people you can help.
P.S. How to stay married for 50 years: “I was wrong; you were right; I love you.” Freely acknowledging you made a mistkae .. and then that you love them, that really hellps get over a lot of the bumps in the road.
Biochemist and millionaire drug designer from MIT, #105
Robert Langer, a professor at MIT, and one of the most icted researchers in history. Specializing in biotech fields, drug delivery systems, and tissue engineering. He has bridged theory and practice in being a key to launching many biotech companies out of MIT. Alas, this was a pre-pandemic interview … since this guy is part of companies that are trying to create a vaccine for Covid-19.
<iframe width="560" height="315" src="
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
L; You have a bit of a love for magic. You see a connection between magic and science?
R: I do. Both can surprise you. There’s something magical about making scientific discovery?
L: On the magic side, is there a scientific side to the tricks themselves. There’s a dual nature to it. From the inside, you’re the only person who knows how it works. The outsider doesn’t know …
R: The duality I see is fascination. When I watch magic myself, I’m always fascinated by it; sometimes it’s a puzzle to see how it’s done. Just to see that something you didn’t know coudl happen, does happen. I think about that sometimes with science too.
L: What is the most amazing magic trick you’ve ever seen?
R: The invisible pack. You have this pack, hold it up, you say to somebody … this is invisible. You say “shuffle it” … You ask them to shuffle their imaginary card … You aks them to pick a card, any card, and show it to me. I look at it. Let’s say it’s the 3 of hearts. I say put it back in the deck, but turn it upside down from any other card. They do that again, imaginary. I say shuffle it … they shuffle it. I say so there’s still one card upside-down, what is that? They say 3 of hearts. I say it just so happens in my back pocket, I’ve got only one card in it … there’s only one card in it, it’s a 3 of hearts.
L: You can do this trick? Beautiful. Ok let’s get into the science. As of today, you have over 295,000 citations, an H_index of 269 … you’re one of the most-cited people of history. And yet nothing great is achieved without failure. The interesting part: what rejected papers had the most impact on you
R: I’ve had plenty of rejection; I suppose one way of thinking about this, when I first started, we made two big discoveries, and they were kind of interrelated. One, I was trying to isolate, with my advisor, substances that could stop blood vessels from growing. Nobody had done that beofre. That was Part A, and Part B was to develop a way to study that. What was critical was to have a way to slowly release those substances for more than a day, maybe month. The first draft, we sent to Nature, and they rejected it; the second draft we sent to Science and they accepted it. When we got the rjeectirons, it was really upsetitng. I thought we’d done some really good work. It was very depressing to get rejected like that.
L: Can you linger on the feeling, when you get the rejections. Now, people know you as a brilliatn scientist, but at the time, I’m sure you’re full of self-doubt. Did you believe, that maybe this idea was actrually quite terrible?
R: You feel depresesd; I felt the same way when I got grants rejected. You have multiple emotions. One is being sad, being upset; also maybe being a little bit agnry. But then as I thought about it more, I thought, “Well, maybe I just didn’t explain it well enough.” I guess maybe I’ll explain ti better next time. You get reviews …
How to come up with big ideas in science
L: You’ve given advice to students to do something big. Rather than something incremental. How did you yourself seek out such ideas. Is it a proces,s or is it more spontaneous.
R: The latter. It’s partly exposure things … My postdoctor advisor was good at that; you could see he had big ideas. … A lot of people might just keep doing derivative stuff. It’s not something that i’v eever done .. systematically.
L: In the space of ideas, how many, when you see them, it’s something that could be impactful?
R: Hard to say; there’s multiple levels of idea. …making tissue from scratch for the first itme … or just deeply understanding something. That may lead to other things. Sometimes you could think of a new technology; other time,s things came just from the process of trying to discover things. You don’t necessarily know l… people talk about hot omoments. .. It’s taken me a long time to go form the thought process of starting a new idea to actually completing it. .
L: If you take drug delivery for example; is the intial notion a very general one—we should be able to do this? Then ask how?
R: I think that’s right … Many many examples. Large molecules we use to study blood vessel inhibitors. … Other times, sometimes it’s understanding what goes on, the mechanisms; it’s not a single thig. There’s many parts to it. Over the years, we’ve discovered different principles … for aerosols
L: Let’s explore some key ideas. ‘First—how complicated is biology and chemistry of the human body, from perspective of trying to affect some parts of it in a positive way. For me, coming from field of CS, engineering, and robotics—it seems the human body is exceptionally complicated
R: I agree; we’re still scratching the surface in many ways. But we are having progress in many ways … Others might invent tech. that might be helpful in exploring. Over years, we have understood things better and better.
L: ARe there things that … are there knobs that are reliably controllable about the human body? … If you start to think about controlling various aspects … chemically, of the human body, iks there a solid understanding across populations of humans?
R: That’s hard to do. Whenever we make a new drug or medicadal device, we are doing that in a small way. But I don’t know they’re great knobs; but if there’s a biological pathway or something you can affect or understand, then that might be sucha knob.
How to make a new drug
R: I’ll give an example: When doing my post-doc work with Foulkman, wwe wanted to make a drug that would affect blood vessel growth.
L: What’s a blood vessel—why make it grow or shrink?
R: Could be an artery, vein, or cappillary; it provides oxygen, nutrients, gets rid of waste. To different parts of your body. The blood vessels end up being important. If you have cancer, blood vessels grow into the tumor, enabling the tumor to get bigger, and enables it to metastasize. So … we wanted to see if we could find substances that could stop that from happening. First we had to develop a bio-essay to study blood vesel growth. Needed polymer systems since blood vessels grew slowly. .. After those two things … I had isolated many different molecules … from cartilage. Almost all didn’t work. But we were fortunate, we found one that did work. That paper, I mentioned in Science in 1976 … those were the isolation of the very first angiogenesis blood vessel inhibitors.
L: A lot of words there: what are polymers? Bioessay? What is the process of trying to isolate this thing so that you can control an experiment with it.
R: Polymers are plastics and rubber;
L: Something that has structure and could be useful for what?
R: For delivering a molecule for a long time; slowly diffusing out of that … to where you wanted it to go
L: So there might be blood vessels you could target, and over a long period fo time … place the polymer there, and it would be delivering a kind of chemical. What’s the bio-essay?
R: That’s what you study blood vessel growth
L: You can control that somehow—do we understand what kind of chemicals control it?
R: That’s what we were going after. We showed such molecules existed, and developed methods for studying them. … What would happen … We publisehd that in 1976. Over the next 28 years, others would follow in our footsteps; ultimately, to make a new drug takes billions of dollars. There were different growth factors people would isolate, sometimes using techniques we’d developed … figure out ways to stop growth factors, ways to stop blood vessels from growing. It took many years and millions of dollars. In 2004, the first of those inhibitors got approved by the FDA; it’s one of the top-selling drugs in history.
L: So in general, one of the key ways you can … What’s the hope in terms of cancerous tumors; what can you help by being able to control the growth of vessels
R: If you cut off the blood supplly; it’s like in a war. If the nutrition is going to the turmo, and you can cut it off .. you starve the tumor, and it becomes very small. Then it’s more amenable to other therapies, because it is tiny.
… it really vafies; sometimes people do know (the drugs they’re making); sometimes it is shooting in the dark …
L: What is the discovery process for a drug? Is it … a mix of … serendipitous discoery and art? OR is there systematic science to trying different chemical reactions.
R: I don’t think there’s a single way to go about something …
Drug delivery
L: Do you think it’s possible there could be robotic-like systems roaming our body long-term, and able to deliver drugs in the future
R: Some day; I don’t think we’re very close to it yet. That’s nanotech., and maybe miniaturizing some of the things I just discussed.
: some of it is just the shrinking in the tech.
R: That’s a part of it
L: What role do you see for AGI; making systems that make intelligent decisions. Do you see any of that data-driven computing systems having a role in any part of this. … Delivery ,drugs, any part of the chain.
R: I do, I think it could be useufl. One, if you get a large amount of info., say you have chemical data, from high-thruput screens … let’s say I’m treating Disease X, whatever it is, and I have a test for that, hopefully a fast test, and I test 10,000 chemical substances. With the right kind of AI, maybe you could look at chem. structures, see if there’s commonalities in what works, in what doesn’t work—use that somehow to predict the next geenration of things you’d test.
L: Thoughts on our society’s relationship with pharmaceutical drugs? Do we overrely on them? Do we propertly subscribe them?
R: Pharmaecuetical drugs are really important to life expectancy and life quality, over many many years, both have increased teremendously. Woudl also be good if we could extend that more and more to people in the developing wor.d. Our lab is doing that with the Gates Foundation. If there was some way to reduce costs, that’s certainly an issue; if there were a way to help people in poor countries. Of course we still need to make better drugs for so many diseases. Herart disease; rare diseases; many situations it would be great to do better …
Tissue engineering
L: Another exciting space … regenerative medicine
R: That has to do with building an organ or tissue from scratch. Some day maybe we can build a liver or make new cartilage. Also this would enable you to … ___ organs on a chip. ..
L: Organs on a chip; that sounds fascinating. What are the various ways to generate tissue? One is of course from stem cells; is there other methods? What are the possible flavors here?
R: Multiple components; one is having some time of scaffold; we started that many years ago. Onthat scaffold you might put … a cartilage cell, a bone cell, could be a stem cell
L: A scaffold, is kind fo like a canvas?
R: I think that’s a good explanation, what you just did. A canvas, that’s good. The chip could be such a canvas; could be fivers made of plastics that you put in the body
L: When you say chip, do you mean electronic chip?
R: Not necessarily, it could be, though. It could just be a structure … Canvas is not a bad word.
L: … a computational component … electronic component … control some aspect of this growth process for the tissue
R: Now people aare working mostlyh on validating these kind of chips; to say, yeah it does work … but some day what you suggested would be possible
L: What kind of tissues can we engineer today
R: Skins already been approved by the FDA; phase 3 trials are at completion or near completion for making new blood vessels
L: Wait so heuman skin can be grown? IT’s already approved through the entire FDA process. One, that means you can grow that tissue and do various kinds of experiments … Does that mean, some kind of healing and treatment of different conditions on humans?
R: Yes, different groups and companies and professors have made them. They’ve been approved for burn victims and … diabetic skin ulcers
L: What else?
R: At different stages … skin, blood vessels; there’s clinical trials to help patients hear better, for patients that might be paralyzed, patients with different eye problems. Groups have worked on new livers, new kidneys. Some of this work is early
L: What about neural tissue? Even the brain?
R: There are people … who have done a lot on neural stem cells … some of our spinal cord work
L: Are there challenges to getting the body to accept the new tissue that’s being generated? how do you solve that?
R: There can be problems; you might mean rejection by the body. There’s multiple ways people are trying to deal with that. One way … we did with Dan Anderson, one of my post-docs; with a pancreas … encapsulating the cells … immune cells and antibodies can’t get in; that’s a way to protect them. Another way is to make the cells non-immutagenic … using some gene-editing approaches. Of course if you use the patient’s own cells or cells from a relative …
L: It’ll increase the likelihood it’ll get accepted
R: Yeah. And finally there’s some suppressive drugs …
L: The fact this whole thing works is fascianting; from my outside perspective. Will we one day be able to regenerate any organ? … Do you see some tissues as more difficult than others?
R: I’m an optimist. Some day could be hundreds of years,s but yeah, some day I think we’ll be able to regenerate many things. There’s different strategies that (might be used).
L: What does that mean for longevity? IF we look, 10, 20 years out. The possibilities of the research that you’re doing; does it have a significant impact of longevity of human life?
R: I don’t know that we’ll see a radical increase, but in certain areas, we’ll see poepple leadd better lives, and somewhat longer ones.
Beautiful idea in bioengineering
R: What’s happening right now with Krisper is a beautiful idea; wasn’t my idea. Very interesting here, what people have capitalized on; that there’s a mechanism by which bacteria are able to destroy viruses; understanding that leads to machinery to sort of cut and paste genes and fix a cell.
L: So that kind of … Do you see a promise for that kind of ability to copy and paste? Like we said, the human body is complicated. That seems exceptionally difficult to do.
R: I think it is … But that doesn’t mean it won’t be done. There’s a lot of companies trying to do it. Some of the ways you might lower the bar … not necessarily doing it directly; you could take a cell that might be useful, you want to give it some cancer-killing capabilities … The way a lot of things have moved in medicine over time … ste-wise. I can see things that might be easier to do than fix the braain; that would be very hard to do.
Patening process
What does it take to build a successful startup?
L: Youv’e been part of launching 40 companies worth an estimated $23 billion. You’ve described your thoughts on a formula for startup success … could you describe that formula
R: There are a couple ctegories. I’m a scientist … but the most important thing is the business people that I work with; when I look at the companies that have done well .. it’s because we have great business people. From a science standpoint, we’ve made some kind of discovery … like a platform, we can use it for different things. Like drug discovery; we could use it for drug A, B, C, D, E, and so forth. .. We’ve written at least one really good paper in a top jouornal; we’ve reduced it to practice in animal model .. we’ve drawn patents … issue pattents … that’s kind of key in a patent. A lot of time it’s somebody in a lab has spent a big part of their life doing it; they have this passion, wanting to see something they did make a difference in people’s lives.
L: It’s funny to hear scientists say there’s value to business folks. What business instinct is valuable …
R: I think … You have to be a good judge of people so you hiree the right people; you have to be strategic, so you figure, if you do have a platform. .. knowing that medical research is so expensive, what are your top priorities? You have to have a good FDA regulatory clinical trial strategy. You have to be able to raise money.
Mentoring students
L; One of the largest academic labs in the world, with $10 illion in annual grants … researchers can be individualistic and eccentric. So what insights into research leadership can you give?
R: I don’t know that I’m any expert; what you do, to me, I just want people in the lab to be happy, to be doing things that I hope will make the world a better place. I guess my feeling is, if we’re able to do that, it kind of runs itself.
L: How do youj make a researcher happy?
R: I think when people feel … they’re working on something really important that can affect many other people’s lives, and they’re making some progerss. They’ll feel good about it.
Funding
Cookies
L: I’ve read that you love chocolate, and mentioned tow places: Ben & Bill’s Chocolate Emporium and (names one other place). I wasted too much time at 3 a.m. staring at Rosie’s Baker’s cookies—they just look incredible! For me, oatmeal white raisin cookies won my heart. Dlo you think chemistry could help us engineer the perfect cookie? Or is cookies something that’s mroe art than engineering.
R: I think there’s some of both; engineering will probably help some day. (“What about chocolate.”) Same thing … you’ll have to check out (mentions some guy’s name).
What are you most proud of?
L: My students. We’ve had close to a thousnad go through the lab; 18 are in the National academy of Engineering, 14 (16?) in another national academy, they’ve beenpresidents of universities, etc., etc., it makes y9ou feel good to think that the people … they’re not my children, but they’re close to my children in a way …
Deep Mind neuroscientist, #106
He doesn’t have a Wikpedia, but he works at Stanford AI
Matt Botvinick, director of neurscience research at Deep Mind. He’s a brilliant mind navigating effortlessly between cognitive psychology, computational neuroscience, and AI.
<iframe width="560" height="315" src="
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
L: How muc of the human brain do you think we understand?
M: I think we’re at a weird moment in neuroscience in which … I feel like we understand a lot aboutt the brain at a high level, but a very coarse level.
L: Elaborate. Functional? Structural?
M: What is the brain for? What kinds of computation does the brain do? What kinds of behaviors would we have to … explain if we were going to look down at the mechanistic level. At that level we understand much much more htan we did a few eyears ago. But we’re seeing it through a fog; it’s a very coarse level. We don’t understand the neuronal mechanisms. … We’re better at asking the functions … At the other end of the specturm, in the last few years, incredible progress has been made in terms of technologies that allow us to see … literally see, in some cases, what’s gong on at the single-unit level, even the dendritic level. Then there’s this yawning gap in between.
L: High level is almost cognitive science. Lower level is neurobiology, neuroscience, studying single neurons …
M: Blanket statement i should probably make—as I’ve gotten older, I’ve become more reluctant to make a distinction between psychology and neuroscience. To me, the point of neuroscience is to study what the brain is for. If you’re a neprhologist and want to learn about the kidney, you start with, what is this thing for? IT takes blood on one side, that has metabolites that shouldn’t be there, sucking them out of the blood … and excreting in the form of urine. The rest of the work is diecicing how it does that. This it seems is the right appraoch to the brain. What’s the brain for? For producing inputs; going from perceptual inputs to behavioral outputs … the outputs should be adaptive. The rest of it is figuring out how those outputs are carried out at a mechanistic level
L: The gap between the electrical signal and behaviroo. … you see neuroscience as the science that touches behavior, how the brain converts raw visual information into understand—-you see cognitive sicence, neuroscience … as all one science … Is that a hopeful or a realistic statmeent? Certainly you’ll be correct in some number of years, but that could be 200-300 years from now. Is that aspirational?
M: It’s both, in the sense that … this is what I … hope and expect will bear fruit over the coming decades. But it’s also, pragmatic in the sense that, I’m not sure what we’re doing in either psychology or neuroscience if that’s not the framing. I don’t know what it means to understand the brain if .. part of the enterprise is not about understanding the behavior that’s being produced.
Psychology
The paradox of the human brain
L: … this very conversation is happening because two brains are comunicating … the subjective nature of the experience … a small tangent into the mystical … consciousness … Or are you talking about the mechanism of cognition? For me, it’s almost paralyzing, the beauty and mystery … that it creates the entirety of the experience.
M: I definitely resonate with that latter thought; I often find discussions of AI to be disappointingly narrow; speaking as someone who’s always had an … There are layers to the human experience … emotion … There’s an incredible scope to what humans go through in every moment. Yes, so that’s part of what fascinates me is that our brains are producing that; at the same time, it’s so mystserious as to how! Our brains are literally in our head producing the experience. The scientific challenge of getting at the actual explanation for that is so overwhelming … certain people have fixations on particular questions, and that’s always been mine …
L: … it always saddens me … when you try to create a. benchmark for the AI community, so much of the magic of language is lost. … Something that makes a rich experience … … I wonder how to get it back in; the moment you try to do really good rigorous science, you lose some of that magic. When yo utry to study cognition in a rigorous way .. you lose some of that magic. At this stage in our history … One thing I find exciting about that first wave of Deep Learning models … The people buildingthese models were focused on richness and cmplexity of cognition. The debate in cognitive science … sounds very dry—the formation of the past tense. Two camps. One said, the mind encodes certain rules .. it also has a list of exceptions; the rule is add ‘ed’ but that’s not always what you do. Then second, there were the connectionists, who evolved into the Deep Leearning peopel … Said if you look carefully at the data, it turns out to be very rich. Yes, there’s … most verbs have that major rule … But there’s also, rules (within the exceptions); the exceptions aren’t just random. There’s certain clues to which verbs should be exceptional. There’s exceptiosn to the exceptions. There was a word deployed in order to cpature this—”quasi-regular.’ There’s rules but it’s messy, structure within the exceptions. You could try to write it down in a closed form, but the right way to understand the way tthe brain handles and produces this, is to build a deep NN … and see how it … The way DL was deployed in cognitive psyche. … that was the spirit of it; it was about that richness, and that’s something I always found very compelling. … still do!
… What I would say is something a lot of people are saying, which is that … one seeming limitation of the systems that we’re building now is that they lack the kind of flexibility, the readiness to turn on a dime, when the context calls for it, that is so characteristic of human behavior.
l: Is that connected, in your mind — which aspsect of the NNs in our brain is that connected to? … mY natural inclination is to separate into three disciplines; neuroscience, cognitive science, and psychology. You’ve … kind of shut that down. But just to look at those layers … is there something about the lowest layer in the terms of how the neurons interact that is profound to you … Or is the key difference at a higher level of abstraction?
M: I often think about .. if you take an introductory CS course, and they introduce the notion of Turing machines; one way of articulating their significance is … it’s a machine emulator. It can emulate any other machine. … That, to me, … that way of looking at a Turing machine really sticks with me. I think of humans as sharing in some of that … character. We’re capacity limited; we’re not Turing machines … but we have the ability to adapt behavioors unlike anything we’ve done before. Some mechanism implemented in our brain allows us to run software.
L: You mentioned a Turing machine … but fundamentally our brains are just computational devices in your view? .. Is there any magic in there, or is it just basic computation?
M: I’m happy to think of it as just basic computation; but iI won’t be satisfied until I know what the basic computations are. … It won’t be enough to know the computations that allow people to do arithmatic or play chess. I want the whole thing.
Cognition is a function of the environment
L: L: Small tangent—we mentioned coronavirus. … anything you have to say about … the behavior of large groups being interesting?
M: I … I have the honor of working with a lot of .. smart people. I don’t want to take any credit for leading the way on multiagent work coming out of Deep Mind recently; I do find it fascinating. It can’t be debated; human behavior arises within communities … that seems self-evident.
L: For me, that seems to be a profound aspect of something that created … You look at 2001: A Space Odyssey … That’s the magical moment … Yuval Harari writes … shaking hands versus bumping elbows; somehow converge, without being in a room all together; distributed convergence toward an idea over a period of time, seems to be fundamental to every aspect of our cognition and intelligence. Humans, it seems we don’t really have a clear objective function undre which we operate … but we sometimes converge toward it nonetheless.
M: I think there’s something to the arugment that that kind of strictly bortom-up approach is wrong-headed. There are basic phenomena that can only e understood in the context of groups. I’m perfectly open to that. I’ve never been particularly convinced by the notion that we should consider intelligence to inhere at the elvel of communities. I’m stuck at the … the basic unit that we ought to understand is the individual human. … I stubbornly define intelligence as an aspect of an individual human.
L: I”m with you, but that could be the reductionists’ drema of a scientist … It could also be that intelligence can only arise, … when there are multiple .. humans. That if there were only one human; it. would not become that intelligent.
M: I think a serious effort to understand human intelligence … may be to build a human-like intelligence, needs to pay as much intention to the structure of the environment as to the structure of the cognizing system, whether a brain or an AI system. That I took away from early studies of the pioneers of NN research; Jay Mclellan and john Cohen … the structure of cognition is only partly a function of the arhcitecture of the brain; the learning algorithms that it implements. What really shapes it is the interaction of those things with the structure of the world in which those things are embedded.
L: That’s made most clear in Reinforcement Learning. You can only learn as much as you can simulate. That’s what Deep Mind made very clear … the self-play mechanism … the other agent becomes the environment. That’s one fo the most exciting ideas in AI—the self-play … thing. …
Prefrontal cortex
L: The dirty mess of the human brain. Is there something for people who might not know; could you discuss the key parts of the brain? The different parts that you’re curious about, that you’ve studied?
M: My area of expertise, if I have one, is the prefontal cortex.
L: What’s that?
M: It depends on who you ask; the technical definition … is anatomical. There are … parts of your brrain that are responsible for motor behavior; they’re easy to identify. The region of your cerebral cortex, the outer crust of your brain, that lies in front of those, is defined as the prefontral cortex.
L: If you say anatomical; that’s referring to the geographic region, as opposed to some functional defintiion.
M: Yes … This is sort of the coward’s way out …
L: It’s the thing in the front of the brain.
M: In fact, the early history of neuroscientific investigation of what this front part of the brain does is sort of funny to read; it was really World War I that started people down this road of trying to figure out what different parts of the brain do. .. there were a lot of people who cmae back from the war with brain damaae! As tragic as thatw as, it gave scientists the ability to learn about brain regions. A frustration was … scientists couldn’t identify exactly what the deficit was that went with this front part of the brain. A couple of neuropsychologists identified, through a lot of clinical experience, started to put their finger on a syndrome associated with this region. One scientist named Luria; he started to figure out that the prefontal cortex was somehow involved in flexibility; in guiding behaviors that required someone to override a habit, or to do something unusual, or to change what they were doing in a very flexible ways.
L: focused on new experiences; the way your brain …
M: What later helped bring this fucnction into better focus was a distinction between controlled and automatic behavior. I.e. habitual behavior versus … ____ behavior. It’s very clear the brain has pathways that are dedicated to habits; things that you do all the time, and they need to be automatized, so you don’t need to concentrate too much. Think about the difference between driving when you’re learning to drive, versus driving when you’re fairly expert. There are brain pathways that slowly absorb those freequently performed behaviors.
L: That’s kind of like the purest form of learning, which is hapepnign there. Which is why it’s perhaps the most useful to focus in on , to see how AI systems can learn.
M: … … thinking about where we are in AI research. But just to finish the dissertation here, the role of the prefrontal cortex, is understood these days, in contradistinction to that habitual domain. The prefrontal cortex is what helps you overrirde those habits. It allows you to say, “What I usually do in this situation is X; but given the context, I should do Y.” The elbow bump is a great example. …
M: … for me, it’s an urgent question … whether to pursue (testing of other organisms). To put it briefly, there’s disagreement. People who study fruitflies will tell you, “Hey fruitflies are smarter than you think.” They’ll point to experiments where fruitflies learn new behavior; they suggest there are abstractions which guide their behavior. I’ve had conversations in which I’ll start by recounting some observation about mouse behavior, where it seemed like mice would take an awfully long time to learn what for a human would be an extremely trivial task. But then a mouse researcher will say to me, ‘Hold on! … Problem was, you asked a mouse to deal wit hsitmuli and behaviors that were unnatural for the mouse. If you’d put it in a way that aligns with what mice are used to dealing with, you might find the mouse has more intelligence than you think.’ They have videos of mice doing strickingly inelligent things in their natural habitats. So I think these are open questions, to sum that up.
L: Taking a small step back; you mentioned we’re taking a shortcut by saying t’s a geographic part of the brain. But … what’s your sense, in a bigger, philosophical view: prefrontal cortex of the brain in general … is it a set of subsystems, as we’ve implied? To what degree is that, or to what degree is it a giant, interconnected mess?
M: Overwhelming evidence of functional differentiation; it’s clearly not the case that all parts of the brain are doing the same thing. This follows from the studies of brain damage. It’s obvious from what you see if you stick an electrode in the brain, measuring neural acivity … Having said that, there’s two other things to add. One is that … when you look carefully, at functional differentiation in the brain,whhat you usually end up concluding is that the differneces between regions are graded rather than being discrete. Thus it’s not easy to divide the brain up into true modules with clear boundaries. …
L: This applies to the prefrontal cortex?
M: Yes, iut’s made up of subregions, the functions of which are not clearly defined; the borders of which seem to be quite vague. Another thing that’s popping up in recent research; which involves application of these new techniques … There are a number of studies that suggest that parts of the brain we’d have previously thought were quite focused in their function are actually carrying signals we wouldn’t have thought would be there. … Looking at the primary visual cortex … The first waystation for processing information; what it should care about is, where are the edges in this scene that I’m viewing? … Turns out if you have enough data .. you can recover info from the visual cortex about all sorts of thingsg! What behavior is being engaged in, how much reward is an offer? Even regions whose function is pretty well-defined at a coarse grain, are nonoetheless carrying some info about … very differnet domains. The history of neuroscience is htis oscillation between the two vies you articulated. .. the modular view, and the mush view. We’ll end up somewhere in the middle. Which is unfortunate for our understanding; it’s difficult to think about this lying-in-between.
Information processing in the brain\
L: Unless we can understand deeply the lower-level neuronal comunication … .soemthing there’s still mystery and disagreement on … HOw does the brain .. .communicate .. signals?
m: I gues I’m old-fashioned; I consider the networks we use in Deep Learning to be a reasonable approximation to the networks that carry information in the brain. What really matters is a rate code—how quickly is an individual neuron spiking? What’s the ferquency at which it’s spiking. Is it fast or slow? Let’s put a number on it … There’s still uncertainty about whether it’s an adequate ..description. There are studies that suggest the precise timing of spikes, matters. There may be computations going on in the dendritic ttree, within a neuron, that are rich and structured. .. Having said that, I feel like we’re getting somewhere by sticking to this high level of abstraction.
L: Bu the way, we’re talking about the electrical signal. I remember reading a paper about the mechanical signal…. The vibration of the neurons also communiucates information. They were arguing the electrical signal .. is actually a side effect of the mechanical signal. … It’s almost an interesting idea that there could be a deeper … It’s always in physics, with quanutm mechanics, there’s always a deeper story that could be underlying the whole thing.
M: Thisis a classical view … The only way in which this stance woudl be controversial .. would be … There are members of the Neuroscience community .. who are interested in alternative.s But this is a mainstream view. Neurons comunicate when neurotransmitters arive; they was hup on a neuron. The neuron has receptros for those transmitors. Receptros + transmitters leads to a change in voltage of the neuron. If great enough, a spike occurs. “This is Neuroscience 101; the way the brain is supposed to work. Now when we build artificial NNs, we don’t worry about those individual spikes. .. The frequency … at which they’re generated .The activity of units in a DL system is broadly analagous to the spike rate of a neuron. There are people who believe there are other forms of communication in the brain … The voltage fluctuations that occur in populations of neurons that are below the level of spike production, may be important for communication. I’m old-school; I think the things we’re building in Ai research constitute. … ___ of how the brain would work. …. ….. I wouldn’t be a scientist if I thought there were any chance we were wrong. If you look at the history of DL research as it is applied to neurscience. If you back to the 1980s, there’s an unbroken chain of resource, in which a particular strategy is taken: let’s train a DL system, a multilayer NN, on this task that we trained our Mac on, or monkey on, or human being on. Let’s look at what units deep within the system are doing? Let’s ask if that resembles what neurons deep in the brain are doing? Again and again and again, that strategy works—the learning algorithms we have access to … center on backpropagation; give rise to patterns of activity and response …
Meta-reinforcement learning
Dopamine
L: You’re co-authoring … a recent paper on dopamine and temporal distance learning.
M: I want to acknowledge my co-authors … on this paper … Lex posts “A distributional code for value in dopamine-based reinforcement learning.” with authors I presume. A lot of the work that we’ve done so far is taking ideas that have bubbled up in AI and asking the question of whether the brain might be doing something related. On the surface that seems mainly of use to neuroscience. We see it also as a way of validating what we’re doing on the AI side. … If the brain is using some technique that we’ve been trying out in our AI work; that gives us confidence that it may be a good idea.
Neuroscience and AI research
Human side of AI
Dopamine and reinforcement learning
Can we create an AI that a human can love?
M: I love that question. It relates closely to things that I’ve been thinking about a lot lately in cotext of human AI research. Social psyche. research by Susan Fisk at Princeton, where I used to work; she dissects human attitudes toward other humans, into two-dimensional scheme; one dimension is about ability—how able, how capable is the other person? The otehr dimension is warmth. You can imagine another person who is very skilled and capable, but who is very cold. You might have some reservations about that person. There’s also a reservation we might have about a person who displays a lot of human warmth, but is not good at getting things done. We rserve our greatest esteem for peoplle who are highly capaable and quite warm. This isn’t a normative statement; it’s empirical. These are the two dimensions along which people size other people up. In AI research; we focus on the capability thing. This can play Go at a superhuman level—that’s awesome. But that’s only one dimension! What would it mean for an AI system to be warm? Maybe there’s easy solutions—put a face on it. Big ears. But it’s also, a pattern of behavior. What would it mean for an AI sytem to display caring, compassionate behavior? … So we didn’t feel it was simulated; like we were being duped! People talk about the Turing test; I feel like that’s the ultimate Turing test. An AI system that cannot only convince us … that it can resason. But we’re confident saying, “That AI system’s a good guy!”
L: We intuitively understand it; but … we don’t understand it explicitly enough yet to be able to engineer it. … You addressed it in the human-AI interaction. That’s a question that should be studied
M: We humans are so good at it! It’s not just that we’re born warm; I suppose some people are warmer than others—genetically. But there’s also learned skills involved. There are ways to communicate to other people that you care, that you enjoy interaacting with them. We learn these skills from one another. It’s not out of the question that we can build … systems … we can hnd-design these sorts of behaviors. Systems that set them out in the right direction; so that they. end up learning what it is to interact with humans in a way that’s gratifying to humans. If that’s not where were headed, I want out!
Unusual philosophy interview: ethicist Peter Singer, #107
Peter Singer, profesor of bioethics at Princeton best known for his 1975 book Animal Liberation that makes an ethical case against eating meat. He has written brililantly forom an ethical perspective on extreme poverty, euthenasia, human genetic selection, sports doping, the sale of kidneys, and generally happiness. His books Ethics in the Real Wrold and The Life You Can Save. He’s an early popularizer of the Effective Altruism movement!
<iframe width="560" height="315" src="
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
L; When did you first become conscious of the fact htat there is much suffering in the world.
P: As soon as I was able to understand aything about my family and its background. 3 of 4 grandparents lost in the Holocaust! I knew why I only had one gradnparent—she was a survivor of the camps.
L: My entiere family comes from the USSR … The suffering the war brought is in the music, the literature, in the culture there. What do you think was hte impact of the war broadly on our society?
P: The war had many impacts. I htink one of them, a beneficial one, is that it showed what racism and authoritarian government can do. At least as far as the west was concerned, I think I grew up in an area where there wasn’t the overt racism and anti-semitism that had existed for my parents in Europe. I grew up in Australia,, and that was seen as something unacceptable. There was a fear of further outbreak of war, that this time, we expected owuld be nuclear; there was this overshadowing of my childhood about the possibility I would never grow up to be an adult because of catastrophic nuclear war. The film On the Beach was made …
L; Much beauty comes from war. Don’t you think? The mirror it put to our society; the ripple effects of it …
P: I find it hard to see positive aspects. Some of the things others think of as positive and beautiful, I’m maybe questioning. The certain kind of patriotism—people say we all pull together against the common enemy. It’s true an outside enemy unites a country. But it also engenders a nationalism that can’t be questioned; I’m more skeptical about that.
L; What about the brotherhood people, soldiers, talk about. The counterintuitive, sad idea, that the closest people feel to each other is in those moments of suffering—the edge of seeing your comrades dying in your arms.
Suffering
P: It may bring people close togehter, but there are other ways of bonding … without the suffering and death.
L: Perhaps you can already hear the romanticized Russian in me; we tend to romanticize suffering … What is suffering? If you tried to define it, …
p: It’s a conscious state; there can be no suffering for an unconscious being. It’s distinguished from other conscious states in terms of being what, considered just in itself, as one we would rather be without. We want to stop it if we’re experiencing it; or we want to avoid having it again. I emphasize, for its own sake. Of course, people will say it has good consequences; sometimes it does, of course. Sometimes we might undergo it—we set ourselves a challegne to run a marathon, .. or even just to go to the dentist. I’m not saying we never choose it. I’m saying that, other things being equal, we would rather not …
L: Is the ultimate goal … You have the 10-year-anniversary of the Life you Can Save book … Do we want to eradicate suffering? Or do we want to keep it to keeep things interesting in the world.
P: It’s moot because, in practice, it likely won’t happen. IT’s going to always be there. If you ask me, in theory, whether if we could get rid of it, should we? I think the answer …. if whether we would be better off, or if eliminating the suffering, we might also eliminate the positive highs. If that’s the case, it might be worth keeping it in order to keep the best posible experiences.
L: When you talk about eradicating poverty in the world … Is there, at the basic, human ethical level—a bar that’s absolute. That once you get above it, we can converge toward feeling like we have eradicated poverty.
P: This is true for poverty as well as suffering. There’s an objective level, with objective indicators, like, you’re constantly hungry, you can’t get enough food. You’re constantly cold, you can’t get warm. You have some physical pains you’re never rid of. I think that’s objective. But if you get rid of those needs; there may still be new forms of suffering that develop; perhaps we’re seeing that in affluent societies we have. They get bored, for example. They lack a sense of purpose. That can happen. That’s kind of a relative suffering, distinctive from objective forms of suffering.
L: But you don’t think about the challenges with suffering that emrege.
P: It would be of interest to me if we had eliminated all the objective forms of suffering, which I think of as generally more severe. Yes, in some future state, when we’ve eliminated objective forms of suffering; I’d then want to take up eliminating the relative forms as well.
L: Sorry to linger; Is elimination a goal, for the affluent society? Do you see suffering as a creative force?
P: It can be a creative force; repeating what I said about the highs. It may be suffering makes us more creative and we regard that as worthwhile. … That brings some of those highs with it … I don’t really know; many have suggested that; I certainly have no basis for denying it. If it’s true, I wouldn’t want to eliminate suffering completely.
Is everyone capable of evil?
P: Most of us have potential for both good and evil. I’m not sure everyone is capable of evil; most of us are certainly susceptible to environmental influences. … look at what the Nazis did during the Holocuast. I think it’s difficult to say, “I know I would not have done those things.” If I had grown up under the Nazi regime; had been indoctrinated with racist ideas … plus of course perhaps the threat that if I didn’t do certain things, I’d be sent to the Russian front. I think it’s hard for anyone to say, “Nevertheless, I know I wouldn’t kill those Jews.”
L: What’s your intuition—how many people could be able to say that?
P: Very few. Less than 10%
L: It seems like a great thing to meditate on. I can’t escape the fact (the thought?) that I would not have been one of the 10%.
P: (Discusses the issue) … It would be interesting to find a way to really find these answers. There’s quite a bit of research on people during the Holocuast; how ordinary Germans got led to do terrible things. There’s also studies of the resistance; people in the White Rose group, for example. But I don’t know whether these studies can really answer your larger question …
L: Well, the reason I think it’s interesting is, in a world as you described, where there are things you’d like to do that are objectively good; it’s useful to think about whether, I’m not willing to do something or acknowledge something as the rihgt thing to do, because I’m simply scared of damaging my life in some kind of way. That thought experiment … what is the right thing? There’s things that are convenient … I’m wondering if there are things that are highly inconvenient. That kind of balance, I feel that in America. … in the current times, it seems easier to put yourself back in history; you can objectively contemplate, how willing you are to do the right thing if the cost is high.
P: True but we do face those challenges today; we can still ask those questions. One stand I took 40 years ago was to stop eating meat; at a time when you hardly met anybody that was a vegetarian. … I know, thinking about that decision, I was ocnvinced it was the right thing to do. I had to think, “Are all my friends going to think that I was a crank?” I had to decide; I still think this is the right thing to do … I’ll put up with that if it happens. One or two friends were uncomfortable with that decision, but that was pretty minor compared to historical examples we’re talking about. … Other issues we have around too, Global poverty and what we ought to be doing about that, is another question where people have the opporutnity to take a stand. Climate change owuld be a third question where people are taking a stand. … Look at Great Thunberg there; it must have taken a lot of courage for a schoolgirl to say, I”m going to go on strike about climate change.
Can robots suffer?
L: Especially in this divisive world; she gets support and hatred both. IT’s a difficult (area) for a teenager to operate in. … In your book, Ethics in the Real World with 80 … anecdotes or chapters … Should robots have rights?
P: … until that happens, they shouldn’t.
L: Is consciousness essentialy a prerequisite to suffering? If so, what is consciousness?
P: I certainly think it is a prerequisiite … But is it ture that every being that is conscious will suffer, or has to be capable of suffering? I supposed we can imagine a consciousness that is constructed to only achieve pleasure … Instant anesthesia if anything is going to cause suffering. So that’s possible, but it doesn’t exist yet. You asked what was consciousness. Philosophers often talk about it as being the subject of experiences. … The kinds of AI we have now … I take out my phone, I ask Google directions to where I’m going; Google gives me the direction; Google tells me to go a different way. I think that’s the indicition that … …. It may be difficult to tell whether certain AI is or isn’t conscious. It may mimmic consciousness. We’re looking for—is there a subject of experience … a perspective on the world.
L: So our idea of what suffering looks like just comes from watching ourselves when…
L: That’s how we infer … about animals. … What if robots, you mentioned Google Maps;, and i’ve done this experiment, because I work in robotics. I have several Rumba robots and I play with voice speech interaction. If the robot shows any signs of pain, or being displeased by something you’ve done. :That, in my mind,. .. I can’t help but immediately upgrade it. … Just having another entitty … That feeling is there. What do you htink about a world whewre Google Maps and Rumbas are pretending to be consicious, and we descendants of apes aren’t able to show … SOMETHING.
P: That kind of capability may be closer, but I don’t think it follows that we have to give them rights. The argument; ir … …. … Kant among others said … we
L: I’d like to disagree; at the risk of sounding crazy; I’d like to say that if that Rumba is dedicated to faking the consciousness and the suffering, it will be impossible for us; I’d like to apply the same argument of animals to robots … They deserve rights. I’m quite surpirsed by the upgrade in consciousness that the display of suffering creates. The difference between animals and other humans is that in the robot case, we’ve added it in ourselves; therefore we can say something about how real it is. But I would like to say that the display of it is what makes it real. … I’d act last like to add that as a possibility.
P: There is a philosophical view, has been held about humans,behaviorism. Was employed in psychology (the writing of B.F. Skinner); it was, “what is it that makes this science.” In philosophy, the view defended by people like Gilbert Rile; in this phase of the 1940s … (this guy) said … the meaning of a term is in its use. … That’s what it is to be in pain, and you point to the behavior. … Malcolm, another philosopher in the staff room. When people wake up and say, I’ve just had a dream; here I was.”
L: Noam Chomsky … took down the behaviorist … claims. I am a unique person in that I have made the Rumba scream in pain. I philosophically want to acknowledge the costumes out there … I think it’s a new world; I was curious what your fhoughts are. for not, you feel the dsplay
P: … maybe it’s a good thing that they do feel it …
L: But there’s this line … how do you know when your system is giving you signs of consciousness ….
Animal liberation
Question for AI about suffering
Neuralink
L: There’s been a lot of exciting innovation in brain-computer interfaces … communicating both ways; from a computer being able to activate neurons, and being able to read spikes from neurons … With the dream of being able to expand or increase the bandwidth your brain is able to look up articles on Wikipedia. ‘
P: I’d love to be able to have that increased bandwidth. I want better access to my memory, I’d have to say, too. (Forgetting stuff that happened 20 years ago) … If I had this extended memory I could search that particular year and re-run those things.
L: In some sense we have that in being able to store so much data online. …
Control problem of AI
L: Value alignment … Making sure we build systems that align to our own values and ethics. How do we go about building those systems that align to our values and human ethics, living being ethics.
P: Presumably it’s possible to do that. I know a lot of people think there’s a real danger that we won’t; that we’ll lose control of AGI. I’m not sure what to think. I talk to Nick Bostrum … they think this is a real problem. Then I talk to people who work for Microsoft or Deep Mind or somebody; they say, “No, we’re not really that close to producing AGI.”
L: … The Nick Bostrum argument … I myself engineer AGI systems … Is there any fundamental reason we’ll never achieve it? Iff not, eventually, there will be a dire existential risk. Do you find that argument appealing?
P: Yes I think it’s a rpoblem; I think that’s a valid point. OF course, when you say “eventually,” that raises the question—how far is that? Is there something we can do about it now? If we’re talking abourt 100 years in the future, and you consider how rapidly our knowledge of AI is growing in the last 10-20 years; it seems unlikely there’s anytthing we could do now. People 80 years in the future will be in a better position to …. know what we could do. I’m all in favor of some people doing research into this … I’m very much of the view that extinction is a terrible thing, and therefore. … even if the risk of extinction is very small, if we can reduce that risk, we ought to!
Utilitarianism
L: Interesting from many perspectives. What is it?
P: The ethical view that the right thing to do is the act that has the greatest expected utility, where what that means is, it’s the act htat will produce the best consequences, discounted by the odds you won’t be able to produce those consequences. … IF there’s certainty, then no discount.
L: A bunch of nuanced stuff you talked to Sam Harris about on his podcast—like two hours of moral philosophy discussion. Is that an easy calculation?
P: No, it’s difficult. I need to add—utilitarians think, by “best consequences” they mena happiness and the absence of pain and suffering. There are consequentialists, who are not utilitarians, who substitute in a bunc hof other “good consequences” that complicate the calculations. … If we are just talking about “well-being,” then the calculation is more manageable in a philosophical sense. We still don’t know how to do it. … So at best, we can use it as a rough guide, to different actions. One way we have to focus on short-term consequnces since we' can’t really predict the long-term ones.
l: Would you say you yoruself are utilitarian?
P: Yes
L: What do you make of the extreme suffering of a few individuals
P: I think it’s possible it gets overridden by a benefit to a great number of individuals, but we need to know (all the dat). I tend to think extreme suffering is further below the neutral level than extreme happiness or bliss is above it. …
Deep Mind, but for robots: Berkeley prof Sergey Levine, #108
Sergey Levine, a professor at Berkeley and a world class researcher at Deep Learning, Reinforcement Learning, robotics, and Computer Vision. Including the development of algorithms for end-to-end training of NN policies, that combine perception and control. Scalable algorithms for inverse reinforcement learning, and in general, Deep RL algorithms.
<iframe width="560" height="315" src="
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
L: What’s the difference between a state-of-the-art human and a state-of-the-art robot.
S: Robot capability is a tricky thing to understand, because there are some htings that are dificult we wouldn’t think are difficult; some that are easy we wouldn’t think are easy. There’s a big gap in capabilities of robots because of hardwaree, and capabilities of what they can do autonomously. … There’s a 2004 study from Stanford that had a PR1, a home assistance robot. There’s a video of the robot tidying up the living room, putting away toys, bringing a beer to a human sitting on the couhc. But the punchline is the robot is entirely controlled by a person. … If we’re willing to spend a bit of money and do engineering, we can close the hardware gap … But the intelligence gap, that’s more difficult.
L: … the hardware of the nervous system
S: … the big bottleneck is currently the mind.
L: How big is that gap? Ability to learn, reason, perceive the world?
S: Very large, and it becomes larger the more unexpected events can happen in the wor.d. The spectrum along which you can measure the size of that gap is the spectrum of how open the world is.
L: Nature vs. nurture; how much of it is a product of evolution, and how much do we learn from scratch.
S: I’m going to read into your question, how important is this for AI.
L: If it’s all about learning, then there’s more hope for AI …
S: The way that I look at this is: Well, biology’s very messy; if you the question, ask how does a person do something, you can come up with a bunch of hypotheses; some of them conflicting … One way we can approach what the implications for AI are—we can think about what’s sufficient; For example, the ability to recognize faces. … We can ask what’s the minimum sufficient thing. We can study this by seeing what people do in unusual situations; we present people with things evolution couldn’t have prepared them for. Our daily lives actually do this all the time; we didn’t evolve to deal with automobiles and space flight and whatever. Theyr’e all these situations we find outrselves in, and we do very well. I can give you a joystick to control a robotic arm; if your life depends on it, turns out you’ll probably learn it, even if you’ve never seen it befoere, nor used it before. That’s not your evolved natural ability, but your flexibility, your adaptibility … That’s where robotic systems kind of fall flat.
L: I wonder how much general what we think of as common sense, pre-trained models underneath all that. That ability to adapt to a joystick, requires you to have … it’s hard for me as a human to introspect all the knowledge I have about the world; there might be an iceberg underneath
S: There’s absolutely an iceberg of knowledge; it’s actually built up over our lifetimes. I makes sense that the right way for us to optimize our eficiency i to utilize all that experience, do build up the best iceberg we can get. .. That sounds an awful lot like like ML, I think for modern ML it’s a big challenge to take unstructured experience and distill out a common sense understanding. .. It’s not because ML is broken or hard ,but because we’ve been a little too rigid. Maybe what we really need to do is view the world as a mass of experience that is not necessarily providing rigid supervision. Distill that into some common sense understanding.
L: Seems an optimistic, beautiful picture. We should invest into better building algorithms … Accumulate that iceberg of knowledge
S: It’s even more than just that; this is where we reached the limits of our current understanding … … One thing the research community hasn’t resolved, is how much it matters where that experience comes from. How important is it that your system actually expeirences the world. .. It may be the world is so complex that simply obtaining a large mass of IID samples of the owrld is a.. … What we associatee with comon sense is often associated with the ability to readson with counter-factual questions. ..
L: Where do you think intelligent systems that would be able to deal with this world fall? Could we do pretty well reading all of Wikipedia? OR do we have to be exceptionally selective about which aspects of the world we train on?
S: This is an open problem; I can only speculate. To speculate, we don’t need to be super- super-careful. We are careful to avoid the useless stuff … … It’s important for you to really try out the solutions your current model of the world says would be effective, and observe what happens. Some of that is really essential to have a perpetual improvement loop.
L: How important is exploration, or out-of-the-box thinking? You mentioned an optimization problem, exploring the specifics of a particular strategy. How important is it to explore outside of tried strategies. …
S: In some ways, that question gets at a big difference btween how the game was called and what the scoreboard said … maximizing utility (is the common paradigm). An interesting alternative way to look at the se problems is ast first as something where you get to explore the world. That might suggest a different solutions. … Build up an arsenal of cognitive tools, if you will.
L: You see that as the modern formulation of the RL problem. The more multitask, general-intelligence formulation.
S: I think that’s one possible vision of hwere we might be headed in the future.
Robotics may help us understand intelligence
L: I like it. What is the goal of robotics? What’s the general problem robotics is trying to solve? What in your view is the big problem of robotics.
s: Maybe two ways I can answer this question. One, there’s a pragmatic problem—what would maximize the usefulness of robots. The answer might be a system that, can perform whatever task a human user sets for it. It probably can’t teleport to another planet; but if you ask it to do something it CAN do, it SHOULD do it … in much the same way a human operator should figure out how to drive the robot. That’s the pragmatic view of solving the robotics problem. A second answer, a lot closer to why I want to work on robotis; it’s less about what would be a really good job in world of robotics; it’s the other way around: what robots can bring to tthe table to help us understand AI.
L: So your dream fundamentally is to understand intelligence
S: yes, I think that’s the dream for many people in this space. I think a lot of people that go into the field, the hintss
L: What is it about robotics that’s different .. from NLP or Computer Vision.
S: A couple of things. One is, with robotics, you have to take away many of the crutches. You have to deal with both the particular problems of perception and control; you also have to deal with the integration of those things. This was traditionally a second problem … that’s the modular engineering approach. One thing w’eve seen over the alst couple decades; maybe studying the parts as individual … The integrative nature helps us see different perspectives. Another part of the answer; robotics puts a certain paradox into relief; sometimes called Morphix paradox … certain things easy for humans are difficult for mahcines, and vice versa. Integral and differential calculus are pretty difficult to learn for people; but computers can ccalculate differential … do it all day long without much trouble. Sometimes when we see such blatant discrepancies, that’s a hint we’re missing something important. The bits that we’re missing … if we study them we might find new insight.
L: … That could be in any space; it doesn’t have to be robotics. IT’s interesting that robotics has a lot of thos d If you were to try to disentangle the ___ paradox, why is there such a gap in our intuition about it; why is manipulating objects so hard, from everything you’ve learned applying RL in this space?
S: I think one reason is maybe that, for many of the other problems we’ve studied in AI, in CS, the notion of input/output and super-vision is much cleaner. Computer vision, for example, deals with very complex inputs. But it’s comparatively easier to cast it as a tightly supervised problems. It’s comparatively much harder hard to do the same with robotic manipulation. Let’s say we get a labeled data set where we know exactly what commands to send … it just doesn’t work. It doesn’t match how humans learn; our parents don’t tell us how to walk; we mostly figure it out on our own
End-to-end learning in robotics
L: There’s perception, the copmputer vision problem, understanding the environment. Then there’s prediction; trying to anticipate what things will do in the future, in order to be able to act in that world. Then there’s this game theoretic aspec,t how your action will change the behavior of others. This is bigger than Reinforcmeent learning; … Or is what you said true that, hwen you stat to look at all of them together, that’s a whole nother thing. …
S: I think when you look at them all together, some things actually become easier. Back in 2014, we worked on end-to-end reinforcement learning for robotic manipulation skills for vision … At the time it seemed inflammatory and controversial. in the robotics world. Other than that part of it, we were trying to make the point htat for the paricular case of combining percpetion and control, you could do better if you put them together than if you tried to separate them_ The robot had to insert a little red trapezoid inside a trapezoidal whole. Various level s of solution. Separated solution and then our “intense” solution … ONe thing we observed; if you used the intense solution; the pressure on the perception part of the model was actually lower. You just distribute the errors in such a way that horizontal errors are different than vertical ones. .. You can trade off errors between the components
L: That’s a profound idea! It’s almost tempting to overlook; seems like an idea that should generalize to all aspects of perception and control.
S: People who have studied perceptual heuristics in humans and animals … the Gaze heuristic is a trick you can use to intercept a flying object. … (Describes AI approach to intercepting) … You could maintain a running speed so the object stays in your field of view … Huanns use it when they play baseball … frogs use it to catch insects; pilots use it …
Canonical problem in robotics
L: Do you have a canonical problem? That you think about? We tlaked about robotic manipulation; that seems intuitively … the robotics community has converged towards it. Do you focus on some particular aspect of that problem? If we solve that problem perfectly, it’d be a major step toward AGI
S: I lack a great answer to that. I think partly, that has to do with the fact that the difficulty is in flexibliity and adapatibility, rather than doig a particular thing really really well. It’s the ability to uqickly figure how to do an arbitrary new thing.
L: The source of new-ness and uncertainty; have you found problems in hwich it’s easy to find new new-ness. New types of new-ness …
S: If you’d asked me this in 2016, I’d have said robotic grasping is a good example of that …
S: Picking up any aspect with robotic hands. You get a lot of money if you do it well because a lot of people want to run warehouses with robots. It’s non trivial. But people have gotten really good at coming up with code to solve this probelm. .. Methods that have worked well in theis space …
L: why is picking stuff up such a difficult problem
S: The number of things, the variety of things you have to deal with is extremely large. Things that work for one class of objects doesn’t work for another class. IF you’re good at picking up boxes, and then you have to pick up plastic bags, you need a way different strategy. It’s not just geometry. Some bits have handles … Some are flexible. some you can turn upside down and some you can’t. But the task can still be characterized as one task.
Commonsense reasoning in robotics
L: So in terms of spilling things, there creeps in this notion that starts to sound like common sense reasoning. Do you think solving the general problem of robotics requires general intelligence, like human level capability?
S: I’ll dodge the question. Rather, studying robotics can help us put common sense into our AI systems. Common sense is an emergent property of having to interact with a particular world, or universe. And get things done in it. An image-captioning system maybe looks at pictures of the world nad types out English sentences. IT kind of deals with our world. You can have captioning systems defy common sense; what’s happneing is, the system doesn’t actually live in our world; it lives in a world consisting of pixels and English sentences. Perhaps the reason for the disconnect is that the systems we have now inhabit a different universe. If the AI system is forced to deal with the messiness andcomplexity of our universe .. .will have to develop common sense.
L: You’ve reframed the role of robotics in this whole thing. I thought we need to solve intelligence in order to solve robotics. You’re saying, robotics is one of the best ways to study AI … It’s the right space to explore fundamental learning mechanisms, multimodal task aggregation of knowledge mechanisms. .. it’s an interesting way to think about it. About
Can we solve robotics through learning?
L: Can it be solved through learning, end-to-end learning, as opposed to injecting human expertise and heuristics and so on?
S: I would say yes; in some ways, it may be an overly sharp dichotomy. In some ways when we build algorithms, at some point a person does something. A person turned on the computer; a person implemneted TensorFlow. But yeah, in terms of the point you’re getting at, I think the answer is yes. We can solve many problems that have previously required maanual engineering … through optimization techniques. I don’t think this is a new idea; people have been thinking about automated optimization techniques as a way to do control for a long time. Today we would say, “My robot does machine learning,” whereas in the 1960s we would say, “My system does optimal control.” So maybe it’s not such a large differnece; it’s pushing the optmization deeper and deeper into the thing
L: But with Deep Learning, the accumulation of experiences in data form, to create deep represetnations … starts to feel like knowledge
S: I think that is a good poeint. One big difference between learning-based systems; they should in principle get better and better the more they do something. That is a powerful difference. ..
L: .. Looking back at using logic to accumulate human expertise, human-encoded expertise. Do you think that will have a role? Deep Larning has shown incredible results and breakthroughs and has inspired thousands of researchers, maybe millions. but, there’s symbolic AI still—can it have a role?
S: The descendants of symbolic AI already have a role. The highly biased history from my perspective; initially we though trational decision-making involves logical manipulation; you have a modle of the world in logic; you have a query, what action is needed to get X to be true, and then you ______. . What that turned into in the 1990s; instead of building predicates with true and false values, we did a probabilistic thing with probabilities … and that led to Beyes nets … … .. People said, let’s learn the probabilities inside these models … then let’s just put a neural net in there. But the seed of the main idea is … … … So yes it has a place, and it already holds that place. There are some things we can think about that make this a little more obvious. …
L: There’s a human desire for these autonomous systems to convey in ap oetic way why they chose what they chose … Perhaps we shouldn’t expect that of intelligent systems.. If I were to psychoanalyze the researchers at the time; I’d say expert systems connected to that part … Maybe on that topic, do you have a hope that … inference systems, learning-based systems, will be as explainable as the dream was with expert systems?
S: That’s a complicated question—explainability; it’s closely tied to the question of performance? Wh y do you want your system to explain itself? So when it screws up, you can figure out why it did it? But that’s a bigger problem; it might screw up its explanation! So, maybe a good way to view that problem is as a bigger problem of verification and validation, of which explainability is one component.
L: I see it differently; I see explainability … Another aspect of explainability, it’s story telling. IT has nothing to do with errors; it uses errors as elements of its story … as opposed to a fundamental need to be explainable when errors occur. We seem to want to tell each other stories …
S: Maybe one specific story I can tell you about in that space is by my former collaborator Jacob Andreas. He had this idea in Reinforcement Learning on how natural language can restructure the internal policy of RL. He set up a model that attempts to perform tasks defined by a reward function. It reads in a natural language instruction. You tell it, “go to the red house.” But he treated the sentence not as a command from a person, but as a representation of the internal state of the mind of this policy. Given a new task, it would think of possible language descriptions … and then try to do it. It owuld kind of think out loud. It would go till it got a reward. You could basically incorporate langauge as internal state …
What is reinforcement learning?
L: Also if you add ot the reward function, the convincing-ness of that story. So if you add to the reward function … it could be a hyperparameter … An interetsing notion, right? (Lex is proud of his notion.) … Let me ask the basic question … you’re a world class researcher … what is RL?
S: What it refers to today is just the modern reincarnation of learning-based control. Classically it has a more-narrow definition; learning from reinforcement. But really, it’s used more broadly for learning-based control. Some system that’s controlling something, and it uses data to get better. … (“What is control,” Lex) … It means … making rational decisions … that maximize a measure of utility
L: And sequentially … … It’s easy to see that idea in the space of games, in teh space of robotics. Is it applicable in more places?
S: Rational decision-making is teh encapsulation of the AI problem viewed through a particular lens. Any problem we want it to do can be classified as a decision-making problem. Controlling a chemical plant, decidint what videos to recommend on YT … one of theappealing things about RL, if it encapsulates the range of all these problems … then it’s a way to reeach a broad swath of AI problems.
L: What’s the difference between RL and Supervised ML.
S: RL can be viewed as a generalization of the latter. … You can cast Supervised Learning as a RL problme; your loss function is the negative of your rewrad. But you have stronger assujmptions; that you were told the correct answer. You can view RL as relaxing osme of those assumptions …
L: Mathematically that’s correct … but down the line, everything will be an RL problem. … Today, the tools and ideas, the way we think about them are different; supervised learning has been used effectively to solve basic, narrow Ai problems. RL represents the dream of AI; it’s very much in the research space now, captivating the imagination of people, what we can do with intelligent systems. My question more practically is what is the gap between general RL and the very-specific, …
S: From pracical standpoint, one thing that’s a little tough now, and we’ll perhaps see this gap closing, is the ability of RL algorithms to use large amounts of prior data. wOne ereason it’s difficult today to use __ learning in all things we want to use it for; it’s tough to just startt using new policy and personnel …
Tesla Autopilot
Simulation in reinforcement learning
Can we learn gravity from data?
Self play
Reward functions
Bitter lesson by Rich Sutton
Advice for students interested in AI
Meaning of life
#109: Unix guy from the 70s
<iframe width="560" height="315" src="
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
Brian Kerighan sits with Lex; July 2020, #109
Brian Kernighan, a professor of CS at Princeton; he was a key figure in the CS community in the early Unix days. He co-authored the C programming language with Dennis Ritchie; and has written a lot of books on programming, computers, and life; including The Practice of PRogramming, The Go Programming Language, and his latest, Unix: A History & A Memoir. He co-coreated AWK, the text-processing language used by Unix folks. He co-coreated Ample, an algebraic modeling language … that is used for large-scale optimization. Given all that, he’s one of the most humble and kind people I’ve spoken to.
L: Unix startedd being developed 50 years ago, more than that. Can you tell the story of how it was created?
B: If I can remember that far back! The gist of it is that at the labs in 1969 there were a group who had just finished working on “Multix", itself a follow-on to CTSS; CTSS was a time-sharing system, very nice to use. I used it during a summer at Cambridge in 1966. .. (Lrex: some questions) … CTSS was a time-sharing system ..
L: What’s that?
B: IF we go back to the 1950s and 60s; most computing was done on very big computers, maintained in very-large rooms. You used thingsl ike punch-cards to write your program on … You sent it on a counter over to an operator. Something would come back that would say, “Oh you made a mitake.” The idea of time-sharing was, take that same computer, and connect to it with … an electric typewriter. The OS gave each person who was connected and wanting to do somethign a small slice of time to do a particular job. I might be editing a file; every time I typed something, the OS woke up and say, “oh he typed a character; let me remember that.” (Lex posts a wiki link to Compatible Time-Sharing System (CTSS): first demonstrated on an MIT computer in 1961.)
L: Without individual people being aware, in a sense-the illusion that the machine is your own.
B: That was the idea; if it were well-done and fast enough, you had the illusion you had the whole machine to yourself; so much better than the punch-card model. CTSS was the first of these; it ran on an IBM 7094; modified to have twice as much memory as the norm; it had two banks of 32K words instead of one. So 150KB times 2. At the time, that was a lot of memory, and memory was expensive. IT was a wonderful environment to work on; led by Fernando Corbitov at MIT. I spent the summer of ‘66 working on it; met a lot of nice people, and indirectly knew of people at Bell Labs who were also working on a follow-on to CTSS called Multix (Lex posts: Multics: Multiplexed Information and Computing Service, a time-sharing operating system based on the concept of a single-level memory.) It was going to replicate CTSS but better, for a larger population.
L: What’s the algorithm that performs the scheduling? What’s that look like?
…
L: What was your dream of computers at that time? Could you have predicted how computers are today?
B: IT was a dream job in that I enjoyed what I was doing; Cambridge was a good city to live in …
… Multix didn’t live up to this goal of being an information utility. Its promise was delivered too late. So in the beginning of 1969, Bell Labs pulled out of the project that they had been in on with MIT and General Electric. Bell Labs realized this wasn’t going anywhere; this left several people with an acquired taste for nice computing environments, but no computing environment! What could you do to design a new OS that could have the best components fo CTSS and Multix … Ken Thompson found a little-used PDP7; where he started doing experiments with file systems; how to store on a computer in an efficient way. His wife went away to CA for three weeks; their 1-year-old son; he sat down and wrote an operating system.
l: What’s a PDP7
B: An early machine made by (DEC) Digital Equipment Corporation … It had a very small amount of memory (Lex posts the Wiki) … IT was $100,000; it was expensive, but by 1969 it was getting obsolete.
L: What do you think it’s like to write an operating system; from design to reality?
B: Let me correct one thing: I had nothing to do with it. I have never written OS code. An OS is simply code; this first one wasn’t very big … it lets you store info for periods of time so it doesn’t go away when you turn the power off and review. There’s kind of a core set of tools that are not part of it … You needed a text editor; a file system ttuff he had already been working on …
L: What was the code written on? In Assembly?
B: Yeah, PDP7 was an assembler … it was that way until 1974 or something like that
L: IT feels like a daunting task to write any kind of complex system in assembly. Seems impossible to do what we call software engineering, in assembly.
B: It does seem hard. It’s true if you make a mistake, nobody tells you; just, nothing happens. There’s no debuggers … How do you get something that will help you debug it? Part of it is, you need an ability to see the big picture. These pictures weren’t big the way today’s pictures are. … There’s enormous variation in the capabilities of programmers. Ken Thompson is kind of the singularity in my experience of programmers, with no disrespect to you. Or even to me.
L: I know there’s levels. It’s fascinating that there are unique stars, in a programming space and at a particular time. … A wife does have to leave to create space …
L: Amazing, now most of the systems in the world run on a Unix-like system. And they started from the seed of what he did ein three weeks.
B: It’s quite surprising. There’s no way you could predicrt that evolution … I don’t know whetehr it was inevitable, or just a whole sequence of blind luck. I look at it and think, Gee that’s kind of neat! (lauhgs) I guess a real question is what does Ken think about it? He’s the guy from whom it really came. Tremendous contributions from Dennis Ritchie and then others areound Bell Labs, but if you had to pick a single person, that would be Ken.
L: Your book on Unix, the history and a Memoir. Any funny human stories that stand out?
B: There’s a lot of them! I think part of it was that Bell Labs at the time was a special kind of place to work; a lot of interesting people; the environment was open and free, a cooperative, friendly environment … If you had an interesting problem you went and talked to somebody … and they might help you with the solution. It was kind of a fun environment, too; people did strange things, and often tweeking the bureaucracy in one way or another
L: So rebellious in some ways
B: I think most didn’t take too kindly to the bureaucracy; the bure. put up with a tremendous amount it didn’t want to.
Unix philosophy
L: What is it? The design … Not just the initial, but what carried through the years
B: One aspect was to provide an environment that made it easier to write programs; it was meant as a programmer environment. Not meant for any other kind of job. It was used extensively for word processing but that’s not what it was designed for. Same with lab control; same with as a front end for other big systems. It was meant to be an envrionment where it was really easy to write programs. Part of that was to be a community. Dennis Ritchie notes, in the end of the book, that the real goal was to build a community for programmers to work … For many eyars ti worked. Technically, it made it easier to write programs; people wrote interesting programs; those got used by other programmers.
L: And you were part of that community. What was it lke writing programs on that early Lunix.
B: It was a blast, it really was. I like to program. It was fun to write code; in the early days there was a lot of low-hanging fruit; things people hadn’t done before. The combination of nice tools, a responsive system, and tremnendous colleauges made it possible to write code.e Have an idea in the morning, do an experiment, people react to it the next day. People would say, “wonderful—but you screwed up here!” A lto of things were developed quickly that in many cases still are there today. … Programming is fun, it’s puzzle-solvgin .. but more fun is to do something that somebody else then uses.
L: What was the method of communication; the feedback, before the Interent.
B: Mostly physical right there; somebody comes into your office and says something …. Bell Labs was one giant building; most of the Unix people were in 2-3 corriddors. Tehre was a room, call it 50 x 50 feet, which had some access to computers there. People hung out there; it had a coffee machine. It was mostly very physical. We did use email of course, but it was fundamentally, for a long time, all on one machine. There was no need for Internet.
L: Fascinating to think about what computing tgoday would be like without Bell labs. People being in the vicinity of each other, so many brilliant people; I don’t know where else that could have existed in the world. How does it make you feel?
B: That’s very nice. In a sense, it’s survivor bias. There were other places doing interesting stuff: Xerox Park an obvious one; they created a lot of things … I don’t think they capitalized in the long run as much; their parent company was perhaps not as lucky. Butj that’s certainly another place where there was a lot of influence. There was also a lot of university activities; MIT was no slouch
L: Unix turned out to be Oepn Source because of the various ways AT&T operated; their focus was on telephones
B: That’s a mischaracterization in a sense. It wasn’t open source; it was very definitely propietary; but it was licensed freely to universities in source code form. So generaitons of university students grew up knowing about Unix; expertise in the community made it possible for people to go off and build things that were Unix-lik.e. the Berkeley version started with Unix code, but picked up contributions (from people like Bill Joy) that it eventually became entirely free of any AT&T code. Tehre was legal jokeying around this. I guess the open Source movement might have started when Richard Stahlman started to think about this in the late 80s; when Turdalz decided he was going to do a Unix-like OS; there was enough expertise … that in the communikty he had a target; he could see what to do, because the Unix system interface was there. He was able to build an OS that at this point, when you say Unix, what you’re really thinking is Linux.
L: From my distant perception I thought Unix was Open Source. What’s actually true was that it was freely licensesd. IT felt Open Source in the sense that you could get access if you wanted.
B: Technically not open; technically belonging to AT&T. Pragmatically, pretty open.
L: Those students grew up, worked throughout the world, and it permeated with them. What kind of features make for a good OS, if you take lessons from Unix. Make it easy for programmers, seems like a starting place. But Unix turned otu to be robust and efficient. Is that an accident?
B: Part of the reason for efficiency was that it began on such modest hardware; you couldn’t get carried away, doing a lot of complex things—you didn’t have the resources, neither processor speed nor memory. That enforced minimality, and maybe a search for generalizations, to find one mechanism that could serve for a lot of different things. I think the file system in Unix is a good example; it’s extremely straightforward; that means you can write very effective code for the file system. One of those generalizations is, that file system interface works for a lot of other things. That gets carried further in other parts of the world; processes become, in effect, files in a file system.
L: You said you’re not a very good programmer (laughs). Your’e the world’s most modest human being. You do raadiate a sort of love for programming …
Is programming art or science?
B: I think it’s some of each. Some of the art is figuring what you really want to do; what hould the program be? What would makea good program? Know the audience; I think that’s art in many respects. The science part is trying to figure out how to do it well. Some of that is CS stuff; what alagorithms to use!? Make usre the ones you use work and scale properly; avoid quadratics when linear will do. … Also, I think it’s an engineering field; that’s not quite the same as science. You’re working with constraints. You have to figure out not only the best algorithm, but also adjust for time limitations on compute, on programming, issues having to do with maintneancne … All of those things, if you’re an engineer, you get to worry about; if you think about yourself as a scientist, you push those things over the horizon.
L: On your personal level, what’s your process like in writing a program? do you start coding right away, and evolve iteratively with a loose notion? Or do you plan on paper first, and then design … what they teach you in undergrad software engineering class?
B: It’s more incremental. I don’t write big programs at this point … Many of the programs I write are experiments for something I’m curious about or want to talk about in a class. those necessarily are relatively small. A lot of the code I write these days .. those programs tend to be very small. ..
AWK
B: A language that was meant to make it easy to do quick and dirty tasks like counting things, or selecting interesting info from text files.
L: It runs a command on each line of a file. Its still widely used today. So simple and elegant; the way to explore data, is to write a script that does something seemihgly trivial on a single line; that slice of the data reveals osmething fundamental about teh data. … That seems to work still!
B: It’s good for that kind of thing; That’s sort of what it’s meant for; what wedidn’t appreciate is the model was good for a lot of data processign kinds of tasks … iT’s kept going as long as it has; it’s over 40 years old … It’s still, I think, a suseful tool. I think in terms of programing languages; you get most bang for your buck by learning AWK. It doesnt well on little things … I probably write more AWK Thhan anyhting else at this point.
L: What do you love about it? What gies you joy in it?
B: I think it’s mostly the selection of defautl behaviors. What AWK does is read throuhg a set of files and in each file it writes through each. of hte lines; each has a set of patterns that it looks for; that’s your AWK program. Once it matches, there’s an action you might perform. It’s kind of a quadrupally nested loop or something … That’s all completely automatic; you just write the pattern adn the action and run the data by it … That paradigm ofr programming is a natural and effective one; I think we captured that reasonably well in AWK. IT also splits the data into fields … for free. It collects info as it goes along; what line are we on? What field? Lots of things that make it so that a program in another language, like Python, mabye 5, 10, 20 lines, is only 1-2 lines in AWK.
L: So you can do it on the shell, you don’t need a spearate file. .. Are there Shell commands that you love over the years?
B: Oh, grep! Grep does everything.
L: So Grep is a simpler version of AWK .. What is that?
B: IT basically searches the input for particular patterns, regular expressions. It has same paradgim that AWK does; pattern-action. It reads through all the files; it has a single patern it’s looking for, and a single action … paired with it. I use grep more than anything else; it’s so convenient and nautral.
L: Why doy ou think OS systems like windows don’t have Grep or AWK.? Windows naturally, as part of the GUI, the simplicity of Grep, searching through a bunch of files; why is that unique to the Unix and Linux environment?
B: I don’t know. It wasn’t strictly that unique. .. I think some of it’s the weight of history. Windows comes from MS-DOS; MS-DOS was pretty pathetic OS; although common on a large number of mahcines. Somehwere in roughly the 90s, Windows became a graphical system; McIntosh spent a lot of energy making their graphical interface what it is. It’s a different model of computing; a model that, where you point and click and sort of experiment; it’s a model where it’s worked rather well for non-programmers, who just want to get somethign done. Whereas teaching the command line to non-programmers turns out to be … an uphill struggle. .. Now you mentioned “WSL” or whatever it’s called (they laugh over how he pronoucnes it “whistle”) … Cygwin, a wonderful collection of tools fro mUnix and Linux made to work on Windows; that’s been going on for at least 20 years. I use that on my Windows machine routinesly. Ifyou’re doing something that is Batch computing suitable for Command Line, that’s the way to do it. The windows equivalents aren’t familiar for me.
L: I recommend that people try WSL. I was excited using BASh to write scripts quickly in Windows; it’s changed my life. Ok; what’s your perfect prgoamming setup.
B: My default, I have a 13-inch MacBook Air, which i sue because it’s a reasonable baslance of things I need; I do most of my computing on that. I have a big iMac in my office that I use from time to time, esp. if I need a big screen. ..
L: Editor
B: I use mostly SAM, an editor Rob Pike wrote long ago at Bell Labs.
L: does that proceed VI?
B: It post dates both VI and e-Max; it is derived from Rob’s experience with E-D and VI; E-D is the original Unix editor, dated probably befre you were born.
L: what’s the history of editors? I use eMax, I’m sorry to say.
B: In ancient times, call it the first time-sharing systems, there was an editor on CTSS where you could type program text, or document text. You could save it. The usual thing you’d get in an editor. Ken Thompson wrote an editor called QED;; these are all totally command or cursor-based; it was before mice or even before cursors; they ran on terminals that printed on paper … No CRT type displays, let alone LEDs. Ken eventually took QED and stripped it way down, so that it became just ED …
L: So you can work on parts of a file
B: IT was entirely command-line based, and it was entirely on paper. So that meant, if you changed the line, you had to print that line, using up another piece of paper … The editor I was using was VI; that dates from the late 70s; I suspect eMax was from roughly hte same time. I stopped using ED; I use VI sometimes, and I use Sam when I can. … Sam, you have to download yourself from the PlanMine operating system …
L: When I get home tonight I’ll try it. Although my love is with Lisp and eMax; I went into that hippie world.
B: It’s a lot of things; what religion were you brought up with?
L: Most programming I do is in C++ and Python, but my upbringing is in Lyft.
Programming setup
History of programming languages
B: Tehy started in the late 40s; people used to program computers putting in 0s and 1s using switches on a console or holes on paper tapes. I think the first progamming languages were relatively crude assembly languages. People would write a program that would convert pneumonics like “add” into whatever the bit pattern that corresponded to an add instruction; do the clerical work of figuring out where things were; you put a name on alocation, and the assembler figures it out., where it is in memory. Early on, in the early 50s … there were assemblers written for various machiens people used. .. There was a thingi n Manchester called AutoCode; it sounds like it was a flavor of assembly language; a little higher in some ways… Replaced a lnaguage Alan Turing work where you put in 0s and 1s, but you did it backwards. So let’s clall that the early 50s; everybody had their own assembly language
L: And assembly languages have their own comands, including addition, subtraction, etc.
B: Right. They have exactly, in their simplest form at least, as many instructions as there are operations the machine can do in its repertoir. … You have to know the machine intimately to be able to write programs in it. If you write assembly language in one type of machine, it’s not ready to use in another type of machine. What happened in the late 50s was people realized, you could play this game again and move up a level in creating languages that were closer to the way people might think about how to write code. There were, I guess, 3-4 languages. ForTran came from IBM: Formula Translation … There was COBOL, worked on Grace Hopper, business-type of tasks. There was AlWell, described algorithmic computations. BASIC was arguab ly in there somehwere; arrived a little later. They were all a level up.. They were focused on different domains …
L: ForTran still used today
B: Cobol too. Once you moved up that level, you had a language that wasn’t tied to a particular kind of hardware. Two ramifications: one, you only had to write the program once. Second, you could write it yourself as an engineer, you didn’t need to hire a program to do it for you. It democratized the ability to write code.
L: Puts hte power of programming in the hands of people like you.
B: Yeah, anyone willing to spend some time writnig a programming language. IN the 70s, you get system programming languages, in which C is the survivor.
L: What are those?
B: Programing languages that would take on the kinds of things necessary for systme programs, like text editors, OSystems themselves, those kind of things
L: They have to be feature rich.
B: It’s a different flavor what they’re doing; they’re more in tuoch with the machine ina positive way. You can talk about memory, about the different data types the machine supports; more ways to structure and organize data. The system programming languages; a lot of effort in that in the late 60s, early 70s ,of which C is the only survivor. AFter that, you get OOP … when you write in a program like C, at some point scale gets to you. So C++ comes out of that tradition …
C programming language
L: Then it took off from there. Slightly parallel track, fuctional stuff, Lisp and so on. There’s the Java story, the Java Script—an explosion of languages. The cool kids these days are doing Rust. You wrote a book on the C programming language; C is one of the most important languages in history. What is the most elegant or powerful part of C; why did it survive and have such an impact?
B: I think it found a sweet spot in expressiveness; you could write things in a natural way, and efficiency; particularly important when computers were less powerful, as they were 50 years ago. That’s four, five generations of Moore’s Law. Experesivenss and efficiency, and perhaps the environment it came with which was Unix; if you wrote a program, it could be used on all computers that ran Unix. So the OS itself was portable, as were all the tools. Things fed on each other in ap ositive cycle.
L: What did it take to write a definitive book on programming. That book popularized the language (at least from my perspective) and created a standard of how this language is supposed to be used and applied. Did you have those kinds of ambitions in mind?
B: Is this some kind of joke? Of course not!
L: It’s an accident of timing, skill, and luck.
B: Clearly, timing was good. Dennis and I wrote the book in 1977, and at that point, Unix was starting to spread. Dozens to hundreds of Unix systems. C was also availble on computers that had no Unix. The language had some potential. There were no other books on C; Bell Labs was the only source for it; Dennis was authoritative; he had written the reference manual for the language. I twisted his arm until he agreed to write a book. The virtue of our advantage of going first; other people have to follow you. I think it worked well because Dennis was a superb writer; the reference manual in that book was his, period. Crystal-clear proise; very very well expressed. I wrote most of the expository material. We ping-ponged it back and forth, refining it. I tried to find examples … I’m not sure it completely succeeded, but it mostly worked out.
L: What is the power of example? You’re one of the first to do the “Hello, world” program. That’s like the example. If aliens discover our civilization, it’ll probably be “Heloo, world” programs.
B: A good example will tell you how to do something. It will be representative of … you might not want to do exactly that, but you’ll want to do something in that vein. … We had a lot of texst-processing problems that are representative of Unix, representative of what people want to do. Spell those out so they can take those and see the core parts, and modify them to their taste. A lot of programming books, I don’t look at them a tremendous amount; a lot of them don’t do that; they don’t give you exaples that are both realistic, and also something you might want to do.
L: Magical … doing something that feels useful.
B: The attempt in all cases was doing something that would be directly useful, or representative of useufl things .. In that vein, text processing …
Go language
L: I have to admit, I’ve never used Go. Go and Rust are two languages I hear spoken very highly of. There’s a lot of them; there’s Julia, many incredible modern lagnuages. Could you comment on where Go sits in this spectrum of languages. How do you feel about this wide range of powerful, interesting languages …
B: So Go … Go comes from that first Bell Labs tradition, in part. Two of the three creators, Ken Thompson and Rob Pike … This useful influence from the European school … in particular through Robert Griesber .. a student at ETH; that’s an interesting combination. Go captures the good parts of C; it looks sort of like it; it’s characterized as C for the 21st century. At the same time, it has some data structure capabilities; I think the part I’d say is particularly useful … I’m not a Go expert; 90% of the work was done by Alan Donovan, who is the expert. It provides a nice model of concurrency; communicating sequential processes … Go routines are to my mind a very natural way to talk about parallel computation. They’re easy to write, typically it’s going to work, and they’re very efficient as well. That’s one way Go stands out.
L: To comment on that, do you think C foresaw threads and massively parallel computation?
B: I woudl guess, not really. For a long time, processors got faster. Then they stopped getting faster because of power consumption and heat generation. So instead, there started to be more processors, and that’s hwere parallel thread stuff comes in.
L: Does it break your heart not to be able to explore the full variety of languages?
B: No, but I’d love to be able to explore more of them. I teach a programming class, and I have one small example, I’ll write it in as many languages as I can; I do it in twenty-odd languages at this point. I have a trivial task; it takes 15 lines in AWK; not much more in other languages. What pain did I go through to learn how to do it (in each language)? That’s like anec-data …
L: Still, it’s a little sample. I think the hardest step of the programming language is probably the first step. There, you’re taking the first step.
B: My experience with most languages is very positive. Lua, for example …
Learning new programming languages
Javascript
L: What do you think of JavaScript? Let me comment on it. When I was brought up, JS was seen as probably the ugliest language possible. And yet, it’s quite arguably, quite possibly, taking over not just the front-end, but the back-end of the Internet, but possibly in the future taking over everything. They’ve now learned to make it very efficient.
B: When it first came out, it was deemed to be irregular, and an ugly language; in the academy, people would ridicule it, as not fit for academics … The language itself has evolved; certainly the tech. of compiling it is fantastically better. So it’s absolutely a viable solution on back ends as well as front ends. Used well, it’s a pretty good language; I’ve written a modest amount of it. I’m not a real expert; it’s hard to keep up with the new things that come along with it. I don’t know if it will ever take over the world, I think not. But it’s certainly an important language, worth knowing more about.
L: JS and actually most languages, Python, such a big part of the experience of programming includes libraries; building on top of the code others have built. That’s quite different from the experience of Unix and C days. What do you think of this world of building libraries on top of each other? Leveraging them
B: Yeah, part of the fun was in the old days, you were building it all yourself. That is not the case today; if you want to do something in Python or JavaScript, you have to typically download a boatload of other stuff, and you have no idea what you’re getting. I’ve been playing Maachine Learning over the last couple of days; something doesn’t work, you “Pip Install this” …Down comes a gazillion megabytes of something, you have no idea what. If you’re lucky it wokrs. If it doesn’t, you have no recourse; no idea which of these thousand different packages. I think it’s worse in the NPM environment for JavaScript.
L: There’s also security issues; robust-ness issues. You don’t want to run a nuclear power plant using JavaScript.
Variety of programming languages
L: Do you think variety is good, or ought we to converge to 2-3 languages. You mentioned the community of Bell Labs; the more languages you have, the more you separate the community. Do you hope they’ll unite one day?
B: I don’t hope it; I don’t think one language will suffice. Are there too many at this point? Well, arguably. If you look at the distribution of how they are used; there’s something like a dozen languages that account for 95% of all programming; that doesn’t seem unreasonable. There’s another 2000 languages nobody uses. I think new languages are a good idea in many respects; they’re a chance to explore an idea of how a language might help. They’re a god place where people have explored ideas that at the time didn’t seem feasible but became part of mainstream languages as well. …
AMPL
Graph theory
AI in 1964
Future of AI
Moore’s Law
Computers in our world
Life
L: If you could relive a moment in your life .. are there moments that jump out at you?
B: There were lots of good times at Bell Labs where you would build something and—it worked! And somebody used it; those kinds of things happened quite off in that golden era in the 70s when Uix was young, theree was all this low-hanging fruit; this group of people; we were all together in this. That was in some sense a really really good time. (“Was AWK an example of that?”) Yeah, absolutely. And all your stupid mistakes are all right there for people to look at. …