Open-endedness and AI

A conversation with Dr. Lisa Soros

I’m trying out audio! The full transcript is below, if you’d like to read instead of listening, but I recommend listening! The transcript was automatically cleaned up and might not exactly match the audio. As always, comments are highly appreciated to let me know what you think about this post and format.

So this week, the Artificial Life Conference is taking place in Copenhagen. While this newsletter is about artificial intelligence, I wanted to take the occasion to celebrate this conference by highlighting it through an interview. Artificial life, or ALIFE, is a distinct field from AI, but there are many common themes and intersections. Research in ALIFE studies the fundamental processes of living systems in artificial environments, ranging from studies of the origin of life to understanding properties of complex systems that may or may not fit the definition of living.

ALIFE has had a number of ties to AI, including biologically inspired learning methods, evolutionary robotics, and the recent field of open-endedness. Even big tech is interested. Google DeepMind has an open-endedness team, which recently published an article titled "Open-Endedness is Essential for Artificial Superintelligence."

To chat about open-endedness, I met up with Dr. Lisa Soros. Lisa is a leading researcher in artificial life and specifically the subject of open-endedness. She is currently a Roman Family Teaching and Research Fellow in Computer Science at Barnard College. Since 2017, she has served on the board of directors for the International Society for Artificial Life, which promotes research on the simulation and synthesis of living systems. Her research has covered generative systems in virtual worlds, evolutionary robotics, content generation in video games, and applications of evolutionary algorithms to open-ended search. Her work has helped determine what open-endedness is, established criteria for open-ended evolution, and proposed platforms like Chromaria, which enable the study of open-endedness.

So, the following is a conversation with Dr. Lisa Soros.

DW: So to get things started, I wanted to ask about the biological sort of inspirations for all of this. And for open-ended evolution the source of inspiration can often be biological evolution. So a maybe simple question is, is biological evolution on Earth open-ended?

LS: Yeah, so that's a question that people have differing opinions on. It's kind of been traditional in the artificial life community - again, our sort of tagline, we like to use the synthesis and simulation of living systems, particularly in this open-ended evolution community, which is sort of a subfield of the larger artificial life community - there is the idea that we want something that feels like it would go on forever. So when we think about normal computer algorithms, we want to basically prove that they're going to converge, or we want to guarantee that they're going to get some output. We just want a mathematical guarantee of what's going to happen, and that the algorithm will stop. That's literally like the definition of an algorithm is that it will halt in a finite amount of time. With open-ended evolution, we're kind of trying to do the opposite thing, which is make an algorithm that will do interesting things forever. And what a lot of people say is that evolution on Earth, like biological evolution, is an example of at least one such process because from what we've seen, it doesn't look like things have stopped. It doesn't look like we've started creating new kinds of creatures, like evolution is still happening, things like that. So one idea is if we were to create such an algorithm in a computer, maybe we should look to biological evolution. My personal opinion, which is not necessarily reflective of what the community as a whole thinks, is that biological evolution is at least effectively open-ended. What that means is that from our human perspectives, we're probably not going to see the end of evolution in our lifetimes. Now, is it mathematically completely going to be open-ended and go on for an infinite amount of time? I don't know. Probably not. Just thinking about the fact that the universe probably won't continue forever, but for all practical purposes, sure. So it kind of depends on who's asking the question, I think.

DW: So that then motivates, I guess, the use of computers or artificial systems to study evolution instead of biology. So could you speak to that motivation and why that makes sense?

LS: Yeah, absolutely. So there are a couple of different purposes, I think, work in open-ended evolution serves. One can be to actually use computers to study biological evolution. And the reason computers are useful for that purpose is we can create hypotheses about why we think evolution or biology or really physics, chemistry, anything works the way it does, and we can sort of change the rules of the game. We can almost do a bit of, I hesitate to call it something like science fiction or speculative fiction, but we can ask questions like, well, what if things were a little bit different and with our simulations we can kind of create little knobs we can tune and say, well, what if we really amped up how mutation works or something like that? Or what if we imagined that the basis for being alive wasn't carbon, was something else? That's actually something we usually do. We are very rarely actually trying to recreate literal aspects of biological evolution. We're generally creating abstractions and that's something that you can't do with, if you're a biologist. And that's not to say anything that biologists do is necessarily bad, but if we look at the world that we inhabit, we can't sort of say, well, what if there was just like no hydrogen? What would things look like then? So that's one thing that we can do with computers to sort of study open-endedness. Now, apart from the science, the other thing that we can do, the other purpose that this field serves, is more of an engineering application. And I think that's where it starts to get more into why modern AI researchers have become interested in this. So that's more towards fields like evolutionary robotics, where we can create algorithms that will generate, for instance, increasingly complex neural networks or controllers for robots or things like that. And especially on the AI angle, I think, my background is a bit in this field called neuroevolution, which is applying evolution-inspired algorithms to generating increasingly complex, increasingly more powerful, and I'm not going to say increasingly more intelligent. I'm going to strongly say not increasingly more intelligent, but some people do, so just we can get back to that. But the idea is if we did have an open-ended process that generated a bunch of really cool stuff, like the way that biological evolution did, maybe that would help us solve some bigger problems in AI.

DW: Yeah, and that does seem to be in line with some of the recent research in AI. I want to just get back to the first part of your response about biological evolution and sort of stripping away the complexity or opening up possibilities, because I think that that's also something that we do in AI research, where we have artificial forms of computation and information exchange that we use instead of biological ones that don't have the same constraints. I really liked a quote that I found from your thesis. "One of the challenges with open-ended evolution is to avoid conflating biological evolution, which is a specific instance of evolution, with evolution in general, which is an abstract process that could be implemented in many different domains. The properties and dynamics of biological evolution are not necessarily essential to open-ended evolution, even if they are observed phenomena in nature." So I wanted to ask, after having done research on open-ended evolution, what are sort of the principles of evolution, biological or artificial, that cause it to be open-ended? What are the properties or what makes this process open-ended? And have those sort of been defined or can those be defined?

LS: Right, so one, I was going to say actually related to the question of definitions, I will note that not everybody in this field entirely agrees on definitions of what is open-endedness, but I think many of us are trying to do work despite that. So in terms of what is important about evolution in general, biological evolution, it might actually, instead of first thinking about what are the important properties, might be also thinking about what may not be an important property. In terms of what's necessary, almost every definition of evolution, whether we're talking about biological evolution or evolution that we're creating in a computer or if we're thinking about, to give more instances of how we might think about evolution in general, like evolution of language, evolution of technology, all of these different kinds of things, you have a process where there is heritable change happening. So you have, basically the heritable part is usually we talk about there being parents, what it really means is you have something that leads to something else. So mathematically we might think about some sort of search base that we're looking at, where if you think about a 2D grid you have an x-y coordinate and you have another x-y coordinate next to it, we're moving from point to point, those two points are related. If we're thinking about biological evolution, we just think about for instance like humans having children, we think about, for instance, like memes, you can think about, well we developed some sort of, and memes is something that can be broader than just like our memes on the internet, but to use memes on the internet as an example, you see some new formats, a new meme come up and this one says, well that's funny, and they sort of change it a little bit and that leads to more memes. So the important part of that though is also thinking about the fact that, again, there is change. So there must be some force in your algorithm or your process that is going to make sure that what comes next is not identical always to the thing that came before.

DW: So change is a necessary part, but you could have change in a closed system, right, that eventually doesn't give anything sort of novel or surprising. What is it about biological evolution that leads to it continuously being open? Could you narrow down what this property of evolution even means?

LS: The short answer is it is related to individuals, I don't want to say individuals, in biology, it could be like creatures, we could think about humans, that they have to interact with each other and these interactions between things in the world have to lead to new opportunities for ways of doing things or new possibilities for ways of being, eventually, may not be immediate. And that's something I talk about in my dissertation, so I would say maybe if people want to go take a look at that, feel free, it is not a lecture about that, but the important thing is you have components in the world that are interacting with each other and we call that co-evolution in some ways, where the success of one creature or something like that can affect the success of others. So an example that my PhD advisor constantly used to use a lot was talking about giraffes and trees. So giraffes would not be sort of successful as animals if they didn't have trees to eat. The example that's classically given, so that's again how these sort of creatures or animals or whatever sort of create again new opportunities for each other to succeed or new ways that are possible to be in the world. To then build an abstraction though, and I'll do that briefly, we can talk about a definition. It actually comes, and again this is where the connection to sort of creativity is, but there's an abstract notion of possibility spaces and this is something that a few people have been exploring in the artificial life community but also outside of it, where you have a set of things that are again possible and as your system, which could be again an algorithm, a virtual world, whatever, as it evolves over time the set of things that could exist becomes larger and if anybody is interested in finding out more about that I would highly recommend looking into Stuart Kaufman's work on like the adjacent possible, is the key word you want.

DW: So what are the possible spaces, just to give a sort of concrete example, in something like biological evolution? I mean maybe there's a type of living system where the examples are even more clear, but I like this idea of giraffes and trees, but you know what are the possible spaces around that?

LS: I feel like it's actually easier to think about this in a non-biological context, but something that's concrete is actually technological innovation and evolution where, or scientific evolution, and where we discover, let's see, so for instance discovery of thinking about technology, discovery of fire basically led the way to a lot of things that are possible in modern life. Before we had fire we couldn't really heat things in the way that we do now. If we think about something more modern we can think about just innovations that happened for instance in hardware. So if people have heard of, well yeah actually we could think about even just the fact that a lot of modern deep learning and things like that was made possible with advanced developments with basically GPUs that once, like before that innovation was there, we just couldn't do the things that we can do now. So as soon as, so we would think of that as being yeah like a more closed space that becomes open once we make those kinds of discoveries.

DW: Yeah and I guess that that's a part of the sort of surprising aspect of it.

LS: That's right and I think it's actually harder to think about that in terms of biology because of the fact that biological happens so much slower than for instance technological innovation or something like that. So yeah if you're a person like I would challenge people to kind of think, if you're a person you're a person, I would challenge people to think about like what are sort of yeah things that have developed where it's like you get a sort of aha and then that leads to something else. That's kind of like the feeling of what we're trying to do but again it's from a more like computationally and mathematically rigorous way.

DW: Yeah I mean you're quoted in the DeepMind paper as saying that "observers of an open-ended system will be surprised but will be surprised in a way that makes sense in retrospect."

LS: Yep that's what I believe the feeling of open-endedness and really any sort of system whether it's evolving virtual creatures biologically based or again more towards the AI side. That's I think what we want to feel as humans when we're looking at these kind of simulations.

DW: So even though there's an evident example through biological evolution there are many types of systems out there that undergo open-ended change. Maybe would it be appropriate to call it open-ended evolution or are there other processes besides evolution that are open-ended?

LS: Yeah that's a good question. That's something we're kind of trying to tease apart in our paper and also that other people in the artificial life community have been talking about. A couple of years ago there was a paper by Howard Pattee and Hiroki Sayama called Evolved Open-Endedness Not Open-Ended Evolution, I believe. It might have been the other way around. But that's the paper where we're kind of talking about well there could be an idea of open-endedness that is not necessarily evolutionary. And I think that might be sort of the flavor that the sort of modern say adoption of this language that's been used a lot by the machine learning community and the AI community more specifically. So there's an open-endedness workshop at a NeurIPS called ALOE which is like agent learning, open-endedness, that kind of thing. Where a lot of what I think people are showing are not necessarily evolutionary. And I think what's important to recognize though is that the difference in saying there might be a difference between open-ended evolution and open-endedness in general is at least opening up the possibility of that existing. So we're not strictly requiring that systems have to be bio-inspired in a certain way in order to have these sort of possibility spaces that expand. And sometimes we can have an example of that as like what's the thing that's making change, what's the thing that's again pushing that narrative forward that says well we don't know what's going to happen but we might sort of in retrospect think that it makes sense is if we're thinking about sort of the future of humans and AI there might be a human in the loop that might be where some of the change comes from. But yeah so it's a possibility for possibilities I suppose.

DW: Yeah I mean it seems like some of the defining characteristics of evolution or at least Darwinian evolution of having some sort of stochastic part in the change happening and then also having a selection pressure those are things that happen also in learning. In training a deep neural network you do a similar thing. So are there other processes like learning that could be considered open-ended?

LS: I would almost say that anything that's going to be creative is probably open-ended. So even if we think about human creativity and that's you know the question of how might be a little bit different there. But I would argue that creativity is very much not learning. I mean you might learn things along the way right but the idea isn't necessarily to train a model that's going to get you to a specific point which is again why I will admit I hedge a little bit on like or I rather want to make clear that there are probably oh that's something I probably should mention earlier right. So we've been kind of discussing so far about open-endedness is something that like are something is or is not open-ended and it's probably important to recognize that there are degrees of open-endedness or there could be degrees of open-endedness. I think especially that the AI community and more sort of taking these ideas and applying them towards engineering purposes and trying to create AI might have a process that's in part going to be doing this like expansive exploration and encouraging diversity and messing with selection pressures and stuff in the way that is inspired by biological evolution. But the end goal might still be to achieve a particular outcome which is a bit apart from the spirit of what we're looking for in the artificial life community.

DW: Yeah so it seems like there are a few attempts. I mean you mentioned the ALOE workshop at NeurIPS and there's an article that I'd like to get to its definition of open-endedness at some point. But in the Alife community it seems like there's a sentiment that open-endedness can't fully be defined or quantified. So there's an article by Susan Stepney about the quantification of open-endedness saying "The nature of open-endedness is such that an open-ended system will eventually move outside its current model of behavior and hence outside any measure based on that model. This presents a challenge for analyzing artificial life systems leading us to conclude that the focus should be on understanding the mechanisms underlying open-endedness not simply on attempting to quantify it." So I was wondering if you could give your thoughts on that. Is open-endedness a characteristic that we can say is or is not in a system or can we quantify the degree to which a system is open-ended or is that impossible given the nature of open-endedness?

LS: Just as a side note I'd also like to acknowledge Tim Taylor's work in the past few years about thinking about open-endedness in terms of models and models becoming meta models and things like that. I don't want to speak too much and perhaps mischaracterize the work but he has somebody who's been sort of thinking along similar lines. In terms of quantification historically people have come up with different ways of measuring if something is open-ended and most times kind of just as we as humans looking at the systems that might mathematically say they are open-ended because things keep changing in some way just aren't entirely satisfying. So I don't think it's entirely impossible to quantify open-endedness. However, I'm not sure right now there doesn't seem to be a consensus that anyone's way of measuring or quantifying it is correct. To the work that you mentioned, however, I do think that the idea of generating that looking at the underlying processes are at least more interesting. Some people in the community do have a feeling, especially if we're talking about these simulations where we can look at them and we can see things moving around on the screen. It might look kind of like a video game or something like that, like a zero-player game. There is something to be said for maybe we'll know it when we see it. I'm not saying that's an entirely scientifically rigorous thing to do, but in my own work I've shifted a little bit more towards at least using that as a tool for exploring a bit. We actually did have a blog post that was published on the Cross Labs blog about subjective open-endedness and exploring the idea that might not be possible just because we're humans. Also, there is a question of, well, what would it mean if we ultimately, we the scientists doing this work are humans and we are doing it for humans? I think it's hard to get the feeling of what we want if we presuppose what it's going to look like. If we establish some quantitative signpost for when we see this particular number go up, we're going to be impressed by it. I don't think that's the way that we work. That being said, when we do scientific experiments, we do want to be able to measure what's going on. We want information about what's going on. We want to know if, especially people that are doing more of the scientific work rather than we still want to make sure that we're testing things in a rigorous way. A lot of that might be thinking about what kind of information are you measuring, which to give one pointer that again I don't know entirely much about, there are tools from information theory that would allow us to explore mathematically if something open-ended in a way that also might not just be talking about biology but really any system like technology evolution of memes whatever that would tell us if something is evolving. I'm not an expert on that. What I will say, however, is there's a phenomenon that this is thinking more about like a phenomenon that exists rather than specifically a number, and this might be related to information theory. A lot of people look for something called a phase transition. This is not just about biology. It can be about other domains as well. We want to basically say that our level, and this is a very non-mathematical my understanding of a definition, I'm not saying don't quote me on this, but like don't take this as an extremely technical definition, but from the human perspective of looking at a system with a phase transition, we might say that it might be that the model's changing in some way, but it also just might be that our level of description is changing. We might think of going from looking at us at least my understanding and maybe this is what I want to see is at the beginning physics is interesting, and then once we start seeing different atoms or chemicals coming together, atoms coming into molecules, then we can start thinking about things in terms of chemistry. Once we've got the molecules and all that, and then once we kind of get things getting complicated enough and we don't entirely know how to do that, that's what we're figuring out with this research is suddenly it makes sense to think about these collections of atoms and molecules as creatures or plants or whatever. Once we start going up that ladder from our simple description of things is no longer sufficient to capture the complexity of the world or the system that we're looking at, that's the phenomenon that we're trying to hope to get to. There are tools for recognizing that. I'm not an expert in that though.

DW: I think that that points to some of the limitations of studying this field is that it depends on our viewpoint of it, and we're looking at it as humans with a certain level of understanding of certain systems where if we had greater tools of analysis for some systems, we might recognize these phase shifts going on in them that we haven't seen. That gets to this concept of an observer that I want to get into because the definition of open-endedness, if there is one out there, seems to hinge on this idea of an observer. I just want to read two posed ones more from the machine learning community about what open-endedness could be. The article from DeepMind says "The definition of open-endedness hinges on a system's ability to continuously generate artifacts that are both novel and learnable to an observer. Open-endedness is observer dependent, and novelty and learnability alone is not enough." An article by Olivier Sigaud posits that "an observer considers a process as open-ended if for any time t, there exists a time t' greater than t at which the process generates a token that is new according to this observer's perspective." There are definitely elements of novelty and learnability in there, but I want to get your perspective on what this observer characteristic of open-endedness is about. Why is an observer necessary for the definition or for the understanding of open-endedness?

LS: I'm going to read a quote from the subjective open-endedness blog post where we talk about observers and then maybe explain a bit where that's going. Things might get a little more complicated before they become simpler, but talking about this idea of subjective open-endedness, which again we're saying you might need a person in the loop or just that generally speaking, maybe the quantification is not entirely the way to go. We have how might we make the concepts like interestingness and subjective open-endedness more concrete. We take the point of view of an observer either, and this is the interesting part, either within the system or without, and then look at the relationships formed between observers and how those relationships might evolve along with the system. When we talk about something outside of the system, that one's easier to think about because we might think about a literal human observer, somebody looking at one of these systems and saying, hey, this creature looks different from the things that came before. We can observe that it's its body. We suddenly have more legs or something like that or creatures have a different way of walking or something like that. But talking about an observer within the system, if we're thinking about this from a biological perspective, we kind of have an implicit algorithm or an emergent algorithm. There's not one point we would pinpoint as an observer. Instead, we might say that something like a population of creatures is evolving in a particular way because we have things like selective pressures that are going to say, okay, there used to be a lot of food. There's no longer that type of food, so things are going to die. That's one sense we could think of an observer in. To bring it to something that's easier to understand or make concrete, we would think about then shifting to open-endedness in the sense of machine learning type algorithms where we might think about something like a reward function, something that's looking at how things are changing and deciding what's good, what's bad. That would be one sense we can think of observers in that context.

DW: Could a human be that observer to systems that are as complex as something like information going through a large network? Thinking about a social network and how information spreads there or technological change, can we actually be enough of an observer over longer time scales for things that might last multiple decades? If not, are those systems still open-ended?

LS: First of all, I would say I think that we could. It becomes a bit messy thinking about humans observing technological change because we're also in the system as people that are making the technological change happen, which is not a bad thing. It's just perhaps an important thing to think about, an important perspective. To focus on the technological innovation bit, we might not think of there's one observer of a system. There are many observers of a system probably because there are many forces interacting with the system and exerting change on the system. When we think about technological evolution, it's not just the humans that are coming up with the ideas and responding to the ideas and buying things because they like them. But there's also economic factors that impact things. There are factors impacting the economic factors, so this comes back to the question of can we objectively quantify something as open-ended? It depends on what the question you're asking is and why you're looking at the system in the first place. I don't think that there's always exactly one optimal observer or one position that should be taken or one question that should be asked.

DW: To get a little bit into the relation between this and artificial intelligence, can a human serve as an observer of open-ended change in an AI system?

LS: I would ask what you mean by an AI system.

DW: Something like a neural network, a machine learning model trained on human data that could potentially be continuously changed in order to enact a change.

LS: This comes back to the perspective I mentioned earlier that I like to take, which is that some things can be defined as being practically open-ended. That's actually exactly where that comes from. We, as human observers, especially something that might evolve over decades, something can feel open-ended for a while and that might be good enough. We might not need mathematical guarantees that something is going to be open-ended forever, depending on what purpose we're trying to channel this idea of open-endedness or an open-ended process for. There's a distinction between what a human observer might feel and what might be mathematically true in the long term, but I think depending on the question you're asking and the purpose you want this to serve, it kind of doesn't matter if there's a difference.

DW: There are two aspects to the learnable question at least as posed in some of these articles. There's novelty and there's learnability. Could you give short explanations of what those mean to you?

LS: I can talk about novelty and I can talk about learnability to a limited extent because my gut reaction is that learnability is not something, again, coming from the artificial life perspective that I think we are naturally trying to channel. It might be related to the idea of adaptation. Starting with novelty, what that would mean to me is the idea that there's a new kind of thing and what we want is new kinds of things. A counterexample, what some people do and what an earlier algorithm called novelty search did was define mathematically how different from something in this case we were talking about. It started out talking about agents, AI agents solving a maze, how different must something be from what something else did to qualify as novel. In this case, it was the robots that were navigating a maze ended up in different final xy positions in the maze and that's how you would measure if this was different or novel enough. One of the big questions is figuring out, is there a way of describing how new something is? To bring another counterpoint into that way of looking at things mathematically in terms of how do we define what's novel, there's a researcher named Kate Compton who talks about this idea called the ten thousand bowls of oatmeal problem. In the ten thousand bowls of oatmeal problem, we can imagine that there are ten thousand bowls of oatmeal and all of the oats in those bowls might be mathematically distinct in terms of their orientation. Even though you can describe each of those oats as being different, we as humans looking at it are just seeing ten thousand bowls of oatmeal. We don't really care. What that kind of goes to say is novelty is not enough. This also kind of brings it back a bit to the observer relative position. If we're thinking about technological evolution or how we might talk about things in terms of AI, the fact that we as humans might not see something as novel might also not mean there might be something going on with the learning process under the hood that leads to something that's actually kind of cool. Human perception of novelty is not always the same as what might be important to eventually getting there. Talking about learning, the other idea that you mentioned was learnability. That's something where it might actually be helpful if you could give the perspective that the DeepMind folks are coming from so I can tell you ours is a bit different perhaps.

DW: I think that it definitely comes from a machine learning point of view where the observer needs to be able to understand the phase shift. They give an example of technological innovations in aircraft where going between different aircraft systems from very early planes to modern ones to ones that might be designed in the future, a human observer would find the phase shifts novel and be able to learn and understand even the more complex ones that come after the current design. If we had access to them, we could see them and maybe understand them. Whereas an organism that's less able to understand and learn like a mouse would look at those changes and might consider them novel because they look different but wouldn't be able to learn the ways in which they are novel or different, wouldn't be able to learn those designs.

LS: That's actually really interesting. I'm not sure if anyone has said this before, but I think it's really important and if people keep hearing my things and quoting me informally that's fine. Let's open up a question here and throw it out there for people to think about. What that suggests is that the observer might need to change over time inside of an evolutionary system if it's going to be open-ended. Mice might not see things as open-ended or learnable in some way because they don't understand it. Just like humans, what evolution is going to give us, let's imagine that society hasn't totally collapsed in 200 years which I'm skeptical about and that's fine to leave in the podcast, but assuming we're still here, or think about a thousand years. The world that exists then might not make sense to us given how we understand the world now. If you took someone from the 1500s and showed them the internet, who knows what that would look like? It's only because we as the observers of the system have also been evolving inside the system that we can say, "Oh yes, things make sense now," because we do see that narrative again of we were surprised by the changes but it made sense later because we learned the steps along the way.

DW: That's a really interesting perspective. I think that people who are looking at this inside the AI community are potentially concerned about the technological shift outpacing the human understanding that will change over time. To get a little bit further into the DeepMind article, then I want to hear your thoughts about the AI connection here, but I want to look at the points that they're making first.

LS: Sure.

DW: The argument of this paper is that open-endedness is necessary for what they're calling artificial superhuman intelligence. There's a lot to deconstruct there already in terms of what is artificial.

LS: I'm going to have to sidestep that entirely and not assume that I have either an agreement or a disagreement with what those words might mean or whether they exist.

DW: To keep it focused on open-ended, let's ignore that part of the title.

LS: Excellent, I'm on board.

DW: They argue here to advance in levels of artificial general intelligence towards artificial superhuman intelligence, whatever that might mean, "we require systems that are open-ended endowed with the ability to generate novel and learnable artifacts from a human observer." Since foundation models, large neural networks trained mostly on human data, "since foundation models are periodically retrained on new data, including data generated by their own interactions with humans in the real world, one could argue that the data distribution is not really fixed. We contend that augmenting foundation models with open-endedness offers a path towards artificial superhuman intelligence." From my understanding, we have these large language models, we have vast datasets of human data that now includes AI data inside of it, and we have an ecosystem of models trained through different means, trained through different reward models with reinforcement learning, with human feedback, training them to act in certain ways, is that already an open-ended system? And is open-endedness possible with that setup? I guess those are two separate and large questions.

LS: Sure, so I don't feel entirely comfortable answering the second question with a yes or no because my work focuses more on the evolutionary side of things, and I am very much not a machine learning researcher. What I would say, which is going to sidestep the question a little bit, is to maybe not think about things as they are open-ended or they aren't. There might be aspects of open-endedness or kinds of open-endedness or degrees of open-endedness that are possible in some way, but I would be careful about engaging in the idea that there's one fundamental key that's going to unlock this AGI superhuman intelligence idea. What I think people should take from this idea of bringing in open-endedness is thinking about questions like novelty, learnability, and other ingredients that we can put in to serve our purposes. Let's assume, just for the sake of argument, AGI exists, and we will recognize it when we see it. If we take that for granted, then I think some kind of open-endedness is at least going to give us something different and hopefully better than algorithms that are supposed to be entirely convergent. To my understanding, this is the way we've been thinking about things in AI for a long time.

DW: Yeah, even current training methods for these large language models, from classic machine learning ones to reinforcement learning with human feedback, even if that human feedback is a moving target, it's still an algorithm that's supposed to converge. It's supposed to train to whatever the current human feedback is. Is open-endedness a good objective for training those models, whether or not it leads to some target like AGI or ASI? Is it worthwhile to pursue training models at that scale with an objective of continuing to create new surprising things?

LS: You know, we in the ALife community have been working on this problem of how to define what open-endedness is and measure it for a very long time. If somebody else wants to add to the conversation, that would be great, but it's been an ongoing conversation for decades. So, sure, maybe if you can do it, if you can figure out a way, let us know because we have a lot of ideas but no consensus yet. For engineering purposes, with specific AI applications, where it is possible to define what we need to be different and in what way, it almost seems like even if we're talking about AGI or ASI, I come from the tradition of calling it AGI, but don't take that as loaded with any particular meaning, something that's human level or superhuman level still seems to me to be focused on getting to a specific point. Especially if it's something that we want to say we, as humans, can recognize as being the next step in human evolution. That's going to have to exist. To talk about this idea of adjacent possible stuff, it can't be too far from what we already know. So, it seems there's a very limited set of what the next step would be that would be recognizable or learnable to us. In that sense, we still have a fixed set of things we're trying to achieve. That means we might actually be able to quantify some ideas because the aims are narrower in scope, even if it's a grand idea to say that we want something that's the next level in human evolution.

DW: Could an artificial system act as an observer for the open-endedness of an artificial system?

LS: I don't see any reason, theoretically, why not, but it may or may not be interesting to humans. One might raise the question, is this going to give us Terminator, where robots get out of control and kill us? I watched Terminator once on a bus when I was eight, and that's my assumption about what happened there. When people talk about these fears of robots getting out of control, that's where they're coming from. Bootstrapping the AI with AI is how it learns to become infinitely powerful and just kill us. Not that you asked that question, but frankly, yes, it's possible. What it might do would be interesting. I don't think we have to worry about the Terminator situation as much as we should worry about other humans, so sure, let's explore that.

DW: I agree, and that's a point I try to convey often.

LS: We have other things to worry about. I live in New York City.

DW: Keeping the human perspective as a part of the observation is necessary because we want these things to be useful tools for us. It is an assumption in this article that using an artificial observer will lead to potential safety problems of super-intelligent killer robots. I want to put that aside for a second and talk about another assumption in this article, which is that foundation models, these really giant neural networks, have somehow changed AI and could also change our view of open-endedness or that they combine well with open-endedness. Do you feel foundation models have or could have an impact on the way we think about open-endedness?

LS: What could happen is that new systems developed outside of the context of artificial life might contribute to our set of ideas or ways of thinking about kinds of open-endedness. I'm not entirely sure how much innovation in open-endedness space in the AI world would come back to how we think about evolution in artificial life. It might, but I don't know. I would love to see something that makes me feel like that's possible. I'm not saying it's impossible, but I'm skeptical. I like to be proven wrong about things I'm skeptical about. I just don't get my hopes up about anything before it happens.

DW: A bit of skepticism is always healthy. One aspect that seems relevant is that generative models are now being used a lot in procedural content generation and video games. Simulation platforms have long been useful for the study of open-endedness, creating sandbox worlds where many different things can happen. Is there potential for these large generative models to open those spaces up further and create testbeds or ways for us to look at open-endedness?

LS: I think that goal is really cool, and they could be useful. Those aren't the tools I use, but I'm skeptical and would love to be surprised. One reason I say that depends on what we mean by generative model. My intuition, coming from ALife, is that I'm skeptical about any system fundamentally trained on data about things that already exist or bootstrapping from things that already exist being truly new, surprising, and interesting to us.

DW: Creativity doesn't seem very in line with predicting the next token correctly in a sequence of tokens. You want that to be accurate, not creative.

LS: That's right. The other thing I meant to mention earlier is when we think about either biological evolution or human creativity or processes like that, we don't generally expect the process of innovation to be linear. A lot of times there are pauses where nothing happens, and then suddenly you have a huge spike in innovation or something like that and radically new ideas. Cambrian explosion happened at a point, but the term we use in biology is punctuated equilibrium. If there is supposed to be this continual production of novelty, which is when people talk about open-endedness, but then bringing it back to AI, that might be a way that we're expecting to see linear progress in terms of neural networks solving some problem. I just wanted to mention that that's something that isn't usually true or doesn't have to be true strictly.

DW: To wrap up, if AI systems observing AI systems in an open-ended process isn't the dream connection between open-endedness and AI for you, what are some avenues in which open-endedness and AI have already benefited from each other or worked together? What are some directions you see going forward where open-endedness can be really useful and interesting for AI?

LS: One direction that I'm not sure about the future, but one idea that's popped up in the past couple of years, has been thinking more about environment generation and having different things that we expect AI to do or different tasks we want AI to solve. When I talk about tasks environments, I'm literally talking about virtual worlds. Some aspects of procedural content generation feed into this idea of how do we create these new worlds. There have been algorithms like POET that have to do with generating. You have an agent that solves one task. Can you have multiple agents solve multiple tasks? That's one direction that I'm not sure if it takes inspiration from the open-endedness community in artificial life or what. The idea that we can come up with new things to do or that we would like to see an agent do is something that I think is important and something we should be exploring more, and that I think we are exploring more. There's an open question of how do we make a world that is resonant to us, a virtual world that is resonant to us as humans. By world, it might be tasks we're achieving, like playing hide and seek. There was the hide and seek paper a couple of years ago, but they do change in interesting ways where the environments themselves are becoming more novel and still being understandable to us. What is a robot or a neural network doing that makes sense to us, but it might not have been something that we thought the robot was going to do? Maybe this feeds into the Terminator situation hype. I think the focus on looking not just at an agent accomplishing a task but thinking about the ecosystem of how the agent is embedded in an environment, how it interacts with an environment, and more systems thinking, that's probably the biggest thing that thinking about ALife and open-endedness can bring to AI.

DW: Great, and I think that in terms of robotics, and I would open that up to machines in general, that sounds very relevant because we often put machines into unknown circumstances and hope they can still operate. Thinking about the Voyager spacecraft that continues to operate years after its intended end of operation date, where we have to fix problems with it, there's a certain level of adaptability and robustness to change that we want from those systems. As they get more intelligent or capable and autonomous, being able to encourage that creativity is going to be very important. Thank you for this conversation. It was really interesting.

LS: For sure. This is good. I like thinking about these kinds of things a lot and how they relate to humans, not just to academic researchers.

DW: Yeah, keeping the human connection in mind in this is very important.