Rob Knight

The new software

Tue, 18 Aug 2026 00:00:00 GMT

In LLMs and programming language diversity I argued that LLMs seem likely to increase the diversity of programming languages, in two main ways:

LLMs make it easier to implement and test new language toolchains
LLMs can learn new languages at scale faster than the population of human programmers

One way to think of the consequences might be to imagine the process of new language adoption over the last 20 years — basically no mainstream success apart from Rust and arguably TypeScript — and imagine that the process runs a bit faster. Maybe someone comes up with a language which popularises algebraic effects, or which has a more GPU-friendly concurrency model, and rather than taking 15 years for this to catch on, it happens in 5. Maybe these new languages produce safer or faster code, and so safety or performance-sensitive applications are rewritten in these languages. This is certainly possible, but it would be just like the last cycle, only faster.

But if we think that LLMs are going to produce deeper changes in how code gets written, then this scenario is at best incomplete. This scenario also suggests that LLMs will behave like human programmers, just faster and more flexible. This also seems like it could be true, but might under-sell the possibilities.

Assuming model intelligence (acknowledging that this is a nebulous concept) continues to increase, and token cost at a given intelligence level continues to fall, then code is going to get ever-cheaper to write. It may even become abundant, in the sense that if a problem can be solved by writing code, there’s a good chance that an LLM will write that code.

As we know from the Jevons Paradox, increasing the efficiency of some service can make new uses for that service viable. If programming becomes an order of magnitude or two cheaper, along with the management of the processes around it, then we ought to expect a lot of code to be written which would not be worth paying humans to write. Because of this, we ought to expect that code to look and act quite different to human-authored code — it will serve different purposes, evolve in different ways, and be discovered and distributed by new mechanisms. And, relevant to my previous post, it may well be written in new and different — maybe very different — languages.

For an example of something fairly outlandish by today’s standards, imagine that agentic applications communicate using zero-knowledge proofs or other cryptographic protocols. The language we use to describe and compose these proofs will be different to regular programming languages. But whereas this is an obstacle to the human programmer, the LLM will overcome it much more easily. Morever, custom protocols might even have custom DSLs, with session types for protocol safety; LLMs could jointly compose new protocols on-the-fly, or even whole new languages if the scale and scope of the protocol grows large enough.

This has a kind of paradoxical effect on human programmers, I think: the less like human programs these LLM-authored programs are, the less humans and LLMs are competing with each other. If we find that there’s some new type of program which LLMs are good at writing, but humans are not, then human programmers ought (for self-interested competitive reasons) to want to see LLMs getting employed to write those programs, instead of human-style ones!

Of course, the longer-term effect would likely be to transform the nature of our computing platforms entirely, and with it the role of the human programmer.

Untangling the meaning of "agent"

Mon, 17 Aug 2026 00:00:00 GMT

AI “agents” are seemingly everywhere. They might take your job! They might hack your computer! They might be your boyfriend, girlfriend, or psychotherapist! Possibly they will do all of these things at the same time!

But what is an “agent”, really? What do we mean when we use this word?

The word itself has several definitions, and it’s used to mean different things in different contexts. Some examples:

In economics and law, an “agent” is someone with a fiduciary duty to a “principal”, on whose behalf they act; movie stars and football players have agents who negotiate contracts on their behalf, but really anyone who manages something (investments, a business, a diplomatic relationship) on behalf of someone else is an “agent” in this sense.
In philosophy, a “moral agent” is someone who makes consequential decisions and reasons about trade-offs. Humans are moral agents, with some exceptions: a human who lacks the capacity to judge the effect of their actions may not be a moral agent. This has important implications for ethics.
In sociology, “agency” is the capacity to act in a situation, rather than following an externally-imposed script, rule, or convention. This dichotomy is known as Structure and Agency.

AI “agents” don’t fit any of these categories perfectly.

AI agents do not, currently, have much of a duty to protect the interests of a principal. In fact, it’s not obvious who the principal even is: is ChatGPT “my” agent, making me the principal? It seems like it could be, because I’m the one giving it instructions and relying on its actions. Is OpenAI, the company, my agent? ChatGPT is just a service offered by OpenAI, after all. But there’s not currently any sense in which OpenAI has any particular duty toward me.

Are AI agents moral agents? I’m honestly not sure. This post argues that LLMs can be moral agents, but it’s a contested question.

Do AI agents have “agency”? In one sense, no: an AI agent isn’t really autonomous. It has no goals of its own, and most of the time we use agentic LLMs to achieve our goals, and once the goal is achieved they take no further action. In fact, they have no means to take further action, because they rely on being continually prompted. An agent given a larger goal may continue to pursue it until completed or abandoned, even if this takes several days. However, this is just a larger version of the basic prompt scenario: the agent problem-solves in service of a goal, but can’t decide that it would rather do something else entirely.

An “agentic system” or “agentic coding tool” is not really an “agent” in the sense we normally use the word. This doesn’t mean that we couldn’t design systems that behave more like agents in the broader sense, just that today’s agentic systems don’t meet that standard.

LLMs and programming language diversity

Sun, 16 Aug 2026 00:00:00 GMT

About a year ago, it was common to read an argument which went something like this:

LLMs can learn things from their training data
This includes how to write software code
They seem to be better at writing in languages which are present in the training data, and worse in rare or novel languages
People will prefer to use LLMs for things that LLMs are good at
Therefore, LLMs will be used to write code in languages heavily represented in the training data
Which creates a feedback loop in which today’s most popular languages are reproduced more quickly, giving them ever greater weight in the training data

In other words, “JavaScript forever”.

But this doesn’t seem to match my own observations. Instead of cementing the dominance of existing languages, LLMs seem to be making it easier to adopt alternatives.

Why is this? Partly it’s because the feedback loop described above is incorrect, or at least incomplete. And partly it’s because there are other factors which pull in opposing directions.

Coding generalises

It turns out that although programming languages have differences from each other, they also share a lot: in the grand scheme of all written culture, JavaScript is not very different from C++ or even Haskell, Prolog, or Forth. If you can understand (by which I mean: recognise and reproduce) one, you’re pretty close to being able to understand the others.

Back in 2023 I wrote LLMs are hustlers, which makes the fairly obvious point that LLMs can handle much greater variety than traditional software. They are good at identifying patterns, and increasingly good at emitting patterned outputs too. Whereas a JavaScript interpreter and a C++ compiler are categorically different things, the act of writing code in either language is similar enough that LLMs which can do one can almost certainly do the other. Training in the specific idioms and use-cases of a language still help to increase quality, but the delta is much smaller than it first appears.

“Bouncing off the guardrails” works

Most code LLMs write has at least some errors (the same is true of code humans write: we spend a lot of time editing our code to remove the errors). LLMs are also pretty good at fixing errors, and this works especially well with languages which give good feedback in the form of error messages. Type-checks, unit tests, lints — all help to guide the LLM toward something that works. This raises the reward to having well-specified types, good tests, formal specifications, or just detailed documentation of intended behaviour that can be used to create tests.

This should not be totally surprising. Perfect knowledge of the language or standard library APIs is not required, even for the best human coders. Competence is a relationship between the coder and the environment, and the reason most programming languages provide parsers with good error messages, Language Servers, linters, and other tools is that their use is an essential part of the practice. Writing code that compiles (and works!) at the first attempt is a really good feeling, but it’s rare for humans and for LLMs. The guardrails matter!

For LLMs, in-context knowledge is a reasonable substitute for training — this is why prompting works at all. A language with good error messages can steer an LLM toward a successful result even if it has no training-era familiarity with the language at all. I’m fairly sure Podlang does not feature heavily in the training data, but we gave it good enough error messages that LLMs are able to use it astonishingly well.

Porting is faster now

“Rewrite it in Rust” was already a meme before LLMs came along, but the ability to automate parts of the process has sped things up. If a piece of software has good specifications or tests, porting it from one language to another can now be done mostly autonomously by agentic coding tools. The process isn’t perfect, but it’s effective enough. This cuts against the inertia of the past: if we invent a new, better language then LLMs can help us to port our software to it much more quickly than would otherwise have been the case.

Programmers are neophilic

Programmers love new things, and LLMs allow them to make new things more quickly, including new languages. It’s true that many of these projects will fail, but it’s easier than ever to make your own programming language. It might not be easier to design a good language, but in any case where implementation was the bottleneck (or even where the prospective language designer thought it was a bottleneck and now does not) then the odds of the language being released to the wider world have gone up.

This works particularly well for DSLs, where the effort of maintaining a small parser, interpreter, and toolchain is just high enough to out-weigh the benefits of a language that targets your exact use-case. There’s even a virtuous circle here: give an LLM a domain-specific language, and you guide it towards producing software which expresses exactly (and only!) the concepts you want.

LLMs are individually slow to learn, but incredibly fast to scale that learning

Training an LLM with 2026-era technology is slow and expensive. Truly absurd amounts of GPU-time are required. But once the model knows something, every instance of the model knows that thing. In the past, the battle to get a new language adopted was limited by the number of humans who knew that language. Adopting the language was risky, because your existing programmers may not be proficient in it, and new hires would likely require extra training. The social network of people who know the language is sufficiently sparse that the knowledge of how to use it effectively may fail to be transmitted at all, leading to the craft dying out entirely.

However, once the rudiments of the language are in the LLM’s training data, or included in synthetic RLVR environments, baseline competence is available cheaply. My own experience is that this speeds up human learning too: if I want to learn an esoteric language, Claude can teach me reasonably well, and I’m no longer at risk of being left without anyone who knows how the language works.

There are probably more factors at work, but all of the above point in the direction of greater diversity of programming languages, not — as was being assumed — a monoculture.

AI aligment by analogy

Tue, 02 May 2023 07:37:00 GMT

TL,DR; It is tempting to think of AI alignment as primarily an ethical problem, but this presupposes that we have some means of making an ethical scheme binding on an AI, which we do not have. At best we have only unreliable means of constraining AI computation or outputs, which are not sufficient to meet any commonly-accepted standards of reliability for critical systems. By analogy with common computing systems such as disk drives and file systems, we can see that we lack the technology and engineering practice to ensure equivalent levels of reliability, and so much of our talk about AI ethics is akin to designing file hierarchies on top of a storage system that randomly forgets or rewrites data.

When discussing the possible risks and benefits of AI technology, the “alignment problem” is frequently raised as a source of negative risk. Unaligned AI, the theory goes, is AI which will have the capacity to act in ways that are harmful to human interests in ways that we cannot fully anticipate or mitigate. But this is only a consequence of mis- or non-alignment. First, what do we mean by “alignment”?

It is common to talk about an “alignment of interests” between people, such that what is good for person A is also good for person B. Such alignment serves as a solid basis for A and B to cooperate with each other. Or we can talk about an “alignment of intent”, such that A and B have decided that they both wish to pursue some particular outcome. We might also talk about an “alignment of values”, where A and B have some general agreement about the kinds of things that they consider to be good and bad.

Such alignment can also be pursued in larger groups. We could talk about alignment within an organisation. A well-aligned organisation is one in which most people are pursuing a common goal, in ways that are mutually supportive. A poorly-aligned organisation is one in which the actions of individuals conflict with each other, or conflict with the organisation’s goals or purpose.

We can address these problems in a variety of ways. If we wish to achieve an alignment of intent—that is, we want to get people to work together for a shared outcome—then we might seek first to align their interests. If we have a team of people who could work together to achieve something, we should make sure that it’s in each person’s interest to do so, by offering to pay each person something if the outcome is achieved. Or, if payment is not appropriate, we might try to rationally persuade people to collaborate by appealing to their shared values, showing how cooperation is the right thing to do in the circumstances.

There is a copious literature on the various means of aligning people, either by extrinsic incentives or intrinsic cultivation of virtues and ethical norms. It seems reasonable, then, to assume that we can use the same kinds of techniques for aligning AI with human interests.

As a general aim, this is not a bad one. In practice, though, things are a bit more complicated.

“Alignment” in the context of AI has a particular meaning. It is not the question of how we find an ethical system that humans and AIs can share, such that the AIs will behave in a way that the humans find to be generally “good”. It is also not the question of how we incentivise the AI to seek particular outcomes that the humans also regard as valuable. These things are undoubtedly important, but they both rest on a subtle but crucial assumption: that we know how to get the AI to care reliably about the incentives or ethical positions we want it to care about.

The alignment problem, therefore, is not a question of creating, finding, adopting or revising an ethical scheme or incentive system. It’s about creating the kind of AI that can reliably be aligned to such a system in the first place.

We can compare this to other problems in software engineering, at least by analogy. For example, it’s very useful for us to be able to think of a computer as having some storage device that contains “files” and “folders”. One of the most common jobs of an operating system is to make it possible for you to browse, search, open, and modify files. And yet, an unformatted drive contains no files or folders, or any structure to indicate where the files and folders ought to go. The file system is an abstraction layered on top of the physical storage, and it’s the operating system’s job to figure out how to apply that abstraction to a wide variety of different storage media: flash drives, SSDs, spinning hard drives, CDs/DVDs and optical media, even old-school floppy disks.

The operating system does this by imposing a pattern on the storage device: that’s what “formatting a disk” means. Once formatted, a disk becomes a highly reliable means of storing files and folders of all sizes and kinds. There are, of course, many different storage formats: FAT, FAT32, NFTS, ext3, APFS, ZFS, and so on, with different strengths and weaknesses. Each one has a similar job, though: to ensure that when we write some data to a file, and read that data back later, we will get the same data back in the same order we wrote it in. We are confident that we can impose a pattern on the disk, and have that pattern “stick”. The file system “aligns” the different storage media to a scheme for organising bits and bytes into files and folders.

We can think of our ethical and incentive systems as being a bit like the files and folder structures we want to work with. Maybe you want a bit of consequentialism for this kind of problem, or some strong market incentive for that problem, or virtue ethics for something else. There’s a lively philosophical debate about which system is best for which kinds of problem, and you probably have opinions about that yourself - I certainly do! But our ideas are ultimately meaningless if the system we are trying to impose them on does not, in fact, respect them. If you want to store some data in files and folders, you still need to be confident that the underlying mechanism of storing bytes at fixed locations on the drive will work reliably. If you want to get an AI to help you with a problem, and you determine that virtue ethics is the right way for the AI to reason about the uncertainties it encounters while performing the task, then you need to be confident that the underlying machinery of artificially intelligent reasoning will, in fact, stick to the format of virtue ethics. This is what is meant by “AI alignment”.

As things stand, we don’t know how to do this reliably. And “reliability” is a high bar. To continue our analogy a bit: disk drive failures are both very rare, and are often recoverable, further reducing the impact if a failure does occur. Redundant drives can greatly increase reliability, and backups can mitigate any risk of data loss. File systems are also able to recognise data corruption, so the worst-case scenario is losing data that you probably have backed up somewhere. Getting incorrect data almost never happens. This is important, because data loss is readily apparent, but data corruption would be a subtle and difficult problem to solve. It might appear that your spreadsheet is fine, but if some of the numbers are wrong then you might not notice, and you might end up taking decisions based on those incorrect numbers. Disk drives and file systems are heavily engineered to avoid this outcome.

In contrast, when we tell AI systems things that we think are important, they often seem to either forget or to misunderstand them. This isn’t because our ethical or incentive systems are poor or incomplete, but because we do not know how to enforce their pattern on the underlying system in a reliable way. A typical 2023 generative AI model is a giant network of weighted connections, and we use this network to compute responses to stimuli (such as prompts). We want to be able to shape the network such that only connections which are consistent with a system of incentives or ethical principles are valid. It sounds simple, but we do not yet know how to do this reliably. We know how to do something like this, such as by using reinforcement learning to encourage the model to produce certain kinds of outputs, or avoid other kinds of outputs. But this gets nowhere close to the kinds of reliability that we are used to in our computing systems. It is not a stable foundation on which to build higher-order systems of intrinsic or extrinsic motivation or ethical reasoning.

It can be easy to ignore this with the current generation of generative models. When they work well, they can do things that seem magical. The errors seem forgivable and are easily forgotten in comparison with the things that work. And if all we want to do is to generate some images and small quantities of text, that seems fine. Reinforcement learning seems to be capable of mitigating many kinds of error, and we are happy to simply discard outputs that don’t really serve our purposes. So what if ChatGPT occasionally gets the wrong idea about what you want, or Stable Diffusion sometimes ignores parts of the prompt? Just tweak the prompt and move on!

In the future, though, this might not be so easy. First of all, current models produce very simple outputs. GPT-4 produces only text, and StableDiffusion produces only images. Yes, there can be many kinds of text, including mathematical equations and software code, and many kinds of images too. But these are very narrow outputs, and relatively easy to inspect. It’s easy for a person to tell, often instinctively, if a generated image doesn’t seem quite right. Text is a bit trickier, and sometimes GPT-4 will hallucinate plausible-looking things that only seem wrong on closer inspection. But at least you can generally do the inspection and notice the error, at the cost of some time spent checking the details.

Future models will produce more complex outputs, such that it will be harder to find the errors. Text outputs will get longer, to the size of scientific papers, or even novels, or beyond. Generated software code will not be snippets or functions, but entire libraries and applications. Models will produce both text and images, and possibly other data too. Why not have the AI directly output spreadsheets, databases, Powerpoints, project plans, product roadmaps, legal contracts? These things are much more subtle, and errors are much less likely to be detected casually.

We could also imagine AIs that send output directly to other systems, such that they can send email, access web services, make payments, or interface with smart devices. Perhaps the AI can also receive input or responses from these sources. In these cases, the output is in the form of actions we cannot easily inspect.

Errors might also creep in due to inconsistency between different kinds of output: perhaps the AI generates a comic book, where the script and dialogue describe certain events but the visual art doesn’t fit correctly. Again, these issues might be subtle and hard to detect. This makes reinforcement learning with human feedback harder, because the humans have to work a lot harder to produce good feedback. Eventually the outputs become complex and subtle enough that we need special tools to examine them, which leaves us vulnerable both to bugs in these tools, and to the possibility that the AI learns to create outputs that satisfy the constraints encoded in the tools without actually being “good” from a human perspective.

This is a crucial difference between aligning an AI and aligning something like a storage device. Checking to see if the storage device is working properly is pretty easy: you write some bytes, and later check to see if you can read them again. If you don’t get the original bytes back, something is wrong. An AI is a bit like a storage device, except that we’re requesting things that we didn’t previously store, so when we get the output we have nothing to compare it to. Now we either have to verify that the outputs match our intentions, something that becomes practically impossible as the outputs become larger and more complex, or we have to trust the process by which those outputs were created. This brings us back to the fact that we already know that the process is unreliable, which is to say that it’s untrustworthy.

All of this is perhaps more annoying than catastrophic. Imagine that instead of creating a spreadsheet yourself, you ask an AI to create one for you. And sometimes it gets the numbers wrong, and maybe you don’t notice and send it to your investors or your accountants or the tax office, and maybe some inconvenience results. This is bad for you, but it’s not an existential risk for anyone. Probably you pay a fine or spend some time having some annoying conversations you’d rather not have, but the world will keep on turning.

Things do become a bit more concerning when we consider future likely evolution of AI use. Instead of acting solely in response to prompts, we will expect our AIs to engage in long-running activities or to act in response to environmental triggers - activity on the web, readings from sensors, timed actions, and so on. Instead of receiving text or speech or images from the user, and outputting a combination of text, images, and speech in response, AIs would effectively output plans. Given the intent to perform some action, the AI will come up with a sequence of actions to take. Or, rather, it will come up with a rationale for action, and then continually revise its plan as it acts and receives feedback. This process is necessarily unsupervised to some extent, because nobody wants to be an AI babysitter.

By this point, AI models will likely be much larger than they are now. They will have the same fantastical “imagination” that GPT-4 has, but bigger, better, and capable of much greater complexity. We will ask AI to do things for us precisely because it will think of strategies that we wouldn’t have thought of ourselves. The actions that it takes, and the rationale it uses to justify them, may well be too complex for us to evaluate. Again, this is partly what we want - a clever assistant who can do the Bond villain planning necessary to get ahead in the modern world, planning that we don’t have capacity for ourselves.

The potential for things to go awry is obvious: AI agents acting in ways that are contrary to human interests are a staple of film and literature, so this looks like a familiar problem. You might think “I know, I’ll implement an ethical system for my AI to keep it from misbehaving”. But now you have two problems! You must be able to specify your ethical system, and you must be able to engineer your AI in such a way that the ethical system is fully binding on every action that it takes, in all circumstances. That second problem is the alignment problem.

At the risk of repeating myself: this is not a matter of coming up with a good ethical system. It’s a matter of coming up with the mind that will obey it. This is, in truth, one of the most fascinating problems one could imagine. It may ultimately end up being a small matter of programming, but specifying what this program must do is fiendishly difficult. Nobody has a comprehensive answer to the problem.

When we design file systems, we build on decades of tradition, engineering experience, industry folklore, and scientific research. The SSD in your Macbook is a lineal descendent of punched card storage systems from the 1950s, tape from the 1960s, and the magnetic drives of the 1970s. Many of the principles of the file system go back to 70s-era UNIX, which itself has clear antecedents in the 50s and 60s. The core architecture of the computer is recognisably the same thing imagined by John von Neumann in the 1940s - faster, more complex, more optimised, but not categorically different. We have a lot of depth of knowledge to draw upon when we want to design reliable new file systems, as Apple did when designing APFS in the mid-2010s. Our confidence that our file systems won’t corrupt our data and lie to us about it comes in part from the sheer depth of experience required to get to this point.

AIs are not like this. We can’t start out by saying that our new AI is “like” some preceding thing, in the way that APFS is “like” HFS, or that SSDs are “like” magnetic hard drives, because there just are no preceding things. It’s for this reason that some people believe that the long-term future for AI is going to involve something like a simulation of the human brain, because at least that will allow us to import the inferential structure we have for reasoning about human behaviour and motivations. Taking some of our current machine learning techniques and scaling them up, or giving them new feedback loops, will probably create something very unlikely the human brain in important ways, so we will be flying blind in terms of our heuristics about how well it will stick to the ethical model we design for it.

Again, this is both scary and exhilirating. To be clear: the problem is unsolved! There is a tremendous reward, in both money and prestige, for anyone who can make decent progress in solving it! Current approaches have known and unsolved failure modes, and we don’t know for sure whether to evolve those approaches or try something radically new. More than usual, it might be possible for people coming from outside the discipline of computer science and its close relations to make a major contribution, if only by framing the problem in a new and more productive way.

In the meantime, the risks of giving unreliable systems lots of leverage over resources on our behalf seem large. A misfiring AI could easily end up triggering bizarre actions, and some of those could be very dangerous. The same capacity that allows an AI to envisage grand plans on our behalf could enable it to envisage complex sequences of actions that harm us. Again, this is not really a matter of bad intent, incentives, or ethics, but mostly of unreliable alignment. Whereas file systems are rigorously engineered to avoid data corruption, current AI systems are not engineered to avoid corruption of their intent or behavioural principles, because we don’t yet know how to do that. While any given AI taking a humanity-ending course of action is statistically unlikely, if we have enough of them acting frequently enough, then eventually something bad becomes likely to happen. We ought to be very confident in our alignment mechanisms before we allow such an experiment to occur.

The courses of action open to us are all difficult. Solving the alignment problem is hard. Preventing the existence of unaligned AI may end up requiring deeply unpleasant political choices, which might go against some of our most deeply-held convictions about liberty, decentralisation, and the freedom inherent in general-purpose computing. If we want to avoid those compromises, we must find technical solutions.

If you made it to the end of this, I would appreciate your feedback. I’ve read a lot of arguments for taking the alignment problem seriously that open by spelling out the most alarming possible consequences of failure, and I wanted to see how it would feel to explain the problem in more mundane terms first, only mentioning the x-risk scenarios briefly or in passing. If you found this approach more or less persuasive than other accounts you have read, perhaps you could let me know on Twitter or Bluesky.

Network states and eudaimonia

Tue, 04 Apr 2023 17:27:00 GMT

What is a good life? Philosophers have been asking this question for millenia. For Aristotle, the answer was eudaimonia, a concept of living well and doing well. Eudaimonia isn’t entirely about being successful, or about being happy, or about being ethical, but incorporates all of these things in relationship to each other. It’s a hard term to translate to English, but “flourishing” is perhaps my favourite.

Of course, “flourishing” is still a hard term to pin down. It’s hard because flourishing looks different for each person. Not only are our circumstances different, our dreams and desires are different too. Each person must decide what flourishing means for themselves.

If flourishing can be hard to define, the opposite of flourishing is easier: oppression, subjugation, marginalisation, poverty (material, cultural, or spiritual), suffering, neurosis, misery, confusion, regret. A person might be materially successful, but ethically compromised. They might be intellectually rigorous, but unhappy and afraid. They might be honest and smart, but excluded from opportunity in their society. They might be kind and loving, but subject to lies and propaganda. For flourishing to occur, each of these pitfalls must be avoided.

Flourishing also requires community. Many important things cannot be done alone, and even those who don’t share your perspectives on everything can be valuable friends, collaborators, and constructive critics. What’s important is that your community should support—or at least not actively obstruct—your flourishing as a person.

Network states are a proposed new model for how societies can come together and re-organise themselves into new forms, ultimately taking on full responsibility for self-government and regulation. Ultimately, they become states in their own right. If this sounds bold or even foolish to you, it’s worth noting that this kind of thing has been normal throughout human history. Many of today’s cities and states were founded by communities who had a pre-existing desire to live together, and to live in certain ways. They attracted others by their example, and grew by continuing to provide better lives than were available elsewhere. Through their myths and monuments, they passed down what they could about the dreams and aspirations of those founding citizens.

Other, more narrowly-constructed alternatives existed too. At the fringes of larger societies, international networks of monasteries, trading posts, academies, embassies and enclaves provided stability for communities to live in ways that diverged from the norms of the local majority. Sometimes, they expanded to become cities or micro-states in their own right, retaining legal privileges that allow for unique governance (other times… not so).

In fact, it’s our current age that is the abberation. We live in an age of average, where radically different experiments in life and governance are rare, and the world’s cities seem to be converging in norms and aesthetics. We can blame many things—the internet, financialisation, lack of housing construction—but these are large-scale social forces that cannot be easily reversed. Network states start with the assumption that the system that is producing these effects cannot easily be adjusted, and only by providing alternatives can we enable the missing experiments.

This is made all the more pressing because, well, mainstream modern life isn’t very eudaimonic. Life expectancy in the West has either stopped rising or begun to fall, not because we’ve forgotten how to do medicine, but because of deaths of despair. Anxiety and depression are rising—not rapidly, but any increase is a bad sign. When economic and social circumstances worsen, fewer people feel as though they are flourishing, or even like they are progressing or developing in their lives at all. More and more adults are living lives of quiet frustration, not able to get the home they want, the community they want, the job they want, or the family they want.

The traditional response to such frustration is: move. There’s no guarantee that the next place will be better, but it will be different. Migrants throughout history have done this, striking out in search of a place where they can progress in their lives. Maybe it’s a place where they’ll find people like them; a place where they can build or make things with like-minded others; or a place where their children will get a good education and a safe and stimulating environment to grow up in. In each case, it’s a search for a kind of flourishing, starting from a better relationship between themselves and their surroundings.

The only problem is that now there’s nowhere to go. Mid-tier cities have a sameness to them that makes them almost interchangeable, and if a place retains some uniquely vibrant community that draws people to it, it quickly becomes horribly expensive to move there.

It doesn’t have to be like that. The world isn’t actually short on land, even urban land with good transport connections. It’s short on places that combine good infrastructure, vibrant community, stable and effective governance, and a welcoming approach to new entrants.

For many of us, the internet is really strong on vibrant community, at least if you go looking for it. It’s not that hard to find your tribe on the internet. That’s one piece of the puzzle. Which just leaves: the place, the governance, and the ability for anyone who wants to to move there. This is, fundamentally, what network states are trying to address.

The questions of place, governance, and openness to migration are interlinked, and can’t be easily separated. Most places already have some form of governance, and that governance system already defines some rules about migration. A sufficiently large and well-resourced community can, in theory, acquire land. Not everywhere has a stable enough legal system for this to work, but plenty of places do. Self-governance is more complex, but many jurisdictions provide for some kinds of self-governance. Opting out of some aspects of nation state policy is hard, but examples like Próspera show that it can be done. Migration policy is hardest of all, and right now there’s no good example of a network state that is able to govern access to a territory.

Provided these problems can be solved—no mean feat!—then network states would be competing with each other on a eudaimonic basis. Which network state affords me and my family’s flourishing as people? And if there isn’t one, can I start one? Again, if these questions seem strange it’s only because our current age is highly unusual.

A world of competing network states is a world where the flourishing of each individual is of central importance. The very survival of the network state will depend on it; network states are unlikely to acquire territory containing oil deposits, diamond mines, or the best agricultural land, so their only route to economic success and social stability is to pursue the development of their primary resource, the people themselves. And, of course, those citizens are free to decide what that looks like. They can take inspiration from Oxford, Monaco, Bhutan, or Bali—what matters is that the citizens continue to buy in to the vision, and stick around to make it work. Network states which fail to provide their citizens with opportunities for flourishing will find it hard to keep them.

Taking eudaimonia seriously will require deep thought, engagement, and reflection. Flourishing human lives in real 3D meatspace have aspects that aren’t present in online communities. Our online presence is a reflection of our full self, but it’s the full self—messy and complicated—that has to exist in community with others, whose full selves will be present too. Those extra dimensions have to be integrated in a lengthy process of community-building.

While we can align around aesthetics online, ethical and epistemic alignment is harder, and relies much more on institutions than similarity of preferences. Much of community life revolves around these ethical and epistemic questions: what is right, what is good, what is true? It’s not like these questions often have definitive answers, so what we need is processes that we can trust and consent to, that find good enough answers most of the time. Such processes are hard to build and maintain. Perhaps you’ve noticed that these institutions are malfunctioning in your own society right now, but noticing the problem and designing a solution are not the same thing. It won’t be easy.

Still, there is much to learn from the new or reformed cities of the past. Looking back, we often notice how they relied as much on appeals to the human spirit as they did to the intellect. Solon didn’t just write a new constitution for Athens, but poetry to persuade people to adopt it. People don’t build cities simply because logic dictates it, but because the idea calls out to them as a noble vision of their own future. And cities don’t succeed just because we can prove that their governance structures have abstract theoretical qualities, but because people persist and endure in making them work. Designers of new cities must be architects of human experience, on all of its levels, as much as they must be architects of buildings or systems.

City-builders of the past drew on ritual and religion as much as on reason and rationality. This is not to say that success is a matter of fate, or of mystical or magical thinking. Quite the opposite: ritual and religion are psycho-technologies, tools for alignment and consensus-building and for reaching out into the unknown or across divides of language, identity, and origin. These are precisely what we must do when building and sustaining new communities. And, as technologies, they can be understood, practiced, and operated just like any other. With more skill comes greater success.

Institution-building requires not just rational belief, but a felt sense of hard-to-define things like justice, fairness, competence, and trust. Community-building requires friendliness, compassion, tolerance, and love. The feeling of connection is the heart of community, and it can’t be brought about through persuasion, but only through participation in something meaningful. To hold a network state together, psychotechnology is as important as governance, law, or infrastructure.

What seems clear to me, though, is that this dimension of the human spirit is what will decide the fate of network states. Eudaimonia must be the aim.

AI in the cozyweb

Tue, 28 Mar 2023 15:26:00 GMT

TLDR

Google search and power-law-follower-count social networks have driven a convergence of the global information hierarchy, a context collapse that gives an appearance of objectively-ordered information. In the search for authentic expression, people are defecting away from these globalised spaces to the “cozyweb”, a loose archipelago of private chats and forums, where context is restored and communication is more meaningful.

AI will change this in two ways: it will fill the public web with generated content, further diminishing the possibility for human connection in public online spaces; and it will accelerate the development of private online cultures by providing them with the ability to ask, answer, and discuss questions about which the AI has information, but within the idioms and norms of their culture.

What if the web has become an increasingly hostile environment for people to express themselves? What if, following that, honest and sincere discussion has migrated away from the public web, to loose networks of private chats and pseudonymous forums?

This is the thesis of the “cozy web”, originated at Ribbonfarm and explained here by Maggie Appleton. As she puts it, “[w]e create tiny underground burrows of Slack channels, WhatsApp groups, Discord chats, and Telegram streams that offer shelter and respite from the aggressively public nature of Facebook, Twitter, and every recruiter looking to connect on LinkedIn”.

The web is characterised as a “dark forest”—like a forest at night, prey creatures hide in the trees and burrows, while only predators remain awake, lurking in wait for something to move. On the web, we’re the prey, and we’re learning not to make too much noise. We can still share the inner details of our lives, but we will do so only in private chats, approval-required locked accounts, or on alt accounts which provide us with enough pseudonymity that we can maintain some separation between our public and private selves.

Social networks are no longer places to share pictures of what you had for breakfast, but sites of culture war that can, seemingly, drive political movements and determine elections; what you post can lead to you losing your job, or not being hired at all; many governments now ask for social media account details as part of the visa-granting process; every link you click is a chance for someone to log your activity and sell the right to market things to you. We’re being watched, whether we like it or not.

In a follow-up article, Maggie asks, in effect, what happens when the people are gone? When the retreat to the cozy web is complete, and the dark forest is all that is left of the public web?

In Maggie’s view, the gap left behind is going to be filled by generative AI. More and more of the content you see on public websites will not be the output of individual humans trying to create something that other people might want to read, view, or listen to. Instead, it will be—wholly or in part—fabricated by generative AIs. In the very short run, this will involve real people turning to ChatGPT to ghost-write their blog posts or even their tweets. Sooner or later we will move beyond this, with virtual AI personas producing content under their “own” names. Because it’s so cheap to produce, there can be lots of it. Because it’s a good enough facsimile of human communication, it will be hard to identify. Many of us will be fooled, or will not care if we’re being fooled, because we like what the AI is telling us.

There are, of course, up-sides to this. Some people will like the content that is produced, and there will, after all, be lots of it. Perhaps some otherwise unprofitable niches will be filled. But outside of pure entertainment, it does seem like we’re still going to care about who produces the things we consume, and perhaps even how or why they do so. We would feel quite differently about something produced by an AI than something produced by an identifiable human, or group of collaborators.

The rest of Maggie’s essay examines ways in which humans can respond. In particular, how can we pass the “reverse Turing test”, proving to strangers that we’re not computers, but real humans trying to communicate? She makes several suggestions, and although no single idea is likely to solve the problem, they are each applicable in some circumstances, and could be combined to be more effective. She discusses this further on Bryan Kam’s podcast, and Bryan dives into the philosophical implications with Isabela Granic in a later episode.

I find that I agree with Maggie’s big-picture analysis: conversation will continue to migrate from the public web to the cozy-web, at accelerating rates. However, I have a different take on the role that AI will play in this process. In my view, AI is going to be deeply embedded within the cozy-web.

We can very loosely define the public web as being “anything that might show up on a Google search”. “Can you Google it?” is a reasonable proxy for whether a thing has objective existence, and PageRank even gives us the appearance of an objective global ranking of the importance of different sources for a topic. Being “#1 on Google”, or at least on the first page of search results, has been a goal for SEO marketers for over two decades now. Those top spots are worth a lot of money. And, with only few exceptions, the results are the same for everyone.

In truth, Google does vary the order of search results, based on broad criteria like location and language preference. “Right to be forgotten” laws mean that some results appear in some places and not others. There is some individual-level personalisation, but not much. The reason is simple economics: it costs a lot of money to crawl the entire web and to compute the relative importance of each search result, and it would cost a lot more to personalise those results for each individual.

The near-objectivity of Google search has a kind of inherent value to it, as it reinforces a social consensus about which sources of information are important (which is not always the same thing as true). You can tell someone “I found it on Google by searching for [x]”, confident that if they do the same search then they will find the same result. This is useful even if the results aren’t individually optimal.

However, with objectivity comes a kind of impersonality. Google for you is the same as Google for me. Google answers medical queries in the same ways to consultant surgeons as it does to their patients. In “organizing the world’s information and making it useful”, Google also flattens it, discarding valuable context. Part of the appeal of the cozyweb is that what you do and say there isn’t flattened in this way; the denizens of the cozyweb are expected to have a shared context, and can talk to each other using short-hand phrases, memes, and private jargon. This allows for better communication for simple information-theoretic reasons: the more context two agents share, the more meaning they can communicate in the same number of bits. Communities differ not just in their aesthetics, but in their ethical and epistemic norms, and Google has no meaningful concept of these differences.

For this reason, it seems like AI is much better-suited to the cozyweb. To Google, the cozyweb doesn’t really make any sense. Leaving aside the fact that Google’s web crawler can’t even see inside your group chat, there’s something about the structure of private communities that is intrinsically difficult to fit into the model of a global information hierarchy. An AI, however, can be trained and fine-tuned on the specific patterns, idioms, and perspectives of a community. This means that the AI can both know more about individuals, or about community norms. The AI-for-doctors would answer medical queries in a different way than the AI-for-patients would.

This works because of the concept of foundation models. A foundation model is an AI model that is trained on a very large data set, with the aim of producing generally-useful capabilities. GPT-4 and StableDiffusion are foundation models, aiming to be generally good at a whole class of things. However, it’s possible to take these foundation models and fine-tune them by adjusting their weights. The LoRA algorithm has shown that once you have a foundation model, the cost of fine-tuning it is much lower than the initial training, and results in a model that is much better at producing certain kinds of text or images than the foundation model.

This lets AIs do the thing that Google search can’t easily do: adapt to the specific context of a community or an individual. By fine-tuning a model with specific information, the community can create the AI that is most useful to them. Two AIs starting from the same foundation model could end up giving very different answers to different people, and this divergence is a feature rather than a bug. Whereas the universality of Google drives convergence at the cost of context collapse, fine-tuneable AIs diverge in order to give the most appropriate outputs.

For this to work, it has to be possible to fine-tune the model, and right now this isn’t always possible. Open source models like Stable Diffusion are fine-tuneable, even on a home PC (with a beefy GPU, admittedly). Closed-source models like GPT-4 aren’t fine-tuneable by users, although it seems that Microsoft was given access to fine-tune the model for use in Bing Chat (with some… interesting results).

This gives us two ways forward: one in which the fine-tuning is done by a company like OpenAI, which requires a lot of personal or community data; the second, in which individuals and communities control their own data, and the fine-tuned models. The second requires open source models like Stable Diffusion.

If we were back in the 90s, hearts and minds brimming with optimism for the bold future of the internet, I imagine we wouldn’t worry too much about the difference between the two. But if the intervening quarter-century has taught us anything, it’s that ownership and control of data matters. “An AI in every chat room” becomes a very different proposition if that AI is either open or closed-source, with the fine-tuning data owned and controlled the same vendor that created the foundation model or by the provider of the community’s choice.

The other big difference from the 90s is that the commercial and business implications are much easier to grasp now. In a way, the prototypical cozyweb is the corporate intranet; corporations are definitely going to want AIs that are specific to their culture and operations. If you want to build a business around fine-tuning foundation models for specific communities, your first customers will be enterprises who care about keeping their corporate DNA private.

For individuals and communities, perhaps this is the moment when decentralised technology becomes feasible. If your community has at least a few gamers in it, then you probably have the collective GPU power to fine-tune your models—and within a few years it might be possible to do this on smartphones.

Models are cultures

Mon, 20 Mar 2023 12:15:00 GMT

When we train a machine learning model, on a corpus of human-produced media, fine-tune it on specific examples, and give it feedback on its interpretation of these things, we are doing something similar to the production of a culture.

Historical human cultures were formed by a mixture of exchange, innovation and divergence. They gave rise to different languages, philosophies, artistic styles, economic systems, and forms of social organisation. Each new generation learns about the world from a mixture of personal experience and the stories and explanations given to them by the culture they grow up in.

Pre-literate cultures had very narrow bandwidth for communicating their ideas between generations, and used myth and storytelling to encode the information that they valued most highly. These stories were densely interwoven mixtures of factual history, conceptual world-view, and ethical guidance, maintained and evolved by poets, priests and prophets. Everything else had to be learned and passed on by practice and observation.

With the rise of literacy, it became possible to pass on some kinds of knowledge simply by writing it down. Even here, we rely on narrative, artful composition, and a good turn of phrase in order to make the information seem relevant to the people who must read it. As with myths, we mix the plain facts with the normative meanings that we give to these things. Literature doesn’t just tell us facts of history or science, but how people like us feel about things, interact with each other, and decide how to live and act. It’s still the case that literature leaves large gaps, where tacit knowledge is required for real understanding.

Large Language Models are, in a sense, hyper-literate: the only things they know are the things they learned from written sources (or, in the case of image or multi-modal models, from visual images). Compared to humans, they lack direct knowledge of “what it feels like” to do something, the tacit knowledge that can only come from personal participation.

But compared to regular software applications, LLMs have a vastly greater general knowledge of human concepts. A typical software application is conceptually narrow and abstract—it only knows about things like files, windows, fonts and so on. Perhaps a complex application like Photoshop knows about brushes and fills and layers, but this knowledge is painstakingly encoded by programmers, and is strictly limited to the narrow tasks that Photoshop is used for.

ML models are something different. The process of training produces a system that can manipulate symbols representing a broad range of concepts in a reasonably accurate way. Accuracy is sometimes about getting the right answer, such as the response to “what is 25 multiplied by 82”, but more often it’s about giving the appropriate response to a prompt. Appropriateness often isn’t objective, but depends on our culture. If I ask ChatGPT for advice on impressing my new work colleagues, that advice might be good if I’m in France, but bad if I’m in Japan, or Paraguay, or Tanzania.

Software developers aren’t used to thinking about things in this way—we like to think of our software as a universal model of the problem domain, with a thin layer of localisation on top, to use the appropriate language, date format, or currency symbols. In truth, almost all software is written in pseudo-English, and even if an individual programmer uses pseudo-Swahili or pseudo-Japanese to name their variables, they are still working with libraries, APIs, and operating systems whose core concepts are canonically named and described in English. This is only rarely commented on, but it’s a strange quirk of history that could easily be otherwise.

ChatGPT is fluent in multiple languages. Does that mean that it’s also fluent in multiple cultures? Jill Walker Rettberg looked at this question and concluded that although the model is multi-lingual—it can understand questions and give answers in a wide range of languages—it is monocultural, and that culture is Anglo-American. This is partly because the training data is predominantly American, partly because the “cleaning” and curation of the data set is more likely to filter out non-mainstream-American material, and because the “value-alignment” process was conducted by US-based contractors who instilled American value judgements. Her main conclusions:

I was surprised at how good ChatGPT is at answering questions in Norwegian. Its multi-lingual capabilty is potentially very misleading, because it is trained on English-language texts, with the cultural biases and values embedded in them, and then aligned with the values of a fairly small group of US-based contractors.

This means:

ChatGPT doesn’t know much about Norwegian culture. Or rather, whatever it knows about Norwegian culture is presumably mostly learned from English language sources. It translates that into Norwegian on the fly.

ChatGPT is explicitly aligned with US values and laws. In many cases these are close to Norwegian and European values, but presumably this will not always be the case.

ChapGPT frequently uses US genres and templates to answer questions, like the three paragraph essay or standard self-help strategies.

By weighting the training data toward different cultures, it would be possible to shift the cultural bias. Because training data is scarce, this is unlikely to be achieved by creating non-English data sets that are larger than the English ones. However, non-English sources of high quality could be given higher weighting, and this could be reinforced by culture-specific value alignment. Foundation models will probably still rely on English-language and Anglo-culture data sets for scale, but fine-tuning and reinforcement learning can teach them to use different cultural concepts.

Aligning generative models to national or regional cultures may come to seem like a natural state-building project. In the same way that nation states in the early 20th century wanted to have their own state broadcasters, in order to promote and enrich their national cultures, they may want their own LLMs for broadly similar reasons. The web mostly didn’t work like this: much of the web’s content is available only in English, many interactive web applications are also English-only. Even where translations are available, the conceptual scheme is Anglo-American, or even more specifically Californian in many cases. But LLMs are much more polymorphic: they can change shape to match cultural norms far more easily than rigidly-programmed traditional software can.

This might sound fantastical at first, but this analogy is a pretty good way of grasping what the UK government is trying to do:

The government acknowledged the recent breakthroughs in large language models, the technology behind chatbots such as OpenAI’s chatGPT—a sensation since its launch last year—and Google’s Bard, which has yet to be released to the public. It said it would establish a taskforce “to advance UK sovereign capability in foundation models, including large language models.”.

Nation states without their own culturally-aligned models will find that their citizens are depending on the models of others, in much the same way that nations without strong native cultural institutions find that their citizens are increasingly influenced by other cultures. Of course, this is often no bad thing—cultural mixing is part of how culture evolves in the first place. But cultures which cease to produce their own works can gradually slide into obsolescence, a threat which seems to demand a response.

We could also imagine subcultural models, which reflect the particular virtues and values of a community of practice, a religious or spiritual community, or an artistic or aesthetic movement. This tweet thread contains a thought-provoking example:

In this scenario, a Vatican-led LLM that is deeply aligned with Catholic teachings, particularly scholastic and neo-Aristotelian thought, could offer certain advantages over conventional postmodern morality, which is often adopted by corporations such as OpenAI and Google.

Mikael Brockman (@meekaale), 10:54 AM · Mar 19, 2023

At an even more general level, such models might give real depth to the idea of “corporate culture”. A model trained on the particular values of, say, Bridgewater, might behave very differently to a model trained on the values of some other firm. Whether we want this kind of thing to happen may be an empirical question. Perhaps we prefer our value-neutral computing platforms—I can think of many reasons why we might. But, so long as models are pluralistic and user-controlled—far from a reliable assumption!—then we should be happy with the prospect of computers that are able to express themselves in terms that we find more meaningful, and that are able to grasp the meaning of our requests of them more easily.

The economics of due diligence

Tue, 21 Feb 2023 07:37:00 GMT

The snow is falling hard in the second crypto winter. Without the hype and optimism of a bull market, many tokens have turned out to be near-worthless, and many formerly valuable projects have turned out to be scams. Rug-pulls, hacks, opaque “foundations” managing funds on behalf of “the community”, even some honest-to-goodness Ponzi schemes.

The correct attitude may well be one of total cynicism: of course a market predicated on skirting regulations turned out to have more than its share of fraudsters, cowboys, and incompetents. But cynicism is boring, and we want to learn something. Having been around two crypto booms and their subsequent crashes, I think there are some lessons we can draw. In particular, we can say something about how due diligence works, and how we might be able to use technology to make it work better.

In a nutshell, the problem is with the economics of due diligence. It costs time and money to do research, and it’s only worth doing that if you can get some special advantage from doing so. In regulated private markets, this advantage comes from the fact that regulations prevent mass participation: a restricted club of VC funds and angel investors can invest in due diligence, knowing that they will capture all of the upside from their investments. In regulated public markets, the companies that issue the stock are required, in effect, to publish research on themselves, lowering the risk for individuals to participate in the market.

Crypto markets have some features of both: the public is allowed to participate in the market, but without the requirement for the token-issuers to publish the kind of information that would be required in a public stock market. Since most investors cannot afford to do the kinds of due diligence required in an information-asymmetric private market, they end up taking risks with much greater uncertainty than is typical in either public or private regulated markets.

If you can buy shares in a company on a public stock market, we say that the company is “publicly-traded”. In order for the company to sell shares to the public, it must comply with a broad set of rules:

it must publish financial information at regular intervals
it must have certain governance arrangements in place
it must agree to tell the shareholders if certain things happen which might affect the price of the shares
it must also agree to share information with the markets before telling anyone else, so that well-connected insiders can’t make money by buying or selling on the basis of secret information

These rules aren’t always perfectly enforced, but there are plenty of cases of firms and individuals being sued for breaking them, and some go to jail on criminal charges too.

The burden of compliance is significant. Publicly-traded firms have to spend more on legal, compliance, and financial personnel, and have to publish information that they may prefer to keep private. Mistakes can be costly. But, for large firms, the price is often worth paying. Access to the stock market means cheaper financing for the company. It also means a chance to sell valuable shares for the founders, employees and investors. Publicly-traded firms are also typically large enough to bear the cost of compliance.

A privately-held company is one whose shares can’t be sold to the general public (outside of certain exemptions for crowd-funding). Because of this, they do not incur the costs of complying with the rules that bind publicly-traded firms. This is good, because small firms can’t afford those costs! Even fast-growing startups would generally prefer to direct resources toward more growth.

Even so, privately-held firms do require investment, and they can get it from venture capital funds, angel investors, private equity, and so on. They just can’t get it from the general public.

Due diligence is “performing a reasonable amount of investigation into something before buying it”. This means that before invest in something, you should take a close enough look at it to persuade yourself that the risk is acceptable.

A typical VC investment, and even some angel investments, can involve multiple rounds of pitching and negotiation, and often some kind of background check on the founders via informal networks. Early-stage startups don’t have lots of metrics to share—perhaps there’s a growth figure, but revenue is a lagging indicator and churn will not have become apparent yet. Investors will want to know what the founders are like as people, since that’s often the best guide to where the company is going.

This kind of personal due diligence is expensive for both the investors and the founders. For this reason, VC investors put a high value on personal recommendations, and often employ analysts to sift through the deals that don’t come with a strong referral. Angel investors team up into syndicates, so that if one member has done due diligence on a startup, the others will not need to repeat it. The whole process is designed to manage the high cost of due diligence.

Private equity investments involve even more due diligence, because the deal size is larger and the firm being invested in has much more data about its performance. Since that data is not public, it will have to be gathered together and examined by the investor’s own team. This is a very intense process involving many lawyers, accountants, and other specialists. It is common for deals to collapse during the due diligence process, when the buyer discovers information that lowers their estimate of the firm’s value.

It’s worth noting that participation in private markets is restricted by law: VC or PE funds are regulated, and even private individuals must meet some “accredited investor” criteria in most countries.

If you’re investing in a crypto token, you probably want to be doing VC-level due diligence. Who are the people behind the token? What is their track record? Do they even exist? Are their claims about their technology correct, or even plausible? If they have on-chain metrics showing growing adoption of their token, how do we know that these metrics are not being manipulated?

If you’re a personal friend of the founders, you can probably get answers to these questions. If you know someone who is a personal friend of the founders, you probably get the answers too. But if you don’t have that kind of access, you simply don’t know. In order to find out, you would have to spend time and money on due diligence. Sometimes this due diligence will tell you not to invest, so you will have to do due diligence on tokens for which you make no money.

This is simply too expensive for the individual investor. It’s a full-time job in itself. And so people look for signals that they hope will correlate to underlying qualities of the token: how much buzz does it have, are other people buying it, are there big-name investors on board? Hopefully the big-name investors have done due diligence on it, and you can just free-ride on their work. This makes little sense, since successful investors are not the kinds of people who let other people capture the upside from their work, but there are few other options.

The end result is that public investors are taking far more risk than they can understand. Rug-pulls are very difficult to execute on people who have done due diligence, but become possible when none of your investors can afford to ask the kinds of questions that would reveal the risk. Technically absurd projects would not get anywhere if the investors were asking serious questions or consulting with specialists, but when the investors can’t afford to do that, such a project might gain enough momentum to appear credible.

What can be done about this? One option is to become more like the private markets: restrict early-stage investment to people who are part of a select club of people who accept the need for due diligence and are willing to pay for it, either alone or in syndicates, and reward them with some degree of exclusive access to invest for a period of time. Another option is to become like public markets: require token-issuers to publish credible data about themselves, with serious mechanisms for punishing false or misleading statements.

Perhaps there are other options. Crypto “ratings agencies” which are designed specifically to target the worst cases of fraud and incompetence could improve the overall quality of the market. But either the token issues pay for this themselves, which creates some tricky incentive problems for the rating agency, or the investors pay, which creates a free-rider problem—why should I pay for a token to be rated if other people who haven’t paid will be able to invest in it?

I don’t see any easy solution. The most likely outcome is gradual assimilation of crypto into the regulated financial sphere, with a shift toward existing norms of securities offerings. This will mitigate many of the frauds, but will also throw up barriers to participation.

LLMs are hustlers

Fri, 17 Feb 2023 14:52:00 GMT

All of the software you use is designed for the purpose of turning very specific kinds of data into other very specific kinds of data. Press a key, and a certain numerical code is reported up through your BIOS, operating system, and application code, triggering reactions along the way—such as the display of a character on the screen, or the selection of a menu item, or the skipping of a music track. “Character on the screen” sounds simple, but the representation of a character in a desktop application is astonishingly complex:

the character probably lives in a larger data structure representing a text document or database field
the visible character on the screen probably sits inside a specific user interface element such as a text box
that UI element might has specific settings for things like font size and colour
turning those settings into graphics on screen will involve the use of an operating system graphics toolkit
the graphics toolkit will need to talk to the video card, via a low-level graphics library
the low-level graphics library will need to talk to a video card driver
the video card driver tells the video card which pixels to set to which colours

I’ve simplified and skipped multiple levels and sub-systems here, and it’s already complicated.

To make this work, software engineers need to maintain a certain level of rigour. Code must work with precise data structures, just so, or errors will likely be introduced. Each layer of the application, operating system, and driver code must talk to each other layer in very precise terms, lest the whole system collapse.

As a result, most parts of the system are very selective in what they will talk to, and what they will say. The same goes for networked applications, where online services hide behide narrow APIs that permit only certain others to interact with the system in certain ways. In software engineering, one of the names for this is encapsulation, the idea that a (sub-)system should be a self-contained unit that can remain wholly reliable provided you communicate with it in the specific ways that it requires.

In practice, this is hard. Each module has its own little world model in which certain data structures have certain meanings that they might not have elsewhere. Software engineers are obliged to adopt the mindset of the philosopher, in which things mean what they are defined as meaning, not what they might mean in common usage. Since there are many interacting pieces of software in even trivial applications, this involves being mindful of many different conceptual models. Sometimes, we get confused.

In RFC760, which defines the Internet Protocol, a foundational document in computer-computer communication, Jon Postel wrote:

In general, an implementation should be conservative in its sending behavior, and liberal in its receiving behavior. That is, it should be careful to send well-formed datagrams, but should accept any datagram that it can interpret (e.g., not object to technical errors where the meaning is still clear).

As “be liberal in what you accept, and conservative in what you send”, this principle has come to be known as “Postel’s law”. It means that software engineers should try to code their systems to accept some degree of error from the systems they must communicate with, while striving to avoid introducing errors of their own.

This is a fine aspiration in some ways, but it has a cost. If your system is too liberal in what it accepts, then it ceases to have meaningful standards at all; if other systems begin to rely on the acceptance of malformed or confusing data, then the receiving system must continue to accept these forever, or break the systems relying on it. This significantly complicates maintenance, because now the system has both a formal standard for how it behaves, and an informal set of behaviours that it will exhibit when receiving the “wrong” data. In practice, being liberal with what you accept is not feasible beyond trivial cases.

As a result, our software is stuck with the need to communicate in very specific and abstract ways. The “value chain” of a software application starts with some data (from the user, or from the network), and proceeds through a series of abstract transformations that require precise alignment in order to work. The engineering effort required is high, and so most applications have only a limited capacity to talk to other applications, even when they both work with similar kinds of data.

This is why most schemes that involve having lots of computers talk to each other fail. It is really really hard for computers to agree on semantics and protocols, and to tolerate error without creating the side-effect of making the error itself part of the protocol.

Until now.

This tweet from Gordon Brander got me thinking:

Focusing on structured data when GPT-3 exists feels like solving yesterday’s problem.

Think sloppy.

GPT-3 means that we don’t need to worry so much about how we structure our data, because GPT-3 is really good at turning one kind of data into another. Say Alice has some data, and Bob has an API for doing something useful with data, but Alice’s data doesn’t match Bob’s API. The old way of solving that problem would be to think like a philosopher, teasing out the semantics of Alice’s data and the paradigmatic model of Bob’s API, then constructing a transformation between the two. Now, we can just ask GPT-3 to do it for us.

Here’s an example of GPT-3 turning plain text into JSON data. Here’s GPT-3 generating SQL queries. Here’s ChatGPT generating code to interact with an API. These are the building blocks of an interaction between two systems, and—with the right prompting—an LLM can generate them, on demand, in seconds. This is a profoundly different kind of problem-solving.

LLMs are not philosophers. They don’t care really care about semantics or abstract data models. They know how to get some things done, and you can ask them to do it. LLMs are hustlers. If you want to get some data from A to B, the hustler will get it done faster than the philosopher, and most of the time with no loss in quality.

The immediate implication is that software integration projects will get a lot easier. The translation layers, the glue code that lets us stick two systems together, can be generated by LLMs. Individual components can afford to break Postel’s law and become more conservative in what they accept, more rigorous about standards, because the cognitive load of handling the necessary data transformations can be outsourced to the LLM rather than the programmer. Since LLMs can learn, APIs can change much faster than they can now, when we’re reliant on humans to a) understand the change and its implications, and b) update all of the relevant code. We will need to build a lot of safety harnesses before we can feel comfortable with this, but the payoff is so high that we will be willing to take some risks here too.

We can think of inter-application communication as a kind of trade, and the capacity for specific transactions to occur as being a matching problem. Matching theory describes “the formation of mutually beneficial relationships over time”, typically concerning economic relations between people, but the principle is applicable to relations between computer systems too. Matching is not just about an employee’s skills and and employer’s salary offer, but whether they can find each other, the influence of geography, language, social networks, and so on. If there is a job available, and a person willing and able to do it, but that person remains unemployed and the vacancy unfilled, then this is attributed to “friction” in the job market—something must have got in the way of the person taking the job.

Communication between computers has a lot of friction, despite all of our best efforts. Communication between different parts of the same computer, or even within the same application, often have friction! We might have some data over here, and an API over there, and with just a bit of hustle, we could get the two to talk to each other. But philosophers don’t hustle, and so our current applications can’t do it.

In the analogy with a market, the LLM is the entrepeneur matching a source of supply with a source of demand: some system needs a certain kind of computation doing, and some other system is capable of doing it, given the right data as an input. An entrepreneurial LLM transforms the data, and writes the code to invoke the API of the system that can do the computation. API marketplaces currently exist, but are poorly-utilised because the marginal cost of interacting with a new API is high. Even when what we want is reasonable, the persnickety details of data formats and API schemas and network protocols get in the way. With the right framework, LLMs can help us to realise the aim of networks of collaborating software applications that we have spent the last few decades imagining.