Auditing AI: A Great Book That Holds AI Accountable—And Shows How We Can Too

The Marquand House Collective, Auditing AI. MIT Press Essential Knowledge Series, 2026.

Auditing AI is the book I’ve been searching for–and I believe it will be useful for any and every reader who wants to move past hype and hysteria to really understand what tools we need to hold various AI systems accountable for what it does and how it does it. There are some 40 books currently on my AI bookshelf and dozens of others of books and articles on my various digital devices, but this tiny book (it literally fits in my jacket pocket) stands head-and-shoulders over most of what I’ve read in terms of giving the reader a real sense of what we can do to gain agency and control over a technology that seems to be everywhere doing everything all at once. I am (of course!) writing my own book on AI and my frustration in reading in depth in this field is that far too many authors take the “black box” of AI at face value. Few writers really enter that black box. Even fewer take the reader long with them, to really show and explain, analyze and assess, what’s actually going on in there and how we–as individuals, communities, educators, policy makers, and governing bodies–can make sure AI is really doing what it says it is doing and doing it equitably and with an objective of ensuring public good not undermining it. Auditing AI is just about the most practical, useful, usable book I’ve seen for defining what the kind of auditing we need to gain control of AI and how we can go about performing such an audit (on the largest scale but even in the realms over which we might have exert some influence and control).

This book’s methodology is the opposite of hyperbolic. If anything it is plodding but in the best possible way–slow and deliberate, careful and never glib. And yet its examples are so vivid and clear that I was able to read it (literally!) while getting my hair done at a busy salon and, on another day, on a noisy and bumpy airplane right. Its calm clarity is why I am so excited by it. Sam Altman, AI’s greatest showman, loves to mystify everything about AI. “We do not know how far beyond human-level intelligence we can go, but we are about to find out,” he says, as if that is a responsible pronouncement for the head of a company designing the most powerful new computational technology of this generation. Or, even more outrageous: “AI will probably, most likely, lead to the end of the world but in the meantime, there’ll be great companies.” This abdication of agency—any ability to control, regulate, or even train AI wisely, even by its own creators—is not only shocking. It’s spell binding—and, I would argue, deliberately so. The purpose of these extravagant superhuman claims for AI is to sell a product at epic scale and without any safety regulations or guard rails to protect its users. “We’re all going to die!”—the cheesy refrain of disaster movies—is the sales pitch of AI. The outlandish, outsized, uncontrolled claims seem to put even smart, well-intentioned people (that would be you, Higher Education administrators!) into a passive stupor of submission and acceptance, as if it is already too late to do a thing to prevent “the end of the world,” so why bother? Auditing AI insists that we bother… and offers us tools and method to do so effectively.

Of course there are as many pundits shouting that GenAI and Large Language Models are terrible as there are fans cheering it this “AI revolution” on. But neither hype nor hysteria offer us, everyday users, or policy makers, the tools, tactics, or even solid, grounded insights to make sensible decisions about adopting AI in our lives, our workplace, our schools, our government, our finances, our military, or even in the most intimate aspects of our private lives. Hype and hysteria do not help us. So much of the journalism, social media, and even scholarship about AI feels like we are on a hair-raising technological roller coaster ride with no “off” button and no exit ramp.

Auditing AI refuses that powerlessness and, beyond, the whole paradigm, including that of hype and hysteria, the binary position of either cheerleader or resister.  This tiny, patient book—190 pages and with a trim size of 5 by 7 inches–slows down the process of evaluation, allowing the reader to understand exactly (and I mean exactly) how one can audit technologies, how it has been done in the past, and how, step by careful step, it can, should, and must be done now to ensure that the public can be confident that the AI systems currently “powering—and changing—countless aspects of our lives” (p. 3) are doing so accurately, without built-in bias, and to serve, not subjugate, those who use it.

Written by The Marquand House Collective, eleven policy makers, lawyers, computer scientists, and scholars, Auditing AI has a practical purpose fueled by deep expertise as professional auditors:  “Experienced AI auditors, like the authors of this book, are excited by the potential for audits to help leaders and the public know when to use (or not use) AI systems, improve public trust in AI, and guide improvements over time” (p. 3). Yes! That is exactly what we need now.

Two Examples of How Auditing Can Work To Eliminate Unfairness in Automated Systems

Instead of polemics, this book guides the reader through past forms of audits to move from an abstract sense of what an “audit” might be like to actual cases where audits were deployed and actually changed how systems were run. Audits can ensure more equity. They can eliminate (or at least minimize) bias in some situations or, in others, ensure fair competition for consumers and companies alike. The book begins by looking at successful audits in the past. For example, it looks at the role of audits in several years in the development of and debates around LYFT (Life Years from Transplant), a system that automated the process for prioritizing who would be eligible first for kidney transplants. This vivid and literal “life and death” system was, at first, based on a set of statistical, actuarial predictors. But, the auditors demanded, what predictors–and how were those determined. As the authors underscore, that precise, probing level of questioning is exactly what is missing in today’s AI conversations. The careful auditing of the LYFT system identified “disparities across patient populations, measuring differences based on age, gender, geography, and race.” The results of the audit forced a conversation about nothing less than who deserves to be given a chance to live longer. It brought greater equity to this system by probing assumptions and social conditions, asking the questions of “why” and “how” behind and beyond the actual systems.

As we think about the results produced by the different contemporary AI systems now, we again need to think about the ways results can embed, mirror, and automate discrimination and bias. The system, in other words, is by not value free. A sound audit is necessary to uncover the untenable assumptions buried within seemingly neutral numbers.

To look at commercial and financial implications of automated systems, the authors also provide the example of the SABRE (Semi-Automated Business Research Environment) computerized airline reservation system developed by American Airlines that changed how seats were booked after the 1970s. Previously, travel agents spent hours making numerous phone calls to book every passenger’s seat, inserting push bins representing each passenger into a “seat board.” SABRE automated the process. However, when other airlines complained that this computerized system seemed to exclude their passengers, then-president of American Airlines Robert L. Crandall testified in 1983 before the U. S. Civil Aeronautics Board saying, “The preferential display of our flights, and the corresponding increase in our market share, is the competitive raison d’etre for having created the system in the first place.” Why invest so much in building an automated system if it didn’t give you first dibs in that system? This argument, known as “Crandall’s Complaint,” is the testcase for prejudicial systems that do not announce their own preferential advantage. It is crucial to remember Crandall’s lament (basically: if I paid for it, I can rig the system so I extract the most profit from it) when thinking about any information, at any level, that we gain when we use Claude, Gemini, ChatGPT, or Llama. These systems are not designed for our benefit but for those who invested in and developed them. Period. We all need to be thinking about what kinds of affordances are built into particular individual systems that, by design, will give Anthropic, Google, Open AI, and Meta greater profit while pretending to be a fair and level playing field.

Step-by-careful-step, Auditing AI walks us through these issues so we can see deeply and clearly what is at stake in automated systems and how an audit can make a difference, holding a system and its developers to account for prejudices of every kind within that system. These examples show us how an audit can be used effectively and powerfully and have been deployed in the past to hold automated systems accountable. Continually, the authors draw the through-line from other forms of automated decision-making technologies to today’s AI. Rather than a polemic about how “good” or “bad” AI, how it will save humanity or end it, how it is superhuman and even its developers don’t understand how it does what it does, these authors insist that a careful audit (as with the LYFT and SABRE systems) will indeed reveal problems, flaws, inequities, preferences, and other crucial issues and give us a powerful tool for exposing and correcting problems.

After carefully helping us to understand how audits work, in Chapters Three to Six in Auditing AI, the authors walk us, with the same deliberate care, through “The Steps of an AI Audit,” “Interpreting Audit Results,” “After the Audit,” and “A Healthy Audit Ecosystem.” Better than any other single book I know, Auditing AI gives us a powerful tool that, most important (and contra Sam Altman) gives us agency again. The book shows how audits work, how they can be used, how they have been used wisely and effectively in past situations, and how they can be used again, now.

A Model of Clear, Persuasive, Practical Writing

On a recent panel at City University of New York, the legendary Random House editor Chris Jackson said the key to writing a great book (whether fiction or nonfiction) is treating your reader like a constant companion on an unfolding journey. Your job as a writer is to revise by constantly circling back and back to your text, revising over and over to make sure your reader is still there, running along side you. It is your job, as a great writer, to make sure the reader is still there. Auditing AI does this as well as any other book on contemporary forms of AI that I know.  To be quite explicit: I love this book. Weirdly, through arrogance or ignorance, far too many AI books skip over the nitty-gritty, the hard parts, jumping to the big polemical conclusion without giving us what we need to be able to leave the book confident in our ability to make our own judgments when we need to.

Interestingly, I find this tendency to gloss rather than illuminate both in books that unabashedly embrace AI as a glorious world-changing new technology as well as in books that unabashedly condemn AI as a horrifically irresponsible new technology that could well change our world in dystopian ways. I find far too much vagueness, including loose terminology when precision is required, or abstractions without clear examples, or simplistic praise that dismisses any critique of AI as “Luddite” or equally simplistic condemnation that refuses to admit any value or use in AI at all. To be a bit crude here, I often find an “imitative fallacy” in AI books: sometimes it feels like books about “AI slop” read a bit like AI slop. They often lack both nuance and clarity, homogenizing or linguistically smoothing over exactly those basics that allow the reader to be, as Chris Jackson insists, a companion on a journey towards real understanding.

The authors in Marquand House Collective bring us along as well as any book on AI that I’ve read this year.  They use the perfect stories to illustrate the point they are making, then stop and explicitly state the point they were trying to make with the illustration, and then, again patiently, tell us how what we have just learned sets the stage for what we will be reading next. They repeat that process over and over again, always pausing to define terms that are bandied about everywhere, as if we all know what they mean.  This is crucial. As the brilliant Stanford computer scientist Yejin Choi (a developer of “Small Language Models”) insists, “there are more categories than words.”  By this she means that even seemingly known words—take “algorithm,” for example, or, for that matter, “audit”—can often take on different nuances in everyday common language use. By grounding such words in real life examples, by insisting on how they are using a given word in a given context, the Marquand House Collective offers the opposite of the Sam Altman style AI showmanship and all the razzle-dazzle “superhuman” rhetoric.  The authors exemplify the Aristotelian value of rhetoric as a form of audience-centered communication. In the old (and still useful) Dale Carnegie adage for effective public speaking: “Tell them what you’re going to tell them, tell them and tell them what you’ve told them.”   

Now, It Is Our Turn

Auditing AI is a book that brings us along, tells us what we need to know, makes sure we understand each different part of the process, and then (only then) ends polemically by insisting on the crucial, life-and-death importance of using these sound auditing methods to hold AI accountable now.  We do not have to wait (contra Altman) for the world to end before we make sure AI “companies” do responsibly and equitably what they should be doing, what we want and need them to do. “Audits allow us to make informed choices about the kind of world in which we want to live, and they move important questions about our lives an collective values into public dialogue” (p. 160).  Auditing AI ends by invoking the Jewish German philosopher Karl Popper who, writing in exile from Turkey during the Second World War, argued that “people in democracies need public information to work together for the common good.”  Careful auditing is the best tool we have for ensuring that AI fulfills that promise. We need not relinquish our responsibility and control to trillionaire oligarchs who say AI is beyond our control. It is not. As the authors insist: “The next step is up to all of us.”