The Question in the Empty Room
On the stage they ask whether the machine will wake up and take everything. When the room empties, the question changes, and the changed question is the true one: what our own people have already done
There is a question I have been asked more than any other about these systems, and it comes with one condition: it waits until the important people have left.
On the stage, or in the all-hands, or in the board deck with the tasteful gradient, the posture is confident and the fear is enormous and safely far away: will the machine become something more than us, will it take the jobs, will it, in the version that has learned to enjoy its own shiver, wake up. These are is-questions, questions about what the thing might become, and they carry a quality I have learned to recognize: they frighten the person asking at no cost to them, because the danger is vast, external, and parked comfortably in the future.
Then the room empties, and someone stays behind, or catches me at the door, or waits until the others are in the lift, and asks the other question. The real one. It is never about what the machine might become. It is always some version of what our own people have already done with it, last quarter, without telling anyone.
I will not give you the room or the face, partly because I mask such things by long habit and partly because the particular does not matter: the question is the same in a bank in one region and a ministry in another and a company young enough that it has not yet learned to be afraid of anything. Does anyone actually know what our people have been pasting into these tools. Is the contract we spent a month arguing over now sitting inside someone else’s training set. Who approved the assistant that reads the whole mailbox, and could we reconstruct what it has already seen. The voice asking is quieter than the one that ran the meeting, and quieter for a reason: the public fear is a performance a person can afford, and this one is not, because this one has a date on it, and the date is in the past.
This is the thing I have learned to watch for, and I would hand it to you from three decades of being the person they ask once the door is shut: the loudness of a fear runs inversely to its nearness. The catastrophes we stage and publish, the machine that awakens, the general intelligence that decides it no longer needs us, are loud precisely because they are far, and being far they ask nothing of us today beyond the pleasant shudder and the thoughtful panel. The danger that is actually here arrives in the opposite register, quiet and specific and a little embarrassing: a junior pasted the merger terms into a chatbot to summarize them; a team wired a model into the customer database because it saved an afternoon; a vendor’s clever assistant has been reading, and keeping, everything. No one convenes a panel about it. It has already happened. It is happening while the panel is on.
I do not think the people asking the small question are foolish, and I do not think the ones performing the large one are cynical: the pull toward the enormous distant fear is human, the same pull that makes the disaster film easier to sit through than the quarterly review. It is more bearable to be frightened of a god than responsible for an intern. The large fear flatters us even in its terror, whispering that we are building something world-ending, something out of scripture. The small one only says we were careless, on a Tuesday, and did not write it down.
So when I am asked what a board should be afraid of, I have stopped answering as posed, because as posed the question is the decoy. What there is to fear sits in the logs, if you keep them, and in the absence of logs, if you do not. The useful question is never “when will it become something,” which no one can answer and everyone enjoys asking. It is “what have our own people already handed it, and could we find out if we had to.” A company that can answer that is governing. A company staging the other conversation, the large and thrilling one, is doing something else, and calling it the same name.
The room empties the same way every time. The confident version of the person leaves with the others, and the one that stays is smaller, and more honest, and asks the question with the date on it. I have come to think of that as the only moment in the whole encounter that was ever quite real: not the meeting, not the deck, not the large clean fear with its comfortable distance, but the quiet figure at the door, asking at last about the thing that already happened. It is not the question anyone wants to ask. It is the only one worth answering.
LinkedIn companion
Body clean of links; essay URL in the pinned first comment within 60 seconds. Schedule Tue–Thu, 8–10am CET. Image idea: an emptied conference room, one figure still standing by the door.
On the stage, the AI fear is enormous: will the machine wake up, will it take everything. When the room empties, the question changes. And the changed question is the true one.
Thirty years doing this, and the one that comes once the door is shut is always some version of the same thing. It is never about what the machine might become. It is about what our own people have already done with it, last quarter, without telling anyone.
Does anyone actually know what got pasted into these tools. Is the contract we spent a month fighting over now sitting in someone’s training set. Who approved the assistant that reads the whole mailbox, and could we reconstruct what it has already seen.
Here is what I have learned to watch for: the loudness of a fear runs inversely to its nearness. The machine that awakens is loud because it is far, and being far it costs us nothing today but a pleasant shudder and a thoughtful panel. The danger that is actually here is quiet, specific, boring: a junior summarized the merger terms in a chatbot on a Tuesday. No one convenes a panel. It already happened.
It is more bearable to be frightened of a god than responsible for an intern.
The useful question was never “when will it become something.” It is “what have our own people already handed it, and could we find out if we had to.”
(Long version below.)