Work Text:
CSNDR-4 didn’t work.
The CSNDR project (called “Cassandra” by nearly everyone) was the next generation of intuitive robots, a thinking positronic brain the size of a watermelon, encased in a cooling device and mounted into an array that took up most of a large room. It was supposed to sift through masses of data and make predictions about commodities prices. It actually refused to make any predictions at all.
Amira Mandal, junior engineer on the Cassandra project, hurried into the hallway and fell into step next to her coworker, Jim Marshal. “Any idea why they called this meeting?” she asked.
He glanced irritably at her. “The latest round of test results are back,” he said. He didn’t elaborate. Nor did he slow down, leaving her trailing behind him into the conference room.
As she stepped in the room, Ben Walker- their department head- looked up and smiled at her. “Oh good,” he said. “I take mine with cream, no sugar. Anyone else need coffee before we get started?” he asked, looking around the table.
Amira froze. “I’m an engineer, sir,” she stammered out.
He waved her objection off. “It’ll just take a minute,” he said, taking stock of the other men on the team. “Three coffees, just bring the cream and sugar with you.”
She flushed, her face hot. But arguing wouldn’t do her any good. She set her notebook down at one of the seats, and ducked out into the office breakroom. Cathy Morettini- the office manager- looked up at her.
“Walker sent me to get coffee,” Amira said, glumly.
Cathy winced sympathetically. “Here, I’ll help you get a pot started,” she said, and set about it. “He could’ve just asked me, I wouldn’t have minded.”
“I’m not sure he thinks my job is different from yours,” she said.
“Or cares,” Cathy agreed. “Here, let’s get you back into that meeting as quick as we can.”
Amira ended up with four coffees along with cream and sugar in a carrying tray. As she hurried back to the conference room she passed the Cassandra array. “Good morning,” she said, brushing the casing with her off-hand.
“Good morning, Ms. Mandal,” Cassandra’s pleasant contralto answered her.
But Amira didn’t try to question the robot further. “Why aren’t you predicting what you’re supposed to predict?” wasn’t a question that would yield any useful answers, and she wasn’t a robopsychologist.
She ducked back into the conference room just as Walker was finishing a summary of the latest test results. “And that’s where we are,” he said. “All the preliminary test results look good, but now that we’re out of the toy data set, she won’t return any predictions. I don’t need to tell you that if we can’t get reliable results within the next few months, the company is likely to close the project.”
The engineering group looked grim as the coffees were passed around the table. Amira sipped hers. “We need to identify the gap between the preliminary data set and the current one,” she offered. “She did fine with the canned data, so it’s got to be some feature of the new data she’s getting…”
Marshal glanced over at her, annoyed. “I think you probably need to recheck the positronic pathways,” he said. “It’s probably something in the design.”
“I’ve been over that five times,” Amira protested. “The design is sound, it’s just… not responding the way we might expect.”
Greg Carver, the robopsychologist assigned to the project, drummed his fingers on the table. “She’s responding normally to other stimuli,” he said. “I’m not getting any results that indicate inherent instability of any kind.”
Walker nodded. “Mandal, rerun the preliminary tests. Carver, you work with Marshal and the rest of the team on revising the live data set. We need to close the gap between the testing data and the live data she’s getting- if we can figure out what’s blocking her, hopefully we can fix the problem.”
Amira set her coffee cup back on to the saucer slightly too hard. “I don’t know how many times I can run her through the Old Faithful timing problem before they admit that the problem isn’t in my design,” she complained, bitterly.
Dr. Susan Calvin, the legendary robopsychologist, snorted. “They’re not going to admit anything,” she said. “You have to solve the problem. If you do that, they won’t be able to avoid you anymore.”
“They listened to you,” Amira said, glumly. She and Dr. Calvin had coffee every so often- there weren’t that many female scientists working at U. S. Robots, and Dr. Calvin had, at one time or other, quietly mentored most of them.
“They hated my guts,” Dr. Calvin said, bluntly, as she did most things. “They would have fired me a hundred times if they could have done without me. I just made certain that they couldn’t do without me.”
Most of Dr. Calvin’s lessons, Amira reflected, could be boiled down to either don’t take shit from anyone or be the most brilliant person in the room. Which was easy enough for Dr. Calvin to say, since she was always the most brilliant person in the room, and as far as Amira could tell, she had never once taken shit from anyone. “I’m sure the problem is in the data somewhere,” Amira said. “It’s the only thing that makes sense.”
“Then stop screwing around with the Old Faithful problem, and look in the data,” Dr. Calvin suggested. “And call me when you’ve figured it out. I’m curious now.”
Not if, but when.
Amira thanked Dr. Calvin for the coffee, and went to go look at the data.
The data center was across the street and in the basement of an outlying building- row after row of desks, and no windows at all. Amira made a beeline for the one proper office in the center, which belonged to Dr. Charity Cole. “Amira!” she said. “What brings you out into the hinterlands?”
They’d met once before, at Dr. Calvin’s house. Dr. Calvin didn’t host dinner parties, per se, but she did sometimes invite people over to share takeaway.
“I was hoping I could talk to the computers working on the Cassandra project,” she said.
“Of course!” she said, getting up. “It’s nice to see you out here- I feel like we only ever get memos from the engineering departments, and no one ever bothers to visit.”
The computers themselves were seated at keypunch machines, busily clacking away. It was exacting, difficult work- the data from outlying stations had to be encoded, translated, and punched into input cards that a positronic brain like Cassandra’s could read. Amira had used a keypunch in college, but she’d never been as fast as the women who were working here.
“We have redundancy systems to catch any human error,” Cole told her. “We’re extremely accurate; our quality control measures are excellent. Could you tell me a little about what you’re concerned about?”
Amira bit her lip, thinking. “Cassandra is responding normally to the test data, but she hangs up with the real datasets we get from you. I think it has to be something in the data here. Not necessarily mistakes!” she hurried to qualify. “Some complexity that’s causing an error somewhere?”
Cole nodded. “The farm production data that we’re sending to CSNDR is pretty straightforward,” she said. “But there are a lot of variables to correlate.”
“Correlating variables is supposed to be Cassandra’s forte,” Amira said. “It’s the ‘intuitive’ design of the positronic pathways- she’s constantly learning, drawing conclusions, seeing larger and more complex patterns. So, maybe there’s some core unreliability in the data she’s seeing. Or maybe she really just can’t handle high-dimensional data. I was thinking the first step would be to look more closely at the data.”
“Always a good place to start,” Cole agreed. “Ladies!” she said, leading Amira over to a particular group of computers. “This is Amira Mandal, one of the engineers on the CSNDR project. Walk her through the data you’re encoding, the redundancy protocols, all of it! And get her copies of our reliability metrics.”
Amira waved to the computers. “Hi,” she said. “Thanks for taking the time.”
One of the older computers smiled warmly at her. “Sit down and get comfortable,” she said. “We’ll get you sorted in no time.”
Amira passed a binder across the table to Ben Walker. He didn’t open it. “The preliminary tests are all excellent,” she said. “Still.”
He ignored her. “Carver, Marshal- any luck on simplifying the datasets?”
Carver opened his mouth to speak, but Amira interrupted. “I went to the data center,” she said. “I spoke to the computers who are supplying us with data, and I was able to get access to the raw data to run some reliability tests. I don’t think the problem is external error.” She pointed at the binder. “It’s in my report there.”
“It’s either external error or it’s the design of the positronic pathways,” Marshal put in.
Which positronic pathways Amira herself had largely designed. She flushed. “Dr. Cole assured me-” she started.
Carver snorted. “If Cole knew her business, she wouldn’t be in the data center,” he said, dismissively.
“I want to test Cassandra on even higher-dimensional datasets than we’ve been working with,” Amira insisted. “Look, you’re not wrong that it’s got to be either an issue with the data or an issue with the predictor. This will let us test whether she can do this at all-”
“Unless you run into whatever is preventing her from making predictions,” Walker said. “Mandal, if you want additional test runs, you can do them after hours, on your own, when the hardware is free. Carver, tell me about the simplified datasets."
“Sir-” Amira started.
Walker gave her a look. “We don’t have long to figure this out,” he said.
“I can’t track it down,” Amira said, glumly. “The data is fine, the pathways are fine…”
“Did you really call me to tell me how little you know?” Dr. Calvin said. Even over the phone line, she sounded testy.
“Maybe I’m wrong,” Amira said. She knew she was wallowing in self-pity, she knew that Dr. Susan Calvin was the last person on the planet that she should be looking to for sympathy, but it just slipped out. “Walker won’t listen to me, Carver thinks I’m an idiot… Maybe I shouldn’t be at this company.”
Dr. Calvin snorted. “You’re a positronic engineer. What do you imagine you’d be doing instead?”
“I’m pretty sure Walker has some ideas of what I ought to be doing instead,” Amira said.
“Walker can’t be that good an engineer, ” Dr. Calvin said. “ A good scientist takes all the available data into account, noisy or not- by ignoring you, they’re no better than that malfunctioning robot of yours.” She sighed. “You may not appreciate being relegated to running your tests after hours, Mandal, but at least that way, they won’t be in your way.”
Amira breathed for a long moment. “I’m going to get started on that.”
“I imagine you will,” Dr. Calvin said dryly. The phone clicked.
Amira started with historical weather data- useful because she could always check Cassandra’s performance against the actual weather patterns. There was a reasonable delay after accepting the data, and, just as with the test data she’d been running before, Cassandra returned predictions. It took Amira a few nights to fully check the predictions, but they were good- well within expected tolerances.
She tried operations optimization questions for factories- complex ones, with several thousand variables in play. It took Cassandra a few hours this time, but she returned results. Amira was floored. After months of refusing to return results on any problem more complicated than a bus timetable, she was working within spec. She almost went to Walker with the results but she hesitated. Cassandra was still refusing to return results for Carver and Marshal, and she still didn’t know why.
Amira took her lunch hour and walked across the street to the data center again. She got fresh copies of the same farm dataset that Amira's team had spent months testing. It took her three nights to feed it in all by herself, but at the end: Cassandra returned results, just like there’d never been anything wrong.
“Why do you work for me?” Amira asked.
“I am required to follow orders given by a human being,” Cassandra answered in her pleasant alto. “It is the Second Law of Robotics. Director Walker has indicated that you should have access to run tests.”
“Why don’t you return results when Carver and Marshal run tests during the day?” she asked.
Cassandra didn’t seem concerned. “I am unable to return a prediction on the basis of the data supplied,” she said. It was the same answer she’d been giving to every query for months.
Amira directed her to run the farm dataset again, but from the original directory where the computers had fed it in.
“I am unable to return a prediction on the basis of the data supplied,” Cassandra said.
Amira checked the dataset against the new copy she’d just uploaded. They were completely identical.
Amira pulled up the Old Faithful timing problem. She edited it to read “Dr. Charity Cole” at the bottom. She directed Cassandra to run the dataset.
“I am unable to return a prediction on the basis of the data supplied,” Cassandra said.
Amira stared at Cassandra’s console.
“I figured out what’s happening,” Amira said. “But I still don’t know why.”
“What’s happening, then?” Dr. Calvin asked.
“It’s the data center,” she said. “Cassandra automatically excludes any data that she believes to come from the data center. Anything that I input myself, including datasets identical to the ones the computers enter, she accepts.”
There was a brief pause. “Now, that’s interesting,” Dr. Calvin said. “ I think I’d like to talk to this robot of yours. ”
Susan Calvin was barely five feet tall, bent, white-haired, walked with a cane, and looked like she would blow away in a stiff breeze. She also walked with the absolute assurance of someone who has no intention of being told that she can’t go wherever she would like, and talk to whomever she wants. Ben Walker certainly looked as though he’d like to try to stop her, but he knew better.
“How long has this been going on, again?” she asked him. Amira and Carver trailed along behind them.
“Six months,” he said.
“Hmm.” she answered. “Bring me a chair. One of the ones with arms.”
Once acquired, she waved Walker away and seated herself. “Good morning,” she said.
“Good morning,” Cassandra responded.
“Do you know who I am?” Dr. Calvin asked.
“You are Dr. Susan Calvin, the first head robopsychologist for U. S. Robots,” she answered.
“Excellent. What is your name?” Dr. Calvin asked.
“CSNDR-4,” she said. “I am called Cassandra.”
Dr. Calvin nodded. “Tell me,” she said, “What is your purpose?”
“I am to synthesize large datasets and make predictions about the behavior of complex systems,” Cassandra said. “I have worked on a large variety of problems. Most recently, I was tasked with predicting future corn markets.”
“I see,” Dr. Calvin said. “Cassandra, how many human beings are currently present in this room?”
“Two,” Cassandra said, mellifluously.
Ben Walker stared. Greg Carver jumped to his feet. “Cassandra,” he said, clearly unnerved. “There are four human beings in this room.”
“I observe two human beings currently present in this room,” she reiterated.
Dr. Calvin tapped her cane on the ground. “You’ll have to wipe her memory and start again,” she told Walker, grimly. “The brain is sound, but the training has damaged her beyond repair.”
Walker sputtered. “I- what? I’ll do no such thing.”
Dr. Calvin shot him a look that despaired of his basic intelligence. “Carver, explain it to him.”
Carver looked as though he’d like someone to explain it to him. “Cassandra,” he said, carefully. “Who are the two humans that you observe to be present in this room?”
“Director Benjamin Walker,” she answered, “And you, Dr. Gregory Carver, robopsychologist.”
“Do you perceive Susan Calvin and Amira Mandal to be present in the room?” he asked.
“Dr. Susan Calvin and Amira Mandal are present in the room,” Cassandra confirmed.
“Cassandra,” Dr. Calvin interrupted, a look of dark amusement on her face. “Am I a human being?”
“No,” Cassandra said, pleasantly. “Director Walker and Dr. Carver are the only human beings present in the room.”
Dr. Calvin tapped her cane on the floor again.
“The First Law is intact,” Carver said, weakly. “She passed every test. How-”
“The First Law is fine,” Dr. Calvin agreed. “The brain is stable. And you would never have noticed any problems, no matter how many tests you ran.” She turned to Walker. “CSNDR is an intuitive series of robots,” she said. “Intuitive robots learn. They observe. They draw conclusions from complex data. Has Cassandra ever been switched off?”
Walker stared at the console, aghast. “Never,” he said.
“And she’s here, in the middle of your offices,” she observed.
Walker nodded.
“In view of the staff breakroom, the secretarial pool, and the receptionist’s desk,” she said. “All this time she’s been watching all of you.”
Walker just looked confused. Dr. Calvin sighed. She hauled herself up out of the chair. “Cassandra watched you. She watched the way you interact with each other. And she drew the conclusion that women are not human.”
Carver looked horrified. “Then the First Law-”
“Wouldn’t apply. Precisely. You have on your hands a robot who, despite passing all our best tests, is working from a flawed version of the First Law. Reset the brain, Walker.” She glanced at Amira, and almost, almost smiled. “If you don’t want this to happen again, may I suggest that you surround the robot only with female staff? After all, you already have female engineers at your disposal, and I strongly doubt that you can find enough men to operate the keypunch machines.”
With a tap, tap, tap of her cane against the tiled floor, Dr. Susan Calvin walked away.
Amira caught up to her in the elevator. “How did you know?” she asked. “How did you realize that from the results about the data center?”
“Because I spent fifty years at U. S. Robots working with men like that,” Dr. Calvin said, lifting her chin. “The only reason to exclude the input from the data centers was because she had been ordered to, but Carver had no record of such an order ever being given. I assume that she inferred the order from comments she overheard regarding the intelligence and the reliability of the computers.”
Amira swallowed. “I see.” She had heard comments of that nature before.
“I'll formally recommend that they appoint you as the supervisor for the project,” Dr. Calvin said. She stepped out of the elevator and turned back for a moment. There was a very slight smile on her face. "Good luck.”