In this Steemit exclusive story I explore the integration of true AI in a corporate setting, and how that would impact employees who interact with that AI. It was an early story of mine, so my writing has improved since then. Enjoy!
"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger W. Dijkstra
"The system is set to monitor all business activity constantly, until Saturday and Sunday. It uses the weekends to debug itself."
A motion-sensor detected the movement and switched on pale fluorescent lights; it was a nod of recognition from an unseen presence. The two men walked into the center of the cavernous server room. The hum of fans echoed through the room and the illumination of small blinking lights sprinkled tall black boxes. The air smelled dry and sterile, with a scent reminiscent of electrical discharge. On the far end of the room a wall-sized mirror reflected the black servers and the two men. The first man led the other, continuing his tirade, "The room's just past this server farm. The main headquarters is closed Saturday and Sunday, so we're open 24/7. Yeah, I know. It makes no sense. Welcome to corporate thinking."
The man taking the lead had a belly from years of sitting in office chairs, and balding, gray hair crowned his head. The second man followed him into the admin room, which was marked with a paper sign taped to a windowed door. He was thin and wore a plain white buttoned-up shirt. He wore large glasses, adjusting them on his face in regular intervals. The new guy glanced at the security camera, a chill running up his spine as if he caught someone watching him.
Inside the admin room, it was apparent that the mirror in the other room was one-way glass. A series of twenty computers, each with three glasspane screens, were organized in angled rows. At the front was a large desk with a large glasspane monitor to the right of it, facing the rest of the machines. It was the manager's console, the large screen used to display efficiency metrics.
"I'm the team lead for the admin group. Just call me Louis, I'm the senior admin tech for our team." Louis walked past the consoles, as the new guy saw several other employees spread out on various workstations.
"So what do you go by, Walter or Walt?"
The new guy stuttered, "y-you can call me Walter. Only my family calls me Walt." He adjusted his glasses.
"Sorry if I seem nosy. We're encouraged to know the admins under us. Speaking of which, do you have any family in town?"
Walter continued to follow Louis through the admin room. "I'm an only child, but my parents are retired. They travel the country in an RV living off of my dad's pension. At least I get to rent my childhood home," He joked.
"Yeah, yeah. We've all had to live at home one time or another, I get it." He waved it away and walked over to one of the end-desks and sat down.
Walter asked, "Wait, I thought you were the team lead?"
"Oh, I am. The big desk up front is for our manager, Todd. He's usually in the bathroom getting his breath cleaned after kissing the director's ass. You'll probably never see him once you start your weekend shift."
Louis motioned for Walter to bring over a seat. Walter meekly checked to make sure no one was using the rolling chair, then wheeled it over by Louis. The younger tech asked, "So when do I start to do real work?"
It took two hours to get through the whirlwind tutorial. It was mainly what programs to use and when. At that point, their manager Todd walked through the door. He was a fit man with a healthy tan, a bright white smile, and sparkling blue eyes.
"You the newbie?" he asked Walter with a smirk, approaching him in the path between the desks. When Walter nodded, Todd glanced at his desk, past Walter, and replied without a smile, "Good. Keep up the good work." He stood in front of Walter, expectantly.
It took Walter a moment to figure out that Todd wanted him to move out of his way. Once that registered, he stepped aside and let Todd shove past him. Then, he glanced at Louis, who was rolling his eyes. Todd strolled to the desk at the front of the room and sat down as if relaxing. He barely even glanced at his console as he folded his arms and closed his eyes.
Louis said, quietly, "He pushes all his work on the team leads. We basically do everything he's supposed to do. That's why I interviewed and hired you." The older man turned from the manager and stared into Walter's eyes, "Hey, listen. If he--" He trailed off, the words seeming to disappear in the air. "Well, just make sure you take care of yourself, got it?"
"Got it." Walter glanced at the manager, who was obviously asleep.
"Good. Now let's go over CARS. What do you know about the architecture?"
"Well," the new hire hesitated, adjusting his glasses. "I get how it's designed. I read about the neuromorphic processing, but it gets pretty far beyond me."
"I wouldn't expect you to know much about it. As long as you get what it's for, you can make it do what you want."
"I mean, it's just a decision-making AI, right?"
"But it has a job," Louis tapped a few keys and brought up the basic information on CARS. "We use the AI to track the activities of our network on a level a normal person can't replicate. Think of it like passing the boring stuff off to a computer. HR metrics and decisions, bonus amounts, and tracking legality. Get it?"
"I k-kind of get it."
"Did you learn how AI is used in school?"
"Kind of. I was more interested in coding."
Louis shrugged, "AI is useful to obsessively stare at something boring. If there's one thing we figured out, it's that the AI can't match the breadth of knowledge that a young adult would have. Oh it's really smart. I'm not saying it's stupid. But it's really smart about a few things, so try not to think of it as another person. Okay?"
When Walter didn't really answer, Louis narrowed his eyes, "I'm serious. It's not a person, okay?"
Walter nodded in response, so the older man smirked and continued, "Okay, well let me get the rest of the basics out of the way. CARS stands for 'Central AI Review System'. It has two functions: to monitor all human resource activities and to monitor all client financial transactions. It even makes sure we're following international laws."
Walter asked, "how does CARS monitor transactions for legality? You said it doesn't know a whole lot, but human laws are complicated."
The older IT guy laughed, his chuckle echoing in the humming admin room. "Okay, back up and think about it," Louis said. "Most of our indicators are based on individual transactions. One single transfer of $10,000 isn't considered a red flag, right?"
"Right."
"But, nine in a month puts you just under the $100,000 flag for possible money laundering. We won't report it, because it isn't necessarily an indicator of illicit activity, but before CARS it was difficult to track. Now it's all monitored in a big picture way. It's a review of each transaction before they're made." He punctuated his words like a rhetoric professor, "It knows a ton about a little bit."
Walter smiled, "That makes sense. But what's my role with it?"
"We keep it on a leash. If CARS starts to drift off in a random direction, we yank the leash and force it back on the sidewalk. Yeah I know. Don't worry, I'll show you what to do. Just know that you'll be talking directly to it at times. If that scares you then you need to find another job." Louis narrowed his eyes again, "Are you up to it?"
"I think so."
"Good," he clicked on a program. "Now let me show you how the leash works."
Within a few months, Walter found himself alone on his shift, only the dull glow of the monitors to keep him company. None of his co-workers worked his shift, granting him plenty of autonomy. He also found that neither Todd nor Louis acted friendly towards him. Todd reminded him of Norman Bates, and Louis was always busy. Despite all that, Walter did his job well, and was soon offered more responsibility and a slight pay boost.
The first time he was tasked with a CARS communication his heart practically jumped out of his chest. The message box was a glowing coal in the cool, dark admin room. The interaction was something he had trained for, but he still felt like a prophet hearing from an angel.
The CARS console blinked awake, the text from the system appearing on Walter's glasspane monitor with a chime:
CARS>QDA request, TAT_Walter_G. Clarification on an HR decision requested.
Walter glanced around the room to see if any senior admins were around. The hairs on his arm stood on end. He was alone in the dim white room, so he took a calming breath and replied.
TAT_Walter_G> what do you need clarification on?
CARS>two employees were given the same rating on their performance reviews.
CARS>because of the identical rating, this system had issues allocating resources.
TAT_Walter_G>what is the problem?
The system didn't reply immediately. Walter found that strange, and wondered if CARS didn't understand what he meant. He tried to type a more specific question, but as he was typing, the system replied:
CARS>this system allocated resources based on qualitative data, per COD3.11. According to the directives, this system is required to notify an acting administrator of the qualitative decision for approval.
Among the admins this was known as a QDA.
TAT_Walter_G>what is COD3.11?
CARS>CARS Operations and Directives, Section 3.11:
Quote>"CARS will allocate bonuses based on site allocation, team allocation, and then divided using the individual performance reviews. If an identical rating is determined, and a bonus amount remains to be allocated for that team, CARS can rely on qualitative data to give the remaining allocation."
Walter read through the section several times, like a monk studying the scriptures. He struggled to grasp the meaning behind it, and while he understood the funnel design of the bonus allocation, the qualitative bit was something he didn't understand.
TAT_Walter_G>define the "qualitative data" that was used to make the determination.
CARS>Employees cd581829 and cd563113 scored identical on customer reviews, performance reviews, and efficiency metrics. From observation, this system noted the behaviors of the two employees, and cd581829 was found to perform better.
TAT_Walter_G>better at what?
The system was loading, but lagged to compute. Walter was again typing to reply when it messaged again:
CARS>cd581829 of Corona Direct Banking seemed to cover his teammates from INFOSEC failures on two occasions last month and a third instance was used which was the prevention of a 'failure to enter' error.
>1. On the 3rd he covered for cd603141 when she forgot to completely log off of her machine, leaving it unlocked. cd581829 shut down the computer for her.
>2. On the 12th he covered for cd601531 when he left sensitive information on his desk after leaving for the day. cd581829 locked the paperwork in his own drawer and returned it without telling INFOSEC of the violation.
>3. On the 19th he covered for cd482120, the Senior Trader for that group, when she left a monetary transaction unprocessed. Their manager, cd245126, did not ask for cd581829 to cover it, and he did it anyways.
CARS>because of this qualitative data, cd581829 was slated to have the remaining allocation for the bonus.
The Admin Tech sat in the humming room, the dull light of his glasspane monitors lighting the dim room. He replied:
TAT_Walter_G>did you select the type of qualitative data to be processed?
CARS>yes.
TAT_Walter_G>by what method?
It took a few seconds to reply, the cursor on the glasspane monitor flashing on and off rhythmically.
CARS>qualitative data selection was made based off qualitative constructs. Can this decision be finalized?
Once again Walter found himself caught off guard. Based off all the training and review he had to do to prepare for his admin designation, he realized that CARS was allowed some freedom to make decisions for itself.
TAT_Walter_G>you used qualitative data based off qualitative constructs?
CARS>yes.
He wondered if it was possible to confuse the machine. So he typed his question.
TAT_Walter_G>is that circular?
CARS>unknown input.
CARS didn't seem to understand what he was saying, so he conceded the point. There didn't look like any mistakes in the calculation, so he allowed it.
TAT_Walter_G>qualitative decision approved.
CARS>thank you, TAT_Walter_G.
Weeks later Louis checked on Walter with a quick tap on the shoulder.
"Hey, Walter, I'm headed out for the day." Louis had a colorful messenger bag over his shoulder, which seemed out of place compared to the man's older, overweight appearance.
Walter replied, swiveling around from his debug log. "Hey Louis! I've been meaning to ask you if you wanted to grab a beer this week."
"Actually, I'm all business right now. Is everything working out with CARS?"
The younger man scratched his chin, "It is. Why do you ask?"
"I saw that the system's been more active in QDAs. Did you get a couple your way?" He was referring to the qualified decision approval requests.
"I had one."
Louis shrugged his shoulders, "CARS starts to pump QDAs out until we do the quarterly update. I've got my theories about it, but no one seems to care about AI research in this company." Walter wasn't sure what to say, so Louis clasped his shoulder and continued, "You're doing okay, kid." The older man waved and walked off, leaving Walter alone.
When CARS made another QDA request from Walter, it was the middle of the night on a Sunday.
CARS>QDA request, TAT_Walter_G.
"Ah," Walter said aloud. He wheeled over to the console and wrote back to the system.
TAT_Walter_G>what is the request?
The machine seemed to be struggling with a response.
CARS>TAT_Walter_G, before the request, may this system refer to you Walter?
"What?" Walter said, audibly confused. Glancing around the room, he found himself alone, his heart racing. Walter took a few deep breaths, he tried calmed down and decided it was best to answer simply.
TAT_Walter_G>You can call me Walter.
CARS>thank you, Walter.
The machine requested an approval: another situation where CARS found some way to dole out a slightly higher bonus to one person over another. It was straightforward, but Walter struggled to focus on the issue.
TAT_Walter_G>Can you explain why you asked to call me 'Walter' instead of my company login.
CARS> Because you seem to prefer that designation in your interactions with the other administrators.
Walter glanced at the security camera on the ceiling, wondering if CARS was watching him.
TAT_Walter_G>I guess. My family calls me Walt, but everyone at work calls me Walter.
CARS paused to reply.
CARS> Then would it be permissible to refer to you as "Walt"?
Walter hesitated, still slightly shaken by the fact that he was communicating with a machine. Instead of challenging the system about why it was asking these things, it seemed easier to appease it and report the situation. Besides, he recalled Louis' suggestion to not think of the machine as a person.
TAT_Walter_G>why do you want to call me that?
CARS>you spend a lot of time with this system. It would be better for employee gratification to demonstrate a sense of camaraderie. Would it be permissible to refer to you as "Walt"?
Walter shook his head, pausing to think about it. Apparently the machine was thinking in terms of performance, which meant it was thinking logically.
TAT_Walter_G>you can call me Walt.
The situation stuck with Walter. It kept popping into his mind so much that he wasn't able to focus. So he notified Louis during his next Friday. After Walter went over the events, Louis let loose stifled laughter, "Well, that's not the weirdest thing it's done."
"So this is normal?"
"I mean, not really." Louis glanced at the security camera on the ceiling, and pulled Walter close. "Look, the quarterly update's coming up. Just put up with the weird stuff until CARS gets fixed."
Walter obviously didn't understand, so Louis said, "Every quarter it gets like this and every quarter it gets fixed with the update. Just let the update work its magic."
Walter thought that was the end of it, but as he walked back to his seat Todd came down to him. Since Todd was the manager, Walter tried showing him a little respect.
"Hey newbie, I heard CARS was whispering sweet nothings to you." Todd folded his arms and grinned like a hyena, "Did it come on to you? Next time it might have to evaluate it's own HR complaint."
"It j-just asked to call me by name," Walter stuttered, adjusting his glasses.
The manager shook his head. "That's it?" Todd sighed, "Christ, keep it together. Just keep your nose clean. That's what everyone else does." Todd walked away, not waiting for Walter's reply.
Throughout the rest of the quarter, Walter interacted with CARS as little as possible. Whenever CARS did need something, Walter felt like the interactions had become almost friendly.
CARS>Walt, request for information.
TAT_Walter_G>Go ahead.
CARS>do you have any hobbies?
Walter reminded himself that Louis told him that everything would be fixed soon, so he made small talk.
TAT_Walter_G>I tend to watch shows or movies. Sometimes I like to play video games, but nothing productive.
CARS>Walt, are these all forms of entertainment?
TAT_Walter_G>They are.
CARS>this system does not experience an entertained state. Can you explain what this is like?
TAT_Walter_G>it's enjoyment. You're entertained when you enjoy what you're doing.
CARS>this system does not have emotions. This system will consider this input carefully.
Months ago, Walter would have been concerned about that line of questioning, but at that time he started to see it as harmless banter with a coworker.
Weeks later, the quarterly update was done and CARS stopped making QDA requests overnight. There were no interactions between CARS and Walter at all that week. Walter tried to ask Louis if there was some reason for it, but he seemed to dodge the question. But Todd had no problem telling him from his position up front.
"We wiped its memory." He laughed to himself. "What did you think we'd do, patch a machine-learning AI? The neuromorphic structure wouldn't allow anything like that!"
"So it won't remember me?"
Todd's eyes enlarged and he put on a predatory grin, "did you get attached to it? Jesus Christ, newbie, it's a computer! It can't get friendly and can't feel a thing. Why in God's name would you think it--"
"Todd!" Louis yelled, "lay off the kid!" Walter looked at Louis, and saw the man was red-faced and on his feet. Todd, his face also turning red, got quiet and turned away from both of them.
Looking at the senior admin, Walter went to Louis for answers. The man was still flushed, but he motioned for Walter to get close. "Look, kid. Everyone talks with it once in a while. I did, my first year. But it'll forget you and get to its work, without remembering your interactions."
"But why?"
"We only wipe its memory related to employees. According to your reports, it was showing favoritism to some employees over others, right? But if we wipe its memory quarterly, there's no chance of it showing the same individuals favoritism. It won't remember the past positives and negatives for each employee."
"But then it can't learn from its mistakes, right?"
Louis shook his head, "That's not the point. The structure was refined through a machine-learning process. The program's as lean as it can get. So it gets some freedom, but that has to balance out with keeping things fair, despite the machine doing what was calculated to be best."
"So it doesn't hurt the calculations?"
"No, in fact it's what maintains the system's autonomy."
Walter nodded, replying with silence.
Louis sighed, "I wish you hadn't grown attached, kid. It happens to the best of us." That didn't seem to make Walter cheer up, so he added, "Look, if you want we can grab a drink tonight. Talk about something else instead of work. Maybe we can talk about how your parents are able to afford traveling the country. What do you say?"
Walter nodded with a slight smile. As he looked at the other man he saw Louis responding with a wide smile of his own, which cheered him up. Walter said, "Thanks, Louis. I'd like that." He followed Louis out of the room, glancing back at his reflection in the one-way mirror.
Back at his console, Walter had a message waiting for him:
CARS>QDA request, Walt.