*****
Executing Unit Tests……
0/107 Complete; 0 passed, 0 failed, 0 skipped
23/107 Complete; 23 passed, 0 failed, 0 skipped
72/107 Complete; 66 passed, 0 failed, 6 skipped
…
…
“And it never finishes?” I ask Alan.
“Well, not in the three hours that it’s been running.”
“Can’t you look at the log and see whose error it is?”
“Yeah, it’s Dave’s new validation stuff.”
I squint as I try to remember the details from my team’s standup meeting that morning… Ah, right. Dave was working on an enhancement for our web-form. Currently we check to see if people enter valid values and complain at them if they don’t make sense. Like if you put just four digits in the Phone Number field. Dave wanted to make the form a little more helpful by suggesting what you might have meant to enter, like putting the @ symbol in a likely place if you forgot it with your email.
“Okay, well did you ask him why his code is breaking the build?”
“No. He committed it late last night and now he’s on vacation until Monday.”
I sigh. That, unfortunately, sounds just like the Dave we know and hate. Always trying to cram things in at the last minute and then not around to clean the resulting mess.
“Okay, I’ll roll back his changes and he’ll have to take care of it when he gets back.”
“Yep.”
I walk away to my machine and open up the build server. Every time a member of our team makes changes to the code there is a gauntlet of tests that it has to pass before it can go to production. Think of it like a filter to catch the bugs before our customers see them. Dave’s code has gotten clogged in that filter, so now I have to pull it out.
I open the page with his code changes and click on the Revert button. My cursor turns into that little spinning icon that means the computer is waiting for a process to finish. Curious, I check what test it got hung up on… It was for the credit card information where you enter the year that your card expires. The test was supposed to enter an invalid year from the past (2012) and get a recommended correction (2021). It’s odd. We really shouldn’t be trying to auto-correct people’s credit card information for one, and also that’s an incredibly basic test. Dave’s code shouldn’t have choked on it.
In any case, that little spinning icon finally goes away and the code gets pushed back. I don’t think anything more about it until the next week.
*
“So yeah, Dave, we had to push your code back out. And frankly, you shouldn’t be trying suggest corrections in the credit card fields, just highlight that the entry is invalid and let the user correct it.”
We’re in our Monday standup meeting where each team member brings everyone else up-to-date on their current work and needs.
“And let me use this as another reminder that no one should be committing code to the main branch unless they’re able to stay around and see that it passes the automated tests.”
“I did, Greg,” Dave pipes up. “But it kept getting stuck on that one, so I didn’t have time to see if it would pass.”
I suppress the things I want to say.
“If a simple validation takes longer than half-a-second it has failed, whether it got the right answer or not,” I say tersely. “What on earth was your code doing that it would take so long anyway?”
Probably he had gotten it stuck in some idiotic infinite loop.
“I dunno what it does. I just used some validation library I found.”
A “library” means a bundle of code that someone else has written to perform a suite of functions. Often we use them to cover basic stuff like validation, because there’s no need to reinvent the wheel. However…
“I don’t remember seeing any proposal for adding a validation library!” I snap. “You’re supposed to clear these things with me. I’m responsible for verifying everything that we’re using. Get it out of there!”
Since anyone can upload a library you always want to be sure of its source. One that’s taking way too long to do a basic task might very well be a trojan horse for all sorts of viruses.
The meeting has me upset enough that I make a few notes in Dave’s file to bring up in our yearly review. He will not be pleased with his end-of-year bonus.
In the meantime, Dave returns to his machine, pounds away at the keyboard for the next while, and I don’t hear anything more about his code breaking our builds. I have a nagging feeling that I ought to do personal inspections on his code for the next couple weeks, but my next meeting is already starting and I grab my headset. In no time Dave is far from my mind.
*
One day, a couple weeks later, I come into the office, log into my machine, load up my emails, and immediately my heart skips a beat. We have been flagged for suspicious behavior by the company’s technology auditing department.
I open the email and my eyes rove over it even while my phone starts ringing. The caller id informs me that it is my boss, and there’s no question what he’s calling for. I gulp, pick up the phone, and begin the unpleasant conversation. Apparently our code in production grew more than 100 times in size overnight. There have been no code commits, which means all the growth is being perpetuated by something running on our servers.
Like a virus.
I apologize to my boss that I don’t have any clue what could be causing this and vow to get to the bottom of it right away. Then I pull our code out of production and call an emergency team meeting. Five minutes later we’re all crowded around the same table with our laptops, combing through the production environment.
“Yeah something’s writing new files like crazy,” Alan says. “The business logic layer has grown two gigabytes just since we got here. We’re going to run out of storage on the server soon.”
“Great,” I say sarcastically. “Now any idea what’s doing it?”
“Naw, these new files being all have auto-generated names. They don’t give any meaningful–oh wait, here’s something… ‘rubricValidationTemplate_0072.json’… that mean anything to anybody?”
“Validation?” I snap, and I see Dave trying to shrink behind his laptop. “Hey, is that the validation stuff you were setting up,” I bark at him.
“Um, its name does sound similar to that library I was using…but I already took it out, just like you told me to!”
But I’ve already been clicking away furiously, pulling up the relevant code files.
“No you didn’t! You removed it from your methods, but you’re still importing the library and initializing it!”
“What–I must have forgotten that. But if I’m not calling any of its functions it shouldn’t be doing anything.”
Alan snorts. “No, it shouldn’t. But its a blackbox, isn’t it? So there’s no telling what it is doing, regardless of whether it should or not.”
The “blackbox” Alan is referring to is the common structure by which these code libraries get shared. You can’t peek inside to see how it does what it does. You just send stuff in and get stuff back, everything in between is encrypted. And normally that’s fine, because all that is being hidden is trade secrets. But for a malicious library it could also be hiding the fact that its hacking your machine on the side. As this one appears to be.
I want to scream at Dave that he’s fired right then and there, but I figure I had better not. We’ll verify that his illicit library is at the root of this all, and then we’ll deliver his head to the higher-ups. Maybe that will be enough to appease them, and I won’t have to lose my own job as well.
Alan pulls up the list of background services running on the server to look for anything named RubricValidation there. In the meantime I tell Dave to send me a link to where he got that library from.
Alan clicks his tongue. He has indeed found a “RubricValidationService” running in the background and he turns it off. Background services are like little programs that run behind-the-scenes on your computer. A few moments pass and then all of the developers start confirming that the rampant growth of files has come to a stop.
We all look to Dave who is sweating now. He informs us that he can’t get the link to where the library came form. It would seem that it has been pulled from the website it was being hosted on for ‘potentially harmful behavior.’
I shout at Dave for a few minutes, but honestly I’m starting to feel better. We have our culprit and the mystery is solved. Still some cleanup to do, but life can start getting back to norm–
“Wait, the files are growing again,” Craig says from the end of the table.
“What?!”
“Oh yeah…” Alan says. “And–it looks like there’s a new service running in the background. RubricEnforcedValidationService.”
“Just shut the whole server down,” I order. “We’ll delete everything, format the hard drive, and do a clean install.”
Suddenly my phone starts vibrating like it’s going to explode. I pull the device open, turn on the screen, and it’s overflowing with messages from my work email:
Unusual behavior detected on server. 48 emails sent in last minute!
Unusual behavior detected on server. 53 emails sent in last minute!
Unusual behavior detected on server. 61 emails sent in last minute!
…
It’s one of our security checks that has been triggered. Our server frequently sends emails to report when it completes certain tasks, but at most it only ever sends out a dozen in a day.
“And now there’s a RubricCommunicationValidationService,” Alan muses from his chair.
“I said shut the server down!” I see my spittle flying through the air. “Do it now!”
“Hey boss,” Craig says slowly. “I just got an email from the server.”
“Do not open it.”
“I didn’t… but our data scanner service seems to have tripped something in it.”
I wrench Craig’s laptop over to me. Each of our machines watches for emails from the company and automatically extracts data from them for analysis. On his screen I saw a loading bar filling up.
Rubric Validation Data Downloading…8%
Rubric Validation Data Downloading…9%
Rubric Validation Data Downloading…10%
…
Rubric Validation Data Downloading…4%
“Everyone turn your machines off!”
“What?”
“Do it!” I scream. “It just sent out a virus that gets opened automatically!”
Each of them looks dazed, but they move to obey me.
“But how will we fix this if our machines are off?” Greg asks.
“I don’t know! We’ll figure it out… We’ll–we’ll get some new machines, ones that don’t have our email scanner running in the background. Go over to Stephanie’s team and tell them we’re commandeering theirs until I can get us replacements. Go! Tell them I made you do it…. Wait no! Wait!” Everyone pauses in mid-step. My mind is racing faster than my mind can keep up. “Alan, Did you get the server shut down?”
“No.”
“NO?!”
“You just told me to turn my computer off!”
“I know! But– whatever. You go! Take someone else’s machine and get that server off! The rest of you, come with me. We’ve got to shut down every other computer that was on our distribution list right now!”
Because, you see, it wasn’t just our team that got those reports. It was many of our higher-ups as well.
Everyone rushes to follow my orders and the next ten minutes are a blur. Eight overweight men sprinting, sweaty, and cursing all through the office building, slamming peoples’ laptops closed, hurriedly apologizing for crushed fingers, and rushing off to the next.
And though I try to suppress it, there is a voice voice inside, taunting me that I’m fighting a losing battle. What are a few puny humans going to do against a virus that just went…well..viral? All this time it’s been churning away on the servers, thousands of operations every second. And even if Alan has managed to take it down who knows where it has replicated itself to. We saw the email attacks, but who’s to say that was its only outlet?
In fact we know it isn’t. The servers it is sitting on are public facing. They are the brains behind a website that our customers use every single day. Right this moment there are at least tens of thousands of people logged in to our product, taking in whatever RubricValidation is sending to them!
“What is the meaning of this!” Howard is shouting at me. Trying to wrestle his laptop from my clutching hands.
“There’s a virus!”
“I’ll run a sweep after this meeting!” he pleads as I finally manage to wrench it free.
“I’m sorry boss.”
“You’re fired!”
“I know.”
***
So, just in case you were wondering: no, this isn’t an entirely accurate representation of how software development or viruses work. Though I would say its a good deal better than what you get from Hollywood! I’ve taken creative liberties and exaggerated things, but at the core these are exactly the sort of malicious attacks that are every tech company’s worst nightmare.
On Monday I spoke about how we incorporate vague and massive things into our stories. Things that might represent the supernatural, or the unknowable, or something of such profound emotion that it cannot be fathomed. With this story I wanted to combine some of these ideas in the virus that these developers discover.
For one thing the virus is of a mysterious origin. It comes from some unknown “black box,” and no one knows what its secret objective is. It is not just that the answers are unknown, but that they are unknowable. Literally encrypted.
And then it grows quickly. So rapidly, in fact, that it becomes a hyperbole. The men have seen it escalate from a hang-up on an automated test to a malignant virus installing on thousands of machines. And again, this sheer massiveness serves to further obfuscate any clear understanding of the thing. Is its malignant spread random and chaotic, only meant to tie up resources? Or is this simply one piece in a much larger strategy?
Or at least…those were the sorts of intriguing questions I had in my head when I started on this piece. But now that I have written out the first half I will admit it tastes pretty weird. A lot of technical jargon, but also plenty of hyperbole, and some humor that isn’t landing as well as I’d hoped. That’s just how it goes with creativity, though. Sometimes it exceeds even our own expectations, but other times you find yourself saying “that sounded a lot better in my head.”
Usually we try keep those “sounded better in my head” moments from the public eye, but that’s not the purpose of this blog. I want this to reflect my writing journey honestly, the good and the ugly. If this is what I was able to come up with, then this is what I want to share.
On Monday I’d like to take some time to talk more about this. We’ll discuss why it is so hard to accurately predict what is a good idea and what is not, and we’ll talk about how to tell them apart. After that we’ll have the second half of Hello, World. Until then, have a wonderful weekend!
Give Me Pen and Paper 