The Plot of Control

In this post I am going to provide a chronologically ordered synopsis of the primary plot thread of Control. It should go without saying that this will be a complete synopsis, so massive, huge, bright-red blinking SPOILER ALERT!

If you have not played Control or are not familiar with this game, it is a third-person shooter developed by Remedy. The game has received high praise for its incredible graphics, being one of the first to earnestly leverage nVidia's RTX ray-tracing capabilties and DLSS AI performance upscaling. It is, without question, a beautiful game.

Control's combat revolves around a collection of unlocked abilities and form-swapping primary weapon, though the depth of combat is not necessarily one of the game's strongest selling points. Combat and game mechanics are fun and remain entertaining throughout, but it is not (and does not attempt to be) a game with a Dark Soul's level of combat detail or complexity.

Most fascinating to me, however, is the mixed reception to its story. Not so much from people who dislike the story, but rather, the number of people who claim the game has no story, or that the story is illogical or lacking in nuance or depth. This is a bit surprising to me because the story of Control is, in my opinion, its absolute best feature. Better than even the graphics, the plot of Control was one of the better video-game stories I've experienced in a long while.

Now, I can completely understand people who simply don't like the story. It is a world based on the surreal supernatural ideals of the SCP, the X-Files, Twin Peaks, and Stranger Things, to name a few obvious inspirational sources. And without question, that is not a style everyone is going to enjoy. Art is subjective, after all. But to dismiss the existence or depth of its story entirely is a confusing stance for anyone to take.

So, to clarify and codify both the existence and depth of the story of Control, I'm providing a complete synopsis of the primary plot thread weaving through the game. Again, MASSIVE SPOILER ALERT. Further, I should also mention that the world of Control is directly connected to, and part of the universe of, other games Remedy has produced; namely, the Alan Wake series. To this day I haven't played these games and I know nothing about their world or story, so I cannot speak to any connections that exist. I can only say with certainty that the plot of Control stands completely on its own and familiarty with these other titles is absolutely no requirement to fully enjoying this game.

Finally, I have done no research. The synopsis presented here is solely from what I remember about my playthrough of the game along with a few trips to the wiki for pretty pictures and to make sure I was getting names and spelling correct. If anything I say is in conflict with more learned sources it's most certainly me who is incorrect. Oh, and for good measure, one final time...

SPOILER ALERT! COMPLETE GAME SYNOPSIS

PROLOGUE

This information is uncovered throughout the game through exploration and various cutscenes, lore objects, etc.

Jesse and her brother, Dylan, are part of an altered world event (AWE) involving objects of power (OOP) that occurs in the town of Ordinary while they are children. There are multiple effects, but the primary impact comes from the use of an object (a slide projector) to travel between dimensions/realities. As part of this event and the children's exploration of it, hostile entities eventually threaten our reality. A benevolent entity from one of the explored dimensions attaches itself to Jesse and her brother in an attempt to help defend our reality from this threat.

Researchers from the Federal Bureau of Control (FBC) learn about this AWE. They find and attempt to contain the objects from Ordinary in order to study the event that occurred there in accordance with the overall directive and purpose of the FBC. As part of these efforts they manage to capture Jesse's brother but do not capture Jesse. As Jesse grows and learns to cope with the events of her childhood she eventually sets out to find and rescue her brother from the FBC. During this journey she is in constant communication with the benevolent entity in her head that crossed over into our reality during these orginating events; an entity she has named "Polaris".

While researching the slide projector from Ordinary, the FBC accidentally re-connects to the reality with one of the hostile entities, the Hiss. This entity is essentially a memetic thought-virus propagated on some kind of paranatural equivalent to EM radiation (transmission/wavelengths). This entity manages to subvert the then-head of the FBC, Director Trench. As a result, Trench makes increasingly suspicious and irrational decisions for the FBC over the ensuing years, putting FBC researchers at odds with each other. This is ultimately revealed to be in pursuit of destabilizing the FBC in order to allow the Hiss to invade our reality.

One researcher of note, Dr. Darling, eventually makes independent contact with Polaris and is made to understand the threat to the FBC and our reality, but too late. He makes emergency preparations for the coming disaster in order to attempt to stave off the Hiss attack in the form of "HRA" devices that can protect people from the Hiss thought-virus. It is insufficient to stop the events that follow, but does provide protection for enough survivors to attempt to repel the Hiss attack.

ACT 1

The game proper begins in media res. With the help of Polaris, Jesse has "concidentally" arrived at the FBC in search of her brother just as Director Trench has opened the floodgates and allowed the Hiss to invade our reality. Jesse finds the FBC in an eeriely quiet and chaotic state as automatic measures, responding to the Hiss invasion, have put the FBC on lockdown. The only surviving staff are those who followed Dr. Darling's advice and began using the HRA protection devices. Details about prior events are revealed throughout the game as you search the building, in the form of visions, dialogues, cutscenes, and from reading, watching, or listening to various lore objects discovered throughout the FBC.

Jesse's first act upon arrival is to find Director Trench's office. It is revealed in a vision from "The Board" that he has killed himself, but Jesse is not told why. At this point Jesse discovers that the FBC building itself is an object of power/altered world event that is somehow deeply connected with entities from another plane of reality, known as The Board. The Board has its own long history and agenda with the FBC (that we will consider worldbuilding and not part of this plot), with the important bit being that the Board chooses the FBC's current Director by assigning a symbol of authority, an OOP that takes the form of a transforming weapon.

Upon discovering Trench's office and his suicide, Jesse takes up the weapon and is therefore assigned the FBC's Directorship by will of The Board. In the beginning this weapon can take only the form of a pistol named Grip, but later you can learn to invoke other forms. Jesse also begins to develop paranatural abilities, each one triggered by interacting with other objects of power contained in the FBC, but augmented in strength and control via Polaris' influence.

ACT 2

Jesse continues to explore the FBC in search of Dylan, encountering enemies that are distorted versions of people, creatures, or items from our reality subsumed by Hiss control. She finds several surviving FBC employees trying their best to understand and repel the Hiss attack and save the FBC, foremost among them the Head of Research, Emily Pope. All of these employees recognize her as the new Director by virtue of her command over Grip, though Jesse is reluctant to assume this title. Her opinion of the FBC at this point is ambivalent; she uses the Director role only in-so-far as it will help her find her brother, and cares little about the FBC or its problems.

She aids these people in various ways in order to further her goal of finding her brother, but we'll consider all of those stories mere side-plots and discount them. Of note is the very first "employee" she meets, Ahti, the janitor. He is the only employee immune to the Hiss without the need for an HRA (other than Jesse). Throughout the game he provides assistance at key points, and it is slowly revealed that he is yet another ancient paranatural entity with a long history of involvement with the FBC, distinct from The Board. The remainder of the details around Ahti we will dismiss as worldbuilding/lore.

Jesse eventually makes contact with and frees her brother from FBC containment, but it is too late. He has been overwhelmed by the Hiss, but in a way that is different from the rest of the FBC staff encountered previously. Polaris, also connected to Dylan, has provided some measure of protection. Dylan appears to channel Hiss communication and intent, but he is able to resist the complete zombie-like takeover that turns other Hiss victims hostile.

With the help of surviving FBC employees, Jesse isolates and contains Dylan and communicates with him from time to time in order to try and understand the Hiss so she can find a way to help him. Part of this process involves fully uncovering the extent of the FBC's study, experimentation, and manipulation of Dylan, as well as their attempts to locate and capture Jesse. This turns Jesse's ambivalence of the FBC into a much deeper distrust, further leading her to reject any implication of Directorship.

Despite her new distaste for the FBC, Jesse's personal goal and the FBC's imposed Directorship goal are now in full alignment; she must uncover the source of the Hiss and stop it in order to save her brother. This will incidentally also save the FBC, which is The Board's goal.

During this period we also learn more about Polaris' nature. Polaris and the Hiss are, in many ways, old archenemies. Polaris has witnessed the Hiss take over other realities in the past and desires to prevent their success in our reality. Polaris is itself a type of benevolent memetic thought-virus/wave-pattern (or at least, capable of producing one) that counteracts the Hiss pattern/influence. Polaris had not revealed any of this to Jesse previously.

ACT 3

Jesse eventually learns that Polaris' goals are not entirely selfless, and that the timing of Jesse's arrival with the Hiss invasion is not at all coincidental. The FBC, during their research into the events of Ordinary, had not just captured Dylan. It turns out they had also captured Polaris, who has been isolated and contained in the FBC more or less since the events of Ordinary and has been under constant study in the same way Dylan has.

Polaris first attempted to use Dr. Darling to prevent Trench from starting the Hiss invasion, but was too late. Now, Polaris has influenced Jesse to arrive at the FBC, again ostensibly in search of her brother, but at least in part also to repel the Hiss and save Polaris.

It is never explicitly stated and, as far as I know, never realized by Jesse, but the insiduous implication here is that Polaris has allowed Dylan to be affected by the Hiss to the extent he has in order to secure Jesse's continued motivation in fighting the Hiss after initially finding Dylan. No other explanation is provided as to why Polaris can fully protect Jesse from the Hiss' influence but not Dylan, despite ample evidence that Polaris' connection to Dylan is just as strong as to Jesse. Research logs reveal Dylan's mastery of similar paranatural powers as Jesse, and Jesse confirms multiple times in her internal conversations with Polaris that Polaris is connected to Dylan in similar fashion.

With the Hiss pouring into the FBC, freeing Polaris becomes Jesse's immediate concern. Unfortunately, Jesse's attempt to free Polaris fails and the entity is killed by the Hiss (or at least, the portion captured/contained by the FBC). With the death of Polaris (or that portion of her), Jesse nearly falls under the sway of Hiss influence. Ultimately she successfully resists, implying at least some of the protective nature of Polaris' pattern remains active in Jesse, preserving both her Hiss resistance and her powerful command over her paranatural abilities.

With her desire to save Dylan unchanged, Jesse finally confronts the Hiss invasion into our reality directly by locating the original OOP recovered from Ordinary, the slide projector that opens the door between these dimensions. The final battle involves defeating the Hiss' representation in our reality and closing the door, stopping the invasion in its tracks. This frees Dylan from Hiss control, but he falls into a coma, his final fate left uncertain.

EPILOGUE

In the denouement it is suggested that, over the course of this journey, Jesse comes to believe that most FBC personnel are in fact working for ethically reasonable goals and that a significant portion of the evil perpetrated on her brother, herself, and the world at large by the FBC came primarily from the Hiss-influenced involvement of ex-Director Trench.

Jesse has grown to the point that she is ready and willing to accept her title as the FBC's new Director in order to lead it down a reformed, positive path. Having saved the FBC from the Hiss attack and having uncovered Trench's treachery, she appears in the end to finally embrace the Director role.

The Board seems mostly pleased with this.

Running Minecraft with a managed console (STDIN) using systemd, tail, and mkfifo+pipes

This post shows you how to control a systemd-managed Minecraft server by connecting its console (STDIN) to a named pipe.

If you have tried to setup a Minecraft server on linux using standard systemd service units you have likely run into the same pain I have. A Minecraft server is a process that receives control commands via console input (STDIN). There is no control application you can use to send messages to the server via IPC; even the simple act of stopping a vanilla Minecraft server requires a "stop" console command sent via STDIN rather than the traditional SIGTERM that you would expect to work for a linux service.

There are many possible work-arounds to this problem. The most common I've seen, and the one I've used in the past, is to wrap the Minecraft Java process using a console "window manager" such as screen or tmux. But this solution has some downsides. These tools are not commonly installed by default, for one. Another issue is that trying to share a screen or tmux session with other users, or automate sending commands to that console, can be difficult and potentially brittle. Yet another problem is that screen or tmux controls the process' output. This means systemd cannot grab STDOUT and direct it as desired. If you prefer journalctl for your logging, for instance, losing access to the STDOUT dump is a bit of a bummer.

Perhaps most importantly, though, is that using a screen or tmux wrapper simply doesn't feel like the UNIX way. When we hear that we want to control a process via STDIN, the UNIX mind immediately jumps to pipes. Instinctually it feels like any solution not using pipes is somehow missing a better approach.

With these issues in mind, I was able to formulate a set of goals for managing my Minecraft servers that were not being met by my original screen-based solution:

• server console input should be handled through a named pipe (mkfifo); this makes it easy to apply access controls and script commands, and feels appropriately UNIX-y
• systemd should know that the "main process" is the Java process so it properly tracks when it dies
• as few non-default tools as possible should be required (ideally, none)
• the server output should remain unmodified on STDOUT so it can be sent wherever desired

I will first provide the files that enable the new solution, then go over the details of how and why it meets our criteria.

The necessary files

This method works to control any vanilla, Spigot, or PaperMC Minecraft server. It will also work on a BungeeCord server, though you will need to modify the stop command to send "end" instead.

Console commands can be issued to the server through the named pipe "/run/mcsrv/stdin". For example, running sudo -u minecraft sh -c 'echo tps > /run/mcsrv/stdin' will execute the tps command on the server console.

The systemd service unit file

[Unit]
Description=Minecraft Server
After=syslog.target network.target

[Install]
WantedBy=multi-user.target

[Service]
Type=forking
User=minecraft
Group=minecraft
Restart=no
KillMode=control-group
WorkingDirectory=/srv/minecraft
RuntimeDirectory=mcsrv
StandardOutput=journal
PIDFile=/run/mcsrv/server.pid
KillSignal=SIGCONT
FinalKillSignal=SIGTERM
ExecStartPre=mkfifo --mode=0660 /run/mcsrv/stdin
ExecStart=/bin/bash start.sh /run/mcsrv
ExecStop=/bin/sh -c 'echo stop > /run/mcsrv/stdin'

The shell script to start the server

Based on the service definition above this script should be saved as "/srv/minecraft/start.sh":

#!/bin/bash

MCDIR=$1
MCJAR=minecraft.jar
MCARG="-Xms2G -Xmx2G"

{ coproc JAVA (/usr/bin/java -server ${MCARG} -jar ${MCJAR} nogui) 1>&3; } 3>&1
echo ${JAVA_PID} > ${MCDIR}/server.pid
{ coproc TAIL (/usr/bin/tail -n +1 -f --pid=${JAVA_PID} ${MCDIR}/stdin 1>&${JAVA[1]}) } 2>/dev/null

The magic explained

Our service unit file specifies that a runtime directory should be created for this service. We create a named pipe inside the rundir using mkfifo as defined in the ExecStartPre directive. The --mode parameter for this call can be modified as needed to control the access permissions that are set on this pipe. This pipe is where we will hook the server's STDIN; anything sent to this named pipe will be received by the server console. Running sudo -u minecraft sh -c 'echo tps > /run/mcsrv/stdin', for instance, would execute the tps command via the server console.

We then start the server by executing the start.sh script. This script has what is probably our most specific requirement; it must execute with a shell that understands the coproc command. Luckily this includes bash, zsh, ksh, and a number of other very common default shell processors.

The startup script uses coproc to capture the STDIN handle when we launch the Java server process. Because this also ordinarily eats the process' STDOUT, we apply a pair of redirects to make sure that the STDOUT for the server remains attached to this script's STDOUT (which is, ultimately, what systemd will use as its interpretation of STDOUT for the overall process group). If you don't want the server output going into the systemd journal then change the StandardOutput directive to your preferred logging method.

Next we dump the server's PID into a pidfile that is also stored in the runtime directory. In our service unit we specify this file using the PIDFile directive. This allows systemd to properly associate the Java process as the main process for this service unit. If the Java process flat out dies for any reason, systemd will know. Change the Restart directive to your preferred failure behavior.

Now we need to hook our named pipe to the server's console. If you have investigated this before you may have realized how difficult it is to do this in the context of a systemd service. This is mainly owing to the semantics of a pipe being automatically closed after a single succesful writer closes its handle. A bit of research into the problem of keeping the pipe open puts us on the tail of, well, tail.

The script starts and backgrounds a tail instance that handles our pipe magic. While our server lives this tail instance constantly reads one line at a time from our named pipe and redirects this output into STDIN for the server. It is also set to "wait" on the PID of the Java process using the --pid parameter. This ensures it will clean itself up when the server stops.

To stop the service we tell systemd to send a "stop" command to the named pipe to shutdown the server cleanly. It will also send SIGCONT to the process group to make sure both tail and Java processes are not suspended. If the server fails to cleanly stop in the allowed time period systemd will then send SIGTERM to the processes. If you are running a vanilla server then this will basically force-stop the server. If you run spigot or papermc these servers will attempt to shutdown gracefully on receiving SIGTERM.

Note that the above signal methods can be modified to your preference. If you prefer to guarantee absolutely no chance of a forcibly stopped server causing data loss then you may want to change the KillMode directive to "none". This ensures that the server processes are left alone if the "stop" command doesn't do its job. The downside is the potential to leave orphaned processes that require manual intervention and cleanup. Also, if your server takes longer than the default service timeout to cleanly stop (90s) you will want to add the TimeoutStopSec directive to tell systemd to wait longer before sending the final kill signals.

Linux router conntrack settings

Linux firewall? Conntrack? NAT? Connection issues?

This is one of those "I'll never remember this again if I don't write it down" types of posts. So entirely for my own purposes, basically :)

I have a linux box acting as a firewall/NAT device for my local network. Among other things, I'm using conntrack modules for NAT connection tracking to handle proper NAT port forwards and also for firewall rules to filter active connections properly.

A few issues cropped up in weird places like Netflix, YouTube, VEVO, and other streaming services where streams would die for no apparent reason. I always just chalked this up to ephemeral internet issues and did not investigate it deeply as it was not common enough to be more than a minor, infrequent, and random inconvenience. However, a consistent, reproducible, and most irritatingly CONSTANT problem trying to watch twitch.tv streams finally got me looking into this in detail.

Now, I don't know the details of WebRTC, HLS, RTP, and all the other protocols under the hood for video streaming tunnels through HTTP. What I do know (now) is that the default timeouts in conntrack in 3.x kernels seem to be too aggressive (at least for my internet connection), causing conntrack to often drop the TCP connection tracking for computers using these streaming protocols.

The result? Random drop-outs and network connectivity problems in HTTP-based video streaming.

The fix ends up being stupid simple. I just doubled (or sometimes tripled/quadrupled) the TCP connection timeouts for conntrack and, at least so far, streaming stability has improved dramatically (and twitch.tv actually works). The new timeouts are still short enough that for my limited network size I'm in no danger of running out of conntrack entries in any reasonable timeframe.

So, to the end of my /etc/sysctl.conf file, I simply added these timeouts (Ubuntu 14 LTS system, btw):

net.netfilter.nf_conntrack_icmp_timeout = 60
net.netfilter.nf_conntrack_tcp_timeout_close = 60
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 240
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 240
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 240
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 60
net.netfilter.nf_conntrack_udp_timeout_stream = 240

Oh, and while I'm on the topic... I also updated my conntrack modules to handle a few of those irritating protocols. These work much more cleanly now with the proper conntrack handling in place (especially PPTP). To my /etc/modules I have added:

nf_conntrack_proto_gre
nf_conntrack_pptp
nf_conntrack_sip
nf_conntrack_h323
nf_conntrack_ftp
nf_nat_proto_gre
nf_nat_pptp
nf_nat_sip
nf_nat_h323
nf_nat_ftp

EVE ruminations

It's been a while since I've posted about EVE Online... or, let's be honest, rambled about it. I felt like posting a new ramble because, well... mostly because I like the sound of my own voice. But also because CCP has been through a lot of ups and downs and their future, and the future of EVE, was looking pretty shaky for a while. Recently, though, it feels as if they have finally found the right track again.

I've always found CCP and EVE to be a fascinating example of game development done... maybe not "right", but at least "well". Since I've been getting hyped about Star Citizen recently, and since it is still so very far away, I thought it might be nice to remind people that there is an excellent sci-fi/spaceship MMO out there that you can play right now :) Not only is it out there, but that it seems (once again) to be getting better every day.

(random self-serving note... if anyone reads this post and is inspired to try EVE, please be a pal and use my referral link)

I want this post to focus on where EVE seems to be going and why I'm excited about it. But it's hard to do that without taking a glance at where EVE has been.

Whereupon I offer a biased and unabashedly opinion-laced view of EVE's history...

CCP -- the creators of EVE -- are an interesting example in game development and the videogame business. They are an Icelandic company with a true "garage startup" genesis story that rose to success based almost solely on the singular strength of their idea; to make a sci-fi spaceship MMO with lasting consequences and an intrinsic PvP focus where everyone is playing in the same (and only) universe/shard. That game -- EVE Online -- launched in 2003 and has been growing ever since.

Starting in about 2011 it became undeniably apparent that the growth and success of EVE may have had as much to do with luck and the power of its niche idea than with any particular genius over at CCP. Floundering and confused choices made by CCP with regard to EVE, combined with spectacular failures in other projects, suggested that many of things CCP did right over the years might, in fact, have been accidental rather than intentional.

Powerful as the niche idea was, in 2011 EVE was already 8 years old. An idea can only take you so far. There was growing concern at this point that CCP might no longer be able to replicate the successful choices that had brought the game as far as it had come, leaving the future health of EVE in doubt. Some of these great chioces, in no particular order: self-published, a downloadable game client, no "box fee", a warm embrace of 3rd party applications and openly available game data, the introduction of PLEX (a free-to-play option that dampens gold farming while avoiding pay-to-win), complete graphic engine overhauls, the CSM, hiring an actual economist to help deal with the player-driven market... the list of intriguing business decisions alone goes on. Sure, some of this stuff seems common-place now, but at the time these things were introduced to EVE they were rarely industry-standard and often bordered on revolutionary. But quietly so.

I'm not going to discuss the things that happened leading into 2011 that made it apparent to players that CCP might not actually know what they were doing. If you are not familiar with EVE you don't care, and if you are familiar with EVE then you already know about this stuff. I will, however, mention in passing that at least one event led to CCP being the only game developer I know with an Internal Affairs division.

What it all led to was a fairly simple conclusion; CCP believed they knew better than the players what EVE was and should be, and always seemed a little bit surprised when there was any kind of player backlash. The issue, however, is that many EVE players seemed to think this was a NEW problem at CCP. They looked back at previous successful choices as compared to some of the more recent disasters and concluded that something at CCP had changed. But it hadn't.

The reality is that most of the previous successful choices made by CCP were made under the same belief that they knew better than the players. They just got lucky with those choices for a variety of reasons, so things turned out OK. There is one classic example that highlights this: wormholes.

In 2009, EVE got one of the most significant and expansive universe updates it's ever seen, leading to a whole new dedicated style of small-group gameplay that revitalized the ability for the "little guys" to make a place for themselves in EVE. It was widely applauded and led to a whole new style of emergent gameplay enjoyed by many, injecting life into the game. But pretty much everything people love most about wormholes? It was a mistake.

Wormholes were essentially intended to be dangerous PvE content for groups. They were effectively a first attempt at what was later re-introduced in the form of incursions in 2011. Nobody was supposed to setup permanent residency and actually LIVE in wormhole space. The fact that you could anchor a starbase in one and create a home for a small band of misfits was entirely accidental and a mere oversight on the part of the devs; they simply forgot to turn off the flag allowing for anchorable structures. And yet claiming wormhole space was exactly what players did, in droves. It offered a welcome outlet to groups interested in PvP and self-reliance but who were simply not large enough or didn't care enough to challenge the sovereignty and political machinations of null-sec.

That is just one instance, but really you can look at the majority of their choices and find similar themes. Titans, capital ships in general, jump drives, "gun mining", moon goo, original faction warfare... the list of poorly implemented game mechanics that were badly abused by players in unintentional ways goes WAY back. That's not even including the poor business choices. Just because most of these things turned out OK prior to 2011 didn't change the basic fact that CCP always assumed they knew better than players and always reacted to player backlash with surprise and a bit of resentment.

Oh sure, they would sometimes get around to fixing things that the players had proven beyond a shadow of doubt were completely broken or imbalanced, but it was always with a tinge of resentment from the developers, and typically MONTHS too late to prevent the exploitation of these broken features from severely impacting all aspects of the game. Further, by the time CCP would wake up to and think about addressing some of these mistakes, often so much time had gone by that the players now saw the broken mechanic as part of the status quo and resisted any attempt to improve or tweak things for the better. This was not helped by the fact that CCP seemed unusually reticient to ever admit that the first pass had BEEN a mistake. By 2011 however, CCP had made a series of bad choices, none of which could be viably salvaged in any reasonable way. In short, their luck had run out.

UphEVEal (yes I know that's not how you spell it)

The realization that they had been lucky and not genius must have come as quite a shock to CCP. Sure, they had always given lip-service to the idea that they embraced player feedback, but reality proved different. They ignored all of the most requested player features, left broken mechanics in place for far too long, introduced new bizarre mechanics from way out of left field, and generally operated in a way that showed their true colors: "CCP knows best". Only by 2011 it was finally obvious to them that if they stayed that course, EVE was going to die (for real this time!)

Watching CCP struggle for the last three years to internalize and act upon this realization has been interesting, to say the least. Again I won't bore with details here; either you follow EVE and have already witnessed the earthquake that is shaking up CCP culture, or you don't care. I'm just going to talk about the result, since the result makes me excited for the future of EVE; enough so that I actually bothered to write this blog post :)

It seems ridiculously simple, but everything that was wrong with CCP and all the hope for their future can be summed up with one simple change; their new patch cycle. Until recently, CCP operated on a 6-month patch cycle, attempting to create two "major content" releases each year. These were accompanied by grand ideas, hype, trailer videos, and all the fanfare they could muster. They have recently changed their patch cycle to a 5-week one, aiming to release 10 updates a year.

It might seem like oversimplification to boil it down to this, but all I can say is that the release cycle has an incredibly insidious and far reaching effect on every aspect of development in a company. For a developer, every feature, every interaction with players, every interaction with your managers; all of it is driven in large part by how it works into the release cycle. Likewise management is evaluating every effort and tradeoff against the release cycle as well. In EVE, the 6-month cycle lead to an 11-year problem; no little features, no big features, no iteration... just "medium" features that sounded cool in trailers.

If a feature was too big to fit into a 6-month cycle, it wasn't even attempted. This led to some of the most serious long-term issues with EVE mechanics to date; null-sec sovereignty problems, poor starbase controls, a general feeling of "static-ness", and ignoring many of the most requested player features going back a decade, such as customizable ship appearances. These are things that the developers have been saying they would like to fix for YEARS, yet somehow they never managed to sort out how to start working on those things within their 6-month cycle.

Likewise, if a feature was too small to help hype up the next 6-month release, it seems to have been thought of as "wasted effort" and was never approved for work. Ditto for iterations and tweaks on recently released features from the previous content update. This led to a horrific absence in quality-of-life improvements to essential aspects of the game, such as the user interface. As for player feedback and "mistakes"; if, as a developer, you have to convince your boss to release a patch off-cycle, you are of course going to be resentful to any player that presumes to point out something broken. You might agree it's broken, but you simply don't want to face the hell of trying to get it fixed to production. So you are probably going to argue (or be dealing with your manager arguing) that it isn't "broken enough" to warrant a patch. Which leads to a broken mechanic becoming status quo, because everyone knows it won't be touched until AT LEAST the next 6-month update. And by that point it's easier just not to touch it at all. If you are an EVE player, this should sound very, very familiar.

CCP's new focus on a more frequent, less-hyped release cycle is the closest thing you will ever see to a silver bullet in development and business choices. Of course in isolation it's not enough, but taken as a whole it's proof-positive that CCP has actually had an internal revolution that it can no longer just be "lip-service" to listen to the players. They actually intend to DO it. I'm certain that the further down the ranks of CCP employees you go, the more you find people who have always wanted to do so while also feeling the futile pain of acturally trying to do so. The new release cycle gives these day-to-day developers the tool they need to finally turn the lip-service into reality. And so far, it seems to be working.

In the last year I've seen more quality-of-life UI improvements than at any point the game's history (FYI: I've been playing since 2004). And I don't mean the "let's overhaul everything UI in Trinity and then leave half of it broken for years" type of update. I mean actual, useful, small improvements that benefit everyone. Things that are finally making it into releases because there is no longer a need to tie every release to a marketing push. More importantly, these improvements have been iterated!

The tooltip update not quite what you wanted? No problem, we'll listen to all that player feedback and actually TWEAK IT with improvements 5 weeks later! I can only imagine how empowering it must feel as a developer at CCP today, being able to finally ACT on the things people are complaining about instead of simply being helpless to respond.

But it's not just little things... for the first time in, well, ever, CCP is finally making tangible progress on some of the largest features that, until now, they had only ever talked about. Customizable ship appearances, starbase overhauls, and changes to nullsec sovereignty. All three of these things had previously been projects so large that they were never attempted.

The funny thing is that at least one of these major changes has EVE players freaking out, because they perceive it as a reversion to the old "CCP knows best" mentality that has burned them so badly over the years. This is the power projection changes being introduced in the next patch cycle (Phoebe) to start working on the null-sec sovereignty issues. Superficially, these changes feel very much like the CCP of old; a set of changes that nobody quite wants or understands, fraught with unpredictable consequences which CCP has not fully thought out, all while players point out significant potential flaws on the forums that devs only seem to tentatively take to heart.

But the likeness is, thankfully, only superficial. What these players are not realizing is two important facets; one, that the devs, after literally years of doing nothing -- paralyzed by the requirement to get a nullsec overhaul done right, the first time, in no more than 6 months -- have finally actually made the first tangible progress in trying to change an aspect of the game that is widely regarded as needing improvement. It honestly doesn't matter if they get it right out of the gate. The fact that they feel the freedom to attempt anything at all is a huge improvement over the previous state of affairs.

Secondly, and more importantly, is the fact that CCP has proven over the last year that they are no longer in the "release and ignore" mentality of previous updates. They have repeatedly iterated on new features rapidly under the new patch cycle, and the 5-week timeframe allows them to do so before those changes become an entrenched part of the status quo. It's far easier as a developer to admit that something was not quite right (or even, gasp, a mistake) when you get to fix it almost immediately than it is when your ego is on the line for the next six months.

What this means for Phoebe is simple; I have no doubt the initial release is going to be not quite perfect. I don't expect CCP to get this one any better than any of the other massive, unpredictable changes they've tried to bring to EVE in the past. What I do believe, however, is that they will finally be able to adapt on the fly. When Phoebe goes live they'll be able to watch the impact and iterate quickly to tweak quickly as needed. That isn't something to freak out over, it's something to be excited about.