Wherever you go, there you are.

Running Minecraft with a managed console (STDIN) using systemd, tail, and mkfifo+pipes

1

This post shows you how to control a systemd-managed Minecraft server by connecting its console (STDIN) to a named pipe.

If you have tried to setup a Minecraft server on linux using standard systemd service units you have likely run into the same pain I have. A Minecraft server is a process that receives control commands via console input (STDIN). There is no control application you can use to send messages to the server via IPC; even the simple act of stopping a vanilla Minecraft server requires a "stop" console command sent via STDIN rather than the traditional SIGTERM that you would expect to work for a linux service.

a crane lifting a pipe

There are many possible work-arounds to this problem. The most common I've seen, and the one I've used in the past, is to wrap the Minecraft Java process using a console "window manager" such as screen or tmux. But this solution has some downsides. These tools are not commonly installed by default, for one. Another issue is that trying to share a screen or tmux session with other users, or automate sending commands to that console, can be difficult and potentially brittle. Yet another problem is that screen or tmux controls the process' output. This means systemd cannot grab STDOUT and direct it as desired. If you prefer journalctl for your logging, for instance, losing access to the STDOUT dump is a bit of a bummer.

Perhaps most importantly, though, is that using a screen or tmux wrapper simply doesn't feel like the UNIX way. When we hear that we want to control a process via STDIN, the UNIX mind immediately jumps to pipes. Instinctually it feels like any solution not using pipes is somehow missing a better approach.

With these issues in mind, I was able to formulate a set of goals for managing my Minecraft servers that were not being met by my original screen-based solution:

  • server console input should be handled through a named pipe (mkfifo); this makes it easy to apply access controls and script commands, and feels appropriately UNIX-y
  • systemd should know that the "main process" is the Java process so it properly tracks when it dies
  • as few non-default tools as possible should be required (ideally, none)
  • the server output should remain unmodified on STDOUT so it can be sent wherever desired

I will first provide the files that enable the new solution, then go over the details of how and why it meets our criteria.

The necessary files

This method works to control any vanilla, Spigot, or PaperMC Minecraft server. It will also work on a BungeeCord server, though you will need to modify the stop command to send "end" instead.

Console commands can be issued to the server through the named pipe "/run/mcsrv/stdin". For example, running sudo -u minecraft sh -c 'echo tps > /run/mcsrv/stdin' will execute the tps command on the server console.

The systemd service unit file

[Unit]
Description=Minecraft Server
After=syslog.target network.target

[Install]
WantedBy=multi-user.target

[Service]
Type=forking
User=minecraft
Group=minecraft
Restart=no
KillMode=control-group
WorkingDirectory=/srv/minecraft
RuntimeDirectory=mcsrv
StandardOutput=journal
PIDFile=/run/mcsrv/server.pid
KillSignal=SIGCONT
FinalKillSignal=SIGTERM
ExecStartPre=mkfifo --mode=0660 /run/mcsrv/stdin
ExecStart=/bin/bash start.sh /run/mcsrv
ExecStop=/bin/sh -c 'echo stop > /run/mcsrv/stdin'

The shell script to start the server

Based on the service definition above this script should be saved as "/srv/minecraft/start.sh":

#!/bin/bash

MCDIR=$1
MCJAR=minecraft.jar
MCARG="-Xms2G -Xmx2G"

{ coproc JAVA (/usr/bin/java -server ${MCARG} -jar ${MCJAR} nogui) 1>&3; } 3>&1
echo ${JAVA_PID} > ${MCDIR}/server.pid
{ coproc TAIL (/usr/bin/tail -n +1 -f --pid=${JAVA_PID} ${MCDIR}/stdin 1>&${JAVA[1]}) } 2>/dev/null

The magic explained

Our service unit file specifies that a runtime directory should be created for this service. We create a named pipe inside the rundir using mkfifo as defined in the ExecStartPre directive. The --mode parameter for this call can be modified as needed to control the access permissions that are set on this pipe. This pipe is where we will hook the server's STDIN; anything sent to this named pipe will be received by the server console. Running sudo -u minecraft sh -c 'echo tps > /run/mcsrv/stdin', for instance, would execute the tps command via the server console.

We then start the server by executing the start.sh script. This script has what is probably our most specific requirement; it must execute with a shell that understands the coproc command. Luckily this includes bash, zsh, ksh, and a number of other very common default shell processors.

The startup script uses coproc to capture the STDIN handle when we launch the Java server process. Because this also ordinarily eats the process' STDOUT, we apply a pair of redirects to make sure that the STDOUT for the server remains attached to this script's STDOUT (which is, ultimately, what systemd will use as its interpretation of STDOUT for the overall process group). If you don't want the server output going into the systemd journal then change the StandardOutput directive to your preferred logging method.

Next we dump the server's PID into a pidfile that is also stored in the runtime directory. In our service unit we specify this file using the PIDFile directive. This allows systemd to properly associate the Java process as the main process for this service unit. If the Java process flat out dies for any reason, systemd will know. Change the Restart directive to your preferred failure behavior.

a graph of nodes

Now we need to hook our named pipe to the server's console. If you have investigated this before you may have realized how difficult it is to do this in the context of a systemd service. This is mainly owing to the semantics of a pipe being automatically closed after a single succesful writer closes its handle. A bit of research into the problem of keeping the pipe open puts us on the tail of, well, tail.

The script starts and backgrounds a tail instance that handles our pipe magic. While our server lives this tail instance constantly reads one line at a time from our named pipe and redirects this output into STDIN for the server. It is also set to "wait" on the PID of the Java process using the --pid parameter. This ensures it will clean itself up when the server stops.

To stop the service we tell systemd to send a "stop" command to the named pipe to shutdown the server cleanly. It will also send SIGCONT to the process group to make sure both tail and Java processes are not suspended. If the server fails to cleanly stop in the allowed time period systemd will then send SIGTERM to the processes. If you are running a vanilla server then this will basically force-stop the server. If you run spigot or papermc these servers will attempt to shutdown gracefully on receiving SIGTERM.

Note that the above signal methods can be modified to your preference. If you prefer to guarantee absolutely no chance of a forcibly stopped server causing data loss then you may want to change the KillMode directive to "none". This ensures that the server processes are left alone if the "stop" command doesn't do its job. The downside is the potential to leave orphaned processes that require manual intervention and cleanup. Also, if your server takes longer than the default service timeout to cleanly stop (90s) you will want to add the TimeoutStopSec directive to tell systemd to wait longer before sending the final kill signals.

Linux router conntrack settings

0

Linux firewall? Conntrack? NAT? Connection issues?

This is one of those "I'll never remember this again if I don't write it down" types of posts. So entirely for my own purposes, basically :)

I have a linux box acting as a firewall/NAT device for my local network. Among other things, I'm using conntrack modules for NAT connection tracking to handle proper NAT port forwards and also for firewall rules to filter active connections properly.

A few issues cropped up in weird places like Netflix, YouTube, VEVO, and other streaming services where streams would die for no apparent reason. I always just chalked this up to ephemeral internet issues and did not investigate it deeply as it was not common enough to be more than a minor, infrequent, and random inconvenience. However, a consistent, reproducible, and most irritatingly CONSTANT problem trying to watch twitch.tv streams finally got me looking into this in detail.

Now, I don't know the details of WebRTC, HLS, RTP, and all the other protocols under the hood for video streaming tunnels through HTTP. What I do know (now) is that the default timeouts in conntrack in 3.x kernels seem to be too aggressive (at least for my internet connection), causing conntrack to often drop the TCP connection tracking for computers using these streaming protocols.

The result? Random drop-outs and network connectivity problems in HTTP-based video streaming.

The fix ends up being stupid simple. I just doubled (or sometimes tripled/quadrupled) the TCP connection timeouts for conntrack and, at least so far, streaming stability has improved dramatically (and twitch.tv actually works). The new timeouts are still short enough that for my limited network size I'm in no danger of running out of conntrack entries in any reasonable timeframe.

So, to the end of my /etc/sysctl.conf file, I simply added these timeouts (Ubuntu 14 LTS system, btw):

net.netfilter.nf_conntrack_icmp_timeout = 60
net.netfilter.nf_conntrack_tcp_timeout_close = 60
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 240
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 240
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 240
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 60
net.netfilter.nf_conntrack_udp_timeout_stream = 240

Oh, and while I'm on the topic... I also updated my conntrack modules to handle a few of those irritating protocols. These work much more cleanly now with the proper conntrack handling in place (especially PPTP). To my /etc/modules I have added:

nf_conntrack_proto_gre
nf_conntrack_pptp
nf_conntrack_sip
nf_conntrack_h323
nf_conntrack_ftp
nf_nat_proto_gre
nf_nat_pptp
nf_nat_sip
nf_nat_h323
nf_nat_ftp

EVE ruminations

2

It's been a while since I've posted about EVE Online... or, let's be honest, rambled about it. I felt like posting a new ramble because, well... mostly because I like the sound of my own voice. But also because CCP has been through a lot of ups and downs and their future, and the future of EVE, was looking pretty shaky for a while. Recently, though, it feels as if they have finally found the right track again.

I've always found CCP and EVE to be a fascinating example of game development done... maybe not "right", but at least "well". Since I've been getting hyped about Star Citizen recently, and since it is still so very far away, I thought it might be nice to remind people that there is an excellent sci-fi/spaceship MMO out there that you can play right now :) Not only is it out there, but that it seems (once again) to be getting better every day.

(random self-serving note... if anyone reads this post and is inspired to try EVE, please be a pal and use my referral link)

I want this post to focus on where EVE seems to be going and why I'm excited about it. But it's hard to do that without taking a glance at where EVE has been.

Whereupon I offer a biased and unabashedly opinion-laced view of EVE's history...

CCP -- the creators of EVE -- are an interesting example in game development and the videogame business. They are an Icelandic company with a true "garage startup" genesis story that rose to success based almost solely on the singular strength of their idea; to make a sci-fi spaceship MMO with lasting consequences and an intrinsic PvP focus where everyone is playing in the same (and only) universe/shard. That game -- EVE Online -- launched in 2003 and has been growing ever since.

Starting in about 2011 it became undeniably apparent that the growth and success of EVE may have had as much to do with luck and the power of its niche idea than with any particular genius over at CCP. Floundering and confused choices made by CCP with regard to EVE, combined with spectacular failures in other projects, suggested that many of things CCP did right over the years might, in fact, have been accidental rather than intentional.

Powerful as the niche idea was, in 2011 EVE was already 8 years old. An idea can only take you so far. There was growing concern at this point that CCP might no longer be able to replicate the successful choices that had brought the game as far as it had come, leaving the future health of EVE in doubt. Some of these great chioces, in no particular order: self-published, a downloadable game client, no "box fee", a warm embrace of 3rd party applications and openly available game data, the introduction of PLEX (a free-to-play option that dampens gold farming while avoiding pay-to-win), complete graphic engine overhauls, the CSM, hiring an actual economist to help deal with the player-driven market... the list of intriguing business decisions alone goes on. Sure, some of this stuff seems common-place now, but at the time these things were introduced to EVE they were rarely industry-standard and often bordered on revolutionary. But quietly so.

I'm not going to discuss the things that happened leading into 2011 that made it apparent to players that CCP might not actually know what they were doing. If you are not familiar with EVE you don't care, and if you are familiar with EVE then you already know about this stuff. I will, however, mention in passing that at least one event led to CCP being the only game developer I know with an Internal Affairs division.

What it all led to was a fairly simple conclusion; CCP believed they knew better than the players what EVE was and should be, and always seemed a little bit surprised when there was any kind of player backlash. The issue, however, is that many EVE players seemed to think this was a NEW problem at CCP. They looked back at previous successful choices as compared to some of the more recent disasters and concluded that something at CCP had changed. But it hadn't.

The reality is that most of the previous successful choices made by CCP were made under the same belief that they knew better than the players. They just got lucky with those choices for a variety of reasons, so things turned out OK. There is one classic example that highlights this: wormholes.

In 2009, EVE got one of the most significant and expansive universe updates it's ever seen, leading to a whole new dedicated style of small-group gameplay that revitalized the ability for the "little guys" to make a place for themselves in EVE. It was widely applauded and led to a whole new style of emergent gameplay enjoyed by many, injecting life into the game. But pretty much everything people love most about wormholes? It was a mistake.

Wormholes were essentially intended to be dangerous PvE content for groups. They were effectively a first attempt at what was later re-introduced in the form of incursions in 2011. Nobody was supposed to setup permanent residency and actually LIVE in wormhole space. The fact that you could anchor a starbase in one and create a home for a small band of misfits was entirely accidental and a mere oversight on the part of the devs; they simply forgot to turn off the flag allowing for anchorable structures. And yet claiming wormhole space was exactly what players did, in droves. It offered a welcome outlet to groups interested in PvP and self-reliance but who were simply not large enough or didn't care enough to challenge the sovereignty and political machinations of null-sec.

That is just one instance, but really you can look at the majority of their choices and find similar themes. Titans, capital ships in general, jump drives, "gun mining", moon goo, original faction warfare... the list of poorly implemented game mechanics that were badly abused by players in unintentional ways goes WAY back. That's not even including the poor business choices. Just because most of these things turned out OK prior to 2011 didn't change the basic fact that CCP always assumed they knew better than players and always reacted to player backlash with surprise and a bit of resentment.

Oh sure, they would sometimes get around to fixing things that the players had proven beyond a shadow of doubt were completely broken or imbalanced, but it was always with a tinge of resentment from the developers, and typically MONTHS too late to prevent the exploitation of these broken features from severely impacting all aspects of the game. Further, by the time CCP would wake up to and think about addressing some of these mistakes, often so much time had gone by that the players now saw the broken mechanic as part of the status quo and resisted any attempt to improve or tweak things for the better. This was not helped by the fact that CCP seemed unusually reticient to ever admit that the first pass had BEEN a mistake. By 2011 however, CCP had made a series of bad choices, none of which could be viably salvaged in any reasonable way. In short, their luck had run out.

UphEVEal (yes I know that's not how you spell it)

The realization that they had been lucky and not genius must have come as quite a shock to CCP. Sure, they had always given lip-service to the idea that they embraced player feedback, but reality proved different. They ignored all of the most requested player features, left broken mechanics in place for far too long, introduced new bizarre mechanics from way out of left field, and generally operated in a way that showed their true colors: "CCP knows best". Only by 2011 it was finally obvious to them that if they stayed that course, EVE was going to die (for real this time!)

Watching CCP struggle for the last three years to internalize and act upon this realization has been interesting, to say the least. Again I won't bore with details here; either you follow EVE and have already witnessed the earthquake that is shaking up CCP culture, or you don't care. I'm just going to talk about the result, since the result makes me excited for the future of EVE; enough so that I actually bothered to write this blog post :)

It seems ridiculously simple, but everything that was wrong with CCP and all the hope for their future can be summed up with one simple change; their new patch cycle. Until recently, CCP operated on a 6-month patch cycle, attempting to create two "major content" releases each year. These were accompanied by grand ideas, hype, trailer videos, and all the fanfare they could muster. They have recently changed their patch cycle to a 5-week one, aiming to release 10 updates a year.

It might seem like oversimplification to boil it down to this, but all I can say is that the release cycle has an incredibly insidious and far reaching effect on every aspect of development in a company. For a developer, every feature, every interaction with players, every interaction with your managers; all of it is driven in large part by how it works into the release cycle. Likewise management is evaluating every effort and tradeoff against the release cycle as well. In EVE, the 6-month cycle lead to an 11-year problem; no little features, no big features, no iteration... just "medium" features that sounded cool in trailers.

If a feature was too big to fit into a 6-month cycle, it wasn't even attempted. This led to some of the most serious long-term issues with EVE mechanics to date; null-sec sovereignty problems, poor starbase controls, a general feeling of "static-ness", and ignoring many of the most requested player features going back a decade, such as customizable ship appearances. These are things that the developers have been saying they would like to fix for YEARS, yet somehow they never managed to sort out how to start working on those things within their 6-month cycle.

Likewise, if a feature was too small to help hype up the next 6-month release, it seems to have been thought of as "wasted effort" and was never approved for work. Ditto for iterations and tweaks on recently released features from the previous content update. This led to a horrific absence in quality-of-life improvements to essential aspects of the game, such as the user interface. As for player feedback and "mistakes"; if, as a developer, you have to convince your boss to release a patch off-cycle, you are of course going to be resentful to any player that presumes to point out something broken. You might agree it's broken, but you simply don't want to face the hell of trying to get it fixed to production. So you are probably going to argue (or be dealing with your manager arguing) that it isn't "broken enough" to warrant a patch. Which leads to a broken mechanic becoming status quo, because everyone knows it won't be touched until AT LEAST the next 6-month update. And by that point it's easier just not to touch it at all. If you are an EVE player, this should sound very, very familiar.

CCP's new focus on a more frequent, less-hyped release cycle is the closest thing you will ever see to a silver bullet in development and business choices. Of course in isolation it's not enough, but taken as a whole it's proof-positive that CCP has actually had an internal revolution that it can no longer just be "lip-service" to listen to the players. They actually intend to DO it. I'm certain that the further down the ranks of CCP employees you go, the more you find people who have always wanted to do so while also feeling the futile pain of acturally trying to do so. The new release cycle gives these day-to-day developers the tool they need to finally turn the lip-service into reality. And so far, it seems to be working.

In the last year I've seen more quality-of-life UI improvements than at any point the game's history (FYI: I've been playing since 2004). And I don't mean the "let's overhaul everything UI in Trinity and then leave half of it broken for years" type of update. I mean actual, useful, small improvements that benefit everyone. Things that are finally making it into releases because there is no longer a need to tie every release to a marketing push. More importantly, these improvements have been iterated!

The tooltip update not quite what you wanted? No problem, we'll listen to all that player feedback and actually TWEAK IT with improvements 5 weeks later! I can only imagine how empowering it must feel as a developer at CCP today, being able to finally ACT on the things people are complaining about instead of simply being helpless to respond.

But it's not just little things... for the first time in, well, ever, CCP is finally making tangible progress on some of the largest features that, until now, they had only ever talked about. Customizable ship appearances, starbase overhauls, and changes to nullsec sovereignty. All three of these things had previously been projects so large that they were never attempted.

The funny thing is that at least one of these major changes has EVE players freaking out, because they perceive it as a reversion to the old "CCP knows best" mentality that has burned them so badly over the years. This is the power projection changes being introduced in the next patch cycle (Phoebe) to start working on the null-sec sovereignty issues. Superficially, these changes feel very much like the CCP of old; a set of changes that nobody quite wants or understands, fraught with unpredictable consequences which CCP has not fully thought out, all while players point out significant potential flaws on the forums that devs only seem to tentatively take to heart.

But the likeness is, thankfully, only superficial. What these players are not realizing is two important facets; one, that the devs, after literally years of doing nothing -- paralyzed by the requirement to get a nullsec overhaul done right, the first time, in no more than 6 months -- have finally actually made the first tangible progress in trying to change an aspect of the game that is widely regarded as needing improvement. It honestly doesn't matter if they get it right out of the gate. The fact that they feel the freedom to attempt anything at all is a huge improvement over the previous state of affairs.

Secondly, and more importantly, is the fact that CCP has proven over the last year that they are no longer in the "release and ignore" mentality of previous updates. They have repeatedly iterated on new features rapidly under the new patch cycle, and the 5-week timeframe allows them to do so before those changes become an entrenched part of the status quo. It's far easier as a developer to admit that something was not quite right (or even, gasp, a mistake) when you get to fix it almost immediately than it is when your ego is on the line for the next six months.

What this means for Phoebe is simple; I have no doubt the initial release is going to be not quite perfect. I don't expect CCP to get this one any better than any of the other massive, unpredictable changes they've tried to bring to EVE in the past. What I do believe, however, is that they will finally be able to adapt on the fly. When Phoebe goes live they'll be able to watch the impact and iterate quickly to tweak quickly as needed. That isn't something to freak out over, it's something to be excited about.

Cyrptocurrency Revisted

0

I just wanted to make a quick follow-up to my original cyrptocurrency post from a few weeks back. This just has a few follow-up links and additional specifics I've come across since that time.

I forgot to mention last week that if you want to participate on the Litecoin or Bitcoin networks and have your own personally managed wallet address (e.g. not a wallet on an external exhange or site, like Cryptsy), then the simplest way to do that is to grab the offical bitcoin-qt client (for BTC) and the official litecoin-qt client (for LTC). These are p2p clients that enable the underlying networks of their respective cryptocurrencies, but they also allow you to create wallet addresses for receiving deposits and making transactions.

If you are particularly concerned about your wallet's security (or are just a very paranoid person :), consider using Armory for your BTC wallet. It pulls data from a locally running bitcoin-qt client under the hood and leaves all of the p2p and networking details to that software. Armory instead focuses on managing your wallet addresses as securely as possible. It has many advanced features such as options for wallet recovery and truly "offline" wallets that can be stored on a computer not connected to the internet at all. These offline wallets require physical access to the machine in order to sign/verify transactions (which you can then send out using a "read-only" online wallet counterpart).

On the mining side of things, Trade My Bit, the multipool I use for mining, has added several new features. Most notable is an automatic exchange that allows you to opt-in to a centralized and managed sale of the multipool mining currencies. Yes, there's a small fee for it, but it means not having to worry about linking up to Cryptsy (or your preferred exchange) for every single currency or having to directly manage a ton of microtrades if you don't want to be bothered with the minutae. You can just get a regular BTC payout from the shared exchange pool at the end of the day. Definitely a very cool feature and the devs there have been very active in updating the pool in general.

If you are using AMD GPU's for your mining efforts, this guide is a detailed step-by-step process for setting up a headless cgminer on Ubuntu. And this optimization guide will help you get the best possible hashrates out of your Radeons.

Lastly, for the AMD crowd using cgminer you are probably aware that 3.7.2 is the final version that supports GPU mining; all further releases of cgminer are dedicated to bitcoin mining on ASIC's and drop GPU mining support. However, someone has taken up the banner and forked cgminer into sgminer. It provides on-going development of and bugfixes for the GPU scrypt mining capabilities in the 3.7 branch of cgminer.