Category Archives: Gaming

Programming Game Review: SpaceChem (2011) by Zachtronics Industries

Introduction

I bought SpaceChem years ago, but only launched it once. It did not immediately grab my attention, so it spent years unplayed in my Steam library. A few months ago, though, I came across a presentation by Zach Barth, the mastermind behind SpaceChem and the subsequent games released by Zachtronics. He talked about the design philosophy behind their games and briefly covered aspects where he thought SpaceChem went wrong. He pointed out that puzzles in Zachtronics games have multiple solutions, which may also have to take tradeoffs into account, which is similar to engineering problems. That is true for all games Zachtronics has released. What is not true for all of them is that they are punishingly difficult. Yet, when Zach Barth mentioned that less than two percent of players were able to finish the story mode of SpaceChem, I was intrigued.

The main reason I am put off by most video games is that they do not offer much of a challenge. Instead, they are a sedative grind, rewarding you for the time spent doing mundane tasks instead of pushing your skill as a player. To illustrate this, think of the vast majority of games that offer “RPG elements”, where the stats that define your character grow. You do not get any better at the game. Instead, your avatar becomes more powerful. In comparison, to advance in a skill-based platforming game — think of Super Mario games — you need to learn how to control your avatar better. Furthermore, back in the days of the Nintendo Entertainment System, games were normally punishingly difficult, i.e. “Nintendo hard”, which was a consequence of memory limitations. Developers could not get a lot of content into a game as memory was limited. That kind of challenge was largely one of execution, however. SpaceChem feels like a throwback to those days, but only in the abstract. While there are almost no demands on mechanical skills, the game tests you in other ways, namely abstract thinking. Just like with a Nintendo-hard game from decades ago, you need a tremendous amount of persistence.

As a full disclosure, unlike the vast majority of people who have reviewed SpaceChem online, I am part of the very small minority of players who did finish the campaign, and then some. I’d say I took the game as seriously as a part-time job as I found it extremely captivating. I played through the story mode, the Corvi 63 DLC, and the Moustachium bonus levels in the span of about two months. During that time, I took two breaks of about one week each, one after a boss fight I thought was poorly implemented, which temporarily sapped my enthusiasm for SpaceChem (details later), and another when I got stuck at what is arguably the most difficult problem in the game (Omega-Pseudoethyne). I took some time off because I did not make any progress on it for a while. I completed all in-game challenges. I did not consult any outside resources, neither solutions nor hints. With very few exceptions, my solutions are far above average, and commonly in the top quintile if not the top few percentiles in terms of cycles used. I don’t mention this to brag, but to give the reader some background. If anything, professional game journalism would be a tad less useless if reviewers stated how competent they are at a given game, or at least how familiar they are with the genre the game they have been reviewing belongs to. I quit playing SpaceChem after solving a few of the curated user-generated puzzles of the ResearchNET section, for reasons which I will mention below.

I put more than 100 hours into SpaceChem, which makes it my second most-played game in my life, after Tetris in various iterations (original Game Boy, NES, TetrisFriends.com, and Puyo Puyo Tetris on Nintendo Switch). I can’t give a precise figure because I played the game offline, so the Steam application could not track the time. Also, even if the Steam client had been online, a significant part of my play time consisted of me closing my laptop and working solutions out with pencil and paper.

Basics of SpaceChem

SpaceChem is a “design puzzle” game, according to Zach Barth. You progress through a series of puzzles that are connected with a story that may as well not exist as it is rather disconnected from the actual gameplay. In the game, you transform inputs into outputs using a visual programming language. The input are atoms and molecules, and so is the output. According to the story, you work for a corporation called SpaceChem that exploits various planets for resources. A lot of people die, and for some reason there are enormously large monsters to defeat as well. Yet, from one stage to another, you either write functions (regular levels) or programs (production levels). Below is a screenshot of a regular level, which looks like nothing as a static image. Yet, you can probably notice that two operators, a red an a blue circle, assemble a molecule from atoms.

This is a relatively straight-forward regular level of SpaceChem

One step above are production levels, where multiple reactors, which all execute one function, are connected via pipes. This adds further difficulties, because pipes can be clogged. The entire execution path crashes if one reactor produces its output faster than the subsequent one is able to consume it.

This is a production level. The reactor on the left produces input for the reactors on the right.

This was the bird’s eye view. In more detail, you work with a (functional) visual programming language. Interestingly, that language, at least when used in production levels, has been shown to be Turing-complete by implementing a Brainf*ck interpreter in it. Each function executes two “processes”, which run concurrently and indefinitely. These processes are visualized by circles. Input shows up in the fields ‘alpha’ and ‘beta’, and output has to be delivered, in most levels, to fields ‘omega’ and ‘psi.’ To do so, you create a path for your process markers to run on. Along those paths, you place operators, which start out being quite limited: you can request an input element, grab or drop it, bond it to other elements, or remove bonds. Within a few levels, though, you end up creating quite complex programs with subroutines, conditional expressions, and flip-flops (and more!). You may not like that the game does not properly explain what some operators can achieve. Instead, you are supposed to figure this out yourself and push those operators to their limits. In fact, if the game told you some of the supposed “tricks” it expects you to discover, quite a few of the stages would be boring routine work.

You will face much more complicated problems than this one.

You may wonder why you might want to play SpaceChem. Well, if you subscribe to the school of thought according to which leaning a new programming language, even one you will never use, will make you a better programmer, then there’s your answer already! In SpaceChem, you have to wrap your head around a different way of expressing computational primitives, break down problems, translate them into the abstractions you have at your disposal, and even do some software architecture by combining reactors to achieve a particular task. You program with a different set of tools. On top, you get to practice your understanding of concurrency and multi-core programming, the former quite concretely, and the latter more in the abstract. True, SpaceChem will not get you a job, but if you manage to play through it, you are most definitely smart enough to be a very effective software engineer. I don’t know much about process engineering, but some people claim that SpaceChem is quite similar to that field as well. In any case, SpaceChem helps you keep your mind sharp, and that is arguably worth a lot.

A Game for the Cognitive 1%?

SpaceChem is brutally difficult. Zach Barth even called it “stupidly difficult.” Fully understanding the programming model is part of it, which probably already excludes a good 90% of the population. Another part, and this is where a bit of trial and error comes into place, is fitting your solution into the available space, which intensively tasks your ability to reason spatially, probably excluding another 5% of the population. Thus, you end up with a game that only the smartest 5% of the population, and that is probably too high a percentage, could play through, due to the demands on abstract thinking and spatial reasoning required. If you enjoy abstract reasoning, chances are that you will love SpaceChem. Otherwise, you will probably drop the game within the first few stages; if you are in doubt, better play the free demo first. In later games, Zachtronics fundamentally changed their approach to difficulty, which I will talk about further down.

Completion rates of SpaceChem, according to data by the developer (Fair Use is assumed).

However, you not only have to be smart, you also have to be very dedicated. In essence, the game presents you with a few dozen problems, one more difficult than the next, and gives you almost no guidance. There are a few text screens, which cover the basics. Yet, the techniques to solve the more difficult levels you have to discover yourself. The game does treat you like an intelligent adult. It gives you a problem, and tells you to figure it out. Modern games do not do that, which might explain why they can be so incredibly tedious to play. For instance, the last time I played mainstream video games, which were GTA V and The Witcher 3, I spent most of my time following a marker on a map, which I found quite off-putting (I put my PlayStation 4 up for sale soon afterward).

Sometimes, you find a solution to a problem that you cannot implement because you are not able to route your processes given the constraints on space. This can largely be overcome with practice. After about one third into the game, I got to the point where I was able to visualize solutions to SpaceChem problems in my head. I could think in the SpaceChem language, but even that has limitations. For later production levels, you will probably have to use pencil and paper to break the problem down because you would juggle too much information in your head. This is not an exaggeration. For one of those levels, I used up five or six pages of scrap paper. That level alone took me about eight hours to solve, at which point I doubted my abilities. Yet, after I learned that less than five percent of players even got to that point (you can eyeball absolute completion rates with the plot above), and that it was quite common that the more difficult levels take many, many hours, I resigned myself to the fact that that level really is that difficult. Some levels take ten hours or more; the infamously difficult last level of the game took some people weeks or months, which probably amounts to a few dozen hours in play time.

While the game does not track how long it took people to solve problems, it takes other metrics. Well, tracking time would be imperfect anyway because you may well find the solution to a problem in the shower, or through working it out with pencil and paper. Instead of time, the game measures the number of cycles your solution takes, the number of symbols you have used, and the number of reactors. This only makes sense because there are many possible solutions to each problem. It would not work with traditional puzzle games that only have one solution, for instance Portal or The Witness. Metrics are illustrated with histograms. Sometimes, the fastest solution has the fewest symbols, but in general this statement does not hold. You can certainly come up with a very slow solution that uses an excessive amount of symbols. I found those histograms quite motivating, because they make you not just want to solve a level, but to also do it well, “compared to other engineers”, as the results screen states. I think the game deliberately breaks the fourth wall at this point. While you are a “reaction engineer” in the game, the game itself is mostly played by engineers, so your performance is compared against real-life engineers.

Making it through the story mode without consulting outside sources is quite a feat. I managed to do so. It may sound bizarre, but playing through SpaceChem on my own was much more difficult than any course I have taken at university. Note that I hold a Master’s degree in Computer Science from a decent technical university, so the issue is not that I have never encountered challenging material. Yet, compared to the dedication and creativity some of the later levels of SpaceChem require, any problem I encountered in upper division courses like Discrete Optimization or Algorithm Design seemed almost trivial in comparison. It is highly likely that your day job, no matter what it is, is not nearly as difficult as SpaceChem. Well, in the end you don’t get anything for finishing SpaceChem, but it if you like solving challenging problems, it will feel highly satisfying.

Sometimes people mention SpaceChem as a good game to introduce people to programming. I would not recommend that at all because an introductory programming course is laughably easy in comparison. If anything, it is a game you may want to introduce a subset of programmers to. If you can make it through SpaceChem, you should have no problem at all to make it through a technical degree, based on demonstrated intellectual ability and perseverance. When I researched what other people thought of that connection, I came across a blog post by an academic in educational research at King’s College London who bluntly stated that he would not use SpaceChem at university as the game is far too difficult. Later Zachtronics games are probably suitable for that purpose. Still, I think if you want to learn how to program, you should just learn how to program. The barrier to entry is basically non-existent at this point in time, given that you can learn a language like Python or JavaScript with free, interactive online tutorials.

The achievement completion percentages on Steam you can ignore, by the way. It currently lists that around five percent of players have completed the story. I would be interested in the real numbers, but the rampant use of tools for unlocking Steam achievements make them relatively useless. In that regard, it arguably did not help that SpaceChem has been included in indie bundles, as this exposes this game to people who are more interested in harvesting Steam achievements, trading Steam cards, and other items on, which can be used for bartering, or even traded for money.

Flaws of SpaceChem

As satisfying as SpaceChem is, it does have a few flaws. I think that the story is superfluous. The game would have worked just as well without it. The soundtrack is a bit odd. It consists of a few pieces that are several minutes long, which are repeated ad infinitum. While you can solve the earlier puzzles in a few minutes, many of the later ones will take multiple hours. Thus, you will quickly get to the point where you will disable the soundtrack. Just like it is with the story, the soundtrack is thus quite superfluous. The same is true for the sound effects.

You may also find some of the later levels tedious. There are a few that are conceptually simple to solve, yet challenge you because it is difficult to accommodate the necessary operators on the restricted playing field. Debugging such levels can test your patience and determination. Still, once you solve them, you will feel every bit as accomplished — in particular when your first working solution is one of the fastest recorded ones. This happened to me a few times, and it served as a powerful confirmation that I had mastered the mechanics of SpaceChem. That being said, the story mode levels almost never felt like routine work. The same cannot be said for some of the curated user-created levels, but those should be viewed as bonus material for those who just cannot get enough of this game.

SpaceChem only allows you to keep one active solution per level. It is not possible to save a solution and reload it later. This may consequently discourage experimentation in more difficult levels where the play area is packed with symbols. Zach Barth has mentioned this flaw in various talks. In later Zachtronics games, it is possible to save and reload solutions. Some players may find that it is a flaw that progress in SpaceChem happens mostly in a mostly linear fashion. In order to access level n, you need to have solved level n – 1. There are very few situations where you are free to choose the order in which you tackle stages. In later Zachtronics games, you have a lot more freedom in that regard. Yet, I view this limitation as a positive as it forces you to focus on one problem at a time. Some take significant dedication. You also get the sense that you have to focus on the task at hand as there is no alternative stage to work on instead. I think there is the risk that some players would jump between puzzles, solve neither, and abandon the game if non-linearity was not enforced as much as it is. Seeing that completion rates in later Zachtronics games are also quite low, this is not just baseless conjecture.

In particular for later production levels, it would have been helpful to have the option to use placeholder inputs to speed up development. Sometimes, it takes a few hundred cycles to get the required inputs to show up in a later reactor, as a reactor earlier in the system first has to produce it. I would have appreciated it very much had I been able, for debugging, to specify any input I wanted and receive it immediately afterward. Of course, for the proper simulation, you would still have to rely on the modules your reactors actually produce.

In terms of gameplay decisions, I am a bit unhappy about the defense missions, as they involve manual intervention. You speed up or slow down the simulation, and when appropriate, you have to hit certain triggers. Some of those missions can be solved without triggers, which may not always have been by design, but in others it seems unavoidable. I don’t quite see what this mechanic adds to the game. I would have been much happier by solving regular problems and then watching my solution being carried out, instead of slowing down the simulation when the boss creature reacts in a particular way, manually interfere with the execution of my pipelined reactors, for instance by triggering the firing of a laser, and then speed up the simulation up. It feels like a burden.

I got quite annoyed with the boss of stage 6, where I was very proud of a solution that extremely quickly produced the necessary modules for firing a laser. Yet, the external trigger element arrived in only irregular intervals, and not nearly quickly enough. Thus, I had to slow down my solution. In the end, I arrived at a fully-automated three-reactor solution that avoids manual triggers altogether. This happened by accident. I first sped up the first reactor a little bit, which led to more erratic behavior. After speeding it up some more, I discovered that the pattern at which the elements that trigger the laser arrive are now antagonistic as they come in at an importune moment. In the end, adding two extra cycles to one reactor happened to enable a fully automated solution. However, it does so by exploiting the arbitrary game logic of this particular level. It would have been much more satisfying had the reactor fired whenever a module was ready, or just more often in general. Coming up with a very fast solution, just to learn that it is too fast for an external process you have no control over, on the other hand, is demotivating. What is worse is that you do not discover this limitation before you have a working solution that is able to fire several times. At that point, I felt that SpaceChem had jumped the shark and I was close to quitting. I was incredibly turned off by that experience and almost would have deleted the game afterward.

There are very few bugs in the game. It crashed once without any warning. Another time I opened a reactor level, and wanted to edit a solution, only to find that it was scrambled: there were two red “start” symbols and the layout did not make any sense. The game then booted me back to the level select screen, and when I reloaded that stage, the previously valid save state was restored. Lastly, in an early production level I encountered a reproducible bug when running the simulation at the fastest speed: one of the process markers was not following its path but instead followed a straight line. This bug is reproducible and appears every other time I run the simulation.

The blue ‘waldo’ is not supposed to leave its path. This bug is reproducible. Thankfully, it only affected one problem.

Additional SpaceChem Content

SpaceChem comes with copious amounts of bonus content, none of which matches the excellent difficulty curve of the main story mode. There is a bonus section in which you research “Moustachium”, which gives you bonus items for Team Fortress 2, or at least did. I haven’t tried if this still works as I don’t own that game. The few problems of that section are comparatively easy. The last one is rather enjoyable as it suggests an approach you may not have used in the main story. That content is free and it doesn’t take long to complete. As a conservative estimate, two hours should be plenty.

Most bonus content is part of ResearchNet, which divides problems into volumes. This section contains hundreds of problems. They are ranked by difficulty, from 1 to 3. The vast majority of those problems were created by users. While ResearchNET has the potential to entertain you for hundreds of hours, I dropped it relatively quickly. The biggest issue is that the content widely fluctuates in quality. Some problems are very easy to solve, others not so much. In comparison, almost every problem in the story mode is more difficult than the preceding one.

Furthermore, the problems in the story mode often have many viable solutions. I lost my interest in ResearchNET almost completely when I encountered a problem that was very restrictive. I solved it with little difficulty, but it was tedious. The reason was that instead of finding your own solution, which is one of many viable ones, you have to find the one solution the puzzle creator intended, or one that is very close to it. You can identify those problems easily because their associated histograms show a very narrow distribution curve, indicating that all solutions are within a tight range. Many ResearchNET puzzles lack histograms, which is quite unfortunate as part of the appeal of the story mode of SpaceChem is that you get to compare yourself with a large number of smart anonymous people and their solutions. Even if you lack a competitive streak, you still get feedback telling you how well you did. In the absence of such feedback, any working solution is a good solution.

The “Corvi 63” DLC is quite good. It’s lacking in presentation, however. Instead of story panels with illustrations, you only get to read a few emails. The graphics are bland, and the absence of a visually impressive boss creature does not help either. On the plus side, the various stages are fun. They introduce a new mechanic, quantum tunneling, which necessitates finding new solutions to problems, sometimes old ones. The first few problems help you get used to that mechanic. The subsequent ones test your understanding in increasingly complex problems. At the end, there is a final mission which is disappointingly easy. I would be surprised if anyone who has finished the story mode got more than eight hours out of Corvi 63. There is not a single level that provided a hefty challenge. Instead, the DLC felt more like a victory lap, after beating the story mode. Compared to the price of the main game, which is $10, the price of the DLC is too high, even though it’s only $2. I got the DLC during a sale at a 75% discount, and SpaceChem at an 80% discount, and I still think that the DLC was not worth it, compared to the fantastic value for money of the main game.

Game Length

SpaceChem is a long game. Zach Barth thinks that the game is too long overall, but I don’t mind the length much at all. In an interview with Gamasutra, he spoke of an “oppressive 40+ hour difficulty curve that only 2 percent of players reach the end of.” Notably, Zach himself has not finished SpaceChem and has no interest in doing so. I think he is missing out. Instead of his harsh judgment towards his game, I would liken it to a very rewarding hike that rarely gives you any respite, but which will give you an incredible sense of accomplishment, the likes of which you probably will not find in any other video game. Granted, I would have been perfectly happy had SpaceChem ended after the fifth or sixth planet. In particular, one of the bonus challenges I found fiendishly difficult. Had that been a regular level, followed by a boss fight, the game would have ended on a very high note for me. Overall, I was quite happy I got so much great content, but there is certainly a bit of fat that could have been cut.

A single play-through through the story mode, including all challenges, may take easily between 80 and 100 hours — if you are good. If you finish SpaceChem in 40 hours, you probably do not spend any time on optimization and skip the few brutally hard optional levels in the story mode. A good chunk of your play time you may spend on optimizing your solutions. (It took some people well over 200 hours to just play through the campaign, ignoring the challenges.) If you need more content, you can purchase DLC levels, do a short bonus mission, or solve curated user-created levels, which will entertain you for hundreds of hours more. There are very few people who have finished SpaceChem completely. Their play time is in the region of 500+ hours, spread over four to six years, based on the few people I was able to track down online who did so.

In terms of bang for your buck, there are few games out there that can compete with SpaceChem. Yet, I think that the story mode has overstayed its welcome. I would, at the very least, have cut the final level as that is a rather tedious boss fight. The most difficult level, in my opinion, was the second-to-last regular level, which is followed by a surprisingly easy one which could easily have been cut. It probably serves as a breather before the boss battle, which is quite straight-forward and one of the less challenging ones. Yet, afterward, you encounter the very tedious final boss of the game. The chemical processing I found quite trivial, in particular compared to how difficult that stage supposedly is. You can sidestep some of the challenges, which I am not going to spoil, and then you have to make it through a slog of a battle that could send you right off to sleep. Thus, for me, SpaceChem ended on a sour note.

Quo vadis, Zachtronics?

We may never see a game like SpaceChem again. Its creators underestimated the difficulty of their game. In fact, several levels of SpaceChem had never been solved by anyone when the game was released. Thus, Zach and his colleagues may not even have been aware of how tough the game gets. Yet, the utterly brutal difficulty is the biggest strength of this game, which is only exacerbated by the lack of instructions. SpaceChem is quite antagonistic. It throws you into the deep end and makes hardly any attempt to give you a helping hand. Many players may not like that. On the other hand, those that do not give up and persist will feel rewarded as they overcome tough intellectual challenges. Arguably, it’s bad enough if you put out a game for the general population and only a few percent complete it, but if your target audience consists of engineer-types and even among those less than two percent complete it, your game is indeed too difficult. I can understand that Zachtronics want to grow their audience. By doing so, they however lose the hard core that embraced SpaceChem for its difficulty, not in spite of it.

Making more and more difficult games is a losing proposition. There are genres that got so specialized that only a handful of players on the planet is able to complete its games. The poster child of that is arguably the bullet hell genre, which caters to a tiny niche, at the cost of alienating a mainstream audience. Then again, the question is whether those games could even reach a mainstream audience in this day and age. Maybe the most apt comparison one could make is between Zachtronics and From Software. The latter put out the game Demon’s Souls in 2009, which was a sleeper hit. It looked like a commercial disaster at first, and in the end it sold millions. In that game, the player has to master multiple roughly 30-minute long sequences, consisting of a difficult level that ends with a challenging boss fight. You have to complete each level in one sitting. Of course, being an action RPG, Demon’s Souls is the polar opposite of SpaceChem. Yet, in both games, players find it very rewarding to eventually beat stage after stage, for which they have to overcome challenges they initially may well have thought impossible. In subsequent games, From Software restructured their games. The spiritual sequel to Demon’s Souls, Dark Souls (2011), was a massive commercial success. Yet, among those who have played the predecessor, that game left something to be desired. You will find various opinions online. I think it boils down to challenge and reward, which were both higher in Demon’s Souls than of the later games in that series. Arguably, the more challenging a game, the higher the reward the player experiences, provided the challenge is perceived as being “fair.”

Zachtronics has released several more games that embrace programming or engineering in one way or another. There are two different kinds of games they have put out: (1) more visually-oriented games with fewer constraints and (2) games in which you write actual code. The first kind is represented by Infinifactory and Opus Magnum. In Infinifactory, you put together assembly lines. Compared to SpaceChem you get to work with an additional dimension. Based on what I have read online, the third dimension makes the game a lot easier and thus this game is often recommended as an entry point for Zachtronics games. It is only about half as long as SpaceChem and takes around 30 to 40 hours. People who have completed SpaceChem tend to report that they found Infinifactory very straightforward and not particularly challenging. The other game in that vein is Opus Magnum, in which you do not deal with fake chemistry like in SpaceChem but with “real” alchemy instead. You operate again on a 2D plane, but this time it is comprised of hexagons. It is an infinite plane. Due to that, Opus Magnum is even easier than Infinifactory. People who were chewed out by SpaceChem report that it took them ten to twelve hours to play through the campaign. On the other hand, people who did finish SpaceChem report that it took them four to six hours to play through the entire (!) campaign. To put this in comparison: A significant number of individual level in SpaceChem will take you about as long, and some could take you twice as long. Among the visual Zachtronics games, I have not the slightest interest in Opus Magnum. I might play Infinifactory because I have already picked it up as part of a bundle some time ago.

The other kind of games Zachtronics has put out more fully embrace programming. You write assembly code and you even have to read manuals. It’s almost like your day job. The games in that vein are TIS-100, Shenzen I/O, and Exapunks. The first game in this series was TIS-100, which has you program interconnected nodes. It’s assembly-programming with a flavor of Erlang. That game was a surprise success, so Zachtronics doubled down on it. For Shenzen I/O, they put together an even longer manual. In addition to writing assembly, but this time no longer for multiple nodes, you have to put together electronic gadgets that perform a certain role. TIS-100 has tight space constraints. This does not seem to apply to Shenzen I/O, with the exception of one stage, according to player feedback I have gathered online. I already own TIS-100 due to a bundle sale, but I do not have much interest in playing it as I find it visually off-putting with its tiny white text on a black background. I tried reversing the colors on my screen, but this didn’t improve matters much as the white turns into a grey-tone.

I have no interest in Shenzen I/O either as the visual presentation is just as off-putting, with its tiny text. From what I have read, the verdict is that if you have studied assembly programming at one point, neither TIS-100 nor Shenzen I/O are particularly challenging. Of course, these games do not use MIPS, ARM or x86 assembly, but the mental model will nonetheless apply. TIS-100 uses Erlang-inspired assembly. Shenzen I/O is conceptually simpler. One credible voice online wrote that those games quickly become a “plug and chuck” exercise. Zachtronics recently released a third game in that vein: Exapunks. You again write assembly code, and it seems there is one mode where you can even write ‘real’ JavaScript. In a tweet, the company shared that there are no constraints on space, meaning that your programs can be arbitrarily long. Upon reading this, I lost all interest in that game. From the first report I found online of someone who finished the game I learned that it took that person 13 hours to play through the campaign, further corroborating that this game is far too easy. Given the comparatively high price point of that game, I’d say that that is not nearly enough value for money. This is a disturbing trend in their games: they get ever more expensive, yet they provide fewer and fewer hours of entertainment, and that is without taking into account the qualitative difference between a hard game that feels rewarding and a game that lacks all challenge, which cannot keep your attention.

Zachtronics have been trying to widen the appeal of their games. Instead of catering to the ultra hardcore, engineer-types who solve difficult problems for fun, they put out games probably the average person could solve if they just dedicated themselves to it. In SpaceChem, the first part of the challenge is solving the problem. Afterward, you can look for a more efficient one. Step 1 is normally quite difficult. Step 2 is an optional bonus. In many cases, just cleaning up your solution makes it a lot more efficient. Sometimes, a different approach helps to make it so. On the other hand, if Step 1 is trivial, then the optimization is basically pointless. It is unfulfilling busy work. Think of the more difficult problems you may have faced in university courses: You solve them, you feel proud of your work, and then you clean up your solution. On the other hand, if the problem was not challenging, you just move on because it was not engaging enough. If you improve it, you do it for a better grade, but not because you feel driven to do so. It seems that Zachtronics has been heavily moving in that direction. Life is full of fairly trivial problems that are not particularly engaging. I do not need to pay for a game to get more of them. What I would like to see more of are “stupidly difficult” problems I can sink my teeth in, not games I can sleep-walk through because they go to great length to eliminate the possibility of failure. I could already get that in triple-A gaming, if I wanted to.

As it stands, SpaceChem is my favorite game after Tetris (NES), which is a perfect game, in my opinion. Both are lightning in a bottle. They are much different games, but there is an interesting parallel I would like to draw your attention to: Nintendo, who implemented that version of Tetris, was unable to come up with a puzzle game that was anywhere near as good, and it was not for a lack of trying. The comparison is a bit off because Nintendo did not create Tetris. Yet, they created gameplay mechanics that have not been reproduced in any other version of Tetris. That game is something special, which is only underscored by the existence of a seemingly endless number of insipid versions of Tetris out there. (Most people are not even aware that there are significant differences between Tetris versions, believing they are all the same.) Zachtronics seems poised to never produce a game that is anywhere near as brilliant as SpaceChem. Their follow-up games similarly underscore how great SpaceChem is, and as long as they do not produce another “design puzzle game” that is not afraid to genuinely challenge the player, SpaceChem may never be dethroned in that genre.

Metacritic and the Legitimacy of Video Game Journalism, Part II

In the last post I wrote about my perception that video game journalism is little more than advertising in disguise. In this post, I’ll discuss numbers that may support my opinion. I use data gathered by the website Metacritic. For those who are not familiar with it: Metacritic collects “expert” reviews of various entertainment media, including video games, normalizes their scores, and posts an average. Users are also able to rate games, which then leads to the games in their database having both an expert and a user average.

One objection you could now voice is that user opinion may be unreliable because it might primarily be people who either dislike a game and give bad reviews, or who love a game and like to see it having many recommendations. Sure, both camps may have their agenda. However, there are PR guys who pose as users, and try to inflate scores, like it happened with the recent Star Trek game. I don’t think any company would pay a PR agency to plant negative reviews, so while you might question the legitimacy of user scores, it’s probably better to view them as slightly inflated due to corporate meddling. Therefore, “true” user scores for big-budget would probably be slightly lower than they are. I do agree that the metrics are hardly reliable, and that it’s questionable whether you could objectively grade a game on a ten scale anyway. People have different tastes, and some people are more forgiving of gameplay flaws than others. Still, as a voice of “the people”, it may be interesting to see how they differ from the alleged experts.

I’ve conducted my analysis with data that was current as of August 2, 2013. Games that were released since then are not considered. Further, there may have been slight changes in some of the user scores, simply due to ratings that were submitted to Metacritic in the mean time. I only focus on Xbox 360 games, but I don’t think there would be a fundamental difference when analyzing ratings for PlayStation 3 or PC games.

Metacritic holds records for 1505 Xbox 360 games. I did start with some rather basic analysis. My aim was to uncover discrepancies between users and “experts”. The first few examples may not be overly interesting, but there is a climax towards the end, so please keep reading.

Here’s a top-level overview of the data:

  • user rating < expert rating: 760 games (50.5 %)
  • user rating > expert rating: 683 games (45.4 %)
  • user rating = expert rating: 63 games (4.1 %)

This doesn’t look like much of a discrepancy, though, so I adjusted my parameters to check how the picture changes if I define “equal” to be a difference of +/- 5 %:

  • user rating < expert rating: 494 games (32.8 %)
  • user rating > expert rating: 443 games (29.4 %)
  • user rating = expert rating: 568 games (37.8 %)

Since this wasn’t particularly revealing either, I then moved on to have a look at the average user and “expert” ratings, considering all 1505 games:

  • expert average: 69.0
  • user average: 68.3

The difference looks miniscule. However, the video game industry is focussing on “AAA games”, i.e. games that cost a lot of money to make, and whose advertising budget is even higher than their astronomical production costs. Since I had the impression that it is primarily this category of games that receives inflated ratings, I changed my script to only consider games that were rated with at least 90 by the “experts”. This applies to just 33 games. Suddenly, the picture gets much more interesting:

  • expert average: 93.3
  • user average: 81.1

I’ll let you draw your conclusions about that when you consider that as a whole the difference between user and expert ratings is 0.7. It seems that as soon as big advertising money comes into play, the “experts” identify greatness where users see just another okay game. A difference of 12.2 points is quite staggering.

If I run the same script but limit the scope so that it only counts games that were rated with 85 or more by the “experts”, a total of 137 games, the discrepancy is still quite startling:

  • expert average: 89.3
  • user average: 79.5

A difference of about 10 points is not to be ignored either, but not quite as damning as the previous subset of games. Let’s now have a look at those fabulous 37 games that the “experts” thought were marvels of digital entertainment. The columns have to be interpreted the following way:

  • column 1: user rating minus expert rating
  • column 2: expert rating
  • column 3: user rating
  • column 4: game title

To make it perfectly clear what the data means, look at this line: “-38 93 55 Mass Effect 3”. This means that the users gave the game a 55, the experts a 93, and that the user score is 38 lower than expert score.

Here is the entire list:

-38 93 55 Mass Effect 3
-33 94 61 Call of Duty: Modern Warfare 2
-20 93 73 Street Fighter IV
-19 98 79 Grand Theft Auto IV
-18 94 76 Halo 3
-17 93 76 Gears of War 2
-15 91 76 Gears of War 3
-15 91 76 Super Street Fighter IV
-14 91 77 Halo: Reach
-13 92 79 Forza Motorsport 3
-12 93 81 Pac-Man Championship Edition DX
-12 93 81 Rock Band 3
-12 96 84 The Elder Scrolls V: Skyrim
-11 91 80 Forza Motorsport 4
-11 92 81 Guitar Hero II
-11 92 81 Rock Band
-11 94 83 Gears of War
-11 95 84 Portal 2
-10 94 84 Call of Duty 4: Modern Warfare
-9 92 83 Rock Band 2
-9 94 85 Batman: Arkham City
-8 92 84 The Walking Dead: A Telltale Games Series
-8 93 85 BioShock Infinite
-8 93 85 Fallout 3
-8 96 88 BioShock
-7 93 86 Braid
-7 94 87 The Elder Scrolls IV: Oblivion
-7 96 89 Mass Effect 2
-7 96 89 The Orange Box
-6 91 85 Far Cry 3
-6 95 89 Red Dead Redemption
-5 92 87 Batman: Arkham Asylum
-4 91 87 Mass Effect

It turned out that across the board users were less impressed than professional reviewers, and oftentimes dramatically less. Maybe you have played some of those games and asked yourself why they don’t live up to their glowing reviews. For instance, I was thoroughly unimpressed by Halo 3, and Grand Theft Auto IV I view as a game with many issues, but with plenty of fun moments nonetheless. The “experts” give the game a 98, the users a 79. The user score is closer to my perception of that game’s quality. The Gears of War games have a ham-fisted plot and okay gunplay, and don’t think they deserved the praise it got. The third part had some really great moments, though, but also much more ham. The by far best third person shooter I played on the Xbox 360 was Binary Domain. The critics gave it a 74, the users an 81.

If you wonder whether you should trust video game reviews, then you now have some good justification why you shouldn’t. Nowadays I ignore mainstream reviews and look for the consensus in niche communities, like on message boards for people who play STGs. This is much more helpful than the writeup of some “expert reviewer” who is forcing himself to type a few hundred words about a game he may not even have finished or hasn’t even understood the controls of. My personal consequence is that I buy fewer games, and if I want to play some “AAA title”, I wait until I can pick it up for cheap to minimize the amount of money I might potentially waste. If there are more people like me, then the money hatting strategy of the big publishers might not be as effective as the MBA types who dreamt it up think. The rise of indie games might suggest that people are getting fed up with “AAA gaming”.

For a more uplifting conclusion that supports the view that fan feedback can be very valuable with regards to games that are under-appreciated by professional reviewers, let’s now look at 20 titles fans enjoyed but critics despised:

33 48 81 Otomedius Excellent
33 44 77 Warriors Orochi 2
31 52 83 Samurai Warriors 2
30 57 87 World of Outlaws: Sprint Cars
29 53 82 Tetris Splash
29 42 71 Venetica
27 49 76 Dynasty Warriors: Gundam 2
26 63 89 Triggerheart Exelica
26 53 79 Samurai Warriors 2 Empires
26 49 75 Puzzle Arcade
25 58 83 Vigilante 8: Arcade
25 53 78 Ecco the Dolphin
24 64 88 Project Sylpheed: Arc of Deception
24 61 85 Dynasty Warriors 5 Empires
24 60 84 Sonic Adventure 2
23 63 86 Serious Sam 3: BFE
23 62 85 WRC 3
23 54 77 Rise of Nightmares
22 53 75 Warriors Orochi

I recognize two STGs, Otomedius Excellent and Triggerheart Exelica, plenty of samurai games, and a couple of classic games, or reinterpretations of classic games. People seem to have a lot of fun with those, even though the “experts” don’t get them. As a cynic, I’m tempted to conclude from those data that it doesn’t make a lot of sense to let hats full of money go around when the game you’re about to release only appeals to a more “hardcore” audience, and its sales potential is comparably low. On the other hand, if you are marketing Halo, then you can be a bit more generous and discredit video game journalism even more. Just look at Geoff Keighley:

Geoff Keighley, a paragon of integrity in video games journalism

Geoff Keighley, a paragon of integrity in video games journalism

But, hey, the man’s got to eat, too! Who knows, maybe he’ll one day figure out how to play a video game despite having one hand buried in a bowl full of Doritos, and holding a Mountain Dew in the other.

Metacritic and the Legitimacy of Video Game Journalism, Part I

I recently bought an Xbox 360 since I was interested in playing a couple of recent games. Some of my buying choices were influenced by reviews on websites. Games like Red Dead Redemption were great, but others didn’t quite live up to my expectations. I’ve now had about a handful of experiences were the glowing reception of the games by reviewers didn’t at all match my subjective experience. Thus, I wondered how legitimate video game journalism really is.

I also noticed that some games I tremendously enjoyed, such as Vanquish, received a rather lukewarm reception. Also, I couldn’t help but notice that gaming websites are plastered with ads, so the obvious assumption is that those sites don’t want to bite the hand that feeds them and therefore promote “AAA titles”, while they spend little attention to games that cater to a niche audience. It’s also quite obvious that many mainstream reviewers don’t understand particular genres or, well, just plain suck at playing video games.

Here is a prime example: the Destructoid review of Vanquish by Platinum Games, written by Jim Sterling. He gave the game a 5 out of 10, and every single word he wrote indicates that he just didn’t get how to play this game. Vanquish isn’t a “cover-based shooter” in the vein of Gears of War but instead puts heavy emphasis on offensive play. The game is a bit more complex than in, say, Call of Duty, but you’ll get amply rewarded if you spend a few minutes to learn the controls. This was seemingly too much effort for Jim Sterling, so he wrote:

Sam [the protagonist] actually needs energy to punch his opponents, and once he’s landed a single successful punch, he can’t glide away since the energy meter completely drains. Several times, I punched an enemy, failed to kill it thanks to Sam’s inability to aim his punches properly, and was killed because I could neither defend myself or swiftly escape.

The issue is that the energy meter that enables you to perform more powerful attacks depletes as you do your little tricks. However, there is a risk/reward mechanism built in. If you completely replete the energy meter, your combat suit overheats. You then have to allow it to cool down, which makes you vulnerable to enemy attacks since you can neither defend yourself properly nor quickly evade. The core mechanic is therefore to find a rhythm for your attacks. This is not as bad as it may sound since the energy meter replenishes very quickly. I found the game mechanic to be highly satisfying, and with some practice, it’s quite easy to get into a state of flow. Frankly, I thought that Vanquish was absolutely fantastic and that it reaches a high-water mark for action games. Enthusiastic reception of this game by actual players, like on NeoGAF, seems to indicate that I’m not the only one who had been very impressed by it.

Jim Sterling’s review of Vanquish may be an egregious example, but the average games journalist is hardly an expert. Especially when it comes to niche games they don’t seem know what they are looking at. For instance, one of my favorite genres are shooting games (STGs). No, not the Call of Duty kind, but the modern descendants of Space Invaders. While most games strive to be “entertainment”, and therefore offer at best a moderate challenge, STGs are designed for repeated play throughs, with the goal being mastery so that you can eventually clear the entire game on just one credit. Depending on your skill level, this may take many months, and with the harder games you may never even get there because you’re just not good enough. This can be a humbling experience, but if you master a game like that, you’ll feel a sense of achievement which you just don’t get from games that hold your hand all the way through. Sure, it’s not for everyone, but those games have enthusiastic fans. Yet, the typical mainstream reviewer is quick to dismiss those games because you can just hit “start” again and “see everything in 15 minutes”. The thread Amusingly bad reviews on shmups.com collects statements like that. You can only shake your head.

I’ve now mentioned some examples of games or genres that tend to get short shrift by mainstream reviewers. Now let’s look at games that receive lavish praise, and whose faults either get ignored or justified. One prime example is one of the greatest commercial successes in recent years: Grand Theft Auto IV. It sold 25 million copies, and it’s the top rated Xbox 360 game on Metacritic. I think it’s a decent game, but it has its flaws, like a repetitive mission structure, poor driving, and clunky weapon mechanics. It’s not a bad game, but hardly the masterpiece it is claimed to be. Less than a quarter of the people who bought the game finished it. The other three quarters probably got bored or frustrated.

Another example is Resident Evil 5. It’s probably not a bad game once you get into it, but it doesn’t make it easy for you to like it. My main gripe is that your character controls like a tank. One of the very first scenes has you enter a shack. Then zombies start attacking you from two and then three sides. They first come running at you, but before they reach you, they seem to hit an invisible wall that makes them stop. From this point onward, they take turns attacking you, which results in incredibly awkward gameplay. This isn’t my idea of having fun, so I have yet to return to this game. When Resident Evil 5 came out, reviewers were defending the controls as “traditional Resident Evil gameplay”, and hordes of obnoxious gaming fanboys were eager to tell anybody who dared to criticize their favorite franchise a variation of “the controls are fine, maybe you just suck at the game.”

Seeing that some of the biggest games have quite startling flaws, I ended up wondering whether “AAA games” that are backed by multi-million advertising campaigns get much more praise than they deserve, and not because they are so great, but because “money hatting” buys good review scores. The big games normally don’t dare to be challenging, so even your average video game reviewer can play them. On the other hand, it seems that many journalist lack the knowledge and skills to appraise niche games, and are therefore quick to dismiss them. A glance at review scores on Metacritc, which contrasts user and “expert” opinions seemed to support this hypothesis. Looking for hard facts, I then made use of my programming skills and analyzed their data. I will share the results with you in the next post.