Feedback/Reinforcement Loops

I’ve been thinking a bit about feedback/reinforcement loops lately, so I figured I would write up a post on them.

Positive feedback loops in games are mechanics that reinforce the success or failure of the player and make future successes/failures more probable. This can be seen in numerous games, especially in RPGs. The cliche often used to explain positive feedback loops is “the rich get richer”. Having money is often a prerequisite to gaining more money: you need capital to start a business,  so the ones making money are the ones already rich. And so on.

But let’s take an example from board games first. In Risk, players control armies trying to take over the world. They do so by battling other armies. When players win battles and control entire continents, they receive additional armies every turn – this in turn “feeds back” into the simple result that having more armies makes you more likely to win battles. So not only does winning a battle give a direct advantage (the opponent has fewer units) but you also feed back that victory into gaining more units, which should ensure further victories.

Continent bonuses.
Continent bonuses.

Quake II faces positive feedback loops. The relevant mechanic is that when you die, you restart with guns that are not as good as the ones that can be picked up on the battlefield. The player with the first kill then has an inherent advantage – he will have better weapons than the newly respawned player and will be more likely to kill him again, given equal skill.

Professional baseball in America has a somewhat muted positive feedback loop. There are no salary caps in baseball, so teams can spend as much money as they have to procure the best players. The best players (one would think) lead to the best success on the field. Success on the field leads to more money as people bid up tickets and buy merchandise. That money can then be fed into buying even better players.

The Facebook game Mafia Wars originally had a pretty severe positive feedback problem. Players could buy properties that were constantly earning rents. Players could dump all of their money into buying “Mega-Casinos” which in turn gave a lot of additional money in rents, for which players could buy additional “Mega-Casinos”. After my first burst of playing the game, I quit. I came back three months later and had hundreds of billions of dollars, enough to buy any object in the game (that could be bought with in-game money, that is).

Sweet, delicious, infinity monies.
Sweet, delicious, infinity monies.

Positive feedback isn’t always bad. For one, it helps direct player actions. Players want to become richer, more powerful, &c., so giving them rewards for succeeding makes sense. It also breaks stalemates. Consider a game like the card game War. In it, the player with the highest card value wins. It is entirely random, so on average, each player should win 50% of the time. Series of War games will always be in stalemate. No previous game affects the current game. But if you added a positive feedback loop that changed that, the game could progress towards an end state. Say that players get to turn over an additional card if they won the last hand. This gives the winning player an advantage, that should cause a positive feedback loop leading to an end-game state where one player has dominance over the other and can be concluded the winner. I’m not saying that will make War fun (Lord knows), but it can end the damn thing.

When I was assigned to NCAA Football 08, there was a positive reinforcement system being implemented called “DPR” – dynamic player ratings. The idea was to implement the most straightforward of positive reinforcements. Player events that were tied to random die rolls were affected by ability ratings (a player with 90 catching ability would catch more often than a player with 89 catching rating), as a player succeeded during the course of a single game, his ratings would be temporarily boosted, causing more successes.  Player ratings converged to either essential perfection or complete inadequacy. That system had to be tweaked the entire cycle, the smallest change could spiral out of control creating perfect passers or punters that couldn’t hit the broad side of a barn. In the end, it was tuned down to look significant, but the probabilities affected were only slightly nudged. The essential nature of the positive feedback dynamic was never addressed.

This problem is endemic to many sports games as “stat boosts” seem to be the only reward structure worth pursuing.

Negative feedback loops in games are mechanics that cause to hinder succeeding players from further success or failing players from further failures. The classic example is Mario Kart.

Mario Kart is a racing game where players have weapons that they can use to speed themselves up or hinder the progress of their opponents. In a later installment, Nintendo added the Blue Shell weapon. The Blue Shell is the ultimate negative feedback loop. It homes in and destroys the player in first place no matter where they are on the map. There is nothing the first place player can do to avoid this. The blue shell is almost inevitable – there is almost one in every race. It is always on the mind of the player in first. Succeeding players are punished – no one wants to be too far ahead of the pack because the blue shell will knock them back. Instead, it encourages a dynamic where players want to be in second place until the final parts of the race – safe from blue shells but also close enough to take the lead by the finish.

Simply the most well-understood negative reinforcement mechanic in games.
Simply the most well-understood negative reinforcement mechanic in games.

Negative feedback is dangerous because it sends mixed signals. We direct our players to succeed – win the race, kill the bad guys &c., but at the same time our negative feedback mechanics do not agree with our stated intent.

Some racing games suffer from this problem. Since most racing games are more interesting when you are jockeying for position with other cars, if you do too well and leave the pack behind, the AI will cheat and make the opponents go supernaturally fast to catch up with player to encourage more nail-biting racing. This punishes the player for succeeding, which goes against all the other mechanics in many of these games which are tailored to encourage the player to race well. Players are almost unanimously against this technique – do a search in racing game reviews for “rubber-band AI” and you will see what I mean. Designers here are faced with a conundrum – have no “rubber banding” and let good players race off into the distance, effectively racing time trails, or have the AI cheat to make the race more exciting.

Even relatively simple games can use negative feedback. In the trivia game Buzz, the player in first place never gets to select the trivia category. The player selecting should naturally try to pick something that she knows that the leader doesn’t and thus this should serve as negative reinforcement.

Positive feedback loops are naturally occurring for a simple reason: players direct themselves towards actions that make them stronger. They buy swords in RPGs that let them kill bigger creatures that drop better loot that allow them to buy even better swords. There’s nothing inherently wrong with that, but it can cause problems. Negative feedback loops are much harder to come by because they can be simply contradictory and non-intuitive. We guide players to help them succeed and then behind the scenes sabotage them with negative feedback so that succeeding is bad. We spend more time figuring out solutions to positive feedback situations than negative because positive loops cause more problems:

  • Interest curves. Even single player games can fall victim to this. What if the player spends all his time trying to level up instead of exploring all the great content we have for him? What if the drops he gets from such leveling up make the rest of the game too easy? How can we reward the player with one hand and take it away with the other such that he keeps progressing through the game?
  • Problems of balance. Multiplayer games need to use positive feedback sparingly. If the entire game is decided by the time the first winning move enters the positive feedback loop (like the first kill in Quake II above), then why play the rest of the game? The loser will be continually beaten by the first winner, and without hope of winning himself, probably won’t be having much fun.
  • Endgame problems. World of Warcraft has a problem. Players get stronger and stronger, but there is a finite amount of content they can offer. Eventually, you have to reach the strongest sword in the game. At that point, players cannot be directed to kill bigger bad guys for the purpose of better drops that have better stats. The designers there have to deal with the problem of “what to give the player that has everything”. Naturally, they’ve done a good job as they have a healthy base of players at the maximum level, but this problem only exists because of the positive feedback loops in the main gameplay.

Disgaea took a different approach to this. They simply scale up the numbers for weapons and monsters to ridiculous levels, encouraging an ever-moving treadmill of progression-escalation. World of Warcraft does not have this luxury because they have to deal with players of uneven levels cooperating in the same area. It is hard enough to do that for a Level 80 and a Level 70, let alone a Level 200 and a Level 70.

One technique to fix positive feedback getting out of control is to decouple what the reward affects and what the tasks require. Farmville and other “social” games are all about walking up a treadmill to get the next doo-dad. But the doo-dads in these games generally aren’t things that help the player succeed. Often they are cosmetic or tangential. For instance, while the harvester in Farmville allows the player to harvest in less clicks, it doesn’t make the plants grow faster. The reward is reinforcing the click-to-harvest mechanic and not the XP-gaining growth mechanic. Thus, players can lust over the harvester and harvest like silly without breaking the game. This is a fine technique and one that I recommend, but it isn’t perfect because the player has to want a reward of something other than what will give him more power to achieve. Thus it can only be applied in some situations.

When I was working on the multiplayer for Superman Returns DS, we started by drafting a pretty clever multiplayer board game where players attempted to take control of the city of Metropolis. Quickly, we fell into problems of positive feedback loops – players who controlled more of the city could move around the city easier and thus could get the bonuses that helped the players towards victory. We could have gimped it by removing the positive feedback from ease of movement around territory you control, but we chose a different method. We added a negative feedback mechanic- taking over an opponents areas gains someone more territory than taking over neutral areas. I don’t remember if this stayed in the final version or not, but since no one actually played the multiplayer of Superman Returns, even though it was far and away the best part, I’ll just make that claim and leave it at that.

There are no pictures of multiplayer SMR on the Internet.
There are no pictures of multiplayer SMR on the Internet.

This is often the best solution if the positive reinforcement is knocking the game out of balance and you cannot simply remove the offending mechanic – pair positive reinforcements with negative ones.

RPGs do this: When you level up, monsters also become tougher. So you get stronger, but your enemies do as well, likely at the same rate. Your power increase is positive feedback. Your enemy’s increase is negative feedback. This would be noticeable to players if put that explicitly, so what we do is allow the player to really beat up on some lower-level thugs after powering-up to get them feeling more invincible before throwing their ass in front of newly souped-up baddies.

But this has to be carefully balanced. In Morrowind, players could choose to level up and if they did so, stronger baddies would appear instantly, rolled from a different table. When I first played it, I stayed as a level Suck player for as long as I could, simply because the quest items I found were positively reinforcing and leveling up was a net-negative reinforcement that was under my control. Certainly, it was not what the designers intended but I’m sure many players chose that path.

“Fixing” positive feedback doesn’t even have to really fix the problem, only the appearance of a problem. Consider our Major League Baseball example above. The National Football League saw that positive feedback problem and added negative reinforcements – salary caps (limits on how much a team can spend on personnel) and reverse draft selection (the worst team gets to choose first the best players coming out of the collegiate system.) It is generally seen as more fair. In baseball, the top four teams (Yankees, Dodgers, Cardinals, Giants) combine for 50% (53/105) of all world championships. In football, the four winningest Super Bowl teams (Steelers, Cowboys, 49ers, Patriots) combine for 43% (19/44) of all Super Bowls. While football is widely considered to be more egalitarian, the statistical difference is slight.

EDIT: And I almost forgot to mention one of the most clever negative reinforcements in all of the games I’ve experienced. In Dominion, the act of buying victory points and putting them into your deck de facto makes your deck weaker since victory cards are empty draws. This means that you are always stuck with the decision of making an efficiency advantage at the cost of points versus a points advantage at the cost of efficiency.

7 thoughts on “Feedback/Reinforcement Loops”

  1. Hmm, interesting examples of feedback loops; although I’m surprised you left out games which have no negative or positive feedback loops. Are they that hard to find? I would have thought sport games especially basically are neutral – for instance, if you play Pool, losing one set doesn’t mean your opponent gets any advantage, and in fact the rotation of who goes first and second means it should be equal over the entire game (same as Tennis and so forth). I’m surprised games don’t use this more often, and go for a round-based winning system with stuff reset between each attempt.

    There are also competitive-sans-battling games which have positive feedback for actions but not necessarily ones that stop your opponents. For instance, Settles of Catan – you buy more road, towns or whatever, but the only thing it can ever do to opponents is stop them placing things nearby there (this is about the only way to stop anyone directly doing anything), and they have their own areas to take over too. You don’t ever literally fight each other to gain advantages over any one player, only indirectly make it so they can’t place items down which they also do to you.

    Civilization 4 also goes a bit this way with positive actions from taking resource points which the enemy also likely has some supply of – although the threat of war and borders colliding make up most of the game, so taking enemy territory can lead to risk-style positive reinforcement and dominance (although it has negative reinforcement – a large army and large amounts of towns can really drain an economy and a long lasting war usually makes your people hate you, cutting production).

    Interesting stuff regardless. Adding in the negative to outweigh the positive is something, I wonder if it is even a good idea to remove it entirely though. For instance would Quake be Quake if you started out with all the weapons each time you respawned? Or does the fact you need skill to also retrieve new weapons add to the game? It’s likely the latter, and while possibly unfair positive feedback for 1v1 combat, it likely is more unpredictable then spawncamping, as a cunning player would know where to safely go for a weapon right away (or be able to fight off the damaged player with his starting weapon). It wouldn’t be as fair as many single player sports mind you though.

  2. Andrew, no, games without are not hard to find at all. I just didn’t think it was worth commenting about.

    Actually, I would argue that 8-ball pool is negative reinforcement. Say I’m playing you and I suck so I have six balls left and you have two. When it is my turn, I have a much easier time sinking my next ball because I have six options and only two obstacles that can get in my way (plus the eight ball).

    Catan does have a positive reinforcement. Winning players have more resources and thus can buy settlements and cities that gain them more resources. The game is balanced such that this doesn’t break it, but there is still little way for a behind player to catch up except for some fortuitous dice rolls. It doesn’t break the game – it is just a result worth mentioning.

  3. Andrew: Some shooters (like Team Fortress 2) do take away the weapon-finding part of Quake-style games.

    There, the main positive feedback loop is territory that your team is controlling with engineer buildings, I think.

  4. Zack; I think it’s interesting to list why some games have no reinforcements – apart from perhaps psychological ones of being ahead or behind. Pool I admit was a mistake – I should have said defiantly just tennis or somesuch where the game board is reset and play is rotated fairly in a game (same with Football, Rugby, etc.). Pool also you’re right is negative – but snooker is an interesting alternative which is actually positive – if you’re able to pot more reds you can effectively narrow the chances of the opponent being able to get enough red-blacks in a row to win, but the advantage is of course just one from playing the game well – you yourself need to get the same straight run without missing.

    Catan’s reinforcement is lessened, you’re right it is there though. I think the economic games tend to have it much less then actual destruction of your enemies forces which is more common – the destruction of something the enemy has is a much much larger game changing reinforcement!

    Mark; obviously there are tons of games which do full weapons – I mean, Call of Duty does in all it’s modes too. I think team based games (even UT/Quake CTF gives you weapons in your base more so then on the map) generally stick to that out of necessity – fighting more then one enemy with a pistol is not fun. Then there is the deathmatch in Call of Duty for instance – it is necessary to have weapons because your movement is so limited (in comparison) and therefore would be impossible to have Quake-like finding weapons, which is intrinsic to that game type – fast paced and very tactical.

Leave a Reply

Your email address will not be published. Required fields are marked *

Human? * Time limit is exhausted. Please reload the CAPTCHA.