Game Theory & Oligopoly: Nash Equilibrium, Prisoner’s Dilemma & Cartels

Game theory in economics provides the essential framework for understanding how firms behave in oligopoly markets — industries dominated by a small number of competitors whose decisions are deeply interdependent. Unlike perfectly competitive firms that simply take the market price, oligopolists must think strategically: every pricing, output, or advertising decision depends on what rivals are expected to do. This guide covers Nash equilibrium, the prisoner’s dilemma, dominant strategies, cartels, and the major oligopoly models that explain how strategic interaction shapes market outcomes. (For the foundational market model, see supply and demand.)

What Is an Oligopoly?

An oligopoly is a market structure in which a small number of firms dominate the industry, selling similar or identical products. The defining feature of oligopoly is strategic interdependence — each firm’s profit depends not only on its own decisions but also on the decisions of its rivals.

Oligopolies typically feature high barriers to entry — large capital requirements, patents, or economies of scale that prevent new firms from easily entering the market. Real-world examples include the U.S. wireless industry (T-Mobile, Verizon, AT&T), major airlines (Delta, United, American, Southwest), and tennis ball manufacturers (Wilson, Penn, Dunlop, Spalding — the example that opens Mankiw’s chapter on oligopoly).

Economists measure market concentration using tools like the Herfindahl-Hirschman Index (HHI), which sums the squared market shares of all firms in an industry. A higher HHI indicates greater concentration — antitrust authorities use HHI thresholds to evaluate whether mergers would create excessively concentrated markets.

Game Theory in Economics: Players, Strategies & Payoffs

Game theory is the study of how people and firms behave in strategic situations — situations where the outcome for each participant depends on the actions chosen by all participants. Every strategic interaction can be described using three elements:

• Players — the decision-makers (e.g., two competing firms)
• Strategies — the actions available to each player (e.g., set a high price or a low price)
• Payoffs — the outcome (usually profit) each player receives for each combination of strategies

The standard way to represent a strategic interaction is a payoff matrix — a table showing the payoff to each player for every possible combination of strategies. This article focuses on simultaneous-move games with pure strategies, the foundation of oligopoly analysis.

Dominant Strategy & the Prisoner’s Dilemma

A dominant strategy is a strategy that yields the best payoff for a player regardless of what the other player does. When both players have a dominant strategy, the outcome is predictable — but it may not be the best outcome for the players collectively.

The prisoner’s dilemma is the most famous game in economics. It illustrates why cooperation is difficult to maintain even when it would benefit everyone involved.

The prisoner’s dilemma appears throughout business. In the airline pricing game above, Low Fare is dominant for both airlines: each earns more by cutting fares regardless of the rival’s choice. Both end up at $3M each — worse than the $5M each would earn if both maintained high fares. The dilemma: individual rationality leads to a collectively inferior outcome.

Nash Equilibrium

The concept of Nash equilibrium — named after mathematician John Nash — is the central solution concept in game theory. It identifies outcomes where no player has an incentive to change their strategy unilaterally.

In the prisoner’s dilemma, (Confess, Confess) is the Nash equilibrium — neither player can do better by switching while the other confesses. In the airline game, (Low Fare, Low Fare) is the Nash equilibrium for the same reason.

A Nash equilibrium is not necessarily the best outcome for the players or for society — it may reduce total surplus relative to the competitive outcome. It is simply a stable point — once reached, no individual player wants to deviate. Some games have multiple Nash equilibria, and some have no Nash equilibrium in pure strategies (requiring analysis of mixed strategies, which is beyond the scope of this article).

How to Identify Nash Equilibrium in a Payoff Matrix

Finding Nash equilibria in a 2×2 payoff matrix is straightforward once you know the method. For each player, identify their best response to each of the other player’s strategies:

1. Fix one player’s strategy — look at one column (or row) at a time
2. Find the other player’s best response — which strategy gives them the highest payoff in that column (or row)?
3. Mark the best response — underline or circle the best payoff in each column/row
4. Identify Nash equilibria — any cell where both players are playing their best response is a Nash equilibrium

Cartels & Collusion

The Nash equilibrium in the Jack & Jill example ($1,600 each) is worse for both firms than the monopoly outcome ($1,800 each). This creates a powerful incentive to collude — to agree to restrict output and raise prices as if they were a single monopolist.

A cartel is a group of firms that explicitly agree to coordinate their output or pricing decisions. A collusive agreement is any arrangement — formal or informal — where firms agree to restrict competition.

Cournot Competition: Quantity Rivalry

The Jack & Jill duopoly example illustrates what economists call Cournot competition — a standard oligopoly model (named after French economist Antoine Augustin Cournot) where firms compete by simultaneously choosing how much to produce. Each firm selects its output to maximize profit, taking the other firm’s output as given.

The key intuition behind Cournot competition involves two opposing forces that every oligopolist weighs:

• Output effect: Producing one more unit at the current price increases revenue
• Price effect: Increased total production lowers the market price, reducing revenue on all existing units

Each firm produces where its output effect equals its price effect. The result is the Cournot-Nash equilibrium, which falls between the monopoly and competitive outcomes. In Jack & Jill’s case, each produces 40 gallons — more than the cartel share of 30 but less than the competitive quantity of 60.

An important insight from the Cournot model: as the number of firms in the oligopoly increases, the price effect on each individual firm shrinks (because each firm’s output is a smaller share of the total). With many firms, the output effect dominates, and the Cournot outcome approaches the competitive result where price equals marginal cost. This provides an economic rationale for policies that increase competition, including international trade that exposes domestic oligopolies to foreign competitors.

Bertrand Competition: Price Rivalry

Bertrand competition is an alternative oligopoly model (named after French economist Joseph Bertrand) where firms compete on price rather than quantity. The result is strikingly different from Cournot.

The Bertrand paradox: with identical products, just two firms competing on price are enough to drive the price all the way down to marginal cost — the same outcome as perfect competition. The logic is straightforward: if Firm A charges any price above marginal cost, Firm B can undercut by a penny and capture the entire market. This undercutting continues until both firms price at marginal cost.

With differentiated products (where consumers see the two firms’ offerings as imperfect substitutes), the Bertrand result changes dramatically. Each firm has some pricing power because a small price increase does not cause all customers to switch. Prices settle above marginal cost, and both firms earn positive profits.

Repeated Games & Tacit Collusion

The prisoner’s dilemma result — where cooperation breaks down — applies to one-shot games (interactions that happen only once). But real-world oligopolists compete against the same rivals day after day, quarter after quarter. This repetition fundamentally changes the strategic calculus.

In a repeated game, firms can use strategies that reward cooperation and punish cheating. The most well-known is tit-for-tat: cooperate in the first round, then in every subsequent round, do whatever the other player did in the previous round. If the rival cooperates, you cooperate. If the rival cheats, you punish by cheating in the next round.

More broadly, trigger strategies sustain cooperation by threatening permanent punishment. If Jill exceeds her quota of 30 gallons, Jack responds by producing at the Nash level (40 gallons) forever. The one-time gain from cheating — producing 40 gallons while the rival sticks to 30, yielding total output of 70 gallons at $50 per gallon and a cheater’s profit of $2,000 versus the cooperative $1,800 — is outweighed by the permanent loss of future cooperation ($1,600 per period vs. $1,800 per period). When firms value the future sufficiently, the threat of punishment makes cooperation self-enforcing.

Oligopoly vs. Other Market Structures

Oligopoly occupies a unique position in the spectrum of market structures. The key distinguishing feature is strategic interdependence — only in oligopoly must firms actively consider competitors’ reactions when making decisions.

Common Mistakes in Game Theory & Oligopoly Analysis

1. Confusing oligopoly with monopoly. Oligopoly involves multiple firms whose behavior is interdependent — each must consider rivals’ reactions. Monopoly involves a single firm with no strategic interaction. The analytical tools are entirely different: game theory for oligopoly, standard profit maximization for monopoly.

2. Assuming Nash equilibrium is always unique. Many games have multiple Nash equilibria, and some have no Nash equilibrium in pure strategies. The prisoner’s dilemma happens to have a unique equilibrium, but this is not guaranteed in all strategic situations.

3. Ignoring repeated-game dynamics. The prisoner’s dilemma result (cooperation fails) applies to one-shot games. When firms interact repeatedly, strategies like tit-for-tat can sustain cooperation. Analyzing an oligopoly as a one-shot game when firms compete over many periods leads to inaccurate predictions.

4. Confusing explicit collusion with tacit interdependence. Explicit agreements to fix prices or divide markets are illegal under antitrust law. But parallel pricing behavior — where firms independently arrive at similar prices through rational strategic analysis — is a natural feature of oligopoly, not evidence of illegal activity. The distinction is legally and economically important.

5. Assuming cartels are stable. Cartels face constant pressure from the incentive to cheat. OPEC’s history — dramatic price increases followed by quota violations and price collapses — demonstrates that cartel discipline erodes over time as individual members pursue their own interests.

Limitations of Game Theory Models

Rationality assumption: Game theory assumes all players are perfectly rational and can accurately calculate optimal strategies. In practice, decision-makers may be influenced by emotions, cognitive biases, or incomplete information. Behavioral economics has documented systematic departures from the rational-player assumption.

Information requirements: Finding a Nash equilibrium requires knowing all players’ strategies and payoffs. Real firms face significant uncertainty about competitors’ costs, capacity, and strategic intentions. The precise numbers in textbook payoff matrices are pedagogical tools — real-world payoffs are uncertain and estimated.

Static vs. dynamic: Basic game theory models analyze a single decision point. Real competition is ongoing, with firms learning, adapting, and innovating over time. Dynamic models exist but are substantially more complex than the introductory frameworks covered here.