Chi-Square Tests: Independence, Goodness of Fit and Financial Applications

Table of Contents

    Do stocks with a Buy rating beat their benchmark more often than stocks rated Hold or Sell? Do large daily losses occur as often as a risk model predicts? Chi-square tests compare observed counts with the counts a hypothesis would predict. Use the test of independence for two categorical variables and the goodness-of-fit test for one set of counts against a probability model.

    What Are Pearson Chi-Square Tests?

    The Pearson statistic gets larger as observed counts move farther from their expected counts under the null hypothesis. Both tests use the same measure of discrepancy. The question you ask determines how you calculate those expectations.[1][2]

    Choose the null hypothesis before counting

    For independence, ask whether the probability of an outcome is the same across groups. For goodness of fit, specify the probabilities your model assigns to the categories. Give each observation one category per variable, using categories that do not overlap and cover every possible outcome.

    The Chi-Square Formula

    Pearson chi-square statistic
    χ2 = ∑ [(O − E)2 / E]
    O is the observed count and E is the expected count under the null. Sum over the cells or bins, excluding totals.

    Squaring makes a shortfall and an excess both count toward the discrepancy. Dividing by the expected count means a gap of 10 contributes more where you expected 20 than where you expected 100. Expected counts can be fractional; keep their precision until you report the result.

    Expected Counts for Independence

    Expected count in row i and column j
    Eij = (row totali × column totalj) / N
    Multiply the number of observations in the row by the column’s share of all N observations.

    Under independence, each row has the same expected column proportions as the whole sample. For a table with r rows and c columns of categories, the usual reference distribution has (r − 1)(c − 1) degrees of freedom. Once the margins and the top-left (r − 1) by (c − 1) block are known, the margins determine the remaining counts.[1]

    Expected Counts for Goodness of Fit

    Expected count in bin i
    Ei = N × pi
    pi is the probability the benchmark assigns to the bin. For a continuous model, subtract its cumulative probability at the lower edge from its cumulative probability at the upper edge.

    For k bins with positive, fully specified probabilities, use k − 1 degrees of freedom: the count in the last bin is determined by the total and the other bin counts. If you fit parameters using the same sample, the calibration can change.[2][3]

    How to Interpret a Chi-Square Result

    Result What it tells you
    Chi-square statistic How far the counts are from their null expectations, summed over the table.
    p-value Under the null and the sampling assumptions, the approximate probability of a test statistic at least this large. Compare it with a significance level chosen in advance.
    Effect size and proportions How much the categories differ in the sample. For a contingency table, use Cramér’s V for the overall association and compare row proportions to see its direction.

    Even small differences can yield a low p-value with enough data. The p-value does not give the probability that the null is true or tell you whether a result is useful for an investment decision. Failing to reject means the evidence did not clear your threshold under this test.[5]

    Independence Example: Analyst Ratings and Outperformance

    Suppose a research team records one rating per issuer before a 30-trading-day return window. It compares each issuer’s total return with a benchmark selected in advance. An issuer that matches or trails the benchmark counts as “Did not beat.” The 200 observations below are hypothetical; assume independent observations to see how the test works.

    1. Count outcomes by rating

    Null hypothesis: The probability of beating the benchmark is the same across Buy, Hold and Sell. The alternative is that at least one group’s probability differs.

    Observed issuer counts by initial rating and subsequent benchmark outcome
    Rating Beat Did not beat Row total
    Buy 48 27 75
    Hold 32 43 75
    Sell 20 30 50
    Column total 100 100 200
    2. Work out what independence would predict

    Half the issuers beat the benchmark overall (100 / 200). Under the null, expect half of each row to beat it. For Buy and Beat, that is 75 × 100 / 200 = 37.5. The other cells use the same column share:

    Expected counts under independence
    Rating Beat Did not beat
    Buy 37.5 37.5
    Hold 37.5 37.5
    Sell 25 25

    Buy and Beat contributes (48 − 37.5)2 / 37.5 = 2.94. Within each row, the two counts depart from equal expectations by the same amount in opposite directions. Summing the six terms gives 2 × (2.94 + 0.80667 + 1) = 9.4933 after rounding. With (3 − 1)(2 − 1) = 2 degrees of freedom, the right-tail p-value is 0.00868. At a prespecified 5% level, reject independence in this example.

    What the Association Means for the Analyst

    Buy beat the benchmark in 48 / 75 = 64% of observations, versus 32 / 75 = 42.7% for Hold and 20 / 50 = 40% for Sell. The Buy–Sell sample gap is 24 percentage points. These rates describe the direction of the sample differences; the omnibus test does not tell you which pairs differ statistically.

    Cramér’s V: overall strength of association
    V = √[χ2 / (N × min(r − 1, c − 1))]
    Here V = √[9.4933 / (200 × 1)] ≈ 0.218.

    V measures magnitude without indicating which rating tends to win.[3] Even a large gap in win rates would leave a different question unanswered: how much did the winners gain and the losers lose? A strategy could beat the benchmark more often and still have weak average returns or lose its edge after trading costs.

    Goodness-of-Fit Example: Return Bins Against a Normal Model

    Now suppose a risk analyst takes 200 hypothetical independent one-day returns. Before the test, the analyst fixes a normal benchmark with a mean of 0% and a standard deviation of one percentage point. The four bins split the observations at −1%, 0%, and +1%, with each boundary value assigned to the bin on its right. The null says the four population bin probabilities equal those of this specific model.[2][6]

    Turn model probabilities into counts

    The benchmark puts about 15.87% of returns below −1%, since that cutoff is one standard deviation below the mean. The expected count is 200 × 0.1586552539 ≈ 31.73. The observed count of 42 contributes (42 − 31.73105)2 / 31.73105 ≈ 3.3233 to the statistic.

    Hypothetical returns against a fixed normal benchmark; displayed values rounded
    One-day return Observed Model probability Expected Contribution
    Below −1% 42 15.8655% 31.73 3.3233
    −1% to below 0% 60 34.1345% 68.27 1.0016
    0% to below +1% 61 34.1345% 68.27 0.7740
    +1% or above 37 15.8655% 31.73 0.8749
    Total 200 100% 200.00 5.9737

    Adding the unrounded contributions gives χ2 ≈ 5.9737. With 4 − 1 = 3 degrees of freedom, the p-value is 0.1129. At 5%, fail to reject the model’s bin probabilities. Losses just below −1% land in the same bin as much larger losses, so this result does not validate the model for extreme losses.

    What Changes When You Fit the Benchmark?

    If you use the evaluation sample to fit the model, the expected counts adapt to the same data. With a regular fit by maximum likelihood to the grouped bin counts, estimating q parameters gives the usual k − 1 − q degrees of freedom, provided the parameters are estimable, expected counts are adequate, and degrees of freedom remain positive. Estimating a normal mean and standard deviation that way uses two degrees of freedom. If you fit to raw returns or choose bins after looking at them, do not just subtract two and assume the reference distribution is right. One option for independent observations is to simulate from the fitted null and repeat the actual fitting and binning procedure for every simulated sample.[2][7]

    Pearson Chi-Square vs. Likelihood Ratio and VaR Tests

    Pearson Chi-Square

    Sums squared observed-minus-expected count gaps, scaled by expected counts. The test uses two categorical variables for an independence table or one set of categorized observations against a specified model.

    Likelihood Ratio (LR)

    Compares how well a restricted model and a more flexible one fit the data through their likelihoods. An LR test can use a chi-square reference distribution under suitable conditions, as in tests of nested logit and probit models. It computes a different statistic.[3]

    For value-at-risk backtesting, a 99% VaR model targets a 1% exception rate. The Kupiec proportion-of-failures LR test compares the observed exception rate with the target and ignores the order of the exceptions. Christoffersen’s framework adds a check for first-order dependence and can test the rate and independence jointly. A Federal Reserve staff study used both tests to evaluate 99% one-day VaR forecasts over test periods from 2006 through 2023. Even if a model gets the total rate right, the ordering can reveal problems that the total conceals.[8][9]

    How to Run a Chi-Square Test in Financial Research

    1. Define the unit and the decision. State what counts as one issuer, borrower or day, and what you want to learn. Set the benchmark, horizon, categories, boundary treatment and significance level in advance.
    2. Tally actual observations. Include each unit once. Start from counts rather than percentages, return amounts or portfolio weights. Document missing observations so the reader knows what population the table represents.
    3. Compute expected counts and check the design. Use table margins for independence and model probabilities for goodness of fit. Check expected counts, how probabilities were estimated and whether the units may share information or shocks.
    4. Report enough to reproduce the result. Give the observed counts, expected-count method, statistic, degrees of freedom and p-value. For an association, report row rates and effect size with the test. For a model, point out which bins account for most of the discrepancy.

    Common Mistakes

    • Using equal expected counts by default. Independence preserves the observed margins. Goodness of fit uses the specified probabilities, which need not be equal.
    • Misreading a zero count. Keep a zero observed count if the expected count is positive. A zero expected count makes the usual Pearson term undefined. In a contingency table, check for an empty margin or a combination ruled out by design. In goodness of fit, if the prespecified model assigns a category zero probability and a verified observation falls there, the data contradict the model. Report that contradiction rather than dropping the observation to calculate a p-value.[3][7]
    • Searching for a low p-value. Trying many sectors, return windows or cutoffs and reporting the smallest p-value as a single planned test overstates the evidence. Disclose the searches and account for false positives from multiple testing, or test the selected specification on new data.[5]
    • Reading causation or a trading signal into an association. Ratings can reflect sector, size or prior performance. An unadjusted table does not isolate their role. A model such as logistic regression can help compare binary outcomes while accounting for measured characteristics, subject to its own assumptions.

    When the Usual p-Value Can Mislead

    Separate variable independence from observation independence

    The null asks whether the two variables are independent within an observation. To use the ordinary reference distribution, the observations themselves also need to be independent of one another. Repeated issuer or borrower records, shared market shocks and overlapping return windows can violate that sampling condition. Counting each ticker once does not remove a common calendar shock.

    This matters even when the table looks well populated. In published research on earnings announcements, A. Craig MacKinlay showed why inference based on uncorrelated abnormal returns can break down when windows overlap across securities. For a ratings study, consider a sampling design or analysis that accounts for relevant issuer or time dependence. If you cannot justify the inference, report descriptive rates rather than an ordinary Pearson p-value.[10]

    Sparse cells. Checking that each expected count is at least five is a conservative screening rule. For larger contingency tables, a common rule of thumb is no expected count below one and roughly 80% or more at least five. Check what your chosen method assumes. Fisher’s exact test may suit a sparse 2 × 2 table with independent units; larger tables or one-way counts need a method calibrated to their own null. An exact test does not fix dependence between observations.[1][4]

    Lost information within categories. Two samples can have identical bin counts and very different return magnitudes. If the question is whether average returns differ, use a test for mean returns suited to the data. If the question is extreme-loss risk, choose a diagnostic aimed at that tail rather than relying on broad bins.[2]

    Frequently Asked Questions

    Use actual counts. A table with 6 of 10 issuers beating a benchmark has the same proportion as one with 600 of 1,000, but treating both as 60 would erase the sample sizes. Use percentages after the test to describe differences.

    Yes, if you turn them into counts in nonoverlapping bins and test the probabilities assigned to those bins. This discards detail within bins. The right test depends on whether you care about the average return, the probabilities in specified ranges or the overall return distribution.

    Inspect each contribution, (O − E)2 / E, and check whether its observed count was above or below expectation. That locates the largest discrepancies. A separate claim about a cell or pair needs inference appropriate to the table and to the number of comparisons.

    It means the test did not detect a sufficiently large departure from the tested probabilities. Different distributions can assign very similar probabilities to a few broad bins, and a sample may have too little information to distinguish them.

    Sources

    1. NIST Dataplot, Chi-Square Independence Test.
    2. NIST/SEMATECH e-Handbook, Chi-Square Goodness-of-Fit Test.
    3. SAS/STAT User’s Guide, PROC FREQ: Chi-Square Tests and Statistics.
    4. R stats, Pearson’s Chi-squared Test and Fisher’s Exact Test.
    5. American Statistical Association, Statement on Statistical Significance and P-Values announcement, March 7, 2016.
    6. NIST/SEMATECH e-Handbook, Normal Distribution.
    7. SciPy documentation, Pearson’s chisquare, goodness_of_fit, and multinomial, Notes.
    8. Peter Christoffersen, Evaluating Interval Forecasts, International Economic Review, 1998.
    9. Federal Reserve Board, Linear and Nonlinear Econometric Models Against Machine Learning Models: Realized Volatility Prediction, FEDS 2025-061, Section 6.
    10. A. Craig MacKinlay, Event Studies in Economics and Finance, Journal of Economic Literature, 1997, Sections 2 and 5D–E.
    Educational Use

    This article is for informational and educational purposes. Its worked examples use hypothetical data and do not constitute investment advice.