Chapter 7 · The Mathematics of Trading

7.3 Backtesting, Costs and Algorithmic Trading

How to test a trading rule honestly: positions without look-ahead, the equity curve and its CAGR, Sharpe ratio and drawdown, what brokerage, taxes and slippage do to an edge, how execution algorithms work, why the best of many backtests is usually luck, and how a t-test and the bootstrap separate skill from noise.

📍 The situation

Kabir, a B.Com student in Indore, has found a rule on a trading forum: "buy the index when it closes above its 50-day moving average, go to cash when it closes below". A video claims the rule "returned 22% a year with half the drawdown of the market". Kabir has some savings and a broker account with an order API, and he is tempted to automate the rule. Before he risks a rupee, he decides to test it himself on data — to backtest it.

A backtest is a simulation: you pretend to have followed the rule in the past and record what would have happened. Done carelessly it is the most persuasive way ever invented of fooling yourself. This section answers Kabir's questions in order. How do you compute a backtest so that the rule never uses tomorrow's prices? How do you read the equity curve — its CAGR, Sharpe ratio and maximum drawdown? What do brokerage, statutory charges and slippage do to the result, and how do algorithms actually execute orders? Why does the best of many tested rules nearly always disappoint on new data? And how sure can he be that an edge is real rather than luck — the job of a t-test and the bootstrap? The mathematics is the statistics of Chapter 3 put to work: compounding, volatility, drawdown, the Sharpe ratio and its standard error.

⚠️ About the numbers on this page

Every price series, backtest result and strategy on this page is simulated or hypothetical; none describes a real index, stock, fund or trader, and nothing here is a recommendation to trade. Brokerage, securities transaction tax (STT), exchange charges, GST, stamp duty and SEBI fees are shown with illustrative rates to explain the mechanism — they change, and differ by broker and segment; check your broker's current charge sheet, the exchange circulars and the Income Tax Department before relying on any figure. Rules for algorithmic trading by brokers and their clients are set by SEBI and the exchanges and are revised from time to time.

The Anatomy of a Backtest

Every backtest, however sophisticated, has the same four parts: a price history, a signal computed from information available at each moment, a position that the signal implies, and the returns that the position earns. The one rule that must never be broken is about timing: a decision taken with today's closing price can only earn tomorrow's return. Our examples use the moving averages and crossovers of §7.1 (Technical Indicators as Mathematics), but the method is the same for any rule.

📘 Definition 7.12 — Backtest: signal, position, strategy return and equity curve

Let $P_0,P_1,\dots,P_N$ be daily closing prices and $r_t=P_t/P_{t-1}-1$ the daily returns (§3.1, Definition 3.1). A trading rule turns the information available at the close of day $t$ — prices up to $P_t$, never later — into a signal $s_t$: the fraction of capital to hold in the asset from the close of day $t$ to the close of day $t+1$ (for a long/flat rule $s_t\in\{0,1\}$; $s_t=-1$ would be a short position). If idle cash earns $r_f$ per day and each change of position costs a fraction $c$ of the capital traded, the strategy return on day $t$ is $$R_t=s_{t-1}\,r_t+(1-s_{t-1})\,r_f-c\,|s_t-s_{t-1}| .$$ Starting from capital $E_0$, the equity curve is $$E_t=E_0\prod_{k=1}^{t}(1+R_k).$$ A backtest is the computation of $R_t$ and $E_t$ over a past (or simulated) price history.

Using any information from after the close of day $t$ in $s_t$ — tomorrow's price, a figure published later, a revised data series — is called look-ahead bias, and it makes a backtest worthless.

Notice the subscripts: the signal decided at the close of day $t-1$, $s_{t-1}$, multiplies the return $r_t$ of the next day. The cost term uses $s_t-s_{t-1}$: it is paid at the close of day $t$, when the position is changed. Writing $s_t\,r_t$ instead — letting the signal "know" the day it is applied to — is the commonest bug in home-made backtests, and it can turn a useless rule into a spectacular one (Exercise I.7).

🧭 Running a backtest in a spreadsheet
  1. Prices. Column B: daily closes, oldest first, adjusted for splits, bonuses and dividends (§2.1) — otherwise a 1:1 bonus looks like a 50% crash.
  2. Indicator. Column C: whatever the rule needs, computed only from rows at or above the current row (for a 50-day average, =AVERAGE(B2:B51) in row 51, copied down).
  3. Signal. Column D: the position to hold from today's close to tomorrow's, e.g. =IF(B51>C51,1,0).
  4. Return. Column E, one row lower: =D51*(B52/B51-1)+(1-D51)*rf/252-cost*ABS(D52-D51) — yesterday's signal times today's return, cash otherwise, minus the cost of any change made today.
  5. Equity. Column F: =F51*(1+E52), starting from your capital. Add a buy-and-hold column for comparison.
  6. Scorecard. CAGR, volatility, Sharpe ratio, maximum drawdown, number of trades (Theorem 7.15), each for the strategy and for buy-and-hold.
✏️ Example 1 — A ten-day backtest by hand
A simulated share closes at ₹100, 102, 101, 104, 106, 105, 103, 107, 110, 108, 111 on days 0 to 10. The rule: at each close from day 2 on, hold the share for the next day if the close is above its 3-day simple moving average (the average of the last three closes, today's included); otherwise hold cash earning 0%. Each buy or sell costs 0.1% of the amount traded. Start with ₹1,00,000 in cash. Compute the equity curve and compare it with buying and holding.
Signals Day 2: SMA $=(100+102+101)/3=101$, and 101 is not above 101, so $s_2=0$. Day 3: SMA $=102.33$, $104\gt102.33$, $s_3=1$. Continuing: $s_4=1$, $s_5=0$ (105 is not above 105), $s_6=0$, $s_7=1$, $s_8=1$, $s_9=0$, $s_{10}=1$.
Returns $R_t=s_{t-1}r_t-0.001\,|s_t-s_{t-1}|$. Trades happen at the closes of days 3 (buy), 5 (sell), 7 (buy), 9 (sell) and 10 (buy).
dayclose$r_t$SMA$s_t$held$R_t$equity (₹)
3104+2.97%102.3310−0.10%99,900
4106+1.92%103.6711+1.92%1,01,821
5105−0.94%105.0001−1.04%1,00,759
6103−1.90%104.670001,00,759
7107+3.88%105.0010−0.10%1,00,658
8110+2.80%106.6711+2.80%1,03,480
9108−1.82%108.3301−1.92%1,01,495
10111+2.78%109.6710−0.10%1,01,394
(Days 1–2: no position, equity ₹1,00,000.)
Compare The rule ends at ₹1,01,394 (+1.39%); buy-and-hold ends at ₹1,11,000 (+11%). Without costs the rule would have made +1.91%; the five trades cost about 0.5% of capital.
Why The rule bought after the big up-days 3 and 7 and was out of the market on days 7 and 10, two of the best days, while it was in on the down-days 5 and 9. On a choppy series a trend rule keeps buying high and selling low ("whipsaw"). Ten days prove nothing either way — but the mechanics are exactly those of a ten-year backtest.

Once the equity curve exists, it is judged with the tools of Chapter 3. Because a strategy is just another return series, everything about volatility (§3.1), drawdown (§3.1, Definition 3.4) and risk-adjusted performance (§3.2) applies unchanged — and so do their standard errors.

📐 Theorem 7.15 (Formula) — The strategy scorecard

For a backtest with $N$ daily strategy returns $R_1,\dots,R_N$ (252 trading days a year, by convention), mean $\bar R$, sample standard deviation $s$ and daily cash rate $r_f$:

  1. CAGR: $\ \text{CAGR}=\left(\dfrac{E_N}{E_0}\right)^{252/N}-1$ (the years are $T=N/252$).
  2. Annualised volatility: $\ \sigma_{\text{ann}}=s\sqrt{252}$ (square-root-of-time rule, §3.1).
  3. Sharpe ratio: $\ S=\dfrac{\bar R-r_f}{s}\sqrt{252}$; against a benchmark with daily returns $b_t$, the information ratio uses the active returns $d_t=R_t-b_t$: $\ \text{IR}=\dfrac{\bar d}{s_d}\sqrt{252}$ (§3.2).
  4. Maximum drawdown MDD of the equity curve (§3.1), and the Calmar ratio $\text{CAGR}/\text{MDD}$.
  5. Exposure (fraction of days invested) and turnover (round trips — a buy and its sell — per year).

Each number is an estimate from one history: by §3.2 (how precise is an estimated Sharpe ratio?) the standard error of $S$ is about $\sqrt{(1+S^2/2)/T}$.

Where the formulas come from. If the equity grew at a constant rate $g$ a year for $T=N/252$ years, then $(1+g)^{T}=E_N/E_0$; solving for $g$ gives the CAGR. Volatility and Sharpe are the definitions of §3.1–3.2 applied to the strategy's returns, annualised with the square-root-of-time rule (which assumes the daily returns are roughly independent). ∎  A scorecard without a benchmark column is half a scorecard: "12% a year" means little until you know that buy-and-hold made 11% or 18% on the same data.

✏️ Example 2 — Reading a five-year scorecard
Kabir's spreadsheet backtest of a rule covers 5 years (1,260 trading days) of a simulated index. The equity grows from ₹10,00,000 to ₹17,70,000. The daily strategy returns have mean 0.050% and standard deviation 0.95%; cash earns an illustrative 6% a year. The highest equity before the worst fall was ₹14,80,000, and the lowest value after it ₹11,90,000. Compute the CAGR, annualised volatility, Sharpe ratio, maximum drawdown and Calmar ratio, and the standard error of the Sharpe ratio.
CAGR $1.77^{1/5}-1\approx12.10\%$ a year.
Volatility $0.95\%\times\sqrt{252}\approx15.08\%$ a year.
Sharpe Daily cash $=0.06/252\approx0.0238\%$, so $$S=\frac{0.0500-0.0238}{0.95}\times\sqrt{252}\approx0.44 .$$
Drawdown $\text{MDD}=1-11.9/14.8\approx19.6\%$; Calmar $=12.10/19.6\approx0.62$.
Precision $$\text{SE}(S)\approx\sqrt{\frac{1+0.44^2/2}{5}}\approx0.47 .$$ The Sharpe ratio is $0.44\pm0.47$: five years cannot distinguish this rule from one with no edge at all. (Check: daily growth $\approx\bar R-s^2/2$ (§3.1, volatility drag) gives $e^{1260(0.0005-0.0000451)}\approx1.77$, consistent with the equity numbers.)
Interactive 1 — Backtest Lab: A Moving-Average Rule Before and After Costs

Ten years of a simulated index (drift 12%, volatility 18% a year) start at ₹1 lakh. The rule holds the index while its close is above its moving average and sits in cash at an illustrative 6% otherwise. Grey: buy-and-hold; green: the rule before costs; red: after the cost per buy or sell you set (brokerage, charges and slippage together). The strip at the bottom shows the days in the market. "Random walk" has no pattern for the rule to find; "persistent trends" adds slowly changing drift. Press Play to backtest one new simulated market after another and keep a tally. (Log scale.)

0.15%
CAGR buy-and-hold—
CAGR rule, before costs—
CAGR rule, after costs—
Sharpe: hold / rule net—
max drawdown: hold / rule net—
round trips a year · in market—
rule (net) beat holding—

Transaction Costs, Slippage and Execution Algorithms

A backtest that trades at the closing price for free describes a market that does not exist. Every real trade pays explicit charges and loses a little to the market itself — and a rule that trades often pays these many times a year. In Indian equity markets the explicit charges on a trade typically include the broker's brokerage, the securities transaction tax (STT), the exchange's transaction charges, a small SEBI turnover fee, GST on the brokerage and charges, and stamp duty on the buy side; delivery sales may also attract a depository charge. Which apply, and at what rate, differs between delivery, intraday and F&O trades and changes over time.

📘 Definition 7.13 — Transaction costs, slippage and market impact
  • Explicit costs: brokerage, taxes and fees actually charged on the contract note.
  • Spread cost: a market buy pays the ask and a market sell receives the bid (§2.1), so a round trip loses the bid–ask spread; relative to the mid-price each side costs the half-spread $\dfrac{\text{ask}-\text{bid}}{2\cdot\text{mid}}$.
  • Slippage: the difference between the price at which the backtest (or the trader) decided to trade and the average price actually obtained, as a fraction of the decision price; it includes the spread, price moves while the order waits, and
  • market impact: the price move caused by the order itself, which grows with the order's size relative to the volume traded.

A round trip is a buy and the sell that closes it. The round-trip cost $c$ is the total of all these, as a fraction of the capital traded.

📐 Theorem 7.16 (Formula) — Cost drag and break-even cost

Suppose a strategy's capital grows by the factor $1+g$ in a year before costs, and it makes $n$ round trips in the year, each with the whole capital and costing a fraction $c$ of it. Then

  1. the net growth is $$1+g_{\text{net}}=(1+g)(1-c)^n,\qquad\text{so}\qquad g_{\text{net}}\approx g-nc(1+g)\approx g-nc\ \ \text{for small }nc;$$
  2. against a benchmark that grows by $1+b$, the strategy still wins after costs only if $c$ is below the break-even cost $$c^{*}=1-\left(\frac{1+b}{1+g}\right)^{1/n}\approx\frac{g-b}{n\,(1+g)} .$$

Costs scale with turnover: doubling the number of trades doubles the annual drag, while the gross edge per trade usually does not grow.

Why. Each round trip leaves a fraction $1-c$ of the capital invested, whatever the trade earned; $n$ of them multiply the capital by $(1-c)^n$, and multiplication does not care about the order in which gains and costs arrive. For small $nc$, $(1-c)^n\approx1-nc$ (the first terms of the binomial expansion), which gives the approximation. Setting $(1+g)(1-c^*)^n=1+b$ and solving gives $c^*$. ∎

✏️ Example 3 — What one round trip really costs
Kabir buys shares worth ₹1,00,000 for delivery and later sells them for ₹1,00,000. Use these illustrative charges (not current rates): brokerage ₹20 per executed order; STT 0.1% on the buy and on the sell value; exchange transaction charges 0.003% of turnover; SEBI fee ₹10 per crore of turnover; GST 18% on brokerage plus exchange and SEBI charges; stamp duty 0.015% on the buy value. Add slippage of 0.05% on each side. Find the round-trip cost in rupees and as a percentage.
Turnover Buy value + sell value $=$ ₹2,00,000.
Charges
itemcomputation₹
brokerage2 orders × ₹2040.00
STT0.1% × ₹2,00,000200.00
exchange charges0.003% × ₹2,00,0006.00
SEBI fee₹10 per crore × 0.02 crore0.20
GST18% × (40 + 6 + 0.20)8.32
stamp duty0.015% × ₹1,00,00015.00
explicit total269.52
Slippage $0.05\%\times$ ₹1,00,000 on each side $=$ ₹100.
Answer About ₹369.52, i.e. $c\approx0.37\%$ of the capital per round trip. Under these illustrative rates the tax on the transaction is by far the largest item; for a small account the flat brokerage matters more. Your own contract notes, not a forum post, are the right source for your costs.
✏️ Example 4 — When turnover eats the edge
A rule earns 16% a year before costs in a backtest, against 12% for buy-and-hold, and makes 20 round trips a year with the whole capital. (a) With the 0.37% round-trip cost of Example 3, what does it earn after costs? (b) What is the break-even cost? (c) A slower version of the rule trades only 4 times a year with the same gross return. What does it earn net?
(a) $$1.16\times(1-0.0037)^{20}\approx1.0771,$$ so about 7.7% a year — well below buy-and-hold. The approximation $16\%-20\times0.37\%\times1.16\approx7.4\%$ is close.
(b) $$c^*=1-\left(\frac{1.12}{1.16}\right)^{1/20}\approx0.175\%$$ per round trip. Above that cost the rule loses to simply holding the index.
(c) $1.16\times0.9963^4-1\approx14.3\%$. Same gross edge, a fifth of the costs.
Lesson The 4-point gross advantage is real money only if it survives a cost line in the backtest. Fast rules need very cheap execution; most of their "edge" in free-trading backtests is a bill that has not been sent yet.

How orders are actually executed. When a fund or a trader has to buy a large quantity — far more than is offered at the best ask — sending one market order would walk up the order book and pay a large impact cost. Instead the parent order is cut into many small child orders sent over minutes or hours. Computer programs that decide the size, price and timing of those child orders are execution algorithms; more broadly, algorithmic trading means any trading in which orders are generated or placed automatically by a program. In India, algorithmic orders are placed through brokers and are subject to SEBI and exchange rules on approval, testing, risk checks (price and quantity limits, order-rate limits, "kill switches") and audit trails; these rules are being revised for retail use of broker APIs, so check the current SEBI circulars and your broker's terms.

📘 Definition 7.14 — Execution algorithms and their benchmarks

If during a period the market trades quantities $q_1,\dots,q_m$ at prices $p_1,\dots,p_m$, the volume-weighted average price is $$\text{VWAP}=\frac{\sum_i p_i\,q_i}{\sum_i q_i}.$$

  • A TWAP algorithm splits the order into equal slices at equal time intervals (time-weighted).
  • A VWAP algorithm trades in proportion to the volume expected in each part of the day, aiming to buy at about the market's VWAP.
  • A percentage-of-volume (POV) algorithm trades a fixed share (say 5%) of whatever the market trades.
  • The implementation shortfall of a buy order is $$\text{IS}=\frac{\bar p_{\text{fill}}-p_{\text{decision}}}{p_{\text{decision}}}$$ (with the sign reversed for a sell), plus the opportunity cost of any shares not bought. It compares the result with the "paper" trade at the price when the decision was made.

Costs are usually quoted in basis points (bps): $1\text{ bp}=0.01\%$.

✏️ Example 5 — VWAP and implementation shortfall of a sliced order
A fund decides to buy 30,000 shares of hypothetical Kestrel Infra Ltd when the price is ₹500.00 and gives the order to a VWAP algorithm. In four parts of the day the market trades 2,00,000, 1,20,000, 1,00,000 and 1,80,000 shares at average prices ₹500.40, ₹501.10, ₹501.60 and ₹502.00. The algorithm buys 10,000, 6,000, 5,000 and 9,000 shares at average prices ₹500.50, ₹501.15, ₹501.70 and ₹502.05. Compute the market VWAP, the fund's average price, its slippage against VWAP and its implementation shortfall.
Schedule The child orders are exactly 5% of each period's volume — the algorithm followed the volume pattern.
VWAP $$\frac{2\times500.40+1.2\times501.10+1\times501.60+1.8\times502.00}{6}=501.22 .$$
Average fill $$\frac{10\times500.50+6\times501.15+5\times501.70+9\times502.05}{30}=501.295 .$$ Against VWAP: $(501.295-501.22)/501.22\approx1.5$ bps, about ₹2,250 on the order.
Shortfall Against the decision price: $(501.295-500)/500\approx25.9$ bps, i.e. $30{,}000\times1.295=$ ₹38,850.
Interpret The algorithm did its narrow job (almost exactly VWAP), yet the fund paid ₹38,850 more than its "paper" price because the price drifted up all day — partly, perhaps, because of its own buying. Which benchmark you use decides whether an execution looks good. A backtest that assumes fills at the decision price silently assumes zero shortfall.

Overfitting, Data Snooping and Out-of-Sample Testing

Kabir's rule has two numbers in it — the length of the moving average and the threshold — and it is tempting to try every combination and keep the best. That is exactly how false discoveries are manufactured. With enough attempts, some rule will look brilliant on any data, including data with no pattern at all, as Interactive 2 shows.

📘 Definition 7.15 — Overfitting, data snooping and out-of-sample testing
  • Overfitting: choosing a rule or its parameters so that it fits the accidental features (the noise) of one particular history; such a rule does well on that history and not on new data.
  • Data snooping (data mining bias, multiple testing): testing many rules on the same data and reporting the best, without allowing for the number of attempts.
  • The in-sample period is the data used to design and choose the rule; the out-of-sample period is data kept aside and used once, after all choices are frozen. In a walk-forward test the choice is repeated on a rolling in-sample window and each choice is evaluated on the following, unseen window.
  • Survivorship bias: testing only on instruments that still exist today (current index members, surviving funds), which leaves out the failures.
📐 Theorem 7.17 (Rule) — The best of many: what luck alone produces

Suppose $M$ strategies with no true edge are tested independently on $T$ years of data. By §3.2 (how precise is an estimated Sharpe ratio?) each estimated annualised Sharpe ratio is then approximately $N(0,1/T)$. Then:

  1. if each is tested at significance level $5\%$, the chance that at least one looks significant is $1-0.95^M$ (40% for $M=10$, 99.4% for $M=100$);
  2. the expected best Sharpe ratio is $\dfrac{e_M}{\sqrt T}$, where $e_M$ is the expected maximum of $M$ independent standard normals: $e_{10}\approx1.54$, $e_{100}\approx2.51$, $e_{1000}\approx3.24$ (roughly $\sqrt{2\ln M}$ for large $M$, which slightly overstates it);
  3. (Bonferroni) requiring each test to pass at level $\alpha/M$ keeps the chance of any false discovery at most $\alpha$: for one-sided tests at $\alpha=5\%$ the hurdle rises from $t\gt1.645$ to $t\gt z_{1-0.05/M}$, e.g. $2.58$ for $M=10$ and $3.09$ for $M=50$.

Tested variants of one rule are correlated, so the effective number of independent trials is smaller than $M$ — but it is never 1 once you have looked at more than one.

Why. (1) Each strategy fails the test with probability $0.95$; independently, all $M$ fail with probability $0.95^M$. (2) Each $\hat S_j=Z_j/\sqrt T$ with $Z_j\sim N(0,1)$, so $\max_j\hat S_j=\max_jZ_j/\sqrt T$ and its mean is $e_M/\sqrt T$; the values of $e_M$ come from numerical integration of $E[\max Z_j]=\int z\,M\varphi(z)\Phi(z)^{M-1}\,dz$. (3) The probability of a union is at most the sum of the probabilities: $P(\text{any false pass})\le M\cdot\alpha/M=\alpha$, whatever the correlation. ∎

strategies tried $M$$P(\text{some } p\lt5\%)$$e_M$luck's best Sharpe, 5 yearsluck's best Sharpe, 10 years
15%00 (± 0.45)0 (± 0.32)
1040%1.540.690.49
10099.4%2.511.120.79
1000≈ 100%3.241.451.03
✏️ Example 6 — 200 parameter combinations: skill or luck?
A forum member tested 200 combinations of moving-average lengths on 5 years of data and reports that the best one has a Sharpe ratio of 1.4, "much better than the market's 0.5". Treating the combinations as independent strategies with no edge, (a) what best Sharpe ratio would luck alone be expected to produce? (b) What is the chance that at least one of 200 no-edge strategies reaches 1.4?
(a) Each estimate has standard error $1/\sqrt5\approx0.447$; numerically $e_{200}\approx2.75$, so luck's expected best is about $2.75\times0.447\approx1.23$.
(b) One no-edge strategy reaches $1.4$ when $Z\ge1.4/0.447\approx3.13$, probability about $0.00087$. For 200: $$1-(1-0.00087)^{200}\approx16\% .$$
Interpret A best-of-200 Sharpe of 1.4 is only slightly better than what pure noise produces, and noise alone would produce it about one time in six. The reported number is evidence that the forum member searched hard, not that the rule works. Since the combinations are correlated, the effective number of trials is lower and the chance smaller — but the honest test is out of sample.
Interactive 2 — The Data-Snooping Machine: Pick the Best Backtest, Then Watch It Fail

A simulated market with no pattern (daily returns independent, mean zero). Each "strategy" switches between long and short at random moments, so none has any edge. The machine tries $M$ of them on the in-sample years, keeps the one with the highest in-sample Sharpe ratio (green), and then runs it on the next 3 years it has never seen (blue). Grey lines are some of the losers. Raise $M$ and watch the in-sample winner get more impressive — and the out-of-sample part stay a coin toss. Press Play for one new market after another.

winner's in-sample Sharpe—
its t-statistic—
luck's expected best—
winner's out-of-sample Sharpe—
average over markets: in / out—

What a good out-of-sample test looks like. Decide before looking which period is out-of-sample (for example the last 30% of the data), do all exploring and parameter choosing on the rest, write the final rule down, and run it on the held-out period once. If you then change the rule and test again on the same held-out data, it has become in-sample. A walk-forward test repeats this honestly through time: choose on years 1–5, trade year 6; choose on years 2–6, trade year 7; and so on — and judge only the stitched-together "trade" years. Other guards: prefer rules with few parameters and a reason to work, check that neighbouring parameter values give similar results (a lone spike in a parameter grid is a sign of noise), and test on several markets or periods.

Is the Edge Real? The t-test and the Bootstrap

Suppose Kabir has done everything right — no look-ahead, realistic costs, a genuine out-of-sample period — and the rule still made money. Could that be luck? Statistics answers with a hypothesis test: assume the rule has no edge and ask how surprising the observed result would be.

📘 Definition 7.16 — Null hypothesis of no edge, t-statistic and p-value

Let $d_1,\dots,d_n$ be the returns being tested: the strategy's excess returns over cash ("does it beat cash?") or its active returns over a benchmark ("does it beat buy-and-hold?"), with mean $\bar d$ and standard deviation $s_d$. The null hypothesis $H_0$ says the true mean is zero: no edge. The t-statistic is $$t=\frac{\bar d}{s_d/\sqrt n},$$ the observed mean measured in standard errors (§3.1, standard errors). The (one-sided) p-value is the probability, if $H_0$ were true, of a t-statistic at least as large as the one observed. A small p-value (conventionally below 5%) is evidence against "no edge" — it is not the probability that the rule is useless.

📐 Theorem 7.18 (Rule) — The t-test for an edge, and how many years it takes

If the $n$ daily returns $d_t$ are independent with mean $\mu$ and finite variance, and $n$ is large, then under $H_0:\mu=0$ the t-statistic is approximately standard normal (for small $n$ use Student's $t$ with $n-1$ degrees of freedom). With $T=n/252$ years and annualised Sharpe or information ratio $\hat S=\dfrac{\bar d}{s_d}\sqrt{252}$, $$t=\hat S\,\sqrt T .$$ So a one-sided test at 5% rejects "no edge" when $\hat S\sqrt T\gt1.645$, and a strategy whose true ratio is $S$ needs about $$T\approx\left(\frac{t^*}{S}\right)^2\ \text{years}$$ of data to reach a t-statistic of $t^*$: for $t^*=2$, 16 years at $S=0.5$ and 4 years at $S=1$.

Why. By the central limit theorem $\bar d$ is approximately normal with standard error $s_d/\sqrt n$, so under $H_0$ the ratio $t$ is approximately $N(0,1)$. Substituting $n=252T$: $$t=\frac{\bar d}{s_d}\sqrt{252T}=\left(\frac{\bar d}{s_d}\sqrt{252}\right)\sqrt T=\hat S\sqrt T .$$ Setting $S\sqrt T=t^*$ and solving for $T$ gives the years needed. ∎  This is the same fact as the standard error $\approx1/\sqrt T$ of a Sharpe ratio (§3.2), seen from the other side. It explains why so few trading edges can be proved: a respectable Sharpe ratio of 0.5 needs about 16 years of out-of-sample evidence to reach $t=2$.

✏️ Example 7 — Three years of daily returns: significant?
Over 3 years (756 trading days) out of sample, a strategy's daily excess returns over cash have mean 0.06% and standard deviation 1.1%. Compute the annualised Sharpe ratio, the t-statistic and the one-sided p-value. How many years of similar results would give $t=2$?
Sharpe $\hat S=\dfrac{0.06}{1.1}\sqrt{252}\approx0.866$ — a ratio many funds would be proud of.
t-statistic $t=\dfrac{0.0006}{0.011/\sqrt{756}}\approx1.50$; equivalently $0.866\times\sqrt3\approx1.50$.
p-value $P(Z\ge1.50)\approx6.7\%$. Not significant at 5%: a no-edge strategy would do this well about one time in fifteen.
Years $T=(2/0.866)^2\approx5.3$ years at the same Sharpe ratio. Note the p-value does not say there is a 6.7% chance the strategy is useless; it says how often luck alone would produce this.

The t-test leans on approximate normality. Strategy returns are often skewed (many small losses and a few big wins, or the reverse) and fat-tailed (§3.1), and with few trades the normal approximation can mislead. The bootstrap avoids the formula: it uses the computer to rebuild the sampling distribution from the data themselves.

🧭 Bootstrapping a strategy's edge
  1. Take the $n$ observed returns $d_1,\dots,d_n$ (daily, monthly or per trade).
  2. Resample: draw $n$ values from them at random with replacement (some appear twice, some not at all) and compute the statistic — mean, Sharpe ratio, CAGR, even maximum drawdown.
  3. Repeat $B$ times (say $B=10{,}000$). The $B$ values form the bootstrap distribution.
  4. Confidence interval: the 2.5th and 97.5th percentiles of the bootstrap distribution give an approximate 95% interval for the statistic.
  5. Test of "no edge": subtract $\bar d$ from every observation (so the data obey $H_0$), resample as above, and take as p-value the fraction of resampled means at least as large as the observed $\bar d$.
  6. If returns cluster (calm and wild spells, §3.1), resample blocks of consecutive days instead of single days (a block bootstrap).
✏️ Example 8 — Bootstrapping two years of monthly active returns
Kabir's paper-traded rule produced these 24 monthly returns over buy-and-hold (in %):
1.1, −0.7, −0.4, −5.5, 5.1, 3.5, −0.2, 2.5, 1.3, −0.8, 3.0, −0.2, −0.2, −1.4, 1.7, 0.4, 2.0, −0.9, 0.9, −1.6, 2.7, 1.1, 1.4, 1.6.
(a) Compute the mean, standard deviation, t-statistic and annualised information ratio. (b) A bootstrap with 10,000 resamples (seeded) gives a 95% interval for the mean of about $[-0.17\%,\ 1.50\%]$, and 5.4% of the resampled means are $\le0$; the centred bootstrap test gives $p\approx4.8\%$. Interpret. (c) What happens to $t$ if the best month (5.1%) is removed?
(a) Sum $=16.4$, so $\bar d=16.4/24\approx0.683\%$ a month; $s_d\approx2.117\%$; standard error $2.117/\sqrt{24}\approx0.432\%$; $t\approx0.683/0.432\approx1.58$ (one-sided $p\approx6.4\%$ with Student's $t$, 23 degrees of freedom). $\text{IR}\approx\dfrac{0.683}{2.117}\sqrt{12}\approx1.12$.
(b) The interval contains 0, and about 5% of resampled histories show no edge: the data are borderline — consistent with a real edge of about 0.7% a month, and also with none. The bootstrap and the t-test agree here because 24 monthly values are not far from normal.
(c) Without the 5.1% month: mean $\approx0.491\%$, $t\approx1.22$. One month carries a large part of the evidence. An impressive-looking IR of 1.12 from two years is a reason to keep paper-trading, not to borrow money.
⚠️ Common mistakes and myths
  1. Trading on the same bar the signal used. Multiplying today's return by a signal computed from today's close is look-ahead bias; the rule must earn the next period's return (Definition 7.12).
  2. "Costs are small, I'll add them later." At 0.37% per round trip, 20 round trips a year cost about 7% a year (Example 4). Costs belong in the first version of the backtest, with slippage.
  3. Using unadjusted prices or today's index members. Unadjusted splits and bonuses create fake crashes; testing only today's survivors ignores the companies that failed or were dropped (survivorship bias, Exercise III.6).
  4. "The backtest returned 22% a year." Without the benchmark, the drawdown, the number of trades and the number of variants tried, a CAGR is advertising, not evidence.
  5. Optimising until it looks good. The best of many backtests is biased upwards (Theorem 7.17); the more combinations tried, the higher the hurdle must be.
  6. Re-using the out-of-sample period. Once you tweak the rule after seeing out-of-sample results, that period is in-sample.
  7. Reading a p-value as the probability the strategy works. $p=0.04$ means "luck would produce this 4% of the time", not "96% chance of an edge"; with many strategies tried, many such p-values are expected.
  8. "A high Sharpe over one year proves skill." $t=S\sqrt T$: a Sharpe of 1.5 over one year is $t=1.5$, not significant.
  9. Assuming fills at the close or the decision price. Real orders pay the spread and impact, and large orders move the market (Example 5).
  10. "An algorithm removes risk." Automation executes a rule faster and more consistently; it does not make the rule profitable, and a bug or a missing risk check can lose money very quickly.
📈 Market case — Kabir's crossover rule, from backtest to verdict

Kabir tests a two-average crossover rule (long when a fast average is above a slow one, else cash) on 10 years of simulated daily index data. Before looking at anything he splits the data: years 1–7 in-sample, years 8–10 out-of-sample. All figures below are hypothetical outputs of his spreadsheet.

  1. Search. He tries 48 combinations (fast 5–50 days, slow 50–250 days). The best in-sample, before costs, has CAGR 17.8% against 12.1% for buy-and-hold, maximum drawdown 14% against 28%, and 6 round trips a year.
  2. Costs. At 0.37% per round trip (Example 3): $1.178\times0.9963^6\approx1.152$, so the net CAGR is about 15.2%. The rule survives costs in-sample.
  3. Edge against the benchmark. Net of costs the active return is 2.9% a year with tracking error 4.7%, so the in-sample $\text{IR}\approx2.9/4.7\approx0.62$ and $t=0.62\sqrt7\approx1.64$, one-sided $p\approx5\%$.
  4. Allow for the search. If the 48 variants were independent no-edge rules, luck's expected best IR would be $e_{48}/\sqrt7\approx2.23/2.65\approx0.84$, and Bonferroni would ask for $t\gt z_{1-0.05/48}\approx3.08$. The variants are correlated, so the truth is less severe, but $t=1.64$ is nowhere near a convincing hurdle.
  5. Out of sample, once. Years 8–10: strategy CAGR 10.9%, buy-and-hold 10.4%, $\text{IR}\approx0.10$, so $t=0.10\sqrt3\approx0.17$. The edge has essentially vanished.
in-sample (7 yrs)out-of-sample (3 yrs)
CAGR rule (net) / hold15.2% / 12.1%10.9% / 10.4%
information ratio0.620.10
t-statistic1.640.17
hurdle allowing for 48 tries≈ 3.11.645 (one test)

Verdict. The mathematics does not say the rule is worthless — the smaller drawdown may matter to Kabir — but it says the evidence for a return edge is weak: the in-sample result is what a search of 48 variants could easily produce, and the out-of-sample test shows almost nothing. He decides to keep paper-trading it and to write down, today, the rule and the test he will apply in two years. That written plan is the cheapest protection against fooling himself.

Exercises

All prices and results are simulated or hypothetical, all charges illustrative. Use 252 trading days a year. Give percentages to two decimals unless stated.

I
Backtest mechanics and the scorecard.
  • 1 A share closes at ₹50, 51, 50, 52, 53, 52, 54 on days 0–6. Rule: at each close from day 1, hold the share for the next day if today's close is above yesterday's, else hold cash (0%). No costs. Find the signals, the strategy's total return and buy-and-hold's return.
  • 2 Repeat part 1 with a cost of 0.2% on every buy and every sell.
  • 3 An equity curve grows from ₹5,00,000 to ₹8,40,000 in 6 years. Find the CAGR.
  • 4 Daily strategy returns have mean 0.04% and standard deviation 0.8%; cash earns 6% a year. Find the annualised volatility and Sharpe ratio.
  • 5 Month-end equity (₹ lakh): 10, 11.2, 12.5, 11.0, 9.8, 10.9, 13.1, 12.4, 13.6. Find the maximum drawdown and the gain that was needed from the trough to recover.
  • 6 Rule A: CAGR 9%, MDD 15%. Buy-and-hold: CAGR 12%, MDD 35%. Compute both Calmar ratios. Is A "better"? Explain what else you would want to know.
  • 7 In part 1, compute the "return" a careless spreadsheet would show if each day's signal were (wrongly) applied to that same day's return. Why is the result so good?
1 Signals $s_1,\dots,s_6=1,0,1,1,0,1$. Held on day 2 (return $50/51-1\approx-1.96\%$), day 4 ($+1.92\%$) and day 5 ($-1.89\%$): total $0.98039\times1.01923\times0.98113-1\approx-1.96\%$. Buy-and-hold: $54/50-1=+8\%$.
2 Trades at the closes of days 1, 2, 3, 5, 6: five costs of 0.2%. Total $\approx-2.94\%$.
3 $(8.4/5)^{1/6}-1=1.68^{1/6}-1\approx9.03\%$.
4 $\sigma_{\text{ann}}=0.8\%\times\sqrt{252}\approx12.70\%$; $S=\dfrac{0.04-0.0238}{0.8}\sqrt{252}\approx0.32$.
5 Peak 12.5, trough 9.8: MDD $=1-9.8/12.5=21.6\%$. Recovery needed $12.5/9.8-1\approx27.55\%$ (achieved at 13.1).
6 A: $9/15=0.60$; hold: $12/35\approx0.34$. A gives more return per unit of drawdown but less return. You would want the benchmark-relative statistics, the number of trades and costs, the number of variants tried, out-of-sample results, and the investor's horizon and tolerance for drawdowns. There is no single "better".
7 Same-day signals hold the share exactly on the up-days 1, 3, 4 and 6: $1.02\times1.04\times1.01923\times1.03846-1\approx+12.28\%$, more than buy-and-hold. The "rule" knew each day's close before trading it — look-ahead bias. A backtest that beats the market on every up-day is almost always a timing bug.
II
Costs, slippage and execution. Use the illustrative charges of Example 3 unless told otherwise.
  • 1 An intraday round trip: buy and sell ₹2,00,000 the same day. Illustrative intraday charges: brokerage ₹20 per order; STT 0.025% on the sell value only; exchange charges 0.003% of turnover; SEBI fee ₹10 per crore; GST 18% on brokerage, exchange and SEBI charges; stamp duty 0.003% on the buy value. Find the total and express it in basis points of ₹2,00,000.
  • 2 A rule earns 15% gross with 30 round trips a year at 0.30% each. Find the net return.
  • 3 In part 2, find the break-even round-trip cost against a 12% buy-and-hold.
  • 4 A scalper makes 4 round trips a day for 250 days, each costing 0.05% of capital. Find the annual cost drag. What gross profit per trade is needed just to break even?
  • 5 The best bid is ₹249.90 and the best ask ₹250.10. In basis points of the mid-price, what does a small market buy pay in half-spread, and what does a round trip lose to the spread?
  • 6 The market trades 3,00,000, 1,00,000 and 2,00,000 shares at average prices ₹240.00, ₹241.50 and ₹242.00. An algorithm buys 6,000, 2,000 and 4,000 shares at ₹240.10, ₹241.60 and ₹242.20. Find the market VWAP, the algorithm's average price and its slippage against VWAP.
  • 7 A fund decides to buy 12,000 shares at ₹820. By the close it has bought 10,000 at an average ₹823.50, and the price has risen to ₹830, so it gives up on the rest. Find the implementation shortfall: execution cost, opportunity cost and total in basis points of the paper trade.
1 Brokerage ₹40; STT $0.025\%\times2{,}00{,}000=$ ₹50; exchange $0.003\%\times4{,}00{,}000=$ ₹12; SEBI ₹0.40; GST $18\%\times52.40\approx$ ₹9.43; stamp ₹6. Total ≈ ₹117.83, about 5.9 bps. Much cheaper than delivery here — which tempts people to trade far more often.
2 $1.15\times0.997^{30}-1\approx5.09\%$.
3 $c^*=1-(1.12/1.15)^{1/30}\approx0.088\%$ per round trip.
4 1,000 round trips: $1-0.9995^{1000}\approx39.4\%$ of capital a year (the linear estimate $1000\times0.05\%=50\%$ overstates it because costs are taken from a shrinking balance). Each trade must earn at least 0.05% gross just to stand still.
5 Mid ₹250.00, half-spread $0.10/250=4$ bps; a round trip loses the full spread, 8 bps — before any charges.
6 VWAP $=\dfrac{3\times240+1\times241.5+2\times242}{6}\approx$ ₹240.917; algorithm average $=\dfrac{6\times240.10+2\times241.60+4\times242.20}{12}=$ ₹241.05; slippage $\approx5.5$ bps above VWAP.
7 Execution cost $10{,}000\times3.50=$ ₹35,000; opportunity cost $2{,}000\times(830-820)=$ ₹20,000; total ₹55,000 on a paper trade of $12{,}000\times820=$ ₹98,40,000, i.e. about 55.9 bps. Not trading is not free when the price runs away.
III
Overfitting, data snooping and out-of-sample testing.
  • 1 50 independent strategies with no edge are each tested at the 5% level. Find the probability that at least one passes and the expected number that pass.
  • 2 Find the Bonferroni one-sided hurdle for the t-statistic with 50 tests at overall level 5%.
  • 3 With 4 years of data, about what best Sharpe ratio do 100 independent no-edge strategies produce on average?
  • 4 A rule has 3 parameters and each is tried at 10 values. How many combinations are there? With 8 years of data, what best Sharpe would luck give if they were independent? Why is the true figure lower?
  • 5 A rule's in-sample Sharpe is 1.6 over 5 years; out of sample it is −0.1 over 2 years. Compute both t-statistics and interpret.
  • 6 Of 100 equally weighted shares in a hypothetical universe, 20 were delisted over 10 years with an average total return of −60%, while the 80 survivors returned +150% on average. Compare the true return of the equal-weight portfolio with that of a backtest on survivors only.
1 $1-0.95^{50}\approx92.3\%$; expected number $50\times0.05=2.5$.
2 $z_{1-0.001}\approx3.09$.
3 $e_{100}/\sqrt4\approx2.51/2\approx1.25$.
4 $10^3=1000$ combinations; $e_{1000}/\sqrt8\approx3.24/2.83\approx1.15$. Neighbouring parameter values give nearly the same trades, so the results are highly correlated and the effective number of independent trials is much smaller than 1000 — but still far more than one.
5 In-sample $t=1.6\sqrt5\approx3.58$; out of sample $t=-0.1\sqrt2\approx-0.14$. The in-sample result looks highly significant, but only if it was the only rule tried; the out-of-sample failure is what overfitting usually looks like. Two years is short, so it does not prove the rule worthless either — but there is no evidence for it.
6 True: $\dfrac{80\times2.5+20\times0.4}{100}-1=+108\%$. Survivors only: $+150\%$. Leaving out the failures adds 42 percentage points over 10 years.
IV
Testing the edge: t-test and bootstrap.
  • 1 Over 5 years of daily data a strategy's active returns have mean 0.03% and standard deviation 0.6%. Find the t-statistic and the annualised information ratio.
  • 2 A strategy's true Sharpe ratio is 0.4. How many years are needed, on average, for $t=2$? For $t=3$?
  • 3 A fund manager reports a Sharpe ratio of 1.2 over 2 years. Find $t$ and the one-sided p-value. Is it significant at 5%? At 1%?
  • 4 36 monthly active returns have mean 1.0% and standard deviation 4.0%. Find $t$ and the one-sided p-value (normal approximation, then Student's $t$ with 35 degrees of freedom).
  • 5 The returns of 5 trades are 2%, −1%, 3%, 0%, −2%. Random draws pick positions 3, 1, 3, 5, 2. Write down this bootstrap resample and its mean; compare with the original mean.
  • 6 Why does resampling single days destroy volatility clustering, and how does a block bootstrap fix it?
  • 7 Two variants of a rule each show a Sharpe ratio of 0.8 over 6 years, and their daily returns have correlation 0.9. Is "two independent confirmations" a fair description? Explain.
1 $n=1260$: $t=\dfrac{0.03}{0.6}\sqrt{1260}\approx1.77$; $\text{IR}=\dfrac{0.03}{0.6}\sqrt{252}\approx0.79$ (check: $0.79\sqrt5\approx1.77$).
2 $(2/0.4)^2=25$ years; $(3/0.4)^2\approx56$ years.
3 $t=1.2\sqrt2\approx1.70$, $p\approx4.5\%$: just significant at 5% for one honest test, not at 1% (which needs $t\gt2.33$). If the manager is the best of many, it means little.
4 $t=1.0/(4.0/6)=1.5$; $p\approx6.7\%$ (normal), $\approx7.1\%$ (Student's $t$, 35 d.f.). Not significant at 5%.
5 Resample $\{3\%, 2\%, 3\%, -2\%, -1\%\}$, mean $1.0\%$, against the original mean $0.4\%$. The trade at position 4 (0%) was not drawn and the 3% trade was drawn twice — that variation, repeated thousands of times, is what the bootstrap distribution measures.
6 Shuffling individual days scatters the wild days evenly through the resample, so calm and stormy spells disappear and drawdowns look smaller than they were. Drawing blocks of, say, 20 consecutive days keeps short-range dependence (clusters, short trends) inside each block.
7 No. With correlation 0.9 the two return series are nearly the same bet, so the second adds little information. Each has $t=0.8\sqrt6\approx1.96$; together they are roughly one test with $t\approx2$, not two.
🎯 Practice corner NISM-styleCase studyInterview

E1. NISM-style Which of the following is an example of look-ahead bias in a backtest? (A) Including brokerage and STT in the returns (B) Buying at a day's close using a signal computed from that day's closing price and earning the next day's return (C) Using that day's closing price to decide to buy at that same day's opening price (D) Testing the rule on a separate out-of-sample period.

Idea Look-ahead bias uses information not yet available when the trade is made.
Check (A) and (D) are good practice; (B) uses only information available at the time of the trade; in (C) the closing price is not known at the open.
Answer (C).

E2. NISM-style A strategy earns 15% a year before costs and makes 25 round trips a year with its whole capital, each costing 0.2%. Its return after costs is closest to (A) 14.8% (B) 12.5% (C) 9.4% (D) 5.0%.

Compute $1.15\times0.998^{25}-1\approx9.39\%$ (the shortcut $15\%-25\times0.2\%=10\%$ is close). (A) charges one round trip; (B) charges half the trips; (D) charges the cost 50 times.
Answer (C).

E3. NISM-style During a session a share trades 100 shares at ₹500, 300 shares at ₹502 and 100 shares at ₹507. Its VWAP for the session is (A) ₹502.60 (B) ₹503.00 (C) ₹502.00 (D) ₹504.50.

Compute $\dfrac{100\times500+300\times502+100\times507}{500}=\dfrac{2{,}51{,}300}{500}=502.60$. (B) is the simple average of the three prices, ignoring volume.
Answer (A).

E4. Case study A hypothetical backtest over 6 years (1,512 days): equity ₹4,00,000 → ₹7,20,000; daily excess returns over cash: mean 0.045%, standard deviation 1.0%; 15 round trips a year; the backtest charged no costs. (a) Find the CAGR, Sharpe ratio and t-statistic. (b) Re-estimate the CAGR with a realistic 0.35% per round trip. (c) The rule was the best of 30 variants. What best Sharpe would luck give (independent variants; $e_{30}\approx2.04$)? Conclude.

(a) CAGR $=1.8^{1/6}-1\approx10.29\%$. $S=\dfrac{0.045}{1.0}\sqrt{252}\approx0.714$; $t=0.714\sqrt6\approx1.75$.
(b) $1.1029\times0.9965^{15}\approx1.0463$: about 4.6% a year net — roughly what cash earns.
(c) $2.04/\sqrt6\approx0.83$: luck's expected best Sharpe exceeds the 0.71 reported.
Conclude Before costs the result is unremarkable for a best-of-30; after costs it is close to cash. No evidence of an edge.

E5. Case study A walk-forward test chooses a parameter on each 4-year window and trades the following year. The four out-of-sample years give strategy returns +14%, −6%, +9%, +2% against buy-and-hold +10%, −12%, +15%, +6%. (a) Find each series' CAGR over the 4 years and the strategy's mean annual active return. (b) Which year did the rule help most, and what does that suggest about what it really does?

(a) Strategy: $(1.14\times0.94\times1.09\times1.02)^{1/4}-1\approx4.48\%$; hold: $(1.10\times0.88\times1.15\times1.06)^{1/4}-1\approx4.22\%$. Active returns $+4,+6,-6,-4$ points: mean $0\%$.
(b) The falling year (−6% against −12%). The rule reduces exposure in down-trends and gives up part of the up-years: its value is lower drawdown, not extra return. With four years and mean active return zero, there is no return edge to test at all.

E6. Interview 1,024 fund managers with no skill each have a 50% chance of beating the index in any year, independently. How many do you expect to beat it 10 years in a row, and what is the chance that at least one does?

Expected $1024\times(1/2)^{10}=1$ manager.
At least one $1-(1-1/1024)^{1024}\approx1-e^{-1}\approx63\%$.
Point A ten-year unbroken record is exactly what a large crowd of coin-flippers produces. Before admiring the winner, ask how many started — the data-snooping problem with people instead of parameters.
🧑‍🏫 For the teacher

Suggested plan — 2 lectures × 60 min.

  1. Building and costing a backtest. Start with the 📍 situation and ask the class whether "22% a year" is believable and what they would need to know. Definition 7.12 with the timing rule; Example 1 on the board, then Exercise I.7 to show what look-ahead does. Scorecard (Theorem 7.15) and Example 2, ending on the standard error. Project Interactive 1: set costs to 0, then 0.30%; switch between random walk and trends; let Play run and read the tally. Costs (Definition 7.13, Theorem 7.16), Example 3 and 4, then execution and VWAP (Definition 7.14, Example 5).
  2. Is it real? Data snooping (Definition 7.15, Theorem 7.17) with Example 6. Project Interactive 2 at $M=1$, then 1000: ask the class to predict the out-of-sample Sharpe before pressing New market. The t-test (Theorem 7.18), Example 7 and the "years needed" table; the bootstrap procedure and Example 8. Finish with the 📈 Market case and Practice corner E6.

Class activity. Give each student a different seed and the spreadsheet of the 🛠️ project with a pure random-walk series. Each tries ten parameter values and reports the best in-sample Sharpe. Collect them on the board: the class will "discover" many good rules on data with nothing to find. Then everyone evaluates their best rule on a common held-out period.

Misconceptions to watch for. Signals applied to the same day's return; costs as an afterthought; the p-value as the probability the rule works; "my backtest has a Sharpe of 2" from one year; confusing the VWAP benchmark with a good execution; believing automation adds an edge.

Suggested homework. I(1, 2, 5, 7), II(1, 2, 4, 6), III(1, 4, 6), IV(1, 2, 5, 7); Practice corner E2, E4, E5.

🛠️ Do it yourselfSpreadsheet + Python mini-project · about 90 minutes

Project. Backtest a moving-average rule on a simulated price series with costs, split the data into in-sample and out-of-sample, and test the result with a t-test and a bootstrap.

Why it matters. Doing it once yourself — and watching a "great" rule evaporate out of sample — protects you from every polished backtest you will be shown later.

What to do.

  1. Simulate 10 years of prices (2,520 rows): in A2:A2521 the day number; in B1 the starting price 100; in B2 =B1*EXP((0.12-0.18^2/2)/252+0.18/SQRT(252)*NORM.S.INV(RAND())), copied down (GBM, §3.3). Copy and Paste Values so the series stops changing.
  2. Signal: with the average length $n$ in cell H1, in C(n+1): =AVERAGE(OFFSET(B2,ROW()-1-$H$1,0,$H$1,1)) or simply =AVERAGE(B2:B51) for $n=50$; in D: =IF(B51>C51,1,0).
  3. Returns with cost per trade in H2 and cash rate in H3: E52 =D51*(B52/B51-1)+(1-D51)*$H$3/252-$H$2*ABS(D52-D51); buy-and-hold in F52 =B52/B51-1; active return G52 =E52-F52.
  4. Scorecard separately for rows of years 1–7 and years 8–10: CAGR =PRODUCT(1+E52:E1815)^(252/COUNT(E52:E1815))-1 (array formula), Sharpe =(AVERAGE(E52:E1815)-$H$3/252)/STDEV.S(E52:E1815)*SQRT(252), IR from column G, t-statistic $=\text{IR}\sqrt T$, and the maximum drawdown of the equity column.
  5. Snoop on purpose: try $n=$ 10, 20, …, 200 in-sample and note the best. Then evaluate only that $n$ on years 8–10.
  6. Bootstrap in Python: import numpy as np; d=np.loadtxt('active.csv'); rng=np.random.default_rng(1); b=np.array([rng.choice(d,len(d)).mean() for _ in range(10000)]); print(np.percentile(b,[2.5,97.5]), (b<=0).mean()).

Hand in: the equity chart (hold, gross, net), the two scorecards, the in-sample best $n$ with its out-of-sample IR and t-statistic, the bootstrap interval, and three sentences on what you conclude.

Going further. Add the "persistent trends" ingredient of Interactive 1 (a slowly varying drift) and see whether the rule now survives out of sample; replace single-day resampling by blocks of 20 days; or download an index's daily closes from the exchange website and repeat — remembering that every look at the held-out years uses them up.

Instructor note: on pure GBM data the rule's expected active return is negative (it is out of a rising market part of the time), so most students should find in-sample "winners" that fail out of sample — that is the lesson. With 21 values of $n$ the variants are strongly correlated; students can estimate the effective number of trials by comparing the spread of their in-sample Sharpe ratios with $1/\sqrt7\approx0.38$.

🧪 Self-check quiz

0/5 correct

Q1. A strategy has an annualised Sharpe ratio of 0.5 measured over 4 years of daily data. Its t-statistic for "no edge" is about

Q2. A rule makes 50 round trips a year, each costing 0.3% of capital. Its annual cost drag is roughly

Q3. An out-of-sample period is

Q4. 20 independent strategies with no edge are each tested at the 5% level. The probability that at least one appears significant is about

Q5. In a bootstrap of a strategy's daily returns, each resample is formed by