World markets: method

Code: pipeline/tipsheet/compute/world.py (build) and publish/world.py (bundles). Tests: pipeline/tests/test_world.py. Data bank adapters: src/databank/connectors/kite_global.py and ecb_exr.py in the Data bank.

The licence rule

Kite Connect data is licensed to the account holder for personal use (PLAN.md, decision 2), and index levels belong to their providers (S&P, Dow Jones, Nasdaq, FTSE Russell, Deutsche Börse, Euronext, Nikkei, Hang Seng Indexes, SSE, ASX). So this section publishes only derived statistics:

• percentage returns over fixed horizons, in local currency and in rupees;
• calendar-year returns;
• rolling correlations and betas of weekly returns;
• drawdown from peak, in percent.

No index level, price, quote or rebased growth curve is published for any index here. A growth-of-100 curve is a level series in disguise, so there is none. Every bundle repeats the rule in meta.licence_rule, and the licence gate (publish/licence_gate.py) refuses any world/ column whose name suggests a level, close, price or rebased value. The Data bank contract for the Kite dataset carries licence.redistribution: internal_only.

MSCI index data: internal use only (validation/research). MSCI’s terms prohibit redistribution and derived works; not published (decision 2026-10-02). An earlier owner approval to publish MSCI-derived returns was withdrawn the same day on reading msci.com/terms-of-use. The licence gate refuses any bundle whose name or columns contain “msci”, whose meta.source* mentions MSCI, or that names the msci_index_levels dataset. See “MSCI (internal only)” at the end.

What is measured

For Nifty 50 and 11 overseas indices (S&P 500, Dow Jones Industrial Average, Nasdaq Composite, Nasdaq-100, FTSE 100, DAX, CAC 40, Nikkei 225, Hang Seng, Shanghai Composite, S&P/ASX 200):

• Return grid (world/returns_grid): 1 week, 1 month, 3 months, year to date, 1 year, and 3 and 5 years annualised, in local currency and in rupees. Plus the current drawdown, the worst drawdown since 2004 and its date, and the date of the last peak. Each row says whether its latest data is official or a provider_quote (see the break below), and whether it is stale.
• Calendar years (world/calendar_years): every year from 2005, local and rupee terms. The current year runs to ytd_through.
• Correlation and beta (world/correlation_weekly, world/beta_weekly): Nifty’s correlation with each market over the last 52 weeks of Friday-ending weekly returns in local currency (at least 40 weeks with both trading), and Nifty’s beta to each market (covariance over the market’s variance: the % Nifty moved per 1% move in that market). Weekly from 2005.
• Drawdown (world/drawdown_weekly): % below the highest close since 2004-01-01, weekly. A common start keeps the markets comparable; Nasdaq and Nifty histories reach back to 1990 but their peaks are searched from 2004 like everyone else’s.
• FX and commodities (world/fx_commodities, world/fx_commodities_calendar_years): % moves over the same horizons for the dollar, euro, pound and yen in rupees, the ICE dollar index, the Fed broad dollar index, gold in rupees and in dollars, and Brent in dollars and rupees.
• Stat bundles (world/stats/*): Nifty’s 1-year correlation with the S&P 500, and its median correlation across the 11 markets.
• Global proxies (world/proxies_grid, world/proxies_calendar_years, world/proxies_correlation_weekly): 20 more markets and asset classes measured through the official daily NAVs of US-listed ETFs, taken from the issuers. See the section below.

Sources

Where an official public series exists it replaces Kite: the S&P 500 uses FRED from 2016-05-31 and Kite before (Kite equals the official close on all 2,125 overlapping days before the break), and both Nasdaq indices use Nasdaq’s own history throughout. The other eight overseas indices are Kite only.

Nifty 50 is the price index, not the total-return index, so it compares like for like with the overseas price indices. The DAX is the exception among those: it is a total-return index by construction, so its returns include dividends.

Rupee terms. Rupee return = (1 + local return) × (1 + change in rupees per unit of the local currency) − 1. Rupees per unit come from ECB reference rates: (INR per euro) ÷ (currency per euro), taking the rate on or before each close, within 4 days. ECB-derived USD/INR agrees with CCIL’s: median gap 0.04% over 5,136 days, 99th percentile 0.56%. The ECB fixing is at 14:15 CET, so for Asian and US closes the FX rate is a few hours off the close. That matters only for 1-week figures and only at the margin.

Kite: access, storage and cleaning

Access. The Data bank adapter reads the Kite session from the Aftermarket Report’s .env (pointed to by KITE_ENV_FILE in the Data bank .env), so both projects share one set of tokens. The daily access token expires around 06:00 IST. Before each fetch the adapter checks it and, if expired, renews it from the stored long-lived refresh token and writes the new token back. This runs unattended. Verified on 2026-10-01: the stored token had expired and was renewed without a login. If the refresh token is ever revoked, renewal fails, only this dataset fails in the 06:00 Data bank run, the warehouse keeps the previous snapshot, and this section keeps publishing from it with stale flags set. Fix: run amr login once in the Aftermarket Report.

Storage. Data bank dataset kite_global_index_history: raw JSON artifacts, contract, audit, a daily update policy (10-day look-back, ending yesterday because today’s candle is still forming), and the 2004 backfill. The authorization header is redacted from stored metadata and is not part of any fingerprint.

Cleaning (counts from the 2026-10-01 build, 59,000 rows):

1. Weekend rows dropped (918). None of these markets trades on a weekend. Most are US30’s forward-filled weekends from 2004-12; others are stray quotes, such as JAPAN225 Sundays in 2017 and a HANGSENG Sunday print 26% above the Friday close in 2008.
2. Rows on exchange holidays dropped, using each exchange’s own session calendar (exchange_calendars: NYSE, LSE, Xetra, Euronext Paris, Tokyo, HKEX, Shanghai, ASX). Before the break this removes only real closures (Hong Kong typhoon days, Whit Monday on Xetra). After it, it removes holiday quotes, 1 to 42 per instrument.
3. Forward-filled bars dropped (open = high = low = close = the previous close).
4. Bars dated on or after the day they were fetched (IST) dropped: the candle is still forming.

Two source bars whose high-low range excluded the close (UK100 2011-08-05, USCOMPOSITE 2023-08-04) keep their close; the Data bank nulls their range and flags range_nulled.

The series break

Kite’s GLOBAL series changed character on 2024-11-07, the day US100 and US10YRYIELD first appear. The date is found from the data: the first run of five days on which Kite’s US500 departs from the official S&P 500 by more than 0.01%. KITE_AUDIT.md put the change at 2025-03; the data shows it starts in November 2024.

Since then the values are provider quotes, at times CFD prices on a fixed tick grid, not official closes. Measured where an official close is public:

Before the break, US500 matches the official close on 100% of 2,125 days and USCOMPOSITE on 97.4% of 5,246 days.

The three US series above use the official closes, so this error does not reach them. For the eight Kite-only markets (Dow, FTSE 100, DAX, CAC 40, Nikkei, Hang Seng, Shanghai, ASX 200), every current figure carries an error of about this size. Their rows are marked basis: provider_quote. The share of closes on a 0.25 tick grid shows where the quotes are clearly CFD-like: US500 59% of the time since the break against 9.5% before, Hang Seng 41% against 4%, Nasdaq-100 always, and the DAX in some recent months. The other markets look like official closes on this test, but the Nikkei check below shows they can still drift.

Checks

Calendar-year returns against the providers’ published figures (price returns; the DAX is total return):

2023 matches exactly everywhere. In 2024 the year-end close falls after the break: the Dow is 0.07 pp off, the Nikkei 0.17 pp and Shanghai 0.66 pp, because Kite’s last 2024 bar is a provider quote (Shanghai 3,371.56 against the official 3,351.76; Nikkei 39,951.90 against 39,894.54). The others match. These comparisons run on every build and are stored in world/returns_grid under meta.checks.

Other checks: ECB-derived USD/INR against CCIL (above); the pipeline-wide licence gate; strict JSON (every bundle is serialised with NaN and infinity forbidden).

Caveats

• Post-break figures for the eight Kite-only markets carry the provider-quote error above. Treat 1-week moves under about half a percentage point as noise.
• Weekly correlations: US markets close after India each day. Weekly returns absorb most of that lag, but part of a US week’s move lands in Nifty’s next week, which pulls measured correlation down a little.
• Rupee-term returns use a 14:15 CET exchange rate, not a rate at each market’s close.
• Calendar years start in 2005 because Kite history starts in 2004. Drawdowns are measured from 2004, so a market still below a pre-2004 peak (the Nasdaq after 2000, the Nikkei after 1989) is shown against its post-2004 high.
• Gold in rupees is the domestic price including import duty; gold in dollars is the LBMA price.
• The Shanghai and Hang Seng figures are in yuan and Hong Kong dollars; the Hong Kong dollar is pegged to the US dollar.

Global proxies: US-listed ETFs’ official NAVs

Rebuilt 2026-10-01. Code: build_proxies, total_return_index, alignment_check and their helpers in compute/world.py, _publish_proxies in publish/world.py. Data bank dataset etf_issuer_nav_history (source etf_issuers).

The first version of this section used Indian international funds’ NAVs from AMFI. The owner rejected them as stale. Many are struck a day late against overseas closes, several are active portfolios, and the Indian-listed international ETFs trade at large premiums under the overseas investment limit. That code path is removed. Every proxy is now the official NAV of a US-listed ETF, taken from the issuer, with basis: issuer_nav.

Where the NAVs come from

Probed on 2026-10-01, in the order the brief set:

Terms. BlackRock’s website terms grant personal, non-commercial use, forbid redistributing site content for public purposes, and forbid robots that copy site materials without permission. SSGA’s material says it may not be reproduced without consent. The dataset is therefore marked internal_only in the Data bank, like Kite. Only derived statistics are published: returns, drawdown in percent and correlations. No NAV, level or rebased curve is published. The Data bank fetches one screener file and two or three workbooks a day. The owner should decide whether this use is acceptable or whether to ask BlackRock for permission.

The proxies

All iShares, all passive. History starts at the fund’s inception.

FXI (China large caps, from 2004) is also held in the Data bank but not published; MCHI is the MSCI fund. SPY and GLD (SSGA) are held for cross-checks.

Building a total return

Each fund’s total-return index chains the workbook day by day: (NAV + the distribution going ex that day) × split factor ÷ the previous NAV. It is held in memory only. Checked against the issuer’s own monthly total NAV returns in the same workbook, the chain matches to within 0.03 pp a month for most funds. The largest gap is 0.37 pp, in one EWW month in 2010. One SLV and IAU month (November 2014) is wrong in the issuer’s Performance sheet, which repeats December’s value.

Days after a fund’s last workbook come from the screener. Their distributions are not yet known, so they are chained from the issuer’s year-to-date total NAV return: value = value at the prior year-end × (1 + YTD). That figure matched the workbook chain to 4 decimals for TLT, IEF, TIP, EWJ, IVV, EEM and IAU. On every overlapping day the build compares the two (ytd_check_max_abs_pp: 0.0 on 2026-10-01). Workbooks are refreshed on a 10-business-day rotation, so distributions_through trails as_of by at most two weeks.

Data quirks (Data bank parser):

• Bare & in hyperlinks makes the workbook ill-formed XML; it is escaped before parsing.
• -- means missing.
• The issuer fills weekends and US holidays with the previous NAV and -- shares, for example 2024-03-29 and 2024-03-31. These rows are flagged carried_forward and dropped: 1 to 5 per fund.
• The Distributions sheet lists announced future ex-dates, which are ignored until they appear in the history.
• Splits. iShares histories are already split-adjusted by the issuer: no NAV step in 1996-2026 matches a share split. The parser still flags a day when the NAV step is within 5% of 1/k and shares outstanding move within 15% of k-fold (k = 2, 3, 4, 5, 8, 10, 20 or their inverses), and the build multiplies that day’s step by k. A Data bank audit fails on any unexplained day-on-day move beyond ±35%. The largest real move is 28% (EWY, 2008-10-30).

Returns, drawdown and rupee terms

Returns are in US dollars, the fund’s own terms, and in rupees using ECB-derived rupees per dollar (14:15 CET). Drawdown runs from each fund’s inception, in both currencies. Calendar years run from 2005, and a year needs a value in the last 10 days of the prior year.

Time zones and the Nifty correlation

NAVs are struck at 4 pm New York time, after India’s close. Non-US funds value their holdings at local closes, adjusted to the US close by iShares’ fair-value pricing. The pipeline measures two pairings against Nifty from 2016: the fund’s NAV and Nifty’s close on the same calendar date, and the NAV of day t against Nifty’s next session. Same-date pairing gives the higher weekly correlation for 14 of 20 funds (IVV 0.524 against 0.506; EWJ 0.478 against 0.427; INDA 0.929 against 0.778). The six exceptions are the three Treasury funds (correlations near zero; the gap is 0.04 to 0.07), the two REIT funds (0.012 and 0.025) and GSG (0.007). So world/proxies_correlation_weekly uses Friday-ending weeks on the same calendar date, each side in its own currency (fund in dollars, Nifty in rupees), the same convention as world/correlation_weekly. A rolling correlation needs at least 40 of 52 weeks. Both pairings are stored per fund in world/proxies_grid under meta.checks.nifty_alignment.

Validation against indices

The warehouse holds indices for four proxies. All comparisons are in US dollars and use monthly returns. The indices are price returns. The fund’s NAV price return (distributions left out) isolates tracking and index-definition gaps; the total-return gap adds the yield.

IVV’s NAV tracks the S&P 500 to within 0.05 pp a year, and its total-return lead (about 1.3 to 1.6 pp) is the dividend yield. IAU’s tracking error against the WGC series reflects month-end timing; its long-run gap is the 0.25% fee. The Japan and Hong Kong gaps are index-definition differences, not data errors. The Hang Seng and Nikkei come from Kite and are internal. Cross-issuer checks on daily NAV price returns since 2016: IVV against SPY, correlation 0.99779 and median daily gap 0.0008 pp (ex-dates differ, so the 99th percentile is 0.45 pp); IAU against GLD, 0.99977 and 0.0012 pp. These checks are rebuilt on every run and stored under meta.checks in world/proxies_grid.

What the bundles hold

• world/proxies_grid: one row per proxy. Columns: key, market, asset_class, region, ticker, proxy, index, issuer, fund_inception, source, basis, currency, as_of, days_behind, stale, first_date, distributions_through, nifty_alignment, ret_{1w,1m,3m,ytd,1y,3y_ann,5y_ann}_usd_pct and _inr_pct, dd_now_usd_pct, max_dd_usd_pct, max_dd_usd_date, dd_now_inr_pct, max_dd_inr_pct and corr_nifty_1y.
• world/proxies_calendar_years: indices[] with key, name, currency, ticker, proxy, basis, returns_usd and returns_inr, from 2005.
• world/proxies_correlation_weekly: corr_<key>, Nifty’s rolling 52-week correlation with each proxy.

Caveats

• These are fund returns, not index returns. They include distributions, the expense ratio (0.03% for IVV to 0.75% for GSG) and the withholding tax the fund pays abroad.
• MSCI country indices are not local benchmarks. Read EWJ as “Japanese large and mid caps”, not “the Nikkei”.
• Fair value. For non-US funds, iShares adjusts stale foreign closes to the US close. Daily moves can therefore lead the local market by a day; weekly and longer figures are little affected.
• Freshness. US NAVs post in the New York evening. The 06:00 IST Data bank run usually carries the previous US session, so as_of is a day behind Indian data.
• GSG is a futures fund. Its return includes the futures roll and T-bill interest, and differs from spot commodity prices.

MSCI (internal only)

Code: pipeline/tipsheet/compute/world_msci.py (build_msci_internal, called at the end of build_world; a failure there never blocks the world bundles). Tests: the MSCI block in pipeline/tests/test_world.py, and test_msci_derived_bundles_are_refused in test_licence_gate.py. Nothing here reaches publish/world.py or the manifest.

Source. Data bank msci_index_levels (the owner’s scraper of MSCI’s public end-of-day index search): 313 codes, USD, daily, three variants (STRD price, NETR net total return, GRTR gross total return). STRD runs from 1997; NETR and GRTR start 2000-12-29 for most codes, so calendar-year net returns start in 2001, not 1998. 224 codes have a blank index_name; names come from the comments in the Data bank’s configs/update/msci_index_levels.yaml (Cou country, Reg region). MSCI market class (DM/EM/FM/standalone) is a hardcoded table in the module; re-check it after each June review.

Outputs (.cache/derived/, never published): msci_countries_grid.parquet (79 countries and 11 regions; NETR returns 1w to 10y annualised in USD and INR, drawdown, 1y volatility, 1y correlation and beta with MSCI India in USD and with Nifty 50 TRI in rupees, ranks by 1y and YTD rupee return among fresh countries), msci_calendar_years.parquet, msci_india_relative.parquet (India against EM, ACWI and ACWI ex India: rolling 1y and 3y return gaps in pp and % change of the ratio; there is no EM ex India code in the dataset), msci_correlation_weekly.parquet (India’s 52-week correlation with USA, China, Japan, Korea, Taiwan, Brazil, EM, World), and msci_internal_checks.json. Every table is checked for level-like columns before it is written.

Coverage (2026-10-01 build). 23 DM, 24 EM, 21 FM and 11 standalone country indices. NETR from 2000-12-29 for all DM and most EM; Gulf markets from 2005, Saudi Arabia from 2014, frontier markets 2002 to 2016. Ghana (last 2017-11) and Botswana (last 2023-02) are dead series and are flagged stale.

Rupee terms. Rupees per dollar from ECB reference rates (14:15 CET), ratio-linked before 2000-01-13 to FRED DEXINUS. MSCI converts at WM/Reuters 4 pm London rates on the same date, so the two are about two hours apart.

Staleness. Measured from every warehouse snapshot: MSCI data lands one day old (three days after a weekend), and the Data bank refreshes it about weekly, so it is 6 to 13 days old just before the next run, and once 22 and once 34 days when runs were missed (August and June-July 2026). Rule: stale after 5 days, and no 1-week return then. If MSCI were ever published, a light daily refresh of the ~40 codes on display would be needed; for internal research the weekly run is enough.

Validation (2026-10-01, monthly returns in USD).

Gross ≥ net ≥ price over 2000-2026 for India, USA and Japan (India: 1,050%, 990%, 458%).