Machine-Readable by Design: How Automated Data Collection Is Reshaping Prediction Markets

Public order books, open APIs and on-chain settlement have made Kalshi and Polymarket the most machine-readable betting venues ever built. The effect on price formation is only now becoming measurable. 

A prediction market is, at bottom, a probability printed in public. Every contract on Kalshi or Polymarket resolves to one dollar or nothing, so the quoted price is a number anyone can read, store and compare. That transparency is what makes the forecast credible, and it is also what makes the venue easy to observe at scale. Both properties follow from the same design decision. 

Over the past two years an infrastructure layer has grown up around both exchanges. Collectors poll order books, mirror trade tapes, index wallet histories and normalise the output into a common schema. Some of this supports academic research and market monitoring; some of it feeds trading systems that consistently take money from slower participants. The difference lies less in the tooling than in what is done with the result. 

Why these venues are unusually machine-readable 

Most betting operators guard their odds. Prediction exchanges do close to the opposite, because open pricing is what makes the probability signal credible in the first place. On Kalshi, market data is public and only account actions — orders, positions, balances, fills — require a signed request. Polymarket settles on a public blockchain, so trade history is retrievable without asking the platform for anything at all. 

That architecture produces three distinct collection surfaces. There are the documented REST and WebSocket endpoints, the undocumented internal endpoints that serve each site’s own front end, and, in Polymarket’s case, the chain itself. A collector blocked at one layer can often fall back to another, which is why access controls alone rarely stop a determined operator. 

A commercial layer has formed around this. General-purpose scraping marketplaces list off-the-shelf collectors for both venues, covering order-book depth, candlestick histories, leaderboard rankings and wallet-level profit and loss. Assembling a cross-venue price feed is now closer to a procurement decision than an engineering project. 

Price inconsistency is measured, not assumed 

The most direct form of automated extraction is arbitrage. Prices that must logically sum to one dollar sometimes do not — within a single market, across linked markets, or between two venues whose order books and participant bases barely overlap. Capturing that gap is a matter of watching continuously rather than forecasting well. 

The scale is now documented rather than anecdotal. Researchers at IMDEA Networks and the Oxford Internet Institute analysed on-chain order-book data covering roughly 86 million Polymarket bets placed between April 2024 and April 2025, and estimated about $40 million in realised arbitrage profit over that year — see Unravelling the Probabilistic Forest and the accompanying coverage in DL News. The paper separates market rebalancing arbitrage, which occurs within a single market or condition, from combinatorial arbitrage spanning several linked markets. 

Two caveats matter for interpretation. The figure covers Polymarket alone and does not measure cross-venue activity, for which no comparable public dataset yet exists. And none of that profit came from forecasting skill: it came from noticing, faster than anyone else, that a set of prices was internally inconsistent. 

Latency, rate limits and the price of speed 

Speed turns a data problem into a plumbing problem. Polling a REST endpoint every few seconds is adequate for a dashboard and useless for trading, so serious operators move market data onto streaming connections and reserve request budget for order placement. Kalshi’s WebSocket feed pushes incremental order-book deltas but is read-only, so orders and cancellations still travel over REST — which is precisely where the limits bind. 

Kalshi’s published rate limit documentation describes a token-bucket system with separate read and write budgets, tiered by account level. Requests carry a token cost, and exceeding the available budget returns a 429. Higher tiers are earned automatically through trading volume or assigned by the exchange, so throughput scales with standing on the venue rather than with client-side cleverness. 

The practical consequence is a two-speed market. Participants who have accumulated volume sit on higher request ceilings and can therefore react to their own data faster, which in turn generates more volume. A Finance Magnates review of Polymarket’s public leaderboard found that 14 of the 20 most profitable wallets appeared to be automated. 

Comparisons with foreign exchange should be handled carefully. That same report cites BIS-derived estimates putting algorithmic execution at roughly 70–80 per cent of spot FX activity, a figure widely repeated in trade coverage. The BIS’s own analysis of the 2025 Triennial Survey reports electronic execution at 59 per cent of FX turnover, which measures something narrower. The direction of travel is clear; the destination is not. 

Public wallets and counterparty tracking 

Polymarket’s on-chain settlement creates a form of exposure with no equivalent in conventional betting. Every position, entry price and exit is permanently attributable to a wallet address, and public leaderboards make the best-performing addresses easy to identify. Third-party services index those wallets, reconstruct their historical returns and monitor them for new activity. 

The strategies built on this vary widely in character. At one end, a trader treats a proven wallet as one input among many. At the other, a system detects a large order from a tracked address and trades ahead of the price impact elsewhere. Polymarket’s market integrity policy prohibits front-running, information misuse and disruptive practices, while stating that data a participant collects and analyses independently is their own work product. 

That is where the genuine ambiguity now sits. Observing a public blockchain is not misuse of confidential information on any ordinary reading of the term. Whether systematically shadowing identified counterparties amounts to a disruptive practice is a question the rulebooks have not yet answered in detail. 

Resolution rules: the least visible source of risk 

The least discussed source of automated edge involves no price data at all. Every contract carries a resolution rule: the source that decides the outcome, the timestamp that counts, the wording that governs an edge case. Two venues listing what looks like the same event frequently resolve it against different sources or different cut-off times. 

Archived resolution text can be compared automatically across thousands of contracts to flag pairs that only appear equivalent. A cross-venue position built on mismatched rules is not risk-free at all; it is an unhedged directional bet presented as a locked one. This is the one area where better data collection reduces risk rather than transferring it from one participant to another. 

Where collection becomes evasion 

Volume is what turns data collection into an adversarial exercise. Sub-second coverage of thousands of contracts exceeds any documented allowance, and the standard workaround is to spread requests across many identities and network addresses. That shifts the activity from reading public data to circumventing a control, which is a different act. 

Bot-management systems resist this in layers: address reputation, protocol-level handshake fingerprinting, browser fingerprinting and behavioural scoring across a session. Each layer is individually beatable and collectively expensive, which is the actual defensive mechanism — not prevention, but cost. 

The compliance position is less settled than either side tends to claim. Reading public market data is generally permitted; deliberately defeating a rate limit or a detection control is a different matter, and platform terms typically treat it as such regardless of whether the underlying content was public. Attribution is difficult, and that difficulty appears to be what many operators are relying on. 

Does automation harm the market? 

The honest answer is that it cuts both ways. Arbitrage systems enforce internal consistency, tighten spreads and pull mispriced contracts back toward coherence — precisely the function that makes prediction market prices worth citing as forecasts. A venue with no arbitrageurs would produce noisier and less reliable probabilities.

The cost lands elsewhere. The $40 million identified in the IMDEA study did not appear from nowhere; it came from counterparties who posted stale quotes or crossed a spread a fraction of a second too late. When those participants conclude that the game is structurally unwinnable, they leave, and the liquidity that made the venue informative leaves with them. 

There is also a reflexivity concern worth naming. Prediction market prices are increasingly quoted by newsrooms and traders as real-time probability estimates. A price set largely by automated systems reacting to other automated systems is a less independent forecast than the same number would have been two years ago. 

What to watch next 

Three developments will determine how this settles. The first is tiering: as exchanges tie request budgets more tightly to volume and identity, the informal collection layer gets pushed toward either paid access or outright evasion. The second is disclosure — whether venues begin labelling automated accounts, as equity and crypto markets have debated. 

The third is regulatory, and it has moved quickly. Kalshi operates as a CFTC-designated contract market, and Polymarket has run a CFTC-regulated US exchange since late 2025, so both now sit inside a supervisory perimeter with well-developed language for front-running and disruptive trading. In April 2026 the CFTC brought its first insider-trading enforcement action involving event contracts, and in June it issued a new proposed rule governing them. 

Applying that vocabulary to automated observation is the harder question. A system that reads a public blockchain very quickly is not obviously doing anything the rulebooks prohibit, and that interpretive gap is the one most likely to shape the next two years. 

For participants, the practical lesson is narrower. On venues this transparent, the edge has largely migrated from knowing the answer to observing the market faster and more completely than the person on the other side. That is a data engineering contest, and it is not currently a close one. 

References 

Saguillo, Ghafouri, Kiffer and Suarez-Tangil, “Unravelling the Probabilistic Forest: Arbitrage in Prediction Markets” (IMDEA Networks / Oxford Internet Institute, arXiv, 2025) — https://arxiv.org/abs/2508.03474 

DL News, coverage of the IMDEA arbitrage findings — https://www.dlnews.com/articles/markets/polymarket-users-lost-millions-of-dollars-to-bot-like-bettors-over-the-past-year/ 

Kalshi, Rate Limits and Tiers (API documentation) — https://docs.kalshi.com/getting_started/rate_limits 

Finance Magnates, “Prediction Markets Are Turning Into a Bot Playground” — https://www.financemagnates.com/trending/prediction-markets-are-turning-into-a-bot-playground/ 

Bank for International Settlements, “The FX trade execution landscape through the prism of the 2025 BIS Triennial Survey”, BIS Quarterly Review, December 2025 — https://www.bis.org/publ/qtrpdf/r_qt2512v.htm 

Polymarket, Market Integrity Policy — https://integrity.polymarket.com/ 

Polymarket US, Market Integrity — https://www.polymarketexchange.com/market-integrity.html 

Lowenstein Sandler, “CFTC and Kalshi Announce Enforcement Actions Targeting Prediction Markets”, April 2026 — https://www.lowenstein.com/news-insights/publications/client-alerts/cftc-and-kalshi-announce-enforcement-actions-targeting-prediction-markets-fctm 

Congressional Research Service, “CFTC Issues Proposed Rule Regarding Prediction Markets”, June 2026 — https://www.congress.gov/crs-product/LSB11441