History Management for Midnight

MIP	MIP-0026
MPS	0032
Status	Proposed
Category	Core
Authors	Dominik Zajkowski @dzajkowski
Created	14-MAY-2026

Abstract

Midnight’s transaction throughput generates a continuous stream of proof data that accumulates over the chain’s lifetime. At target throughput levels, the storage requirements for maintaining full history grow to a point where long-term node operation becomes increasingly expensive, raising barriers to participation and making reliable light clients infeasible. Without a strategy for managing history size, the network’s operational viability and decentralization degrade as the chain ages. Moving data sharing and availability concerns to the mempool stage of operation does not change this discussion — the data still accumulates and must be managed regardless of when or how it is disseminated.

Vision

Midnight nodes operate with bounded storage requirements regardless of chain age. Pruned nodes fully participate in consensus while retaining only a recent window of history. Archival storage is a voluntary choice, not a protocol requirement. New participants bootstrap into current consensus state quickly, without syncing full chain history. Historical data can be verified through efficient retrospective proofs without requiring the full dataset.

Problem

Every transaction carries proof data that cannot be re-derived from public state — the proofs are the only evidence that transactions were valid. Discarding them without a trusted compression procedure means nodes must take each other’s word that history was verified correctly. This makes history management fundamentally harder than in transparent chains, where any node can re-execute transactions to verify state independently.

At the throughput levels targeted in MPS-xxxx (throughput), proof data accumulates fast enough that history management is a continuous operational concern. Without it, growing storage costs shrink the pool of viable node operators and committee candidates, undermining decentralization.

The problem has four dimensions:

Proof compression. Reducing history requires a verifiable procedure that compresses or replaces proof data while preserving the guarantee that all prior transactions were valid. Without such a procedure, pruning breaks the trust model.

Nullifier and commitment pool integrity. These structures grow monotonically and are actively queried by the protocol — nullifier sets to prevent double-spends, commitment pools for transaction construction and verification. They cannot be replaced by a summary proof; they are live state. It is not obvious how to compact or partition them without compromising the guarantees they provide.

User-facing data availability. History management affects not only consensus but also the data available to users constructing and verifying transactions, and to external services querying historical state. Reducing what nodes retain limits what they can serve — a pruned node cannot answer queries outside its retention window, and the protocol must define the boundary between consensus-required data and query-serving data so that indexers and other consumers can identify appropriate data sources.

Privacy under transformation. Any procedure that compresses, reorganizes, or discards historical data must not leak information about the transactions involved.

Use Cases

UC1: Pruned Node Operator

Scenario: A node operator running with bounded history, continuously shedding old data as new blocks arrive to maintain a stable storage footprint.
Limitations: The current architecture requires full history for consensus participation and block validation. There is no mechanism for a running node to discard historical data while maintaining its ability to participate in consensus, serve peers, and preserve nullifier and commitment pool integrity.
Desired Outcome: Nodes can continuously prune history during normal operation without interruption to consensus participation or degradation of protocol guarantees.

UC2: Archival Node

Scenario: A node operator voluntarily storing complete historical data for network redundancy and analysis.
Limitations: The current architecture forces all nodes into this role by default.
Desired Outcome: Archival is a voluntary choice, not a protocol requirement.

UC3: Light Client Bootstrap

Scenario: A new validator or light client joining the network and bootstrapping into current consensus state. “Light client” encompasses a range of device profiles — from laptops to low-end mobile devices — with significantly different memory, CPU, bandwidth, and trust constraints. MIPs addressing this use case must define concrete resource profiles for each client class they target.
Limitations: Joining requires syncing full chain history.
Desired Outcome: New participants can verify current state legitimacy and join consensus without syncing full chain history.

UC4: Historical Data Verification

Scenario: Proving that a specific historical data point existed on the live network.
Limitations: Verification requires access to the full historical dataset.
Desired Outcome: Efficient retrospective proofs and audits without storing full history.

UC5: Indexer and Arbitrary History Queries

Scenario: An indexer or analytics service ingesting blockchain data to support queries against arbitrary historical state — for example, past balances, historical transaction patterns, or contract state at a given block height.
Limitations: The protocol currently does not distinguish between data needed for consensus and data needed for historical queries. A pruned node retains only a recent window of history and cannot serve as a data source for queries outside that window — an indexer requiring arbitrary historical state must ingest from an archival node. However, whether the retention window is protocol-defined or operator-configured is unspecified, and nodes do not advertise their retention capabilities, so an indexer has no protocol-level way to discover which nodes can serve which historical ranges.
Desired Outcome: The protocol defines a clear boundary between consensus-required data and historical query data. Nodes advertise their retention profile (pruned with window size, or archival), enabling indexers to discover appropriate data sources. A pruned node can serve as a data source for indexers querying within its retention window. Archival nodes or dedicated history-serving infrastructure support arbitrary historical lookups. An indexer that tracks the chain tip continuously can operate against pruned nodes alone, ingesting blocks as they arrive; only indexers that need to backfill historical gaps or query arbitrary past state require access to archival nodes.

UC6: Regulatory Audit Under Privacy Preservation

Scenario: A regulator or third-party auditor with a legal mandate to verify a specific compliance property of historical transactions, such as confirming that funds did not pass through sanctioned addresses or that an asset trail matches a declared origin. Limitations: The current architecture leaves the auditor with two extremes: full archival access that breaks the privacy of all users, or no access at all. There is no protocol-level mechanism for selective disclosure of historical state under a defined legal basis. Desired Outcome: The protocol supports selective disclosure of historical data to authorized parties, allowing a regulator to verify compliance properties on specific transactions without compromising the privacy of unrelated users.

Goals

Primary Goal

Enable bounded history operation: The protocol must allow nodes to participate in consensus, verify current state, and maintain security and privacy guarantees without storing or accessing full chain history.

Secondary Goals

Support multiple node profiles: The architecture must accommodate pruned nodes, archival nodes, and light clients as first-class participants with clearly defined capabilities.
Preserve privacy under reduced history: History management must not introduce new privacy leakage vectors or weaken existing guarantees.
Maintain nullifier and commitment pool integrity: History management must not compromise the structures essential for transaction validity.

Expected Outcomes

Nodes with bounded history can fully participate in consensus and block validation.
Running nodes can manage storage growth over time without loss of protocol functionality.
New nodes can bootstrap into current consensus state and verify its legitimacy without syncing full chain history.
Historical data points can be verified without requiring the full dataset.
Nullifier and commitment pool integrity is maintained under history management.
The network maintains BFT safety and liveness guarantees when nodes hold incomplete history.
History management does not conflict with goals defined in companion MPSs on throughput, censorship resistance, and decentralization.

Open Questions

What influence does the desired finality time have on history management? The throughput MPS targets hard finality under 5 seconds. Are there facets of history management that will be affected by such a finality window?
Should the protocol operate as a standalone layer one, or anchor finality to another blockchain, effectively operating as a sidechain? Anchoring provides external finality guarantees affecting synchronization and history management story.
Should the protocol support general purpose zero knowledge execution, or optimize specifically for layer two rollup verification on top of a post-and-verify layer one mechanism?
How does the protocol prove or verify data availability when history is reduced, and what minimal guarantees from that proof suffice to maintain consensus integrity and safety?
What invariants must hold during live history pruning? When a running node sheds historical data, what constraints govern the pruning process to ensure uninterrupted consensus participation, peer synchronization, and integrity of nullifier sets and commitment pools?
What data availability guarantees should pruned nodes provide to indexers? Can an indexer operate against a pruned node for recent data, or must it always ingest from an archival node? What protocol-level distinction, if any, should exist between data required for consensus and data required for historical queries?
What incentivizes archival node operation? If archival storage is voluntary, what economic model sustains it as storage costs grow over the chain’s lifetime?

Recommended MIPs

To be determined as the problem space is explored further.

Acknowledgements

Brian Bush (@bwbush)

Copyright

This MPS is licensed under CC-BY-4.0.