How Filecoin performs storage and retrieval transactions

This post explains how Filecoin transactions work from the perspective of storage providers (also known as miners) and clients who want to store data on the Filecoin network. The technical explanations in this article use the JSON-RPC API to interact with the Filecoin network when querying data, and this API is mainly used by applications. You can use the user-friendly lotus command-line tool, which uses the JSON-RPC API in the background.

The Filecoin network achieves economies of scale by allowing anyone to participate as a storage provider. Currently, the network consists of hundreds of storage providers across the world. Content addressing and cryptographic storage proofs verify that data is correctly and securely stored on miners' hardware over time, creating a robust and reliable service.

This post introduces the basic stages of two Filecoin transactions, storage transactions and retrieval transactions, and details their lifecycles. It also explains the cryptographic proofs used to verify that system participants have fulfilled their responsibilities as promised.

In order to store a file on Filecoin, the client must first import it into its local Filecoin node. This step generates data CID - content identifier, an ID that uniquely describes the content. After that, the data is transferred to the miners. Another way to store files on Filecoin is through offline transactions, which is not covered in this article.

Data can be imported locally into a Filecoin node using the lotus client import command. It is important to remember the resulting data CID (which can also be used later on the local node) because it must be used to retrieve the data from the miners later. After the data is imported into the local node. The user must initiate a transaction. This can be done with the lotus client dea: command. This command takes the data CID as input, generates a Filecoin Piece, and interactively walks the user through the storage transaction process detailed below.

Filecoin Piece is the main unit of negotiation for data stored by users on the Filecoin network. Filecoin Piece has no specific size, but is instead limited by an upper limit on the sector size, which is controlled by network parameters. If a Filecoin Piece is larger than the size of a sector supported by a miner, it must be split into more Pieces in order to fit into a sector.

A Filecoin Piece is a CAR file that contains an IPLD DAG and its own data/payload CID and piece CID.

CAR stands for Content Addressable Archive - a CAR file is a serialized representation of any IPLD DAG as a concatenation of its blocks, plus the graph (with the root CID) described in the file.

When a client wants to store a file in the Filecoin network, they first use UnixFS to generate an IPLD DAG of the file (this is what the lotus client import command does). The hash representing the root node of the DAG is an IPFS-style CID, called the data/payload CID.

UnixFS is a file description format based on protobuf. Directories and symbolic links in IPFS. In Filecoin, UnixFS is used as the file format quideline for files submitted to the Filecoin network. The generated CARfile is padded with extra zero bits to make the file generate a binary merkle tree.

UnixFS is a protobuf-based format for describing files, directories, and symbolic links in IPFS. Filecoin uses UnixFS as a file format guide for files submitted to the Filecoin network. The generated CAR file is padded with extra zero bits to make the file a binary merkle tree.

Users can store data in the Filecoin network and retrieve data from the Filecoin network through transactions. Participants in the network, miners (supply side) and clients (demand side) interact with each other through storage transactions and retrieval transactions.

The life cycle of a storage transaction is as follows:

1. Discovery

The client identifies the miner and determines their current asking price, which is the price per GiB in attoFIL (per block height (30 seconds)) that the miner wants to receive in order to accept the transaction (1 attoFIL equals 10^-18 * FIL). Currently, the minimum term for trading with Filecoin is 180 days.

All currently active miners can be listed using the Filecoin.StateListMiners method by querying the synced node's JSON RPC API (for testing purposes, the https://api.node.glif.io public endpoint is used):

Developers may need to decide based on the reputation or network capacity of a particular provider. The Filecoin protocol does not currently include reputation metrics for miners.

After selecting a specific miner, you need to obtain its PeerID (for example, using the Filecoin.StateMinerInfo method) to establish a secure connection with it via the libp2p protocol:

Then query the signed StorageAsk using the Filecoin.ClientQueryAsk method. This will establish a direct libp2p connection to the selected miner and ask for a storage quote:

2. Negotiation and data transfer

At this stage, both parties reach an agreement on the terms of the transaction, such as transaction cost, transaction duration, transaction start time, etc., and then the data is transmitted from the client to the miner.

3. Publishing

Deals are published on-chain via the PublishStorageDeals message, making storage providers publicly accountable for deals.

4. Handover

After the transaction is published on-chain, it is handed over to the Storage Mining subsystem, packaged into a sector, which is then sealed and subsequently continuously proven.

The Storage Mining subsystem ensures that miners can efficiently submit storage to the Filecoin network and is regulated in two ways:

1. Participate in the Filecoin storage market, obtain data on the client, and participate in storage transactions.

2. Participate in the Filecoin storage efficiency consensus, verify and generate blocks to grow the Filecoin blockchain, and earn block rewards and fees for this.

It oversees the following processes:

1. Submit new storage and register new department

In order to register a sector in Filecoin, miners must seal it. Sealing is a computationally intensive process that generates a unique representation of the data in the form of a proof, called a Proof of Replication, or PoRep. Once the proof is generated, the miner compresses it and submits the result to the blockchain. This is a proof that the miner has indeed made a copy of the data they agreed to store.

2. Continuously verify storage

Each storage miner must continuously submit on-chain proofs to prove that they continue to store their sector.

3. Declare and recover from storage failures

Failure to submit the above certification from the given department will result in an error and a fine will be imposed on the miner.

As mentioned above, storage deals are published on-chain before they become active and sealed. This is important because publishing a deal locks the client’s funds on-chain in custody. Therefore, miners can guarantee that if they do seal a sector of data, they will be paid. It helps to think of the transaction published on-chain as signing a contract, and the conclusion and activation of a contract as starting to do the work that the miner has committed to.

From the perspective of a client who wants to store data on Filecoin, a transaction roughly goes through the following stages:

1. Funding the transaction - the client locks funds in escrow

2. Propose an agreement to miners

3. Check if you are willing to accept the transaction

4. Transmitting the transaction's data to the miners - This is done via the GraphSync protocol. GraphSync is a protocol for synchronizing IPLD graphs between peers. It allows a host to make a single request to a remote peer to facilitate an IPLD selector on the remote peer's local IPLD graph. Lotus uses the ipfs/go-graphsync implementation of the GraphSync protocol.

5. Check for acceptance - Make sure the miner has accepted the transaction and published it on the chain.

6. Sealing - The transaction is in progress and miners are currently sealing the sector containing the transaction.

7. Valid - The transaction is sealed and valid. From now on, storage providers/miners should regularly prove that they continue to store the data. See the Proof of Spacetime section below for more details.

From the perspective of a miner who provides services to clients by storing their data, a transaction generally goes through the following stages:

1. Verify the transaction - receive a transaction proposal and check its parameters (size, price, etc.)

2. Check if there are locked funds - Make sure the customer has locked funds and can pay for the transaction.

3. Waiting for data - receiving transaction data from the customer.

4. Collateral for Chain Transactions

5. Publish transactions on the chain

6. Sealing department

7. Activation Transaction - From here on, storage providers/miners will periodically submit WindowPoSt proofs to prove that they are continuously storing data.

Unlike storage transactions, retrieval transactions are primarily conducted off-chain through payment channels. Data transfers are metered, and clients pay miners incrementally as data is transferred. Creating payment channels and redeeming vouchers are the only parts of the process that interact with the Filecoin blockchain.

The whole process is as follows:

1. Discovery: The client identifies the miners that own the required data and requests a retrieval quote from them - price per byte, unsealed price, payment interval.

2. Payment channel setup: The client establishes a payment channel between them and the miner (if one does not exist yet).

3. Data transfer via payment: Miners send data to clients until payment is required. When a certain threshold is reached, payment is requested and then data transfer continues. Depending on whether the miner has the data in the block stored or not, they may need to unseal it first - a non-trivial and non-instantaneous operation, which is the opposite of the sealing described in the section about storage transactions. The client has not yet successfully retrieved a full copy of the data.

The above section went over many of the details that make Filecoin unique and provide users with probabilistic guarantees about their data. This section covers the two proofs that Filecoin uses and explains how they fit into the protocol and the problems they solve.

Proof of Spacetime (PoSt) is a process by which storage miners can prove to the Filecoin network that they continue to store the only copy of certain data on behalf of the network. Proof of Spacetime manifests itself in two different types in Filecoin today: WindowPoSt and WinningPoSt.

1. WinningPoSt

WinningPoSt is a mechanism by which storage miners are rewarded for their contribution to the Filecoin network. At the beginning of each epoch, a small number of storage miners are elected to mine a new block for each. To do this , each miner is tasked with submitting a compressed storage proof for a designated sector . Each elected miner who successfully creates a block receives FIL (block reward) and has the opportunity to charge other Filecoin participants for including messages in the block. Storage miners that fail to do this within the necessary window will forfeit their chance to mine a block, but will not be penalized for doing so.

2. WindowPoSt

WindowPoSt is a mechanism by which storage miners' commitments are audited by the Filecoin blockchain. Each storage miner should maintain their committed sectors, which contain transactions with clients or empty sectors. The latter is called committed capacity, i.e., miners can make capacity commitments to fill a sector with arbitrary data instead of client data. Maintaining these sectors allows storage miners to prove that they have reserved space on behalf of the network.

Each day is divided into a series of windows, currently 48 windows, with a duration of 30 minutes (60 cycles, since 1 cycle equals 30 seconds). Each storage miner's set of guaranteed sectors is divided into subsets, one subset per window.

Within a given window (30 minutes), each storage miner must submit a proof of spacetime for each sector in their respective subset. This requires that each sector challenged is immediately accessible and will result in a zk-SNARK proof being published to the Filecoin blockchain as a message in a block. In this way, each sector that pledges storage is audited at least once every 24-hour period, and a permanent, verifiable public record is kept to prove the ongoing commitment of each storage miner.

In the image above, you can see that the example miner should submit WindowPoSt proofs within deadline 0 (>16TB), deadline 1 (<8TB), and deadline 2 (<8TB), and most of their sectors submitted within deadline 0. The deadlines for each miner are random, and for this particular miner, the deadlines start at block height 1635, 1695, and 1755. These deadlines and more details about the miners can be viewed on the SpaceGap tool.

The Filecoin network expects the continuous availability of stored data. Failure to submit WindowPoSt for a sector will result in a failure and will reduce the number of storage miners providing the sector, which incentivizes healthy behavior of storage miners.

Failure to include a proof in the Filecoin blockchain within the proof period results in a failure, whether due to loss of network connectivity, storage failure, or malicious behavior.

When a fault is registered for a sector, the Filecoin network will slash the storage miners that were supposed to store that sector. That is, it will assess penalties against miners (to be paid by the collateral the miners are facing) for not honoring their storage guarantees.

There are three types of departmental breakdown charges:

1. Sector Failure Fee: This fee is paid per sector every day while the sector is in a failed state. The size of the fee is slightly larger than the amount of block rewards that the sector can expect to receive each day. If a sector remains faulty for two consecutive weeks, the sector will pay a termination fee and be removed from the chain state.

2. Sector Fault Detection Fee: If a miner dishonestly reports a fault and instead the blockchain discovers an unreported fault, this is a one-time fee paid when a fault occurs. Given the probabilistic nature of PoSt checks, this will be set to the block reward that is expected to be obtained by a specific sector in a few days.

3. Sector Termination Fee: A sector can be terminated before its expiration date by automatic failure or by a miner’s decision. A termination fee is charged, which in principle is equal to the revenue a sector has earned so far, but is subject to a certain cap so as not to affect its longer useful life.

This post covers some of the concepts related to storing and retrieving data on Filecoin, the protocols that clients and miners participate in to achieve this, and the various proofs and guarantees involved in the process. It details the flow of storage and retrieval transactions from the perspective of clients and miners, as well as the penalties that the Filecoin protocol will enforce in the event that one of the parties misbehaves.

In summary, it outlines some of the foundations of how the Filecoin protocol governs the Filecoin network to form a reliable and trustless decentralized storage network.

——End——

Note: This article is translated and compiled by Interstellar Vision. Please do not reprint without permission.