Skip to main content
info

Please note that zkApp programmability is not yet available on Mina Mainnet, but zkApps can now be deployed to Berkeley Testnet.

note

This tutorial was last tested with SnarkyJS 0.8.0.

Tutorial 6: Off-Chain Storage

Overview​

caution

This tutorial provides a single-server solution to data storage. This is useful for prototyping zkApps and building zkApps where some trust guarantees are reasonable, but should not be used for zkApps that require trustlessness. It is intended as one of several options for data availability on Mina.

If you are interested in improving this library to make it more decentralized and useful in production, see the further development section below.

In Tutorial 5, Common Types and Functions, we ended by learning about merkle trees, and how we can use them to refer to large amounts of data stored off chain, by storing only the tree's root hash on chain

In this tutorial, we will share a library and pattern for working with storing merkle trees off-chain, with only the root of the merkle tree stored on chain.

We intend for this tutorial to be useful towards unlocking a larger set of applications that require off-chain storage. Other solutions are intended to follow that provide more decentralized options for zkApps that need more trustless solutions.

Feel free to skip ahead to the implementation section If you want to jump into how to use single server off-chain storage for prototyping applications. Otherwise, read on to understand more about the different options for off-chain storage, their tradeoffs, and how the single-server solution shared here fits in.

Why Off-Chain Storage?​

When building an application we'll just use locally and for testing, its fine to just build and store a merkle root locally.

However, when building a production-ready, distributed zkApp, we need more than this. All users interacting with our zkApp need to be able to retrieve the latest state, and modify it.

If this isn't the case, and someone were able to modify the tree without sharing their modification, the lack of data availability for the new version means that further writes, by other parties that don't have access to the new tree, become impossible. This requires that any data that modifies a zkApp, must be available somewhere for others users to access.

Off-Chain Storage and Decentralization​

Solutions to storage can be on a spectrum from cheap and more centralized, to more expensive and more decentralized. More decentralized solutions will likely be more expensive due to replicating and proving stored data.

Given that, we expect there to be different solutions on a continuum, offering multiple options for developers to choose depending on the zkApp they are building and the guarantees they want that zkApp to have.

Some likely things to expect on this spectrum and are under exploration:

  1. A single-server storage solution (this is what is presented here).
  2. A multi-server storage solution that can be run by multiple parties for stronger trust guarantees.
  3. A solution leveraging storage on modular blockchains.
  4. A future hard fork to add purchasable on-chain data storage to Mina.
  5. A future hard fork to add data-storage committees to Mina for horizontally scalable storage.

Single-Server Off-Chain Storage​

This tutorial shares and explains an implementation of the single-server off-chain storage solution mentioned as the first option above. The library provides a REST server that anyone can run to store data for one or multiple zkApps, as well as a zkApp library to check on-chain if changes have been backed by the server.

This implementation requires a trust assumption for zkApps using it, that both developers and users can trust whoever is running the server.

This makes it useful for both prototyping applications that need off-chain storage, and putting applications into production where these trust assumptions are reasonable, but not for zkApps where a trustless solution is needed.

See the library here if you'd like to learn more about this implementation and see how it works.

Further Development of Single-Server Off-Chain Storage​

We're looking for a developer to run with the library presented here, improve it, make it more decentralized, and run instances for the community. There is a grant opportunity for taking this on. If interested, please reach out here.

Some suggested improvements:

  • Eliminate DDOS vulnerability by adding a token that limits storage requests.
  • Do not store trees for a smart contract if that contract is misconfigured in a way to prevent cleaning up old data.
  • Add support to the client library for connecting to multiple storage servers, enabling correctness under a majority-honest assumption.
  • Switch the project to a more scalable database implementation (e.g. Redis).
  • Write an implementation for an automatically scalable service (e.g. Cloudflare).

Implementing a Project Using Off-Chain Storage​

As usual, there is sample code for this project, which you can find here.

We will be writing and discussing the src/main.ts and src/NumberTreeContract.ts files in this project.

This project implements a tree, where each leaf is either empty or stores a number, which will be our data. Updates to the tree can update a leaf, if the new number in the leaf is greater than the old number. The root of the tree is stored on chain, while the tree itself is stored on an off-chain storage server.

Project Setup​

To start, create a new project with:

$ zk project 06-off-chain-storage

Now let enter the project, delete the existing files, and create a new smart contract and a main file:

$ cd 06-off-chain-storage
$ rm src/Add.ts
$ rm src/Add.test.ts
$ rm src/interact.ts
$ zk file src/NumberTreeContract
$ touch src/main.ts

Edit index.ts to import and export our new smart contract:

import { NumberTreeContract } from './NumberTreeContract.js';

export { NumberTreeContract };

Now, add the library for the off-chain storage server we'll be using:

$ npm install experimental-zkapp-offchain-storage --save

Also install xmlhttprequest-ts, which we will use to make network requests when running from nodejs, where the browser's XMLHttpRequest is not available by default:

$ npm install --save xmlhttprequest-ts

With that, setup is complete!

Running our project​

As in previous tutorials, you can run main.ts with

$ npm run build && node build/src/main.js

This will fail when first run, but once we start adding code to main.ts and NumberTreeContract.ts, it should run successfully.

Also for this project, we will need to run our storage server. To do this, in a new terminal window, change into the root of your project and run:

$ node node_modules/experimental-zkapp-offchain-storage/build/src/storageServer.js

This will start a storage server and create a database.json file in the current directory to store our data for this tutorial.

Implementing the Smart Contract​

To start, open NumberTreeContract.ts in your editor. You can find a full copy of this file here for reference.

Start by adding our imports:

  1 import {
  2   SmartContract,
  3   Field,
  4   MerkleTree,
  5   state,
  6   State,
  7   method,
  8   DeployArgs,
  9   Signature,
 10   PublicKey,
 11   Permissions,
 12   Bool,
 13 } from 'snarkyjs';
 14
 15 import {
 16   OffChainStorage,
 17   Update,
 18   MerkleWitness8,
 19 } from 'experimental-zkapp-offchain-storage';
...

Notice we import some items from zkapp-offchain-storage:

  • OffChainStorage is object containing functions for interacting with off-chain storage. These are:
    • getPublicKey: A function to get the storage server's public key. This is also stored in smart contracts to identify what storage server is storing the smart contract's off-chain data.
    • get: A function for fetching the data for a merkle tree from the storage server, given the root of the tree.
    • requestStore: A function to request storing a tree on the storage server. Returns a proof that the storage server has stored this tree.
    • assertRootUpdateValid: A function used in smart contracts, to prove updates to the smart contract's currently stored tree root result in a tree root that is being stored by the storage server.
    • mapToTree: A storage function to convert maps to trees. Internally the storage server is using maps from tree indices to leafs.
  • Update: A type for updates to merkle trees. We'll be using this below in the smart contract.
  • MerkleWitness8: The type of our merkle tree witness. Others are available for input, such as MerkleWitness32 and MerkleWitness256. This is necessary for SnarkyJS to use the same instances of the witness cross-library.

Continuing, let's setup our smart contract:

...
 21 export class NumberTreeContract extends SmartContract {
 22   @state(PublicKey) storageServerPublicKey = State<PublicKey>();
 23   @state(Field) storageNumber = State<Field>();
 24   @state(Field) storageTreeRoot = State<Field>();
 25
 26
 27
 28   deploy(args: DeployArgs) {
 29     super.deploy(args);
 30     this.account.permissions.set({
 31       ...Permissions.default(),
 32       editState: Permissions.proofOrSignature(),
 33     });
 34   }
 35
 36   @method initState(storageServerPublicKey: PublicKey) {
 37     this.storageServerPublicKey.set(storageServerPublicKey);
 38     this.storageNumber.set(Field(0));
 39
 40     const emptyTreeRoot = new MerkleTree(8).getRoot();
 41     this.storageTreeRoot.set(emptyTreeRoot);
 42   }
...

Here we add 3 pieces of state to our contract: the public key of the storage server, the "storageNumber"—which is used to ensure the storage server is actively storing states, and the root of the merkle tree.

We also initialize our zkApp's initialize for these three values, by setting them to the public key of our storage server, setting storage number to 0, and storing the root of an empty tree.

Continuing, here is our update function:

...
 42   @method update(
 43     leafIsEmpty: Bool,
 44     oldNum: Field,
 45     num: Field,
 46     path: MerkleWitness8,
 47     storedNewRootNumber: Field,
 48     storedNewRootSignature: Signature
 49   ) {
 50     const storedRoot = this.storageTreeRoot.get();
 51     this.storageTreeRoot.assertEquals(storedRoot);
 52
 53     let storedNumber = this.storageNumber.get();
 54     this.storageNumber.assertEquals(storedNumber);
 55
 56     let storageServerPublicKey = this.storageServerPublicKey.get();
 57     this.storageServerPublicKey.assertEquals(storageServerPublicKey);
 58
 59     let leaf = [oldNum];
 60     let newLeaf = [num];
 61
 62     // newLeaf can be a function of the existing leaf
 63     newLeaf[0].assertGt(leaf[0]);
 64
 65     const updates = [
 66       {
 67         leaf,
 68         leafIsEmpty,
 69         newLeaf,
 70         newLeafIsEmpty: Bool(false),
 71         leafWitness: path,
 72       },
 73     ];
 74
 75     const storedNewRoot = OffChainStorage.assertRootUpdateValid(
 76       storageServerPublicKey,
 77       storedNumber,
 78       storedRoot,
 79       updates,
 80       storedNewRootNumber,
 81       storedNewRootSignature
 82     );
 83
 84     this.storageTreeRoot.set(storedNewRoot);
 85     this.storageNumber.set(storedNewRootNumber);
 86   }
 87 }

We get and assert the current state of the contract - and then we perform the update.

First, we check that the new leaf is greater than the old leaf.

Then, we check the update itself. In this example, we perform a single update to the tree, however with multiple witnesses, you can chain updates together to change the tree more than once in a single call to the storage server.

To assert the update is valid, we use a assertRootUpdateValid call from the OffChainStorage library. This checks that when the update is applied to the tree represented by the existing on-chain tree root, the data for the new tree is being stored by the storage server.

That completes the smart contract!

Implementing main.ts​

You can find a full copy of this file here for reference.

We will not be implementing this full file, instead just discussing a few parts of it, since much is repeated from earlier tutorials. So download it from the above link, place it in your src folder, and then open it in your editor.

Note that it contains logic for both running the contract locally and for deploying and interacting with it on Berkeley. This is a useful pattern when developing a new contract. If you would like to try it on Berkeley, deploy our contract as usual with zk deploy, set useLocal to false in main.ts, and add the name of your config to the end of your call to node main.js.

To start, we will connect to the off-chain storage server:

...
 73   const storageServerAddress = 'http://localhost:3001';
 74   const serverPublicKey = await OffChainStorage.getPublicKey(
 75     storageServerAddress,
 76     NodeXMLHttpRequest
 77   );
...

Here, we connect to the storage server, and get its public key. Running the storage server as described above starts it by default on port 3001, and so this code is configured to connect to it there. In a real application, you would run the storage server on an externally exposed machine, and change this address from localhost to match.

Next we will describe the updateTree function, starting on line 120.

Our goal in this function is to:

  1. Get the currently stored tree from the storage server
  2. Select a random leaf
  3. Change the value at that leaf to a bigger nubmer
  4. Create a transaction that performs this update.

To start, let's get the existing tree:

...
109   async function updateTree() {
110     const index = BigInt(Math.floor(Math.random() * 4));
111
112     // get the existing tree
113     const treeRoot = await zkapp.storageTreeRoot.get();
114     const idx2fields = await OffChainStorage.get(
115       storageServerAddress,
116       zkappPublicKey,
117       treeHeight,
118       treeRoot,
119       NodeXMLHttpRequest
120     );
121
122     const tree = OffChainStorage.mapToTree(treeHeight, idx2fields);
123     const leafWitness = new MerkleWitness8(tree.getWitness(index));
...

Here, we get the current root stored in the contract - and request the data for that root from the storage server.

Then, we convert that data from a map to a MerkleTree - and we get a witness for a random index of the merkle tree.

Continuing:

...
125     // get the prior leaf
126     const priorLeafIsEmpty = !idx2fields.has(index);
127     let priorLeafNumber: Field;
128     let newLeafNumber: Field;
129     if (!priorLeafIsEmpty) {
130       priorLeafNumber = idx2fields.get(index)![0];
131       newLeafNumber = priorLeafNumber.add(3);
132     } else {
133       priorLeafNumber = Field(0);
134       newLeafNumber = Field(1);
135     }
...

Here, we check if the leaf is empty, and shape our update accordingly. If the leaf was empty, we set it to one. Otherwise, we set the leaf to whatever used to be there, plus 3.

...
140     const [storedNewStorageNumber, storedNewStorageSignature] =
141       await OffChainStorage.requestStore(
142         storageServerAddress,
143         zkappPublicKey,
144         treeHeight,
145         idx2fields,
146         NodeXMLHttpRequest
147       );
...

And lastly, we request that the storage server stores our data. If successful, we are returned a new storage number and a signature, which we will use for updating the smart contract.

We call our smart contract as follows:

...
160     const doUpdate = () => {
161       zkapp.update(
162         Bool(priorLeafIsEmpty),
163         priorLeafNumber,
164         newLeafNumber,
165         leafWitness,
166         storedNewStorageNumber,
167         storedNewStorageSignature
168       );
169     }
170
171     if (useLocal) {
172       const updateTransaction = await Mina.transaction(
173         { sender: feePayerKey.toPublicKey(), fee: transactionFee },
174         () => {
175           doUpdate();
176         }
177       );
178
179       updateTransaction.sign([zkappPrivateKey, feePayerKey])
180       await updateTransaction.prove();
181       await updateTransaction.send();
...

That completes our review of the code to interact with the off-chain storage server! As mentioned previously, you can find the full copy of this file here for review.

Conclusion​

Congrats! We have finished building a smart contract that leverages off-chain storage

Checkout Tutorial 7 to learn how to use Oracles, to pull in data from the outside world into your zkApp.