Edit

IPFS W3Auth Pinning Service

IPFS remote pinning services allow users pin their IPFS files to remote stable IPFS nodes to ensure file's reliability and accesibility.

Furthermore, remote pinning service might be useful if:

  • Your local node isn't always online, but you need items to be consistently available
  • You'd like to keep a persistent backup of your local node's files somewhere else
  • You don't have all the disk space you need on your local node
  • You run more than one IPFS node, and would like to use one of them as a "personal pinning service" as your preferred location for permanent storage

Background

Once these remote pinning services adapt to pinning serice API, they can be used by users with standard IPFS pinning APIs.

pin-lifecycle

Currently, remote pinning service mostly storing user's IPFS file in the centralized IPFS nodes(called centralized IPFS pinning service), like Pinata, Infura and Cloudflare.

Generally, the centralized remote pinning service's lifecycle are designed as below:

pin-1

This centralized pinning service divided into 3 parts:

  • Pinner logic: handling the centralized authentication(with email/phone), upserting database, handling some commercial logic(payment/billing) and communicating with centralized IPFS nodes
  • State database: handling IPFS pulling/pinning status and managing user's information
  • IPFS node: managing IPFS nodes, maybe IPFS clusters

However, for the Web3 users, they are more innclined to use web3 identity and pay with blockchain tokens/smart contracts. In this article, we'll introduce a lightweight Web3-Auth pinning service based on Crust Network

Solution

The whole design shows below:

pin-2

The main differences between centralized and decentralized pinning are:

  • Pinner logic: W3Auth pinner authenticating with Web3 identities(Ethereum/Substrate-based chains/solana/polygon/near/...), and only recording on-chain status(CID's replica info)
  • State databse: W3Auth pinner only recording and managing on-chain status
  • IPFS nodes: W3Auth pinner DO NOT needs local IPFS, it is based on Crust Network, which is a totally decentralized IPFS Network guaranteed your file by blockchain protocol, you can check here to learn more detail storage information about Crust Network.

Usage

The IPFS W3Auth Pinning Service(aka. W3Auth PS) is compatible with standard IPFS remote pinning service. You can refer IPFS Docs to learn how to use remote pin.

As for the Access Token of W3Auth PS, you can easily get it with several web3 ways.

Authorization: Bearer <base64(ChainType-PubKey:SignedMsg)>

Let's take ipfs cli as an example

ipfs pin remote service add crustpinner http://localhost:3000/psa base64(ChainType-PubKey:SignedMsg)

Get ChainType

ChainType is:

  1. sub(or substrate)
  2. eth(or ethereum)
  3. sol(or solana)
  4. pol (or polygon)
  5. nea (or near)
  6. ava(or avalanche)
  7. apt(or aptos)

Get PubKey and SignedMsg

1. With Substrate

Get PubKey

PubKey is just the substrate address, like 5Chu5r5GA41xFgMXLQd6CDjz1ABGEGVGS276xjv93ApY6vD7

All substrate-based chains are adapted:

Get SignedMsg

Just sign the PubKey with your private key to get the SignedMsg

2. With Ethereum

Get PubKey

PubKey is just the ethereum address(42-characters) start with 0x

Get SignedMsg

Just sign the PubKey with your eth private key to get the SignedMsg

3. With Moonriver

Moonriver is fully compatiable with the Ethereum, you can just follow the same steps with the Ethereum.

Get PubKey

PubKey is just the moonriver(ethereum) address(42-characters) start with 0x

Get SignedMsg

Just sign the PubKey with your moonriver private key to get the SignedMsg

4. With Solana

Get PubKey

PubKey is just the solana address

Get SignedMsg

You can sign the PubKey with your solana private key to get the SignedMsg

5. With Polygon

Get PubKey

PubKey is just the polygon address(42-characters) start with 0x. It's compatiable with the ethereum.

Get SignedMsg

Just sign the PubKey with your polygon private key to get the SignedMsg

6. With Near

You can sign the PubKey with one of your near private key associated with your account to get the SignedMsg

7. With Avalanche

You can sign the Address without chainID prefix, such as avax1se4e9lvhlfwhcqnzjr0vpswqcnhsy5atn5r0l3, with your X- or P- chain private key associated with your account to get the SignedMsg.

8. With Apots

Get PubKey

PubKey is the aptos's account pubkey, such as 0xaa79510150c3a6753f224ef47a315ea6ae9acd23f4506a866feb25f8995c60c. Please pay attention that it's not the same as the address.

Get SignedMsg

You can sign the PubKey with your aptos private key to get the SignedMsg

Deploy

1. Start MySQL

W3Auth PS uses MySQL as its state database, you can just use docker to start and config the db service.

2. Init State DB

Please execute the sql script under ./sql folder to create database and state tables.

3. Config

3.1 Config pinning service

Please create an .env file baseon .env-example, each config item means:

NODE_ENV // `production` or `dev`
MYSQL_HOST // optional, default is `localhost`
MYSQL_PORT // optional, default is `3306`
MYSQL_DB // optional, default is `pinning_service` created by ./sql/V1_CREATE_DATABASE.sql
MYSQL_USER // optional, default is `root`
MYSQL_PASSWORD // optional, default is `root`
MYSQL_POOL_MAX // optional, default is `10`
MYSQL_POOL_MIN // optional, default is `0`
MYSQL_POOL_IDLE // optional, default is `30,000`
MYSQL_POOL_ACQUIRE // optional, default is `30,000`
CRUST_SEED // required, the pinning pool private seeds
WS_ENDPOINT // optional, crust chain websocket address, default is `wss://rpc-crust-mainnet.decoo.io`
DEFAULT_FILE_SIZE // optional, ordering file size, default is 2GB
CRUST_TIPS // optional, ordering tips, default is 0.00005 CRUs
VALID_FILE_REPLICAS // optional, the successfully pinning replica count, default is 3

3.2 Config supported chains

W3Auth PS allows nodes config supported chains with sql script, default supported chain is substrate, you can config different chain_name with corresponding chain_type(auth way):

chain_type

  • 0: Support all substrate-based chains authentication
  • 1: Support all eth-compatible chains authentication, like ethereum/polygon/bsc/...
  • 2: Support solana authentication

And you can run the following sql to add/delete supported chains:

  • Add
INSERT INTO `pinning_service`.`chain` (`chain_name`, `chain_type`) VALUES ('eth', 1);
  • Delete
DELETE FROM `pinning_service`.`chain` WHERE `chain_name`='eth';

4. Start pinning service

  • Run with docker

    Just run with the compose file in this repo

    docker-compose up -d order-service

  • Run with node native

    # 1. Clone repo
git clone https://github.com/crustio/ipfs-w3auth-pinning-service.git
# 2. Install and build
yarn && yarn build
# 3. Run
yarn start

Then, you can just config the standard IPFS remote pinning service with http://localhost:3000/psa!

Resources

← IPFS W3Auth GatewayOverview →