Parsec #
parsec
is a Distributed Hash Table (DHT) and InterPlanetary Network Intexer (IPNI) lookup performance measurement tool. It specifically measures the PUT
and GET
performance of the IPFS public DHT but could also be configured to measure other libp2p-kad-dht networks. The setup is split into two components 1) a scheduler and 2) a server.
The server is just a normal libp2p peer that supports and participates in the public IPFS DHT and IPNI gossipsub topics, and exposes a lean HTTP API that allows the scheduler to issue publication and retrieval operations to either the DHT or an IPNI. It is important to note that we are not running Kubo. Currently, the scheduler goes around all our deployed server nodes (see below), instructs one to publish provider records for a random data blob and asks the other six to look it up. All servers take timing measurements about the publication and retrieval latencies and/or report back the results to the scheduler. The scheduler then tracks this information in a database for later analysis.
GitHubConcepts #
Next to the concept of servers and schedulers there’s the concept of a fleet
. A fleet is a set of server nodes that have a common configuration. For example, we are running four different fleets with seven nodes each (in different regions): 1) default
2) optprov
3) fullrt
4) cid.contact
. Each of these four fleets is configured differently. The default
fleet uses the default configuration in the go-libp2p-kad-dht
repository, the optprov
fleet uses the optimistic provide configuration to publish data into the DHT, the fullrt
fleet uses the accelerated DHT client, and the cid.contact
fleet publishes to and retrieves from the cid.contact network indexer.
Schedulers are then configured to interface with any combination of fleets. Right now, we have one scheduler for each fleet. As said above, it asks one node to publish content, then instructs the others to find the provider records, and then repeats the process with the next peer. However, it is possible to configure a scheduler that does the same thing but with nodes from multiple fleets e.g., default
+fullrt
to check if content that’s published with one implementation is reachable from another one.
Deployment #
As of 2023-05, we are running parsec
servers in the following AWS regions:
eu-central-1
us-east-2
us-west-1
ap-south-1
ap-southeast-2
af-south-1
sa-east-1
Our privileged region is us-east-1
. That’s where our main database resides and
the schedulers run. We currently have four fleets of seven nodes each:
default
- uses the stockgo-libp2p-kad-dht
configurationoptprov
- uses the Optimistic Provide approach to publish contentfullrt
- uses the accelerated DHT client (full routing table)cid.contact
- uses the InterPlanetary network indexer cid.contact
Each fleet has its own scheduler and they don’t interfere with each other.
Contributing #
Feel free to head over to the GitHub repository and dive in! Open an issue or submit PRs.
GitHub