(De)centralisation
One of the biggest decisions that you need to make regarding the indexer of choice is centralisation or decentralisation. Decentralisation is an essential aspect of any blockchain-based decentralised applications (dApp), and since indexers make up a part of that, you will also need to consider the decentralisation of this important component of your technical stack.
- Performance: The vast majority of the time between doing something in a dApp and getting a response is the latency, or the time it takes for the data to move between the indexer’s server and the application. Decentralised indexers are generally more performant since they are distributed globally, meaning that there is less distance between your user and their data, and that you can spread the load more evenly across different indexers
- Reliability: Everyone that has ever run a production server knows that servers go down and issues happen at some point. With a centralised indexer, if that single server goes down, it could mean that your dApp will cease to work for hours or even days. Decentralised indexers are available 24/7 because they are not reliant on a single server.
- Improved Security and data sovereignty: Decentralised dApp indexers are more secure because they are distributed across multiple nodes, this makes it more difficult for an attacker to compromise the service with a DDOS attack. Additionally, decentralised dApp indexers are not controlled by a single entity or group and you can verify the accuracy of the data in the indexer and ensure that there is no manipulation or bias.
In summary, just as decentralisation is critical for your dApp, it is also a critical aspect for your choice of indexer. A decentralised indexer can help to ensure that the data is accurate, unbiased, and accessible to everyone, and your dApp is performant and reliable.
SubQuery has always designed our multi-chain indexer with a strong belief in the principles of decentralisation. We are launching our decentralised network imminently, and it will bring all of these advantages to projects that currently rely on SubQuery.
Comparing Decentralised Indexers
For developers who require or choose a decentralised indexing offering, the market leaders are The Graph and SubQuery. The Graph was established in July 2017 and is one of the most recognised projects in the Ethereum ecosystem while SubQuery launched in January 2020 with its origins in Polkadot. While there are numerous similarities, it is important to articulate some of the key differences between the two.
Let’s consider each project under the following four key areas; openness, flexibility, speed, and universal support.
Openness
Both SubQuery and The Graph are open-source and welcome contributions from third party developers as well as their own community members. The documentation for both is understandable and easy to follow, and they both have democratic foundation with a shared goal of transitioning to a DAO.
Flexibility
SubQuery is just a tool in the hands of our community, endless opportunities exist limited only by the creativity of the people.
Some indexers, like Covalent and Unmarshal, are general purpose indexers that index standard datasets (e.g. transaction lists and blocks) and provide these indexed datasets to customers via some form of API. These services are great, they have a lot of data, and they are really quick to implement. The problem is that they have no flexibility, you can’t change the shape of your data, and you can’t add more data that you need to make your dApp better. When building a unique application that is faster, more feature rich, and more intuitive than your competition, you need custom data from a Flexible indexer.
Both SubQuery and The Graph were designed to be flexible in a way that they act as a scaffold or framework for each developer’s own custom built API — devs have the freedom to adapt and transform decentralised data to suit their needs. They can also adapt these services to fit into their existing backends, interface directly with other data services like Big Query and Athena, connect to dashboards like Grafana or Metabase, or even download data to analyse in Excel!
SubQuery takes this flexibility a whole lot further than the Graph however, by giving the developers the power to retrieve data from external API endpoints, make non-historical RPC calls, and even import your own external libraries into your projects. For example, you could import real world data from a service like Coingecko, or pull data from another chain — with SubQuery you’re not limited by the sandbox!
Speed
Speed is constantly an issue with indexing, and can be measured in three different ways.
Both SubQuery and The Graph also have automated historical state management, allowing you to roll back your data to a particular block and only perform a partial reindex — this is a huge performance advantage over centralised indexers who require you to reindex your entire dataset on each and every change — making your development iterations much slower.
The Graph and SubQuery on the other hand index directly from RPC providers, meaning one less step in the process. SubQuery takes this to the next level though, and for some networks, supports probabilistic indexing of unfinalised data. Most networks have a finalisation period that can range from 2 to 30 blocks. SubQuery skips this finalisation, and will take the most probabilistic data before it is finalised to provide to the app. In the unlikely event that the data isn’t finalised, SubQuery will automatically roll back and correct its mistakes quickly and efficiently — resulting in an insanely quick user experience for your customers.
Universal Support
It is clear that developers are starting to think more about how they can deploy their dApp or protocol across multiple layer-1 networks to onboard customers in each space, and help interconnect web3 to encourage adoption. When deciding on your indexing infrastructure provider, you should also consider what chains they will support you on.
SubQuery’s origins in the Polkadot ecosystem required us to factor multi-chain support from the outset. After delivering world-class infrastructure services to all of the leading parachain teams as they grew from small teams to unicorns, SubQuery was confident we could offer the same experience to developers in other ecosystems.
In only the past year, SubQuery has added support for all of the leading non-ETH ecosystems, including Cosmos, NEAR, and Algorand. This support has also extended to the leading EVM chains and scaling solutions, including Ethereum, Avalanche, and Flare. In a matter of months, SubQuery has gone from a single-point offering to one that can handle the demands of developers in a myriad of ecosystems. In order to be the true universal Web3 data index toolkit, we plan to have the widest chain support of any indexer.
SubQuery has leveraged this advantage to also provide multi-chain indexing. This gives you the option to index data from across different networks into the same dataset, which in turn provides a single endpoint where you can query all of your data across all chains. For example, you could capture XCM transaction data from all Polkadot parachains or monitor IBC messages across Cosmos Zones in a single project, with a single database, and a single query endpoint. It means that SubQuery is truly a write once, run anywhere multi-chain indexer.
About SubQuery