README
¶
{Un}Pack - Solidity (Ethereum) Smart Contract Analysis Toolkit
!!!DOCUMENTATION IS UNDER DEVELOPMENT!!!
In the meantime, explore our OpenAPI V3 documentation to understand the capabilities of our system. Full godoc documentation and repository wiki will be available soon.
Overview
{Un}Pack is a robust modular toolkit designed for Solidity smart contract analysis across all EVM-compatible networks.
{Un}Pack toolkit, via JSON-RPC and REST (OpenAPI v3.1.0) is enabling users to decode transactions, logs, decompile opcodes, discover used standards and many more features listed here. Additionally, it offers the ability to search for specific functions and events by their signatures, providing a comprehensive toolkit for in-depth contract analysis and auditing. This enhanced search capability makes it easier for developers to quickly locate and analyze specific smart contracts components.
To manage the storage of large datasets without using Git LFS, we have opted to utilize Cloudflare R2. This object storage solution offers zero egress fees and is a cost-effective alternative to Amazon S3. As a result, the entire dataset is seamlessly integrated with this repository, eliminating the need for you to recreate the datasets independently.
Long story short, clone repo, configure only necessary settings, migrate database, download, decompress and import datasets, start docker containers and enjoy playing with JSON-RPC services!
Features
- Modular design: Easily extend {Un}pack with additional services, configuration or states. See examples module.
- Easy dataset management: Import or export whole dataset with ease. More information can be found here.
- Cloudflare R2 datasets: 7-zip datasets will be hosted on Cloudflare as it's much cheaper than S3 and there are no egress traffic charges!
- Efficient Contract Downloading: Streamlined process for downloading Ethereum smart contracts.
- Download Resumption: Capability to pause and resume downloads, ensuring progress isn't lost.
- Local Storage Management: Stores contracts in Hydra (Postgres) and disk for quick access and efficient retrieval.
- Source Code Access: Provides easy access to the source code of verified contracts on Etherscan, Bscscan, Sourcify.dev and internal Solgo.
- JSON-RPC, REST and GraphQL Support: For easier access to the data, including documentation, postman and OpenAPI v3.1.0 schemas!
- Advanced Decoding Tools: Utilize JSON-RPC and OpenAPI endpoints to decode transactions, logs, decompile opcodes, and attempt ABI recreation of any deployed Ethereum smart contract.
- Signature Search: Enables the searching of functions and events based on their signatures, enhancing the ability to analyze specific contract aspects efficiently.
This project is inspired from Smart Contract Sanctuary. For now, it will be focused only on mainnet contracts. Ethereum is the first one, Arbitrum, Binance Smart Chain and Polygon will be added later including others.
IMPORTANT
- GraphQL endpoint is not yet fully finished.
- Otterscan JSON-RPC api is recommended. We use Erigon and QuickNode.
- For dataset compression we use 7-Zip and XZ compressions.
- If you do not have an Erigon node, you will need to have Etherscan, Bscscan, Arbiscan and/or BitQuery account to get contract creation information.
- Due to licensing issues, we are not going to provide information that strictly breaks licenses. Instead, we are going to provide you the tools to extract information yourself if you have access to the 3rd party sources, utilising their respective API keys.
Datasets
We offer access to a collection of complete datasets, invaluable for researchers, developers, and analysts who require extensive real-world data for their projects.
These datasets are stored on R2, ensuring reliable and speedy access via direct download links.
To avoid spams and malicious use, you would need to reach us over Discord to gain access to the database datasets. In the future, datasets will be available on Cloudflare R2. We are still deciding onto what can be published due to licensing terms.
Read more about how to export, import, compress, decompress, upload and download datasets here.
Blockchain datasets
| Dataset | Compressed Size | Decompressed Size |
|---|---|---|
| Ethereum Contracts | 526.99 MB | 17 GB |
| ----- | ---- | ---- |
| Total | 526.99 MB | 17 GB |
Last Revision Date: 2024-04-23 09:45 CEST
Database datasets
Bellow is a short recap on database snapshot.
| Dataset | Compressed Size | Decompressed Size |
|---|---|---|
| Accounts | 2.64 GB | 8.39 GB |
| Contracts | 1.03 GB | 25.63 GB |
| Metadata | 100 MB | 340 MB |
| Tokens | 20 MB | 60 MB |
| AST | 20 MB | 70 MB |
| CFG | 30 MB | 340 MB |
| Constructors | 230 MB | 6.44 GB |
| Standards | 50 MB | 7.67 GB |
| Variables | 42 MB | 7.33 GB |
| Functions | 13.39 GB | 240.84 GB |
| Events | 80.57 MB | 61.08 GB |
| ----- | ----- | ---- |
| Total | 17.64 GB | 358.23 GB |
Last Revision Date: 2024-05-19 09:45 CEST
Documentation (WIP)
Documentation is currently being developed.
Begin your journey at the Welcome. For installation instructions, please consult the Installation guide.
Please check out Examples directory for more information about how to use toolkit.
Visit OpenAPI v3.1.0 documentation for more guides on how to use our public apis! Be aware that public API is limited to 10 requests per minute.
Notes
- For the best performance, it is recommended to run this project on NVMe storage. This is because the database is very large and requires fast read and write to the storage to perform well.
This project is not a new blockchain, due to it, key-value storage is not needed. Even more, it's a problem.
Database Statistics
Bellow is a database overview table, explaining the size of Postgres database as of last database push. This data is only informational and is here to give you a big picture, not the most accurate database size (as it changes from second to second).
Dataset Statistics
| Table | Records |
|---|---|
| Accounts Count | 32,180,294 |
| Contracts Count | 1,213,007 |
| Metadata Count | 1,163,230 |
| AST Count | 441,754 |
| CFG Count | 441,097 |
| Standards Count | 1,408,759 |
| Tokens Count | 337,123 |
| Constructors Count | 378,055 |
| Variables Count | 6,410,657 |
| Functions Count | 30,703,202 |
| Events Count | 13,516,522 |
Extending Inspector (Modules)
Please navigate to the Module to see how you can extend inspector with custom options, commands and services!
LICENSE
I am offering this code, not related to contracts at no cost, under the MIT license. For more details about this license, please refer to the LICENSE file included in this repository.
Please note: The contracts themselves are subject to their respective licenses. These licenses can be found within the source code of each individual contract. It is imperative that you review and adhere to these licenses when using the contracts.
Message to Etherscan
I extend my sincere gratitude to the Etherscan team for your invaluable contributions. After reviewing your licensing terms, I believe that my use of your services aligns with these terms. However, should there be any concerns or issues regarding my usage, I welcome your feedback and guidance. Please feel free to contact me at info(at)unpack.dev.
Documentation
¶
There is no documentation for this package.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cmd provides command-line interface commands for the application.
|
Package cmd provides command-line interface commands for the application. |
|
examples
|
|
|
pkg
|
|
|
accounts
Package accounts provides functionality to interact with blockchain accounts and manage their associated states and transactions using Ethereum and Unpack libraries.
|
Package accounts provides functionality to interact with blockchain accounts and manage their associated states and transactions using Ethereum and Unpack libraries. |
|
app
Package app provides structures and functions to create and manage CLI applications.
|
Package app provides structures and functions to create and manage CLI applications. |
|
datasets
Package datasets provides functionality for managing datasets.
|
Package datasets provides functionality for managing datasets. |
|
db
Package db provides utilities for interacting with BadgerDB v4.
|
Package db provides utilities for interacting with BadgerDB v4. |
|
logger
Package logger provides a simplified interface for creating and configuring loggers using the Uber's zap logging library.
|
Package logger provides a simplified interface for creating and configuring loggers using the Uber's zap logging library. |
|
machine
Package machine provides a flexible and robust state machine implementation for managing states and transitions in applications, particularly useful in scenarios involving Ethereum smart contract interactions.
|
Package machine provides a flexible and robust state machine implementation for managing states and transitions in applications, particularly useful in scenarios involving Ethereum smart contract interactions. |
|
options
Package options provides functionality for managing application configuration options.
|
Package options provides functionality for managing application configuration options. |
|
pprof
Package pprof defines a utility for integrating pprof with custom configurations.
|
Package pprof defines a utility for integrating pprof with custom configurations. |
|
rest
Package rest provides the implementation of a REST API server using the Huma framework.
|
Package rest provides the implementation of a REST API server using the Huma framework. |
|
state
Package state is designed to handle current synchronization state.
|
Package state is designed to handle current synchronization state. |
|
subscribers
Package subscribers provides a comprehensive solution for managing subscriptions to Ethereum blockchain events.
|
Package subscribers provides a comprehensive solution for managing subscriptions to Ethereum blockchain events. |
|
unpacker
Package unpacker provides functionalities to handle and process contracts, including discovery, extraction, validation, and storage of contract data.
|
Package unpacker provides functionalities to handle and process contracts, including discovery, extraction, validation, and storage of contract data. |
|
Package schema contains all of the Schema Definition Files (abbreviated SDL) which define (Un)pack Downloader API.
|
Package schema contains all of the Schema Definition Files (abbreviated SDL) which define (Un)pack Downloader API. |
|
services
|
|
|
acl
Package acl provides functionality for managing API keys with access control lists (ACL).
|
Package acl provides functionality for managing API keys with access control lists (ACL). |
|
indexer
package indexer provides an indexing service that integrates various types of interceptors to handle data synchronization and state management across blockchain networks.
|
package indexer provides an indexing service that integrates various types of interceptors to handle data synchronization and state management across blockchain networks. |
|
status
Package status provides functionality for ...
|
Package status provides functionality for ... |