README
¶
API Design
We are using a documentation driven process.
The API is defined using OpenAPI v2 in indexer.oas2.yml.
Updating REST API
The Makefile will install our fork of oapi-codegen, use make oapi-codegen
to install it directly.
- Document your changes by editing indexer.oas2.yml
- Regenerate the endpoints by running make generate from the
api
directory. The sources at generated/ will be updated. - Update the implementation in handlers.go. It is sometimes useful to consult generated/routes.go to make sure the handler properly implements ServerInterface.
What codegen tool is used?
We found that oapi-codegen produced the cleanest code, and had an easy to work with codebase. There is an algorand fork of this project which contains a couple modifications that were needed to properly support our needs.
Specifically, uint64
types aren't strictly supported by OpenAPI. So we added a type-mapping feature to oapi-codegen.
Why do we have indexer.oas2.yml and indexer.oas3.yml?
We chose to maintain V2 and V3 versions of the spec because OpenAPI v3 doesn't seem to be widely supported. Some tools worked better with V3 and others with V2, so having both available has been useful. To reduce developer burdon, the v2 specfile is automatically converted v3 using converter.swagger.io.
Fixtures Test
What is a Fixtures Test?
Currently (September 2022) fixtures_test.go is a library that allows testing Indexer's router to verify that endpoints accept parameters and respond as expected, and guard against future regressions. app_boxes_fixtures_test.go is an example fixtures test and is the creator of the fixture boxes.json.
A fixtures test
- is defined by a go-slice called a Seed Fixture e.g. var boxSeedFixture which contains request information for making HTTP requests against an Indexer server
- iterates through the slice, making each of the defined requests and generating a Live Fixture
- reads a Saved Fixture from a json file e.g. boxes.json
- persists the Live Fixture to a json file not in source control
- asserts that the Saved Fixture is equal to the Live Fixture
In reality, because we always want to save the Live Fixture before making assertions that could fail the test and pre-empt saving, steps (3) and (4) happen in the opposite order.
What's the purpose of a Fixtures Test?
A fixtures test should allow one to quickly stand up an end-to-end test to validate that Indexer endpoints are working as expected. After Indexer's state is programmatically set up, it's easy to add new requests and verify that the responses look exactly as expected. Once you're satisfied that the responses are correct, it's easy to freeze the test and guard against future regressions.
What does a Fixtures Test Function Look Like?
func TestBoxes shows the basic structure of a fixtures test.
-
setupIdbAndReturnShutdownFunc()
is called to set up the Indexer database- this isn't expected to require modification
-
setupLiveBoxes()
is used to prepare the local ledger and process blocks in order to bring Indexer into a particular state- this will always depend on what the test is trying to achieve
- in this case, an app was used to create and modify a set of boxes which are then queried against
- it is conceivable that instead of bringing Indexer into a particular state, the responses from the DB or even the handler may be mocked, so we could have had
setupLiveBoxesMocker()
instead ofsetupLiveBoxes()
-
setupLiveServerAndReturnShutdownFunc()
is used to bring up an instance of a real Indexer.- this shouldn't need to be modified; however, if running in parallel and making assertions that conflict with other tests,
you may need to localize the variable
fixtestListenAddr
and run on a separate port - if running a mock server instead, a different setup function would be needed
- this shouldn't need to be modified; however, if running in parallel and making assertions that conflict with other tests,
you may need to localize the variable
-
validateLiveVsSaved()
runs steps (1) through (5) defined in the previous section- this is designed to be generic and ought not require much modification going forward
Which Endpoints are Currently Testable in a Fixtures Test?
Endpoints defined in proverRoutes are testable.
Currently (September 2022) these are:
/v2/accounts
/v2/applications
/v2/applications/:application-id
/v2/applications/:application-id/box
/v2/applications/:application-id/boxes
How to Introduce a New Fixtures Test for an Already Testable Endpoint?
To set up a new test for endpoints defined above one needs to:
1. Define a new Seed Fixture
For example, consider
var boxSeedFixture = fixture{
File: "boxes.json",
Owner: "TestBoxes",
Frozen: true,
Cases: []testCase{
// /v2/accounts - 1 case
{
Name: "What are all the accounts?",
Request: requestInfo{
Path: "/v2/accounts",
Params: []param{},
},
},
...
A seed fixture is a struct
with fields
-
File
(required) - the name in test_resources where the fixture is read from (and written to with an_
prefix) -
Owner
(recommended) - a name to define which test "owns" the seed -
Frozen
(required) - set true when you need to run assertions of the Live Fixture vs. the Saved Fixture. For tests to pass, it needs to be set true. -
Cases
- the slice oftestCase
s. Each of these has the fields:Name
(required) - an identifier for the test caseRequest
(required) - arequestInfo
struct specifying:Path
(required) - the path to be queriedParams
(required but may be empty) - the slice of parameters (stringsname
andvalue
) to be appended to the path
2. Define a new Indexer State Setup Function
There are many examples of setting up state that can be emulated. For example:
- setupLiveBoxes() for application boxes
- TestApplicationHandlers() for applications
- TestBlockWithTransactions() setup state consisting of a set of basic transactions
How to Make a New Endpoint Testable by Fixtures Tests?
There are 2 steps:
- Implement a new function witness generator aka prover function of
type
func(responseInfo) (interface{}, *string)
as examplified in this section. Such a function is supposed to parse an Indexer response's body into a generated model. Currently, all provers are boilerplate, and with generics, it's expected that this step will no longer be necessary (this POC shows how it would be done with generics). - Define a new route in the proverRoutes struct.
This is a tree structure which is traversed by splitting a path using
/
and eventually reaching a leaf which consists of aprover
as defined in #1.
For example, to enable the endpoint GET /v2/applications/{application-id}/logs
for fixtures test, one need only define a logsProof
witness generator and have it mapped in proverRoutes
under:
proverRoutes.parts["v2"].parts["applications"].parts[":application-id"].parts["logs"] = logsProof
How to Fix a Fixtures Test?
Supposing that there was a breaking change upstream, and a fixtures test is now failing. The following approach should work most of the time:
- Run the broken test generating a temporary fixture file in the
fixturesDirectory
(currently test_resources) with a name the same as the json fixture except begining with_
(e.g._boxes.json
vs.boxes.json
). - Observe the diff between the temporary fixture and the saved fixture. If the diff is acceptable, then simply copy the temporary fixture over the saved fixture.
- If the diff isn't acceptable, then make any necessary changes to the setup and seed and repeat steps 1 and 2.
Documentation
¶
Index ¶
- Constants
- Variables
- func Serve(ctx context.Context, serveAddr string, db idb.IndexerDb, ...)
- func Verify(dm *DisabledMap, nameOfHandlerFunc string, ctx echo.Context, ...) error
- type DisabledMap
- type DisabledMapConfig
- type DisabledParameterErrorReporter
- type DisplayDisabledMap
- type EndpointConfig
- type ErrDisabledMapConfig
- type ErrVerifyFailedParameter
- type ExtraOptions
- type ServerImplementation
- func (si *ServerImplementation) LookupAccountAppLocalStates(ctx echo.Context, accountID string, ...) error
- func (si *ServerImplementation) LookupAccountAssets(ctx echo.Context, accountID string, params generated.LookupAccountAssetsParams) error
- func (si *ServerImplementation) LookupAccountByID(ctx echo.Context, accountID string, params generated.LookupAccountByIDParams) error
- func (si *ServerImplementation) LookupAccountCreatedApplications(ctx echo.Context, accountID string, ...) error
- func (si *ServerImplementation) LookupAccountCreatedAssets(ctx echo.Context, accountID string, ...) error
- func (si *ServerImplementation) LookupAccountTransactions(ctx echo.Context, accountID string, ...) error
- func (si *ServerImplementation) LookupApplicationBoxByIDAndName(ctx echo.Context, applicationID uint64, ...) error
- func (si *ServerImplementation) LookupApplicationByID(ctx echo.Context, applicationID uint64, ...) error
- func (si *ServerImplementation) LookupApplicationLogsByID(ctx echo.Context, applicationID uint64, ...) error
- func (si *ServerImplementation) LookupAssetBalances(ctx echo.Context, assetID uint64, params generated.LookupAssetBalancesParams) error
- func (si *ServerImplementation) LookupAssetByID(ctx echo.Context, assetID uint64, params generated.LookupAssetByIDParams) error
- func (si *ServerImplementation) LookupAssetTransactions(ctx echo.Context, assetID uint64, ...) error
- func (si *ServerImplementation) LookupBlock(ctx echo.Context, roundNumber uint64, params generated.LookupBlockParams) error
- func (si *ServerImplementation) LookupTransaction(ctx echo.Context, txid string) error
- func (si *ServerImplementation) MakeHealthCheck(ctx echo.Context) error
- func (si *ServerImplementation) SearchForAccounts(ctx echo.Context, params generated.SearchForAccountsParams) error
- func (si *ServerImplementation) SearchForApplicationBoxes(ctx echo.Context, applicationID uint64, ...) error
- func (si *ServerImplementation) SearchForApplications(ctx echo.Context, params generated.SearchForApplicationsParams) error
- func (si *ServerImplementation) SearchForAssets(ctx echo.Context, params generated.SearchForAssetsParams) error
- func (si *ServerImplementation) SearchForBlockHeaders(ctx echo.Context, params generated.SearchForBlockHeadersParams) error
- func (si *ServerImplementation) SearchForTransactions(ctx echo.Context, params generated.SearchForTransactionsParams) error
Constants ¶
const ( ErrNoBoxesFound = "no application boxes found" ErrWrongAppidFound = "the wrong application-id was found, please contact us, this shouldn't happen" ErrWrongBoxFound = "a box with an unexpected name was found, please contact us, this shouldn't happen" ErrNoAccountsFound = "no accounts found for address" ErrMultipleBoxes = "multiple application boxes found for this app id and box name, please contact us, this shouldn't happen" ErrFailedLookingUpBoxes = "failed while looking up application boxes" ErrResultLimitReached = "Result limit exceeded" ErrBoxNotFound = "box not found" )
constant error messages.
Variables ¶
var ErrVerifyFailedEndpoint error = fmt.Errorf("endpoint is disabled")
ErrVerifyFailedEndpoint an error that signifies that the entire endpoint is disabled
Functions ¶
func Serve ¶
func Serve(ctx context.Context, serveAddr string, db idb.IndexerDb, dataError func() error, log *log.Logger, options ExtraOptions)
Serve starts an http server for the indexer API. This call blocks.
func Verify ¶
func Verify(dm *DisabledMap, nameOfHandlerFunc string, ctx echo.Context, log DisabledParameterErrorReporter) error
Verify returns nil if the function can continue (i.e. the parameters are valid and disabled parameters are not supplied), otherwise VerifyFailedEndpoint if the endpoint failed and VerifyFailedParameter if a disabled parameter was provided.
Types ¶
type DisabledMap ¶
type DisabledMap struct { // Key -> Function Name/Operation ID Data map[string]*EndpointConfig }
DisabledMap a type that holds a map of disabled types The key for a disabled map is the handler function name
func MakeDisabledMap ¶
func MakeDisabledMap() *DisabledMap
MakeDisabledMap creates a new empty disabled map
func MakeDisabledMapFromOA3 ¶
func MakeDisabledMapFromOA3(swag *openapi3.T, config *DisabledMapConfig) (*DisabledMap, error)
MakeDisabledMapFromOA3 Creates a new disabled map from an openapi3 definition
type DisabledMapConfig ¶
type DisabledMapConfig struct { // Key -> Path of REST endpoint (i.e. /v2/accounts/{account-id}/transactions) // Value -> Operation "get, post, etc" -> Sub-value: List of parameters disabled for that endpoint Data map[string]map[string][]string }
DisabledMapConfig is a type that holds the configuration for setting up a DisabledMap
func GetDefaultDisabledMapConfigForPostgres ¶
func GetDefaultDisabledMapConfigForPostgres() *DisabledMapConfig
GetDefaultDisabledMapConfigForPostgres will generate a configuration that will block certain parameters. Should be used only for the postgres implementation
func MakeDisabledMapConfig ¶
func MakeDisabledMapConfig() *DisabledMapConfig
MakeDisabledMapConfig creates a new disabled map configuration with everything enabled
func MakeDisabledMapConfigFromFile ¶
func MakeDisabledMapConfigFromFile(swag *openapi3.T, filePath string) (*DisabledMapConfig, error)
MakeDisabledMapConfigFromFile loads a file containing a disabled map configuration.
type DisabledParameterErrorReporter ¶
type DisabledParameterErrorReporter interface {
Errorf(format string, args ...interface{})
}
DisabledParameterErrorReporter defines an error reporting interface for the Verify functions
type DisplayDisabledMap ¶
type DisplayDisabledMap struct { // A complicated map but necessary to output the correct YAML. // This is supposed to represent a data structure with a similar YAML // representation: // /v2/accounts/{account-id}: // required: // - account-id : enabled // /v2/accounts: // optional: // - auth-addr : enabled // - next: disabled Data map[string]map[string][]map[string]string }
DisplayDisabledMap is a struct that contains the necessary information to output the current config to the screen
func MakeDisplayDisabledMapFromConfig ¶
func MakeDisplayDisabledMapFromConfig(swag *openapi3.T, mapConfig *DisabledMapConfig, limited bool) *DisplayDisabledMap
MakeDisplayDisabledMapFromConfig will make a DisplayDisabledMap that takes into account the DisabledMapConfig. If limited is set to true, then only disabled parameters will be added to the DisplayDisabledMap
func (*DisplayDisabledMap) String ¶
func (ddm *DisplayDisabledMap) String() (string, error)
type EndpointConfig ¶
EndpointConfig is a data structure that contains whether the endpoint is disabled (with a boolean) as well as a set that contains disabled optional parameters. The disabled optional parameter set is keyed by the name of the variable
type ErrDisabledMapConfig ¶
type ErrDisabledMapConfig struct { // Key -> REST Path that was mis-spelled // Value -> Operation "get, post, etc" -> Sub-value: Any parameters that were found to be mis-spelled in a valid REST PATH BadEntries map[string]map[string][]string }
ErrDisabledMapConfig contains any mis-spellings that could be present in a configuration
func (*ErrDisabledMapConfig) Error ¶
func (edmc *ErrDisabledMapConfig) Error() string
type ErrVerifyFailedParameter ¶
type ErrVerifyFailedParameter struct {
ParameterName string
}
ErrVerifyFailedParameter an error that signifies that a parameter was provided when it was disabled
func (ErrVerifyFailedParameter) Error ¶
func (evfp ErrVerifyFailedParameter) Error() string
type ExtraOptions ¶
type ExtraOptions struct { // Tokens are the access tokens which can access the API. Tokens []string // DeveloperMode turns on features like AddressSearchRoundRewind DeveloperMode bool // Respond to Private Network Access preflight requests sent to the indexer. EnablePrivateNetworkAccessHeader bool // MetricsEndpoint turns on the /metrics endpoint for prometheus metrics. MetricsEndpoint bool // MetricsEndpointVerbose generates separate histograms based on query parameters on the /metrics endpoint. MetricsEndpointVerbose bool // Maximum amount of time to wait before timing out writes to a response. Note that handler timeout is computed //off of this. WriteTimeout time.Duration // ReadTimeout is the maximum duration for reading the entire request, including the body. ReadTimeout time.Duration // DisabledMapConfig is the disabled map configuration that is being used by the server DisabledMapConfig *DisabledMapConfig // MaxAPIResourcesPerAccount is the maximum number of combined AppParams, AppLocalState, AssetParams, // and AssetHolding resources per address that can be returned by the /v2/accounts endpoints. // If an address exceeds this number, a 400 error is returned. Zero means unlimited. MaxAPIResourcesPerAccount uint64 // MaxAccountListSize is the maximum number of items that can be passed in query parameter account lists. // (e.g.: GET /v2/block-headers?proposer=A,B,C) // // Zero means unlimited. MaxAccountListSize uint64 // Blocks MaxBlocksLimit uint64 DefaultBlocksLimit uint64 // Transactions MaxTransactionsLimit uint64 DefaultTransactionsLimit uint64 // Accounts MaxAccountsLimit uint64 DefaultAccountsLimit uint64 // Assets MaxAssetsLimit uint64 DefaultAssetsLimit uint64 // Asset Balances MaxBalancesLimit uint64 DefaultBalancesLimit uint64 // Applications MaxApplicationsLimit uint64 DefaultApplicationsLimit uint64 // Boxes MaxBoxesLimit uint64 DefaultBoxesLimit uint64 }
ExtraOptions are options which change the behavior of the HTTP server.
type ServerImplementation ¶
type ServerImplementation struct { // EnableAddressSearchRoundRewind is allows configuring whether or not the // 'accounts' endpoint allows specifying a round number. This is done for // performance reasons, because requesting many accounts at a particular // round could put a lot of strain on the system (especially if the round // is from long ago). EnableAddressSearchRoundRewind bool // contains filtered or unexported fields }
ServerImplementation implements the handler interface used by the generated route definitions.
func (*ServerImplementation) LookupAccountAppLocalStates ¶
func (si *ServerImplementation) LookupAccountAppLocalStates(ctx echo.Context, accountID string, params generated.LookupAccountAppLocalStatesParams) error
LookupAccountAppLocalStates queries indexer for AppLocalState for a given account, and optionally a given app ID. (GET /v2/accounts/{account-id}/apps-local-state)
func (*ServerImplementation) LookupAccountAssets ¶
func (si *ServerImplementation) LookupAccountAssets(ctx echo.Context, accountID string, params generated.LookupAccountAssetsParams) error
LookupAccountAssets queries indexer for AssetHolding for a given account, and optionally a given asset ID. (GET /v2/accounts/{account-id}/assets)
func (*ServerImplementation) LookupAccountByID ¶
func (si *ServerImplementation) LookupAccountByID(ctx echo.Context, accountID string, params generated.LookupAccountByIDParams) error
LookupAccountByID queries indexer for a given account. (GET /v2/accounts/{account-id})
func (*ServerImplementation) LookupAccountCreatedApplications ¶
func (si *ServerImplementation) LookupAccountCreatedApplications(ctx echo.Context, accountID string, params generated.LookupAccountCreatedApplicationsParams) error
LookupAccountCreatedApplications queries indexer for AppParams for a given account, and optionally a given app ID. (GET /v2/accounts/{account-id}/created-applications)
func (*ServerImplementation) LookupAccountCreatedAssets ¶
func (si *ServerImplementation) LookupAccountCreatedAssets(ctx echo.Context, accountID string, params generated.LookupAccountCreatedAssetsParams) error
LookupAccountCreatedAssets queries indexer for AssetParams for a given account, and optionally a given asset ID. (GET /v2/accounts/{account-id}/created-assets)
func (*ServerImplementation) LookupAccountTransactions ¶
func (si *ServerImplementation) LookupAccountTransactions(ctx echo.Context, accountID string, params generated.LookupAccountTransactionsParams) error
LookupAccountTransactions looks up transactions associated with a particular account. (GET /v2/accounts/{account-id}/transactions)
func (*ServerImplementation) LookupApplicationBoxByIDAndName ¶
func (si *ServerImplementation) LookupApplicationBoxByIDAndName(ctx echo.Context, applicationID uint64, params generated.LookupApplicationBoxByIDAndNameParams) error
LookupApplicationBoxByIDAndName returns the value of an application's box (GET /v2/applications/{application-id}/box)
func (*ServerImplementation) LookupApplicationByID ¶
func (si *ServerImplementation) LookupApplicationByID(ctx echo.Context, applicationID uint64, params generated.LookupApplicationByIDParams) error
LookupApplicationByID returns one application for the requested ID. (GET /v2/applications/{application-id})
func (*ServerImplementation) LookupApplicationLogsByID ¶
func (si *ServerImplementation) LookupApplicationLogsByID(ctx echo.Context, applicationID uint64, params generated.LookupApplicationLogsByIDParams) error
LookupApplicationLogsByID returns one application logs (GET /v2/applications/{application-id}/logs)
func (*ServerImplementation) LookupAssetBalances ¶
func (si *ServerImplementation) LookupAssetBalances(ctx echo.Context, assetID uint64, params generated.LookupAssetBalancesParams) error
LookupAssetBalances looks up balances for a particular asset (GET /v2/assets/{asset-id}/balances)
func (*ServerImplementation) LookupAssetByID ¶
func (si *ServerImplementation) LookupAssetByID(ctx echo.Context, assetID uint64, params generated.LookupAssetByIDParams) error
LookupAssetByID looks up a particular asset (GET /v2/assets/{asset-id})
func (*ServerImplementation) LookupAssetTransactions ¶
func (si *ServerImplementation) LookupAssetTransactions(ctx echo.Context, assetID uint64, params generated.LookupAssetTransactionsParams) error
LookupAssetTransactions looks up transactions associated with a particular asset (GET /v2/assets/{asset-id}/transactions)
func (*ServerImplementation) LookupBlock ¶
func (si *ServerImplementation) LookupBlock(ctx echo.Context, roundNumber uint64, params generated.LookupBlockParams) error
LookupBlock returns the block for a given round number (GET /v2/blocks/{round-number})
func (*ServerImplementation) LookupTransaction ¶
func (si *ServerImplementation) LookupTransaction(ctx echo.Context, txid string) error
LookupTransaction searches for the requested transaction ID.
func (*ServerImplementation) MakeHealthCheck ¶
func (si *ServerImplementation) MakeHealthCheck(ctx echo.Context) error
MakeHealthCheck returns health check information about indexer and the IndexerDb being used. Returns 200 if healthy. (GET /health)
func (*ServerImplementation) SearchForAccounts ¶
func (si *ServerImplementation) SearchForAccounts(ctx echo.Context, params generated.SearchForAccountsParams) error
SearchForAccounts returns accounts matching the provided parameters (GET /v2/accounts)
func (*ServerImplementation) SearchForApplicationBoxes ¶
func (si *ServerImplementation) SearchForApplicationBoxes(ctx echo.Context, applicationID uint64, params generated.SearchForApplicationBoxesParams) error
SearchForApplicationBoxes returns box names for an app (GET /v2/applications/{application-id}/boxes)
func (*ServerImplementation) SearchForApplications ¶
func (si *ServerImplementation) SearchForApplications(ctx echo.Context, params generated.SearchForApplicationsParams) error
SearchForApplications returns applications for the provided parameters. (GET /v2/applications)
func (*ServerImplementation) SearchForAssets ¶
func (si *ServerImplementation) SearchForAssets(ctx echo.Context, params generated.SearchForAssetsParams) error
SearchForAssets returns assets matching the provided parameters (GET /v2/assets)
func (*ServerImplementation) SearchForBlockHeaders ¶ added in v3.7.0
func (si *ServerImplementation) SearchForBlockHeaders(ctx echo.Context, params generated.SearchForBlockHeadersParams) error
SearchForBlockHeaders returns block headers matching the provided parameters (GET /v2/blocks)
func (*ServerImplementation) SearchForTransactions ¶
func (si *ServerImplementation) SearchForTransactions(ctx echo.Context, params generated.SearchForTransactionsParams) error
SearchForTransactions returns transactions matching the provided parameters (GET /v2/transactions)
Source Files
¶
Directories
¶
Path | Synopsis |
---|---|
generated
|
|
common
Package common provides primitives to interact with the openapi HTTP API.
|
Package common provides primitives to interact with the openapi HTTP API. |
v2
Package generated provides primitives to interact with the openapi HTTP API.
|
Package generated provides primitives to interact with the openapi HTTP API. |