internal/

README

Development and Maintenance Guide

Structure

This internal package encapuslates all the details required for LOOPPs to work. The primary moving pieces are organized into individual packages. Note that in the list below sub packages are omitted for clarity

pb                // protobuf definitions and generated code 
types             // common types
net               // networking for loop-specific communication
core              // abstractions that are served by the core node
test              // top-level test helpers used for multi-process testing
relayer           // abstractions that are served by relayer instances
goplugin          // wrappers for the hashicorp go-plugin library. this is the base layer of all LOOPP implementations
reportingplugin   // abstractions served by product-specific LOOPPs. maybe extended or renamed to encapsulate any application specific logic.

At the current time, we do not foresee the need for new top-level packages. If, after reading this guide, you find yourself wanting one, reach out to the Foundations team

Feature Development

Roughly speaking there are four types of feature development, and they break down largely along the package divisions:

• Foundational LOOPP features
• Core node abstraction features
• Relayer abstraction features
• Product abstraction features

This README is top level guide, see the individual package READMEs for more detailed information.

Foundational LOOP Development

The root of these changes are expected to be in net, for things like connection proxying and brokering, or goplugin to assimilate patches or features from Hashicorp. Changes to either of this packages are likely to propagate to the higher level abstractions in core, relayer, and reportingplugin. They may require new kinds of testing infra structure in the test package to ensure proper functioning across multiple processes.

Core node abstraction Development

There are roughly two kinds of development, adding entirely new services and modifying existing services.

The flow to add a new service is

• define your interface in the chainlink-common/pkg/types tree
• create the proto definition in a sub package of pb.
  • golang interface maps to proto service and struct map to message (see the pb README for details about organization and code gen)
• create a pkg in chainlink-common/pkg/loop/internal/core/services and implement the gRPC server and client
• create a test package (chainlink-common/pkg/loop/internal/core/services/mynewservice/test) that provides a static implementation of your interface. see existing code for examples of the pattern, and documentation in pkg/loop/internal/test

Modifications follow a very similar pattern: update the interface (if needed), update the proto to reflect any interface changes, implement and test the changes. See Protobuf best practices[[Protobuf best practices]] for details ensuring backward compatibility of protobufs.

Relayer abstraction Development

There are three kinds of abstractions with the relayer: common components, such as the chainreader, ocr-specific shared components, such as offchain digester and the ocr-specific, product-specific components, known as Providers. Note that the line between the latter two categories is blurry, ocr-specific shared component should be thought as component that have a well defined and reasonable default value; custom Providers clearly fall outside this domain, yet custom Providers may also implement OCR components that have a default implementation.

Go to [Production abstraction Development] for more information about custom Provider development.

Developing common relayer components such as chainreader follows the same pattern as [Core node abstraction Development]. The primary difference is the package location. Instead of adding code to chainlink-common/pkg/loop/internal/core/services you will work in chainlink-common/pkg/loop/internal/relayer or chainlink-common/pkg/loop/internal/relayer/pluginprovider depending on the scope of your changes.

Product abstraction Development

Product specific development is the most nuanced kind of development. There are at least three different paths:

1. Use the Core Node API, without a custom provider.

   This is our vision, but will not be viable until chain reading and writing abstractions are implemented.

2. Use the Core Node API with a custom Provider.

   In practice this means that you changes will primarily be in chainlink-common/pkg/loop/internal/relayer/pluginprovider/ext, the home of custom extensions to the relayer interface as well as type definitions for your interface in chainlink-common/pkg/types. See the README in chainlink-common/pkg/loop/internal/relayer/pluginprovider/ext/ccip for an example.

   The best practice for testing a custom Provider is to implement a static test implementation NOT to use mocks. The static implementation is important because it can be used consistently both in unit tests and tests that exercise gRPC de/serialization. See the chainlink-common/pkg/loop/internal/relayer/pluginprovider/ext/ccip/test for examples.

3. Do not use the Core Node API and implement a custom Provider.

   This is the current state of the world for everything except Median. You need to develop the custom Provider as above. Additionally, you need to define a gRPC layer for the Factory creation of your custom reporting plugin. The Core Node API provides a reuseable mechanism for Factory instantiation and if you don't use it, you have to roll your own gRPC layer. See chainlink-common/pkg/loop/internal/reportingplugin for examples.

   In order to test your custom factory, you need to modify the test driver main program to support it chainlink-common/pkg/loop/internal/test/cmd/main.go

Protobuf best practices

It's extremely important to be mindful of compatibility when changing protobuf definition. See the official best practices for details.

To update protobuf, update the corresponding .proto file in the pb package. Then run make generate to regenerate the code.

FAQ