sibyl2

README

sibyl 2

An out-of-box codebase snapshot service, for everyone.

中文文档

Status

Name	Badge
Latest Version
Unit Tests
Docker Image
Perf Tests
Code Coverage
Code Style

Overview

sibyl2 is a static code analyze service, for extracting, managing and offering codebase snapshot. Inspired by semantic of GitHub.

• Easy to use
• Fast enough
• Extensible
• Multiple languages in one (Go/Java/Python ...)

What's Codebase Snapshot?

Raw source code:

func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error) {
// ...
}

Code snapshot is the logical metadata of your code:

We can do series of things based on it. Such as logical diff, function relationship analysis.

Purpose & Principles

See About This Project: Code Snapshot Layer In DevOps for details.

Languages support

Languages	Function	Function Context	Class
Golang	Yes	Yes	Yes
Java	Yes	Yes	Yes
Python	Yes	Yes	Yes
Kotlin	Yes	Yes	Yes
JavaScript	Yes	Yes	Yes

Try it in 3 minutes

Deployment

For now, we are aiming at offering an out-of-box service. Users can access all the features with a simple binary file, without any extra dependencies and scripts.

You can download from the release page.

Or directly download with wget (linux only):

curl https://raw.githubusercontent.com/opensibyl/sibyl2/master/scripts/download_latest.sh | bash

Now you can start it:

./sibyl server

That's it. Server will run on port :9876. Data will be persisted in ./sibyl2Storage.

Upload

./sibyl upload --src . --url http://127.0.0.1:9876

You can upload from different machines. Usually it only takes a few seconds.

Access

We have a built-in dashboard for visualization. Start it with:

./sibyl frontend

And open localhost:3000 you will see:

Also, you can access all the datas via different kinds of languages, to build your own tools:

Language	Link
Golang	https://github.com/opensibyl/sibyl-go-client
Java	https://github.com/opensibyl/sibyl-java-client
JavaScript	https://github.com/opensibyl/sibyl-javascript-client

See our examples about how to use for details.

Performance

We have tested it on some famous repos, like guava. And that's why we can say it is " fast enough".

See https://github.com/williamfzc/sibyl2/actions/workflows/perf.yml for details.

Language	Repo	Cost
Golang	https://github.com/gin-gonic/gin.git	~1s
Java	https://github.com/spring-projects/spring-boot.git	~50s
Python	https://github.com/psf/requests	~1s
Kotlin	https://github.com/square/okhttp	~1s

Contribution

This project split into 3 main parts:

• /cmd: Pure command line tool for general usage
• /pkg/server: All-in-one service for production
• others: Shared api and core

Workflow:

• core: collect files and convert them to Unit.
• extract: classify and process units into functions, symbols.
• api: higher level analyze like callgraph

Issues / PRs are welcome!

References

• basic grammar: https://tree-sitter.github.io/tree-sitter/creating-parsers#the-grammar-dsl
• language parser (for example, golang): https://github.com/tree-sitter/tree-sitter-go/blob/master/src/parser.c
• symbol: https://github.com/github/semantic/blob/main/docs/examples.md#symbols
• stack graphs: https://github.blog/2021-12-09-introducing-stack-graphs/

License

Apache License Version 2.0, see LICENSE

Documentation

Index

• Constants
• func Extract(targetFile string, config *ExtractConfig) ([]*extractor.FileResult, error)
• func ExtractClazz(targetFile string, config *ExtractConfig) ([]*extractor.ClazzFileResult, error)
• func ExtractFromBytes(content []byte, config *ExtractConfig) (*extractor.FileResult, error)
• func ExtractFromString(content string, config *ExtractConfig) (*extractor.FileResult, error)
• func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error)
• func ExtractSymbol(targetFile string, config *ExtractConfig) ([]*extractor.SymbolFileResult, error)
• func QueryUnitsByIndexNames[T extractor.DataType](result *extractor.BaseFileResult[T], indexNames ...string) []T
• func QueryUnitsByIndexNamesInFiles[T extractor.DataType](result []*extractor.BaseFileResult[T], indexNames ...string) []T
• func QueryUnitsByLines[T extractor.DataType](result *extractor.BaseFileResult[T], lines ...int) []T
• type AdjacencyMapType
• type ClazzWithPath
• type ExtractConfig
  • func DefaultConfig() *ExtractConfig
• type FuncGraph
  • func AnalyzeFuncGraph(funcFiles []*extractor.FunctionFileResult, ...) (*FuncGraph, error)
  • func (fg *FuncGraph) FindCalls(f *extractor.Function) []*FunctionWithPath
  • func (fg *FuncGraph) FindRelated(f *extractor.Function) *FunctionContext
  • func (fg *FuncGraph) FindReverseCalls(f *extractor.Function) []*FunctionWithPath
  • func (fg *FuncGraph) WrapFuncWithPath(f *extractor.Function) (*FunctionWithPath, error)
• type FuncGraphType
  • func WrapFuncGraph(g graph.Graph[string, *FunctionWithPath]) *FuncGraphType
  • func (fgt *FuncGraphType) GetAdjacencyMap() (*AdjacencyMapType, error)
• type FunctionContext
  • func (f *FunctionContext) ToGraph() *FuncGraphType
  • func (f *FunctionContext) ToJson() ([]byte, error)
  • func (f *FunctionContext) ToMap() (map[string]any, error)
  • func (f *FunctionContext) ToSlim() *FunctionContextSlim
• type FunctionContextSlim
• type FunctionWithPath
• type SymbolWithPath

Constants

const HomePage = "https://github.com/opensibyl/sibyl2"

const Version = "v0.14.0"

Version strictly sync with tag

Functions

func Extract

func Extract(targetFile string, config *ExtractConfig) ([]*extractor.FileResult, error)

func ExtractClazz

func ExtractClazz(targetFile string, config *ExtractConfig) ([]*extractor.ClazzFileResult, error)

func ExtractFromBytes

func ExtractFromBytes(content []byte, config *ExtractConfig) (*extractor.FileResult, error)

func ExtractFromString

func ExtractFromString(content string, config *ExtractConfig) (*extractor.FileResult, error)

func ExtractFunction

func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error)

func ExtractSymbol

func ExtractSymbol(targetFile string, config *ExtractConfig) ([]*extractor.SymbolFileResult, error)

func QueryUnitsByIndexNames

func QueryUnitsByIndexNames[T extractor.DataType](result *extractor.BaseFileResult[T], indexNames ...string) []T

func QueryUnitsByIndexNamesInFiles

func QueryUnitsByIndexNamesInFiles[T extractor.DataType](result []*extractor.BaseFileResult[T], indexNames ...string) []T

func QueryUnitsByLines

func QueryUnitsByLines[T extractor.DataType](result *extractor.BaseFileResult[T], lines ...int) []T

Types

type AdjacencyMapType

type AdjacencyMapType = map[string]map[string]graph.Edge[string]

type ClazzWithPath

type ClazzWithPath struct {
	*extractor.Clazz
	Path string `json:"path"`
}

type ExtractConfig

type ExtractConfig struct {
	LangType    core.LangType
	ExtractType extractor.ExtractType
	FileFilter  func(path string) bool
}

ExtractConfig todo: should not use config ptr for parallel running

func DefaultConfig

func DefaultConfig() *ExtractConfig

type FuncGraph

type FuncGraph struct {
	ReverseCallGraph *FuncGraphType
	CallGraph        *FuncGraphType
}

FuncGraph

It is not a serious `call` graph. It based on references not real calls.

Why we used it: - We can still use something like `method_invocation` - But we mainly use it to evaluate the influence of a method - In many languages, scope of `invocation` is too small - For example, use `function` as a parameter.

func AnalyzeFuncGraph

func AnalyzeFuncGraph(funcFiles []*extractor.FunctionFileResult, symbolFiles []*extractor.SymbolFileResult) (*FuncGraph, error)

func (*FuncGraph) FindCalls

func (fg *FuncGraph) FindCalls(f *extractor.Function) []*FunctionWithPath

func (*FuncGraph) FindRelated

func (fg *FuncGraph) FindRelated(f *extractor.Function) *FunctionContext

func (*FuncGraph) FindReverseCalls

func (fg *FuncGraph) FindReverseCalls(f *extractor.Function) []*FunctionWithPath

func (*FuncGraph) WrapFuncWithPath

func (fg *FuncGraph) WrapFuncWithPath(f *extractor.Function) (*FunctionWithPath, error)

type FuncGraphType

type FuncGraphType struct {
	graph.Graph[string, *FunctionWithPath]
	// contains filtered or unexported fields
}

func WrapFuncGraph

func WrapFuncGraph(g graph.Graph[string, *FunctionWithPath]) *FuncGraphType

func (*FuncGraphType) GetAdjacencyMap

func (fgt *FuncGraphType) GetAdjacencyMap() (*AdjacencyMapType, error)

type FunctionContext

type FunctionContext struct {
	*FunctionWithPath
	Calls        []*FunctionWithPath `json:"calls"`
	ReverseCalls []*FunctionWithPath `json:"reverseCalls"`
}

func (*FunctionContext) ToGraph

func (f *FunctionContext) ToGraph() *FuncGraphType

func (*FunctionContext) ToJson

func (f *FunctionContext) ToJson() ([]byte, error)

func (*FunctionContext) ToMap

func (f *FunctionContext) ToMap() (map[string]any, error)

ToMap export a very simple map without any custom structs. It will lose ptr to origin unit.

func (*FunctionContext) ToSlim

func (f *FunctionContext) ToSlim() *FunctionContextSlim

type FunctionContextSlim

type FunctionContextSlim struct {
	*FunctionWithPath
	Calls        []string `json:"calls"`
	ReverseCalls []string `json:"reverseCalls"`
}

FunctionContextSlim instead of whole object, slim will only keep the signature

type FunctionWithPath

type FunctionWithPath struct {
	*extractor.Function
	Path string `json:"path"`
}

FunctionWithPath original symbol and function do not have a path because they maybe not come from a real file

type SymbolWithPath

type SymbolWithPath struct {
	*extractor.Symbol
	Path string `json:"path"`
}