sibyl2

README

sibyl 2

Bring your plain code alive with tiny effort.

中文文档

Status

Name	Badge
Latest Version
Unit Tests
Docker Image
Perf Tests
Code Coverage
Code Style

Overview

sibyl2 is a static code analyzer, for extracting, managing and offering codebase snapshot. Inspired by semantic of GitHub.

• Easy to use
• Fast enough
• Extensible
• Multiple languages in one (Go/Java/Python ...)

What's Codebase Snapshot?

Raw source code:

func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error) {
// ...
}

Code snapshot is the logical metadata of your code:

Purpose & Principles

See About This Project: Code Snapshot Layer In DevOps for details.

Languages support

Languages	Function	Function Context	Class
Golang	Yes	Yes	Yes
Java	Yes	Yes	Yes
Python	Yes	Yes	Not yet

Examples of Usage

One-file-installation

For now, we are aiming at offering an out-of-box service. Users can access all the features with a simple binary file, without any extra dependencies and scripts.

You can download from the release page.

Or directly download with wget (linux only):

curl https://raw.githubusercontent.com/opensibyl/sibyl2/master/scripts/download_latest.sh | bash

Use as a service (recommend)

Deploy

./sibyl server

That's it. Server will run on port :9876. Data will be persisted in ./sibyl2Storage.

Upload

./sibyl upload --src . --url http://127.0.0.1:9876

You can upload from different machines.

Access

After uploading, you can access your data via http api.

Tree-like storage:

• repo
  • rev1
    • file
      • function
  • rev2
    • file
      • function

Try with swagger: http://127.0.0.1:9876/swagger/index.html#/

Access with sdk

Easily combine with other systems:

• golang: https://github.com/opensibyl/sibyl-go-client
• java: https://github.com/opensibyl/sibyl-java-client

We use openapi-generator for creating clients automatically. Also, you can easily create your own one.

java -jar ~/Downloads/openapi-generator-cli-6.2.1.jar generate -i pkg/server/docs/swagger.yaml -g go -o ~/github_workspace/sibyl-go-client

Use as a commandline tool

Basic Functions

./sibyl extract --src . --output hello.json

You will see:

$ ./sibyl extract --src . --output hello.json
{"level":"info","ts":1670138890.5306911,"caller":"sibyl2/extract_fs.go:92","msg":"no specific lang found, do the guess in: /Users/fengzhangchi/github_workspace/sibyl2"}
{"level":"info","ts":1670138890.5596569,"caller":"sibyl2/extract_fs.go:97","msg":"I think it is: GOLANG"}
{"level":"info","ts":1670138890.5836658,"caller":"core/runner.go:22","msg":"valid file count: 55"}
{"level":"info","ts":1670138890.6657321,"caller":"sibyl2/extract_fs.go:76","msg":"cost: 135 ms"}
{"level":"info","ts":1670138890.669896,"caller":"extract/cmd_extract.go:60","msg":"file has been saved to: hello.json"}

... Result will be generated in seconds.

[
  {
    "path": "analyze.go",
    "language": "GOLANG",
    "type": "func",
    "units": [
      {
        "name": "AnalyzeFuncGraph",
        "receiver": "",
        "parameters": [
          {
            "type": "[]*extractor.FunctionFileResult",
            "name": "funcFiles"
          },
          {
            "type": "[]*extractor.SymbolFileResult",
            "name": "symbolFiles"
          }
        ],
        "returns": [
          {
            "type": "*FuncGraph",
            "name": ""
          },
          {
            "type": "error",
            "name": ""
          }
        ],
        "span": {
          "start": {
            "row": 11,
            "column": 0
          },
          "end": {
            "row": 80,
            "column": 1
          }
        },
        "extras": {}
      }
    ]
  },
  ...
]

Source Code History Visualization

Source code history visualization, inspired by https://github.com/acaudwell/Gource

One line command to see how your repository grow up, with no heavy dependencies like OpenGL, with logic level messages.

./sibyl history --src . --output hello.html --full

https://user-images.githubusercontent.com/13421694/207089314-21b0d48d-00d1-4de5-951c-415fed74c78f.mp4

You can remove the full flag for better performance.

Smart Git Diff

Normal git diff has only text level messages.

./sibyl diff --from HEAD~1 --to HEAD --output hello1.json

And you can get a structural one with sibyl, which contains method level messages and callgraphs.

{
  "fragments": [
    {
      "path": "pkg/server/admin_s.go",
      "functions": [
        {
          "name": "HandleStatusUpload",
          "receiver": "",
          "parameters": [
            {
              "type": "*gin.Context",
              "name": "c"
            }
          ],
          "returns": null,
          "span": {
            "start": {
              "row": 17,
              "column": 0
            },
            "end": {
              "row": 23,
              "column": 1
            }
          },
          "extras": {},
          "path": "pkg/server/admin_s.go",
          "language": "GOLANG",
          "calls": null,
          "reverseCalls": [
            {
              "name": "Execute",
              "receiver": "",
              "parameters": [
                {
                  "type": "ExecuteConfig",
                  "name": "config"
                }
              ],
              "returns": null,
              "span": {
                "start": {
                  "row": 67,
                  "column": 0
                },
                "end": {
                  "row": 96,
                  "column": 1
                }
              },
              "extras": {},
              "path": "pkg/server/app.go",
              "language": "GOLANG"
            }
          ]
        }
      ]
    },
    ...

You can easily build some smart test tools above it. For example, Google 's unittest speed up:

Performance

We have tested it on some famous repos, like guava. And that's why we can say it is " fast enough".

See https://github.com/williamfzc/sibyl2/actions/workflows/perf.yml for details.

Contribution

This project split into 3 main parts:

• /cmd: Pure command line tool for general usage
• /pkg/server: All-in-one service for production
• others: Shared api and core

Workflow:

• core: collect files and convert them to Unit.
• extract: classify and process units into functions, symbols.
• api: higher level analyze like callgraph

Issues / PRs are welcome!

Refs

• basic grammar: https://tree-sitter.github.io/tree-sitter/creating-parsers#the-grammar-dsl
• language parser (for example, golang): https://github.com/tree-sitter/tree-sitter-go/blob/master/src/parser.c
• symbol: https://github.com/github/semantic/blob/main/docs/examples.md#symbols
• stack graphs: https://github.blog/2021-12-09-introducing-stack-graphs/

License

Apache License Version 2.0, see LICENSE

Documentation

Index

• func Extract(targetFile string, config *ExtractConfig) ([]*extractor.FileResult, error)
• func ExtractClazz(targetFile string, config *ExtractConfig) ([]*extractor.ClazzFileResult, error)
• func ExtractFromBytes(content []byte, config *ExtractConfig) (*extractor.FileResult, error)
• func ExtractFromString(content string, config *ExtractConfig) (*extractor.FileResult, error)
• func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error)
• func ExtractSymbol(targetFile string, config *ExtractConfig) ([]*extractor.SymbolFileResult, error)
• func QueryUnitsByIndexNames[T extractor.DataType](result *extractor.BaseFileResult[T], indexNames ...string) []T
• func QueryUnitsByIndexNamesInFiles[T extractor.DataType](result []*extractor.BaseFileResult[T], indexNames ...string) []T
• func QueryUnitsByLines[T extractor.DataType](result *extractor.BaseFileResult[T], lines ...int) []T
• type AdjacencyMapType
• type ClazzWithPath
• type ExtractConfig
  • func DefaultConfig() *ExtractConfig
• type FuncGraph
  • func AnalyzeFuncGraph(funcFiles []*extractor.FunctionFileResult, ...) (*FuncGraph, error)
  • func (fg *FuncGraph) FindCalls(f *extractor.Function) []*FunctionWithPath
  • func (fg *FuncGraph) FindRelated(f *extractor.Function) *FunctionContext
  • func (fg *FuncGraph) FindReverseCalls(f *extractor.Function) []*FunctionWithPath
  • func (fg *FuncGraph) WrapFuncWithPath(f *extractor.Function) (*FunctionWithPath, error)
• type FuncGraphType
  • func WrapFuncGraph(g graph.Graph[string, *FunctionWithPath]) *FuncGraphType
  • func (fgt *FuncGraphType) GetAdjacencyMap() (*AdjacencyMapType, error)
• type FunctionContext
  • func Json2FuncCtx(exported []byte) (*FunctionContext, error)
  • func (f *FunctionContext) ToGraph() *FuncGraphType
  • func (f *FunctionContext) ToJson() ([]byte, error)
  • func (f *FunctionContext) ToMap() (map[string]any, error)
• type FunctionWithPath
• type SymbolWithPath

Functions

func Extract

func Extract(targetFile string, config *ExtractConfig) ([]*extractor.FileResult, error)

func ExtractClazz

func ExtractClazz(targetFile string, config *ExtractConfig) ([]*extractor.ClazzFileResult, error)

func ExtractFromBytes

func ExtractFromBytes(content []byte, config *ExtractConfig) (*extractor.FileResult, error)

func ExtractFromString

func ExtractFromString(content string, config *ExtractConfig) (*extractor.FileResult, error)

func ExtractFunction

func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error)

func ExtractSymbol

func ExtractSymbol(targetFile string, config *ExtractConfig) ([]*extractor.SymbolFileResult, error)

func QueryUnitsByIndexNames

func QueryUnitsByIndexNames[T extractor.DataType](result *extractor.BaseFileResult[T], indexNames ...string) []T

func QueryUnitsByIndexNamesInFiles

func QueryUnitsByIndexNamesInFiles[T extractor.DataType](result []*extractor.BaseFileResult[T], indexNames ...string) []T

func QueryUnitsByLines

func QueryUnitsByLines[T extractor.DataType](result *extractor.BaseFileResult[T], lines ...int) []T

Types

type AdjacencyMapType

type AdjacencyMapType = map[string]map[string]graph.Edge[string]

type ClazzWithPath

type ClazzWithPath struct {
	*extractor.Clazz
	Path     string        `json:"path"`
	Language core.LangType `json:"language"`
}

type ExtractConfig

type ExtractConfig struct {
	LangType    core.LangType
	ExtractType extractor.ExtractType
	FileFilter  func(path string) bool
}

ExtractConfig todo: should not use config ptr for parallel running

func DefaultConfig

func DefaultConfig() *ExtractConfig

type FuncGraph

type FuncGraph struct {
	ReverseCallGraph *FuncGraphType
	CallGraph        *FuncGraphType
}

FuncGraph

It is not a serious `call` graph. It based on references not real calls.

Why we used it: - We can still use something like `method_invocation` - But we mainly use it to evaluate the influence of a method - In many languages, scope of `invocation` is too small - For example, use `function` as a parameter.

func AnalyzeFuncGraph

func AnalyzeFuncGraph(funcFiles []*extractor.FunctionFileResult, symbolFiles []*extractor.SymbolFileResult) (*FuncGraph, error)

func (*FuncGraph) FindCalls

func (fg *FuncGraph) FindCalls(f *extractor.Function) []*FunctionWithPath

func (*FuncGraph) FindRelated

func (fg *FuncGraph) FindRelated(f *extractor.Function) *FunctionContext

func (*FuncGraph) FindReverseCalls

func (fg *FuncGraph) FindReverseCalls(f *extractor.Function) []*FunctionWithPath

func (*FuncGraph) WrapFuncWithPath

func (fg *FuncGraph) WrapFuncWithPath(f *extractor.Function) (*FunctionWithPath, error)

type FuncGraphType

type FuncGraphType struct {
	graph.Graph[string, *FunctionWithPath]
	// contains filtered or unexported fields
}

func WrapFuncGraph

func WrapFuncGraph(g graph.Graph[string, *FunctionWithPath]) *FuncGraphType

func (*FuncGraphType) GetAdjacencyMap

func (fgt *FuncGraphType) GetAdjacencyMap() (*AdjacencyMapType, error)

type FunctionContext

type FunctionContext struct {
	*FunctionWithPath
	Calls        []*FunctionWithPath `json:"calls"`
	ReverseCalls []*FunctionWithPath `json:"reverseCalls"`
}

func Json2FuncCtx

func Json2FuncCtx(exported []byte) (*FunctionContext, error)

func (*FunctionContext) ToGraph

func (f *FunctionContext) ToGraph() *FuncGraphType

func (*FunctionContext) ToJson

func (f *FunctionContext) ToJson() ([]byte, error)

func (*FunctionContext) ToMap

func (f *FunctionContext) ToMap() (map[string]any, error)

ToMap export a very simple map without any custom structs. It will lose ptr to origin unit.

type FunctionWithPath

type FunctionWithPath struct {
	*extractor.Function
	Path     string        `json:"path"`
	Language core.LangType `json:"language"`
}

FunctionWithPath original symbol and function do not have a path because they maybe not come from a real file

type SymbolWithPath

type SymbolWithPath struct {
	*extractor.Symbol
	Path string `json:"path"`
}