sibyl2

package module
v0.7.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 13, 2022 License: Apache-2.0 Imports: 10 Imported by: 3

README

sibyl 2

Take a quick snapshot of your codebase in seconds, with zero cost.

What's it?

sibyl2 is a static code analyzer, for extracting meta-data (functions/classes/parameters/lines ...) from raw source code. Inspired by semantic of GitHub.

  • Easy to use
  • Fast enough
  • Extensible
  • Multiple languages in one (Go/Java/Python ...)

It can translate raw source code:

func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error) {
// ...
}

to logical structure：

all you need is a one-line command:

./sibyl extract --src . --output hello.json

and export them with different formats.

... JSON format for your custom operations 
{
  "path": "extract.go",
  "language": "GOLANG",
  "type": "func",
  "units": [
    {
      "name": "ExtractFunction",
      "receiver": "",
      "parameters": [
        {
          "type": "string",
          "name": "targetFile"
        },
        {
          "type": "*ExtractConfig",
          "name": "config"
        }
      ],
      "returns": [
        {
          "type": "[]*extractor.FunctionFileResult",
          "name": ""
        },
        {
          "type": "error",
          "name": ""
        }
      ],
      "span": {
        "start": {
          "row": 18,
          "column": 0
        },
        "end": {
          "row": 46,
          "column": 1
        }
      },
      "extras": null
    }
  ]
}
... And multiple languages supported 
public class Java8SnapshotListener extends Java8MethodLayerListener<Method> {
    @Override
    public void enterMethodDeclarationWithoutMethodBody(
            Java8Parser.MethodDeclarationWithoutMethodBodyContext ctx) {
        super.enterMethodDeclarationWithoutMethodBody(ctx);
        this.storage.save(curMethodStack.peekLast());
    }
}

after:

{
	"name": "enterMethodDeclarationWithoutMethodBody",
	"receiver": "com.williamfzc.sibyl.core.listener.java8.Java8SnapshotListener",
	"parameters": [{
		"type": "Java8Parser.MethodDeclarationWithoutMethodBodyContext",
		"name": "ctx"
	}],
	"returns": [{
		"type": "void",
		"name": ""
	}],
	"span": {
		"start": {
			"row": 8,
			"column": 4
		},
		"end": {
			"row": 13,
			"column": 5
		}
	},
	"extras": {
		"annotations": ["@Override"]
	}
}

Usage examples

Developers can easily combine it with any other tools. We have already built some cmd tools in use. You can download from the release page.

Default Functions
./sibyl extract --src . --output hello.json

You will see:

$ ./sibyl extract --src . --output hello.json
{"level":"info","ts":1670138890.5306911,"caller":"sibyl2/extract_fs.go:92","msg":"no specific lang found, do the guess in: /Users/fengzhangchi/github_workspace/sibyl2"}
{"level":"info","ts":1670138890.5596569,"caller":"sibyl2/extract_fs.go:97","msg":"I think it is: GOLANG"}
{"level":"info","ts":1670138890.5836658,"caller":"core/runner.go:22","msg":"valid file count: 55"}
{"level":"info","ts":1670138890.6657321,"caller":"sibyl2/extract_fs.go:76","msg":"cost: 135 ms"}
{"level":"info","ts":1670138890.669896,"caller":"extract/cmd_extract.go:60","msg":"file has been saved to: hello.json"}
... Result will be generated in seconds. 
[
  {
    "path": "analyze.go",
    "language": "GOLANG",
    "type": "func",
    "units": [
      {
        "name": "AnalyzeFuncGraph",
        "receiver": "",
        "parameters": [
          {
            "type": "[]*extractor.FunctionFileResult",
            "name": "funcFiles"
          },
          {
            "type": "[]*extractor.SymbolFileResult",
            "name": "symbolFiles"
          }
        ],
        "returns": [
          {
            "type": "*FuncGraph",
            "name": ""
          },
          {
            "type": "error",
            "name": ""
          }
        ],
        "span": {
          "start": {
            "row": 11,
            "column": 0
          },
          "end": {
            "row": 80,
            "column": 1
          }
        },
        "extras": {}
      }
    ]
  },
  ...
]
Source Code History Visualization

Source code history visualization, inspired by https://github.com/acaudwell/Gource

One line command to see how your repository grow up, with no heavy dependencies like OpenGL, with logic level messages.

./sibyl history --src . --output hello.html --full

https://user-images.githubusercontent.com/13421694/207089314-21b0d48d-00d1-4de5-951c-415fed74c78f.mp4

You can remove the full flag for better performance.

Smart Git Diff

Normal git diff has only text level messages.

./sibyl diff --from HEAD~1 --to HEAD --output hello1.json

And you can get a structural one with sibyl, which contains method level messages and callgraphs.

{
  "fragments": [
    {
      "path": "pkg/server/admin_s.go",
      "functions": [
        {
          "name": "HandleStatusUpload",
          "receiver": "",
          "parameters": [
            {
              "type": "*gin.Context",
              "name": "c"
            }
          ],
          "returns": null,
          "span": {
            "start": {
              "row": 17,
              "column": 0
            },
            "end": {
              "row": 23,
              "column": 1
            }
          },
          "extras": {},
          "path": "pkg/server/admin_s.go",
          "language": "GOLANG",
          "calls": null,
          "reverseCalls": [
            {
              "name": "Execute",
              "receiver": "",
              "parameters": [
                {
                  "type": "ExecuteConfig",
                  "name": "config"
                }
              ],
              "returns": null,
              "span": {
                "start": {
                  "row": 67,
                  "column": 0
                },
                "end": {
                  "row": 96,
                  "column": 1
                }
              },
              "extras": {},
              "path": "pkg/server/app.go",
              "language": "GOLANG"
            }
          ]
        }
      ]
    },
    ...

intro-google

You can easily build some smart test tools above it.

PaaS

About how we use in real production. Documentation is on the way.

image

Principle

Many devtools required meta-data inside source code for further processing. For example, code coverage, call graph, etc.

Without a cross-languages shared lib, we have to re-implement this extract layer repeatedly for each tool, each language.

This repo aims at offering a general use shared lib for source code meta-data.

Performance

We have tested it on some famous repos, like guava. And that's why we can say it is "fast enough".

See https://github.com/williamfzc/sibyl2/actions/workflows/perf.yml for details.

Contribution

This project split into 3 main parts:

  • /cmd: Pure command line tool for general usage
  • /pkg/server: All-in-one service for production
  • others: Shared api and core

Workflow:

  • core: collect files and convert them to Unit.
  • extract: classify and process units into functions, symbols.
  • api: higher level analyze like callgraph

Issues / PRs are welcome!

Refs

License

Apache License Version 2.0, see LICENSE

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Extract 

func Extract(targetFile string, config *ExtractConfig) ([]*extractor.FileResult, error)

func ExtractFromBytes 

func ExtractFromBytes(content []byte, config *ExtractConfig) (*extractor.FileResult, error)

func ExtractFromString 

func ExtractFromString(content string, config *ExtractConfig) (*extractor.FileResult, error)

func ExtractFunction 

func ExtractFunction(targetFile string, config *ExtractConfig) ([]*extractor.FunctionFileResult, error)

func ExtractSymbol 

func ExtractSymbol(targetFile string, config *ExtractConfig) ([]*extractor.SymbolFileResult, error)

func QueryUnitsByIndexNames 

func QueryUnitsByIndexNames[T extractor.DataType](result *extractor.BaseFileResult[T], indexNames ...string) []T

func QueryUnitsByIndexNamesInFiles 

func QueryUnitsByIndexNamesInFiles[T extractor.DataType](result []*extractor.BaseFileResult[T], indexNames ...string) []T

func QueryUnitsByLines 

func QueryUnitsByLines[T extractor.DataType](result *extractor.BaseFileResult[T], lines ...int) []T

Types

type AdjacencyMapType 

type AdjacencyMapType = map[string]map[string]graph.Edge[string]

type ExtractConfig 

type ExtractConfig struct {
	LangType    core.LangType
	ExtractType extractor.ExtractType
	FileFilter  func(path string) bool
}

ExtractConfig todo: should not use config ptr for parallel running

func DefaultConfig 

func DefaultConfig() *ExtractConfig

type FuncGraph 

type FuncGraph struct {
	ReverseCallGraph *FuncGraphType
	CallGraph        *FuncGraphType
}

FuncGraph

It is not a serious `call` graph. It based on references not real calls.

Why we used it: - We can still use something like `method_invocation` - But we mainly use it to evaluate the influence of a method - In many languages, scope of `invocation` is too small - For example, use `function` as a parameter.

func AnalyzeFuncGraph 

func AnalyzeFuncGraph(funcFiles []*extractor.FunctionFileResult, symbolFiles []*extractor.SymbolFileResult) (*FuncGraph, error)

func (*FuncGraph) FindCalls 

func (fg *FuncGraph) FindCalls(f *extractor.Function) []*FunctionWithPath

func (*FuncGraph) FindRelated 

func (fg *FuncGraph) FindRelated(f *extractor.Function) *FunctionContext

func (*FuncGraph) FindReverseCalls 

func (fg *FuncGraph) FindReverseCalls(f *extractor.Function) []*FunctionWithPath

func (*FuncGraph) WrapFuncWithPath 

func (fg *FuncGraph) WrapFuncWithPath(f *extractor.Function) (*FunctionWithPath, error)

type FuncGraphType 

type FuncGraphType struct {
	graph.Graph[string, *FunctionWithPath]
	// contains filtered or unexported fields
}

func WrapFuncGraph 

func WrapFuncGraph(g graph.Graph[string, *FunctionWithPath]) *FuncGraphType

func (*FuncGraphType) GetAdjacencyMap 

func (fgt *FuncGraphType) GetAdjacencyMap() (*AdjacencyMapType, error)

type FunctionContext 

type FunctionContext struct {
	*FunctionWithPath
	Calls        []*FunctionWithPath `json:"calls"`
	ReverseCalls []*FunctionWithPath `json:"reverseCalls"`
}

func (*FunctionContext) ToGraph 

func (f *FunctionContext) ToGraph() *FuncGraphType

type FunctionWithPath 

type FunctionWithPath struct {
	*extractor.Function
	Path     string        `json:"path"`
	Language core.LangType `json:"language"`
}

FunctionWithPath original symbol and function do not have a path because they maybe not come from a real file

type SymbolWithPath 

type SymbolWithPath struct {
	*extractor.Symbol
	Path string `json:"path"`
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
sibyl
sibyl/subs/diff
sibyl/subs/extract
sibyl/subs/history
sibyl/subs/server
sibyl/subs/upload
pkg
core
ext
extractor
server
server/binding
server/object
server/queue
server/service
server/worker

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.