goexec

package
v0.10.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 16, 2024 License: MIT Imports: 36 Imported by: 0

README

Package goexec

The package goexec is responsible for executing the notebook cells using Go, as well as keeping the state of all current declarations (functions, variables, types, constants, imports) that once declared, stay alive for subsequent cell executions.

The code is organized in the following files:

  • goexec.go: definition of the main State object and the various Go structs for the various declarations (functions, variables, types, constants, imports), including the concept of cursor.
  • execcode.go: implements State.ExecuteCell(), the main functionality offered by the package.
  • composer.go: generate dynamically a main.go file from pre-parsed declarations. It includes some the code that renders teh various types of declarations, and a writer that keep tabs on the cursor position.
  • parser.go: methods and objects used to parse the Go code from the cell, and again after goimports is run.

Documentation

Overview

Package goexec executes cells with Go code for the gonb kernel.

It defines a State object, that carries all the globals defined so far. It provides the ExecuteCell method, to run a new cell.

Index

Constants

View Source 
const (
	MainGo     = "main.go"
	MainTestGo = "main_test.go"
)
View Source 
const (
	// GoGetWorkspaceIssue is an err output by `go get` due to it not interpreting correctly `go.work`.
	GoGetWorkspaceIssue = "cannot find module providing package"

	// GoGetWorkspaceNote is the note that explains the issue with `go get` and `go work`.
	GoGetWorkspaceNote = `` /* 299-byte string literal not displayed */

)
View Source 
const (
	// GonbTempDirEnvName is the name of the environment variable that is set with
	// the temporary directory used to compile user's Go code.
	// It can be used by the executed Go code or by the bash scripts (started with `!`).
	GonbTempDirEnvName = "GONB_TMP_DIR"

	// InitFunctionPrefix -- functions named with this prefix will be rendered as
	// a separate `func init()`.
	InitFunctionPrefix = "init_"
)
View Source 
const (
	JupyterSessionNameEnv = "JPY_SESSION_NAME"
	JupyterPidEnv         = "JPY_PARENT_PID"

	JupyterFilesSubdir = "jupyter_files"
	CompiledWasmName   = "gonb_cell.wasm"
)
View Source 
const GonbCommentPrefix = "//gonb:"

GonbCommentPrefix allows one to enter the special commands (`%%`, `!`) prefixed as a Go comment, so it doesn't conflict with Go IDEs. Particularly useful if using Jupytext.

View Source 
const LinesForErrorContext = 3

LinesForErrorContext indicates how many lines to display in the error context, before and after the offending line. Hard-coded for now, but it could be made configurable.

View Source 
const NoCursorLine = int(-1)

Variables

View Source 
var (
	ParseError = fmt.Errorf("failed to parse cell contents")
	CursorLost = fmt.Errorf("cursor position not rendered in main.go")
)
View Source 
var NoCursor = Cursor{Line: NoCursorLine, Col: 0}

Functions

func CurrentWorkingDirectoryForPid 

func CurrentWorkingDirectoryForPid(pid int) (string, error)

CurrentWorkingDirectoryForPid returns the "cwd" or an error.

func DeclareStringConst 

func DeclareStringConst(decls *Declarations, name, value string)

DeclareStringConst creates a const definition in `decls` for a string value.

func DeclareVariable 

func DeclareVariable(decls *Declarations, name, value string)

DeclareVariable creates a variable definition in `decls`. `value` is copied verbatim, so any type of variable goes.

func GoRoot 

func GoRoot() (string, error)

func IsEmptyLines added in v0.10.0 

func IsEmptyLines(lines []string, skipLines Set[int]) bool

IsEmptyLines returns true is all lines are marked to skip, or if all lines not marked as skip are empty.

func JupyterErrorSplit 

func JupyterErrorSplit(err error) (string, string, []string)

JupyterErrorSplit takes an error and formats it into the components Jupyter protocol uses for it.

It special cases the GonbError, where it adds each sub-error in the "traceback" repeated field.

func JupyterRootDirectory 

func JupyterRootDirectory() (string, error)

JupyterRootDirectory returns Jupyter's root directory. This is needed to build the URL from where it serves static files.

Unfortunately, this isn't directly provided by Jupyter. It does provide its PID number, but get the "cwd" (current-working-directory) differs for different OSes.

See question here: https://stackoverflow.com/questions/46247964/way-to-get-jupyter-server-root-directory/58988310#58988310

func TrimGonbCommentPrefix added in v0.10.2 

func TrimGonbCommentPrefix(line string) string

TrimGonbCommentPrefix removes a prefixing "//gonb:" (GonbCommentPrefix) from line, if there is such a prefix. This is optionally used to escape special commands.

Types

type CellIdAndLine 

type CellIdAndLine struct {
	Id, Line int
}

CellIdAndLine points to a line within a cell. Id is the execution number of the cell, as given to ExecuteCell.

func MakeFileToCellIdAndLine 

func MakeFileToCellIdAndLine(cellId int, fileToCellLine []int) (fileToCellIdAndLine []CellIdAndLine)

MakeFileToCellIdAndLine converts a cellId and a slice of cell line numbers for a file to a slice of CellIdAndLine.

type CellLines 

type CellLines struct {
	// Id of the cell where the definition comes from. It is set to -1 if the declaration was automatically
	// created (for instance by goimports).
	Id int

	// Lines has one value per line used in the declaration. The point to the cell line where it was declared.
	// Some of these numbers may be NoCursorLine (-1) indicating that they are inserted automatically and don't.
	// have corresponding Lines in any cell.
	//
	// If Id is -1, Lines will be nil, which indicates the content didn't come from any cell.
	Lines []int
}

CellLines identifies a cell (by its execution id) and the Lines corresponding to a declaration.

func (CellLines) Append 

func (c CellLines) Append(fileToCellIdAndLine []CellIdAndLine) []CellIdAndLine

Append id and line numbers to fileToCellIdAndLine, a slice of `CellIdAndLine`. This is used when rendering a declaration to a file.

type Constant 

type Constant struct {
	Cursor
	CellLines

	Key                                      string
	TypeDefinition, ValueDefinition          string // Can be empty, if used as iota.
	CursorInKey, CursorInType, CursorInValue bool
	Next, Prev                               *Constant // Next and previous declaration in same Const block.
}

Constant represents the declaration of a constant. Because when appearing in block they inherit its definition form the previous line, we need to preserve the blocks. For this, we use Next/Prev links.

func (*Constant) Render 

func (c *Constant) Render(w *WriterWithCursor, cursor *Cursor, fileToCellIdAndLine []CellIdAndLine) []CellIdAndLine

Render Constant declaration (without the `const` keyword).

type Cursor 

type Cursor struct {
	Line, Col int
}

Cursor represents a cursor position in a cell or file. The Col is given as bytes in the line expected to be encoded as UTF-8.

func (*Cursor) ClearCursor 

func (c *Cursor) ClearCursor()

ClearCursor resets the cursor to an invalid state. This method is needed for the structs that embed Cursor.

func (Cursor) CursorFrom 

func (c Cursor) CursorFrom(line, col int) Cursor

CursorFrom returns a new Cursor adjusted

func (Cursor) HasCursor 

func (c Cursor) HasCursor() bool

func (Cursor) String 

func (c Cursor) String() string

String implements the fmt.Stringer interface.

type Declarations 

type Declarations struct {
	Functions map[string]*Function
	Variables map[string]*Variable
	Types     map[string]*TypeDecl
	Imports   map[string]*Import
	Constants map[string]*Constant
}

Declarations is a collection of declarations that we carry over from one cell to another.

func NewDeclarations 

func NewDeclarations() *Declarations

func (*Declarations) ClearCursor 

func (d *Declarations) ClearCursor()

ClearCursor wherever declaration it may be.

func (*Declarations) Copy 

func (d *Declarations) Copy() *Declarations

Copy returns a new deep copy of the declarations.

func (*Declarations) DropFuncInit 

func (d *Declarations) DropFuncInit()

DropFuncInit drops declarations of `func init()`: the parser generates this for the `func init_*`, and it shouldn't be considered new declarations if reading from generated code.

func (*Declarations) MergeFrom 

func (d *Declarations) MergeFrom(d2 *Declarations)

MergeFrom declarations in d2.

func (*Declarations) RenderConstants 

func (d *Declarations) RenderConstants(w *WriterWithCursor, fileToCellIdAndLine []CellIdAndLine) (Cursor, []CellIdAndLine)

RenderConstants without comments for all constants in Declarations.

Constants are trickier to render because when they are defined in a block, using `iota`, their ordering matters. So we re-render them in the same order and blocks as they were originally parsed.

The ordering is given by the sort order of the first element of each `const` block.

func (*Declarations) RenderFunctions 

func (d *Declarations) RenderFunctions(w *WriterWithCursor, fileToCellIdAndLine []CellIdAndLine) (Cursor, []CellIdAndLine)

RenderFunctions without comments, for all functions in Declarations.

func (*Declarations) RenderImports 

func (d *Declarations) RenderImports(w *WriterWithCursor, fileToCellIdAndLine []CellIdAndLine) (Cursor, []CellIdAndLine)

RenderImports writes out `import ( ... )` for all imports in Declarations.

func (*Declarations) RenderTypes 

func (d *Declarations) RenderTypes(w *WriterWithCursor, fileToCellIdAndLine []CellIdAndLine) (Cursor, []CellIdAndLine)

RenderTypes without comments.

func (*Declarations) RenderVariables 

func (d *Declarations) RenderVariables(w *WriterWithCursor, fileToCellIdAndLine []CellIdAndLine) (Cursor, []CellIdAndLine)

RenderVariables writes out `var ( ... )` for all variables in Declarations.

type ElementType 

type ElementType int
const (
	Invalid ElementType = iota
	FunctionType
	ImportType
	VarType
	ConstType
)

func (ElementType) String 

func (i ElementType) String() string

type Function 

type Function struct {
	Cursor
	CellLines

	Key            string
	Name, Receiver string
	Definition     string // Multi-line definition, includes comments preceding definition.

}

Function definition, parsed from a notebook cell.

type GonbError 

type GonbError struct {
	Lines []errorLine
	// contains filtered or unexported fields
}

GonbError is a special type of error that wraps a collection of errors returned by the Go compiler or `go get` or `go imports`.

Each error line (`GonbError.Lines`) holds some context stored in its own errorLine object. And it is also a wrapper for the failed execution error.

It can be rendered to HTML in the notebook with `GonbError.PublishWithHTML`.

func (*GonbError) Error 

func (nbErr *GonbError) Error() string

Error implements golang `error` interface. In Jupyter protocol, it corresponds to the "evalue" field (as in "error value").

func (*GonbError) Name 

func (nbErr *GonbError) Name() string

Name corresponds to field "ename" in Jupyter. Hardcoded in "ERROR" for now.

func (*GonbError) PublishWithHTML 

func (nbErr *GonbError) PublishWithHTML(msg kernel.Message)

PublishWithHTML reports the GonbError as an HTML report in Jupyter.

func (*GonbError) Traceback 

func (nbErr *GonbError) Traceback() []string

Traceback corresponds to field "traceback" in Jupyter.

func (*GonbError) Unwrap 

func (nbErr *GonbError) Unwrap() error

Unwrap returns the underlying error, so it can be used by `errors.Unwrap`.

type Import 

type Import struct {
	Cursor
	CellLines

	Key                         string
	Path, Alias                 string
	CursorInPath, CursorInAlias bool
}

Import represents an import to be included -- if not used it's automatically removed by `goimports`.

func NewImport 

func NewImport(importPath, alias string) *Import

NewImport from the importPath and it's alias. If alias is empty or "<nil>", it will default to the last name part of the importPath.

type State 

type State struct {
	// Kernel is set when actually connecting to JupyterServer.
	// In tests its left as nil.
	Kernel *kernel.Kernel

	// Temporary directory where Go program is build at each execution.
	UniqueID, Package, TempDir string

	// Building and executing go code configuration:
	Args         []string // Args to be passed to the program, after being executed.
	GoBuildFlags []string // Flags to be passed to `go build`, in State.Compile.
	AutoGet      bool     // Whether to do a "go get" before compiling, to fetch missing external modules.

	// Global elements defined mapped by their keys.
	Definitions *Declarations

	// CellIsTest indicates whether the current cell is to be compiled with `go test` (as opposed to `go build`).
	// This also triggers writing the code to `main_test.go` as opposed to `main.go`.
	// Usually this is set and reset after the execution -- the default being the normal build.
	CellIsTest        bool
	CellTests         []string // Tests defined in this cell. Only used if CellIsTest==true.
	CellHasBenchmarks bool

	// CellIsWasm indicates whether the current cell is to be compiled for WebAssembly (wasm).
	CellIsWasm                  bool
	WasmDir, WasmUrl, WasmDivId string

	// Comms represents the communication with the front-end.
	Comms *comms.State

	// CaptureFile is the file where to write any cell output. It is closed and set to nil at the end of the cell
	// executions.
	// If nil, no output is to be captured.
	CaptureFile io.WriteCloser
	// contains filtered or unexported fields
}

State holds information about Go code execution for this kernel. It's a singleton (for now). It hols the directory, ids, configuration, command line arguments to use and currently defined Go code.

That is, if the user runs a cell that defines, let's say `func f(x int) int { return x+1 }`, the definition of `f` will be stored in Definitions field.

func New 

func New(k *kernel.Kernel, uniqueID string, preserveTempDir, rawError bool) (*State, error)

New returns an empty State object, that can be used to execute Cells.

If preserveTempDir is set to true, the temporary directory is logged, and it's preserved when the kernel exits -- helpful for debugging.

If rawError is true, the parsing of compiler errors doesn't generate HTML, instead it uses only text.

The kernel object passed in `k` can be nil for testing, but this may lead to some leaking goroutines, that stop when the kernel stops.

func (*State) AlternativeDefinitionsPath 

func (s *State) AlternativeDefinitionsPath() string

AlternativeDefinitionsPath is the path to a temporary file that holds the memorize definitions, when we are not able to include them in the `main.go`, because the current cell is not parseable.

func (*State) AutoCompleteOptionsInCell 

func (s *State) AutoCompleteOptionsInCell(cellLines []string, skipLines map[int]struct{},
	cursorLine, cursorCol int, reply *kernel.CompleteReply) (err error)

AutoCompleteOptionsInCell implements a `complete_request` from Jupyter, using `gopls`. It updates `main.go` with the cell contents (given as Lines)

func (*State) AutoTrack 

func (s *State) AutoTrack() (err error)

AutoTrack adds automatic tracked directories. It looks at go.mod and go.work for redirects to the local filesystem.

func (*State) BinaryPath 

func (s *State) BinaryPath() string

BinaryPath is the path to the generated binary file.

func (*State) CodePath 

func (s *State) CodePath() string

CodePath is the path to where the code is going to be saved. Either `main.go` or `main_test.go` file.

func (*State) Compile 

func (s *State) Compile(msg kernel.Message, fileToCellIdAndLines []CellIdAndLine) error

Compile compiles the currently generate go files in State.TempDir to a binary named State.Package.

If errors in compilation happen, linesPos is used to adjust line numbers to their content in the current cell.

func (*State) DefaultCellTestArgs 

func (s *State) DefaultCellTestArgs() (args []string)

DefaultCellTestArgs generate the default `go test` arguments, if none is given. It includes `-test.v` and `-test.run` matching the tests defined in the current cell.

func (*State) DisplayErrorWithContext 

func (s *State) DisplayErrorWithContext(msg kernel.Message, fileToCellIdAndLine []CellIdAndLine, errorMsg string, err error) error

DisplayErrorWithContext in an HTML div, with a mouse-over pop-up window listing the Lines with the error, and highlighting the exact position.

Except if `rawError` is set to true (see `New() *State`): in which case the enriched GonbError is returned instead, for a textual report back.

Any errors within here are logged and simply ignored, since this is already used to report errors.

func (*State) EnumerateUpdatedFiles 

func (s *State) EnumerateUpdatedFiles(fn func(filePath string) error) (err error)

EnumerateUpdatedFiles calls fn for each file that has been updated since the last call. If `fn` returns an err, then the enumerations is interrupted and the err is returned.

func (*State) Execute 

func (s *State) Execute(msg kernel.Message, fileToCellIdAndLine []CellIdAndLine) error

Execute cell code already prepared to `${GONB_TMP_DIR}/main.go`.

If errors in execution happen, fileToCellIdAndLine helps to map the `main.go` line numbers to cell id and line, so errors can be annotated.

If s.CellIsWasm is true, it passes through State.ExecuteWasm.

func (*State) ExecuteCell 

func (s *State) ExecuteCell(msg kernel.Message, cellId int, lines []string, skipLines Set[int]) error

ExecuteCell takes the contents of a cell, parses it, merges new declarations with the ones from previous definitions, render a final main.go code with the whole content, compiles and runs it.

skipLines are Lines that should not be considered as Go code. Typically, these are the special commands (like `%%`, `%args`, `%reset`, or bash Lines starting with `!`).

func (*State) ExecuteWasm 

func (s *State) ExecuteWasm(msg kernel.Message) error

ExecuteWasm expects `wasm_exec.js` and CompiledWasmName to be in the directory pointed to `s.WasmDir` already.

func (*State) ExportWasmConstants 

func (s *State) ExportWasmConstants(decls *Declarations)

func (*State) GoImports 

func (s *State) GoImports(msg kernel.Message, decls *Declarations, mainDecl *Function, fileToCellIdAndLine []CellIdAndLine) (cursorInFile Cursor, updatedFileToCellIdAndLine []CellIdAndLine, err error)

GoImports execute `goimports` which adds imports to non-declared imports automatically. It also runs "go get" to download any missing dependencies.

It returns the updated cursorInFile and fileToCellIdAndLines that reflect any changes in `main.go`.

func (*State) GoModInit 

func (s *State) GoModInit() error

GoModInit removes current `go.mod` if it already exists, and recreate it with `go mod init`.

func (*State) GoWorkFix 

func (s *State) GoWorkFix(msg kernel.Message) (err error)

GoWorkFix takes all modules in `go.work` "use" clauses, and add them as "replace" clauses in `go.mod`. This is needed for `go get` to work.

func (*State) InspectIdentifierInCell 

func (s *State) InspectIdentifierInCell(lines []string, skipLines map[int]struct{}, cursorLine, cursorCol int) (mimeMap kernel.MIMEMap, err error)

InspectIdentifierInCell implements an `inspect_request` from Jupyter, using `gopls`. It updates `main.go` with the cell contents (given as Lines)

func (*State) ListTracked 

func (s *State) ListTracked() []string

func (*State) MakeWasmSubdir 

func (s *State) MakeWasmSubdir() (err error)

MakeWasmSubdir creates a subdirectory named `.wasm/<notebook name>/` in the same directory as the notebook, if it is not yet created.

It also copies current Go compiler `wasm_exec.js` file to this directory, if it's not there already.

Path and URL to access it are stored in s.WasmDir and s.WasmUrl.

func (*State) PostExecuteCell 

func (s *State) PostExecuteCell()

PostExecuteCell reset state that is valid only for the duration of a cell. This includes s.CellIsTest and s.Args.

func (*State) RemoveCode 

func (s *State) RemoveCode() error

RemoveCode removes the code files (`main.go` or `main_test.go`). Usually used just before creating creating a new version.

func (*State) RemoveWasmConstants 

func (s *State) RemoveWasmConstants(decls *Declarations)

func (*State) Reset 

func (s *State) Reset()

Reset clears all the memorized Go declarations. It becomes as if no cells had been executed so far -- except for configurations and arguments that remain unchanged.

It is connected to the special command `%reset`.

func (*State) SetCellTests 

func (s *State) SetCellTests(decls *Declarations)

SetCellTests sets the test functions (Test...) defined in this cell. The default for `%test` is to run only the current tests, this is the function that given the new declarations created in this cells, figures out which are those tests.

func (*State) Stop 

func (s *State) Stop() error

Stop stops gopls and removes temporary files and directories.

func (*State) Track 

func (s *State) Track(fileOrDirPath string) (err error)

Track adds the fileOrDirPath to the list of tracked files and directories. If fileOrDirPath is already tracked, it's a no-op.

If the fileOrDirPath pointed is a symbolic link, follow instead the linked file/directory.

func (*State) Untrack 

func (s *State) Untrack(fileOrDirPath string) (err error)

Untrack removes file or dir from path of tracked files. If it ends with "...", it un-tracks anything that has fileOrDirPath as prefix. If you set `fileOrDirPath == "..."`, it will un-tracks everything.

type TypeDecl 

type TypeDecl struct {
	Cursor
	CellLines

	Key            string // Same as the name here.
	TypeDefinition string // Type definition which includes the name.
	CursorInType   bool
}

TypeDecl definition, parsed from a notebook cell.

type Variable 

type Variable struct {
	Cursor
	CellLines

	CursorInName, CursorInType, CursorInValue bool
	Key, Name                                 string
	TypeDefinition, ValueDefinition           string // Type definition may be empty.

	// TupleDefinitions are present when multiple variables are tied to the same definition as in `var a, b, c = someFunc()`.
	TupleDefinitions []*Variable
}

Variable definition, parsed from a notebook cell.

There is one special case, where one variable entry will define multiple variables, when we have line like: 

var a, b, c = someFuncThatReturnsATuple()

For such cases, one set TupleDefinitions in order ("a", "b", "c"). And we define the following behavior:

  • Only the first element of TupleDefinitions is rendered, but it renders the tuple definition.
  • If any of the elements of the tuple is redefined or removed, all elements are removed.

This means that if "a" is redefined, "b" and "c" disappear. And that if "b" or "c" are redefined, it will yield and error, that is subtle to track.

type WriterWithCursor 

type WriterWithCursor struct {
	Line, Col int
	// contains filtered or unexported fields
}

WriterWithCursor keep tabs of the current line/col of the file (presumably) being written.

func NewWriterWithCursor 

func NewWriterWithCursor(w io.Writer) *WriterWithCursor

NewWriterWithCursor that keeps tabs of current line/col of the file (presumably) being written.

func (*WriterWithCursor) Cursor 

func (w *WriterWithCursor) Cursor() Cursor

Cursor returns the current position in the file, at the end of what has been written so far.

func (*WriterWithCursor) CursorPlusDelta 

func (w *WriterWithCursor) CursorPlusDelta(delta Cursor) (fileCursor Cursor)

CursorPlusDelta returns the expected cursor position in the current file, assuming the original cursor is cursorDelta away from the current position in the file (stored in w).

Semantically it's equivalent to `w.Cursor() + cursorDelta`.

func (*WriterWithCursor) Error 

func (w *WriterWithCursor) Error() error

Error returns first err that happened during writing.

func (*WriterWithCursor) FillLinesGap 

func (w *WriterWithCursor) FillLinesGap(fileToCellIdAndLine []CellIdAndLine) []CellIdAndLine

FillLinesGap append NoCursorLine (-1) line indices to fileToCellIdAndLine slice, up to the current line.

func (*WriterWithCursor) Write 

func (w *WriterWithCursor) Write(content string)

Write writes the given content and keeps track of cursor. Errors can be retrieved with Error.

func (*WriterWithCursor) Writef 

func (w *WriterWithCursor) Writef(format string, args ...any)

Writef write with formatted text. Errors can be retrieved with Error.

Source Files

View all Source files

Directories

Path Synopsis
Package goplsclient runs `gopls` (1) in the background uses it to retrieve definitions of symbols and auto-complete.
 Package goplsclient runs `gopls` (1) in the background uses it to retrieve definitions of symbols and auto-complete.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.