utgen

README

⚡️ Generate unit tests with LLMs, that actually works ⚡️

🌟 The must-have tool for developers in the AI-Gen era 🌟

---

---

Keploy-gen uses LLMs to understand code semantics and generates meaningful unit tests. It's inspired by the Automated Unit Test Improvement using LLM at Meta.

Objectives

• Automate unit test generation (UTG): Quickly generate comprehensive unit tests and reduce the redundant manual effort.

• Improve existing tests: Extend and improve the scope of existing tests to cover more complex scenarios that are often missed manually.

• Boost test coverage: As codebase grows, ensuring exhaustive coverage should become feasible.

Core Components

Phase	Activities	Tools/Technologies
Code Analysis	Analyze the code structure and dependencies.	Static analysis tools, LLMs
Prompt Engineering	Generation of targeted prompts to guide the LLM in producing relevant tests.	LLMs, Custom scripts
Iterative Test Refinement	Cyclic process of refining tests by running them, analyzing coverage, and incorporating feedback.	Testing frameworks (e.g., JUnit, pytest)

Process Overview

Referred from Meta's research, TestGen-LLM top level architecture.

Prerequisites

AI model Setup - Set the environment variable export API_KEY=xxxx to use:

• OpenAI's GPT-4o directly [preferred].

• Alternative LLMs via litellm.

Installation

Install Keploy-gen locally by running the following command:

curl -O https://raw.githubusercontent.com/keploy/keploy/main/keploy.sh && source keploy.sh

→ Running with Node.js/TypeScript applications

• Set the API key, as mentioned in pre-requisites above:

  export API_KEY=xxxx
  

• Ensure Cobertura formatted coverage reports, edit package.json:

  "jest": {
        "collectCoverage": true,
        "coverageReporters": ["text", "cobertura"],
        "coverageDirectory": "./coverage"
      } 
  

  Usage

• Generate tests using Keploy:

  • For Entire Application use the following command to generate tests across:

    ⚠️ Warning: Executing this command will generate unit tests for all files in the application. Depending on the size of the codebase, this process may take between 20 minutes to an hour and will incur costs related to LLM usage.

    keploy gen \
          --testCommand="npm test" \
          --testDir="test" \
          --coverageReportPath="<path to coverage.xml>"
    

    
    

  • For Smaller Scope Testing: If you prefer to test a smaller section of your application or to control costs, consider generating tests for a single source and its corresponding test file:

     keploy gen \                                                                                                                                     
               --sourceFilePath="<path to source file>" \
               --testFilePath="<path to test file for above source file>" \
               --testCommand="npm test" \
               --coverageReportPath="<path to coverage.xml>"
    

  🎉 You should see improved test cases and code-coverage. ✅ Enjoy coding with enhanced unit test coverage! 🫰

→ Running with Golang applications

• Set the API key, as mentioned in pre-requisites above:

  export API_KEY=xxxx
  

• Ensure Cobertura formatted coverage reports.

   go install github.com/axw/gocov/gocov@v1.1.0
   go install github.com/AlekSi/gocov-xml@v1.1.0 
  

  Usage

• Generate tests using Keploy:

  • For Entire Application use the following command to generate tests across:

    ⚠️ Warning: Executing this command will generate unit tests for all files in the application. Depending on the size of the codebase, this process may take between 20 minutes to an hour and will incur costs related to LLM usage.

    keploy gen \
              --testDir="." \
              --testCommand="go test -v ./... -coverprofile=coverage.out && gocov convert coverage.out | gocov-xml > coverage.xml" \
              --coverageReportPath="<path to coverage.xml>"
    

  • For Smaller Scope Testing: If you prefer to test a smaller section of your application or to control costs, consider generating tests for a single source and its corresponding test file:
    

    keploy gen \                                                                                                                                     
               --sourceFilePath="<path to source file>" \
               --testFilePath="<path to test file for above source file>" \
               --testCommand="go test -v ./... -coverprofile=coverage.out && gocov convert coverage.out | gocov-xml > coverage.xml" \
               --coverageReportPath="<path to coverage.xml>"
    

  🎉 You should see improved test cases and code-coverage. ✅ Enjoy coding with enhanced unit test coverage! 🫰

→ Setup for Other Languages

• Set the API key, as mentioned in pre-requisites above:

  export API_KEY=xxxx
  

• Ensure that your unit test report format is Cobertura(it's very common).

• Generate tests using Keploy:

  keploy gen \
        --sourceFilePath="<path to source code file>" \
        --testFilePath="<path to existing unit test file>"  \
        --testCommand="<cmd to execute unit tests>" \
        --coverageReportPath="<path to cobertura-coverage.xml>"
  

Configuration

Configure Keploy using command-line flags:

keploy gen \
           --sourceFilePath "" \
           --testFilePath "" \
           --coverageReportPath "coverage.xml" \
           --testCommand "" \
           --coverageFormat "cobertura" \
           --expectedCoverage 100 \
           --maxIterations 5 \
           --testDir "" \
           --litellmUrl "" \
           --model "gpt-4o"

• sourceFilePath: Path to the source file for which tests are to be generated.
• testFilePath: Path where the generated tests will be saved.
• coverageReportPath: Path to generate the coverage report.
• testCommand (required): Command to execute tests and generate the coverage report.
• coverageFormat: Type of the coverage report (default "cobertura").
• expectedCoverage: Desired coverage percentage (default 100%).
• maxIterations: Maximum number of iterations for refining tests (default 5).
• testDir: Directory where tests will be written.
• litellmUrl: Set to true if using litellm for model integration.
• apiBaseUrl: Base URL for the litellm proxy.
• model: Specifies the AI model to use (default "gpt-4o").

🙋🏻‍♀️ Questions? 🙋🏻‍♂️

Reach out to us. We're here to answer!

🌐 Language Support

Other language may be supported, we've not tested them yet. If your coverage reports are of Cobertura format then you should be able to use keploy-gen in any language.

Dev Support

Keploy-gen is not just a project but an attempt to make developers life easier testing applications. It aims to simplify the creation and maintenance of tests, ensuring high coverage, and adapts to the complexity of modern software development.

Prompt Generation

Referred from Meta's research, the four primary prompts used in the deployment for the December 2023 Instagram and Facebook app test-a-thons

Prompt Name	Template
extend_test	Here is a Kotlin unit test class: {existing_test_class}. Write an extended version of the test class that includes additional tests to cover some extra corner cases.
extend_coverage	Here is a Kotlin unit test class and the class that it tests: {existing_test_class} {class_under_test}. Write an extended version of the test class that includes additional unit tests that will increase the test coverage of the class under test.
corner_cases	Here is a Kotlin unit test class and the class that it tests: {existing_test_class} {class_under_test}. Write an extended version of the test class that includes additional unit tests that will cover corner cases missed by the original and will increase the test coverage of the class under test.
statement_to_complete	Here is a Kotlin class under test {class_under_test} This class under test can be tested with this Kotlin unit test class {existing_test_class}. Here is an extended version of the unit test class that includes additional unit test cases that will cover methods, edge cases, corner cases, and other features of the class under test that were missed by the original unit test class:

Limitation: This project currently doesn't generate quality fresh tests if there are no existing tests to learn from.

Enjoy coding with enhanced unit test coverage! 🫰

Documentation

Overview

Package utgen is a service that generates unit tests for a given source code file.

Index

• Constants
• func GetCodeLanguage(sourceFilePath string) string
• func RunCommand(command string, cwd string) (stdout string, stderr string, exitCode int, commandStartTime int64, err error)
• type AIClient
  • func NewAIClient(model, apiBase string) *AIClient
  • func (ai *AIClient) Call(ctx context.Context, prompt *Prompt, maxTokens int) (string, int, int, error)
• type Choice
• type CompletionParams
• type Coverage
• type CoverageProcessor
  • func NewCoverageProcessor(reportPath, srcpath, format string) *CoverageProcessor
  • func (cp *CoverageProcessor) ParseCoverageReport() (*models.CoverageResult, error)
  • func (cp *CoverageProcessor) ParseCoverageReportCobertura() (*models.CoverageResult, error)
  • func (cp *CoverageProcessor) ProcessCoverageReport(latestTime int64) (*models.CoverageResult, error)
  • func (cp *CoverageProcessor) VerifyReportUpdate(latestTime int64) error
• type Cursor
• type Delta
• type Message
• type ModelResponse
• type Prompt
• type PromptBuilder
  • func NewPromptBuilder(...) (*PromptBuilder, error)
  • func (pb *PromptBuilder) BuildPrompt(file, failedTestRuns string) (*Prompt, error)
• type ResponseChunk
• type Service
• type Source
• type Telemetry
• type Test
• type UnitTestGenerator
  • func NewUnitTestGenerator(srcPath, testPath, reportPath, cmd, dir, coverageFormat string, ...) (*UnitTestGenerator, error)
  • func (g *UnitTestGenerator) GenerateTests(ctx context.Context) (*models.UTDetails, error)
  • func (g *UnitTestGenerator) Start(ctx context.Context) error
  • func (g *UnitTestGenerator) ValidateTest(generatedTest models.UT) error
• type Usage

Constants

const ADDITIONAL_INCLUDES_TEXT = `` /* 217-byte string literal not displayed */

const ADDITIONAL_INSTRUCTIONS_TEXT = `
## Additional Instructions
======
{{.AdditionalInstructions}}
======
`

const FAILED_TESTS_TEXT = `` /* 250-byte string literal not displayed */

const MAX_TESTS_PER_RUN = 4

Functions

func GetCodeLanguage

func GetCodeLanguage(sourceFilePath string) string

func RunCommand

func RunCommand(command string, cwd string) (stdout string, stderr string, exitCode int, commandStartTime int64, err error)

Types

type AIClient

type AIClient struct {
	Model   string
	APIBase string
}

func NewAIClient

func NewAIClient(model, apiBase string) *AIClient

func (*AIClient) Call

func (ai *AIClient) Call(ctx context.Context, prompt *Prompt, maxTokens int) (string, int, int, error)

type Choice

type Choice struct {
	Delta Delta `json:"delta"`
}

type CompletionParams

type CompletionParams struct {
	Model       string    `json:"model"`
	Messages    []Message `json:"messages"`
	MaxTokens   int       `json:"max_tokens"`
	Stream      bool      `json:"stream"`
	Temperature float32   `json:"temperature"`
	APIBase     string    `json:"api_base,omitempty"`
}

type Coverage

type Coverage struct {
	Path    string
	Format  string
	Desired float64
	Current float64
	Content string
}

type CoverageProcessor

type CoverageProcessor struct {
	ReportPath string
	SrcPath    string
	Format     string
}

CoverageProcessor handles the processing of coverage reports

func NewCoverageProcessor

func NewCoverageProcessor(reportPath, srcpath, format string) *CoverageProcessor

NewCoverageProcessor initializes a CoverageProcessor object

func (*CoverageProcessor) ParseCoverageReport

func (cp *CoverageProcessor) ParseCoverageReport() (*models.CoverageResult, error)

ParseCoverageReport parses the coverage report based on its type

func (*CoverageProcessor) ParseCoverageReportCobertura

func (cp *CoverageProcessor) ParseCoverageReportCobertura() (*models.CoverageResult, error)

func (*CoverageProcessor) ProcessCoverageReport

func (cp *CoverageProcessor) ProcessCoverageReport(latestTime int64) (*models.CoverageResult, error)

ProcessCoverageReport verifies the report and parses it based on its type

func (*CoverageProcessor) VerifyReportUpdate

func (cp *CoverageProcessor) VerifyReportUpdate(latestTime int64) error

VerifyReportUpdate verifies the coverage report's existence and update time

type Cursor

type Cursor struct {
	Line        int
	Indentation int
}

type Delta

type Delta struct {
	Content string `json:"content"`
}

type Message

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

type ModelResponse

type ModelResponse struct {
	ID                string   `json:"id"`
	Choices           []Choice `json:"choices"`
	Created           int      `json:"created"`
	Model             string   `json:"model,omitempty"`
	Object            string   `json:"object"`
	SystemFingerprint string   `json:"system_fingerprint,omitempty"`
	Usage             *Usage   `json:"usage,omitempty"`
}

type Prompt

type Prompt struct {
	System string `json:"system"`
	User   string `json:"user"`
}

type PromptBuilder

type PromptBuilder struct {
	Src                    *Source
	Test                   *Test
	CovReportContent       string
	IncludedFiles          string
	AdditionalInstructions string
	Language               string
	Logger                 *zap.Logger
}

func NewPromptBuilder

func NewPromptBuilder(srcPath, testPath, covReportContent, includedFiles, additionalInstructions, language string, logger *zap.Logger) (*PromptBuilder, error)

func (*PromptBuilder) BuildPrompt

func (pb *PromptBuilder) BuildPrompt(file, failedTestRuns string) (*Prompt, error)

type ResponseChunk

type ResponseChunk struct {
	Choices []Choice `json:"choices"`
}

type Service

type Service interface {
	Start(ctx context.Context) error
}

type Source

type Source struct {
	Name         string
	Code         string
	CodeNumbered string
}

type Telemetry

type Telemetry interface {
	GenerateUT()
}

type Test

type Test struct {
	Name         string
	Code         string
	CodeNumbered string
}

type UnitTestGenerator

type UnitTestGenerator struct {
	Files []string
	// contains filtered or unexported fields
}

func NewUnitTestGenerator

func NewUnitTestGenerator(srcPath, testPath, reportPath, cmd, dir, coverageFormat string, desiredCoverage float64, maxIterations int, model string, apiBaseURL string, _ *config.Config, tel Telemetry, logger *zap.Logger) (*UnitTestGenerator, error)

func (*UnitTestGenerator) GenerateTests

func (g *UnitTestGenerator) GenerateTests(ctx context.Context) (*models.UTDetails, error)

func (*UnitTestGenerator) Start

func (g *UnitTestGenerator) Start(ctx context.Context) error

func (*UnitTestGenerator) ValidateTest

func (g *UnitTestGenerator) ValidateTest(generatedTest models.UT) error

type Usage

type Usage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
	TotalTokens      int `json:"total_tokens"`
}