lioss

README

lioss

License Identification tool for OSS project.

🗣 Overview

Generally, OSS projects have licenses. The licenses grant permissions to users for using, modifying, and sharing the software. The users of the software must follow the terms shown in the licenses.

On the other hand, today's software generally has some dependencies. Additionally, dependant software has some dependencies, too. Therefore, the dependant graph of the OSS becomes complex.

In such a situation, it is a quite tough task for checking the conflicts among licenses. The first problem is to detect a conflict between two given licenses. The second problem is to identify the license of a project. lioss tries to solve the above second problem by identifying the license of the given project.

SPDX is trying to automatically identify licenses, however, it is hard to say that it became common sense. This project detects the OSS licenses from the LICENSE files of the given projects. Then, we aim to detect conflicts by identifying OSS licenses from the license files of dependent libraries.

Usage

lioss

Identifies license name from file and/or project directories.

lioss version 1.0.0
lioss [OPTIONS] <PROJECTS...>
OPTIONS
        --dbpath <DBPATH>          specifying database path.
    -a, --algorithm <ALGORITHM>    specifies algorithm. Default is 5gram.
                                   Available values are: kgram, wordfreq, and tfidf.
    -t, --threshold <THRESHOLD>    specifies threshold of the similarities of license files.
                                   Each algorithm has default value. Default value is 0.75.
    -h, --help                     print this message.
PROJECTS
    project directories, archive files (jar, and zip) contains LICENSE file, and/or LICENSE file.

mkliossdb

Creates the database of lioss from License documents.

mkliossdb [OPTIONS] <LICENSE...>
OPTIONS
    -d, --dest <DEST>        specifies the destination file path. Default is 'liossdb.json'
    -h, --help               print this message.
LICENSE
    specifies license files.

Install

Go-lang

$ go get github.com/tamada/lioss

🍺 Homebrew

$ brew tap tamada/brew
$ brew install lioss

💪 Compiling yourself

$ git clone github.com/tamada/lioss
$ cd lioss
$ make

References

• dmgerman/ninka
  • Daniel M. German, Yuki Manabe and Katsuro Inoue. A sentence-matching method for automatic license identification of source code files. In 25nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2010).
  • This product identifies the license of each source file. However, it does not work on my environment.
• pivotal/LicenseFinder
  • This product finds dependencies from build file, and find license.
• SPDX (Software Package Data Exchange). *

Documentation

Index

• Constants
• Variables
• type Algorithm
  • func NewAlgorithm(name string) (Algorithm, error)
• type Database
  • func LoadDatabase(databaseTypes DatabaseType) (*Database, error)
  • func NewDatabase() *Database
  • func Read(reader io.Reader, name string) (*Database, error)
  • func ReadDatabase(path string) (*Database, error)
  • func (db *Database) AlgorithmCount() int
  • func (db *Database) Contains(algorithmName, licenseName string) bool
  • func (db *Database) Entries(algorithmName string) []*License
  • func (db *Database) Entry(algoirthmName, licenseName string) *License
  • func (db *Database) LicenseCount() int
  • func (db *Database) Merge(other *Database) *Database
  • func (db *Database) Put(algorithmName string, license *License)
  • func (db *Database) Write(writer io.Writer) error
  • func (db *Database) WriteTo(destFile string) error
• type DatabaseType
  • func (dt DatabaseType) IsType(dbType DatabaseType) bool
  • func (dt DatabaseType) String() string
• type Identifier
  • func NewIdentifier(algorithmName string, threshold float64, db *Database) (*Identifier, error)
  • func (identifier *Identifier) Identify(project Project) (map[LicenseFile][]*Result, error)
• type License
• type LicenseFile
• type Project
  • func NewProject(path string) (Project, error)
• type Result
  • func (result *Result) String() string
• type Time
  • func Now() *Time
  • func (t *Time) MarshalJSON() ([]byte, error)
  • func (t *Time) UnmarshalJSON(data []byte) error

Constants

const DatabasePathEnvName = "LIOSS_DBPATH"

Variables

var AvailableAlgorithms = []string{"1gram", "2gram", "3gram", "4gram", "5gram", "6gram", "7gram", "8gram", "9gram", "wordfreq", "tfidf"}

AvailableAlgorithms contains the names of available algorithm for comparing licenses.

Types

type Algorithm

type Algorithm interface {
	Prepare(db *Database) error
	Parse(reader io.Reader, licenseName string) (*License, error)
	Compare(license1, license2 *License) float64
	String() string
}

Algorithm shows an algorithm for identifying the license.

func NewAlgorithm

func NewAlgorithm(name string) (Algorithm, error)

NewAlgorithm create an instance of Algorithm. Available values are [1-9]gram, and tfidf.

type Database

type Database struct {
	Timestamp *Time                 `json:"create-at"`
	Data      map[string][]*License `json:"algorithms"`
}

Database represents the database for the lioss.

func LoadDatabase

func LoadDatabase(databaseTypes DatabaseType) (*Database, error)

LoadDatabase loads the lioss database from system path. This function search the following directories.

* ENV['LIOSS_DBPATH'] * /usr/local/opt/lioss/data * /opt/lioss/data * ./data

If the directory found, this function loads the Base.liossgz, OSIApproved.liossgz, and Deprecated.liossgz as needed.

func NewDatabase

func NewDatabase() *Database

NewDatabase create an instance of database for lioss.

func Read

func Read(reader io.Reader, name string) (*Database, error)

Read reads database from given reader.

func ReadDatabase

func ReadDatabase(path string) (*Database, error)

ReadDatabase reads database from given path.

func (*Database) AlgorithmCount

func (db *Database) AlgorithmCount() int

func (*Database) Contains

func (db *Database) Contains(algorithmName, licenseName string) bool

Contains checks existance with algorithm and license name.

func (*Database) Entries

func (db *Database) Entries(algorithmName string) []*License

Entries returns a slice of licenses built by given algorithm.

func (*Database) Entry

func (db *Database) Entry(algoirthmName, licenseName string) *License

Entry return an instance of license built by given algorithm with given license name.

func (*Database) LicenseCount

func (db *Database) LicenseCount() int

func (*Database) Merge

func (db *Database) Merge(other *Database) *Database

func (*Database) Put

func (db *Database) Put(algorithmName string, license *License)

Put registers the given license to the database.

func (*Database) Write

func (db *Database) Write(writer io.Writer) error

Write writes database to given writer.

func (*Database) WriteTo

func (db *Database) WriteTo(destFile string) error

WriteTo writes data in the the receiver database into the given file.

type DatabaseType

type DatabaseType int

const (
	OSI_APPROVED_DATABASE      DatabaseType = 1
	DEPRECATED_DATABASE        DatabaseType = 2
	OSI_DEPRECATED_DATABASE    DatabaseType = 4
	NONE_OSI_APPROVED_DATABASE DatabaseType = 8
	WHOLE_DATABASE             DatabaseType = OSI_APPROVED_DATABASE | DEPRECATED_DATABASE | OSI_DEPRECATED_DATABASE | NONE_OSI_APPROVED_DATABASE
)

func (DatabaseType) IsType

func (dt DatabaseType) IsType(dbType DatabaseType) bool

func (DatabaseType) String

func (dt DatabaseType) String() string

type Identifier

type Identifier struct {
	Threshold  float64
	Comparator Algorithm
	Database   *Database
}

Identifier is a type for identifying the license.

func NewIdentifier

func NewIdentifier(algorithmName string, threshold float64, db *Database) (*Identifier, error)

NewIdentifier creates an instance of Identifier with the given arguments. The range of threshold must be from 0.0 to 1.0.

func (*Identifier) Identify

func (identifier *Identifier) Identify(project Project) (map[LicenseFile][]*Result, error)

Identify identifies the license of the given project.

type License

type License struct {
	Name        string         `json:"license-name"`
	Frequencies map[string]int `json:"frequencies"`
}

License shows the license data for identifying.

type LicenseFile

type LicenseFile interface {
	ID() string
	Read(p []byte) (int, error)
	Close() error
	String() string
}

LicenseFile shows the path of license in the project.

type Project

type Project interface {
	/* BasePath returns the project path. */
	BasePath() string
	/* Close closes the project, and after call this method, this project is not available. */
	Close() error
	/* LicenseIDs returns the ids for licenses in the project. */
	LicenseIDs() []string
	/* LicenseFile returns the path of the License files in the project. */
	LicenseFile(licenseID string) (LicenseFile, error)
}

Project shows project containing some licenses.

func NewProject

func NewProject(path string) (Project, error)

NewProject creates an instance of Project. Acceptable file formats of this function is zip/jar/war file, and directory.

type Result

type Result struct {
	/*Name shows the license name.*/
	Name string
	/*Probability represents the probability of the license, by range of 0.0 to 1.0.*/
	Probability float64
}

Result shows identified license and its probability.

func (*Result) String

func (result *Result) String() string

type Time

type Time struct {
	// contains filtered or unexported fields
}

Time represents the time for marshaling a specific format.

func Now

func Now() *Time

Now returns now time.

func (*Time) MarshalJSON

func (t *Time) MarshalJSON() ([]byte, error)

MarshalJSON is called on marshaling JSON.

func (*Time) UnmarshalJSON

func (t *Time) UnmarshalJSON(data []byte) error

UnmarshalJSON is called on unmarshaling JSON.