githubPatches

README

github-patches

does what is says on the tin

this tool helps you scrape commit metadata pushed to github (and grab their patch files) in a given time range using the github api. It returns the patch URL, repo type and more!

Usage

cd github-patches-cli/

go run main.go -h

Usage of ./github-patches:
  -from string
        Starting timestamp in '2006-01-02-15' format
  -outputDir string
        the directory to save files to. 'githubCommits/' will be made locally if not specified (default ".githubCommits/")
  -to string
        Ending timestamp in '2006-01-02-15' format

Examples

Note: use the --concurrent flag to download files concurrently (at the same time!)

Patches from the last hour

go run main.go

cat .githubCommits/2024-10-14-15.json

{"id":"42807516937","type":"IssuesEvent","actor":{"id":41898282,"login":"github-actions[bot]","display_login":"github-actions","gravatar_id":"","url":"https://api.github.com/users/github-actions[bot]","avatar_url":"https://avatars.githubusercontent.com/u/41898282?"},"repo":{"id":859486274,"name":"leyu-sports/leyuio","url":"https://api.github.com/repos/leyu-sports/leyuio"},"payload":{"action":"opened","issue":{"url":"https://api.github...
{"id":"42807516940","type":"IssuesEvent","actor":{"id":41898282,"login":"github-actions[bot]","display_login":"github-actions","gravatar_id":"","url":"https://api.github.com/users/github-actions[bot]","avatar_url":"https://avatars.githubusercontent.com/u/41898282?"},"repo":{"id":869028658,"name":"long8guoji/long8ty","url":"https://api.github.com/repos/long8guoji/long8ty"},"payload":{"action":"opened","issue":{"url":"https://api.github.com/repos/long8guoji/long8ty/issues/34119","repository_url":"https://api.github.com/repos/long8guoji/long8ty",...
...

Patches from October 14, 2024 at 3 PM UTC to October 15, 2024 at 2 AM

go run main.go --from=2024-10-14-15 --to=2024-10-15-2

cat .githubCommits/2024-10-14-15.json

{"id":"42807516937","type":"IssuesEvent","actor":{"id":41898282,"login":"github-actions[bot]",...
...

Patches for a single timestamp

go run main.go --from=2024-10-14-15 --to=2024-10-14-15

cat .githubCommits/2024-10-14-15.json

{"id":"42807516937","type":"IssuesEvent","actor":{"id":41898282,"login":"github-actions[bot]",...
...

Patches from October 15, 2024 at 2 AM to October 14, 2024 at 3 PM UTC (in reverse)

go run main.go --to=2024-10-14-15 --from=2024-10-15-2

cat .githubCommits/2024-10-15-2.json

{"id":"42807516937","type":"IssuesEvent","actor":{"id":41898282,"login":"github-actions[bot]",...
...

Documentation

Index

• Variables
• func GetCommitsInRange(outputDirectory, from, to string, concurrent bool)
• func GetISO8601Timestamps(from, to string) []string
• func GetLast100Gists() string
• func MakePatchURL(apiURL string) string
• func PrintGharchiveChunkUrls(from, to string) []string
• type Actor
• type Author
• type Commit
• type CommitInfo
• type Event
  • func ParseGitHubCommits(filename string) ([]Event, error)
• type EventInfo
  • func GetLast100Commits() ([]EventInfo, error)
• type File
• type Files
• type Gist
• type GistData
  • func ParseGistData(data string) ([]GistData, error)
• type Owner
• type Payload
• type PushEvent
• type Repo
• type User

Variables

var GithubCacheDir = ".githubCommits/"

Functions

func GetCommitsInRange

func GetCommitsInRange(outputDirectory, from, to string, concurrent bool)

Function to read and parse the JSON file

Inputs:

outputDirectory (string): the directory to save files to. "githubCommits/" will be made locally if not specified

From (string): from timestamp as string in the format "2006-01-02-15"

To (string): from timestamp as string in the format "2006-01-02-15"

Concurrent (bool): download all the patch files at the same time. Note: this ramps up network, disk and CPU usage. Limit this to short timeframes.

Note: if no timestamp is specified, the previous hour's JSON file will be downloaded.

Output: A JSON file downloaded to the directory as specified

func GetISO8601Timestamps

func GetISO8601Timestamps(from, to string) []string

func GetLast100Gists

func GetLast100Gists() string

func MakePatchURL

func MakePatchURL(apiURL string) string

func PrintGharchiveChunkUrls

func PrintGharchiveChunkUrls(from, to string) []string

Types

type Actor

type Actor struct {
	AvatarURL  string `json:"avatar_url"`
	GravatarID string `json:"gravatar_id"`
	ID         int    `json:"id"`
	Login      string `json:"login"`
	URL        string `json:"url"`
}

type Author

type Author struct {
	Email string `json:"email"`
	Name  string `json:"name"`
}

type Commit

type Commit struct {
	Author   Author `json:"author"`
	Distinct bool   `json:"distinct"`
	Message  string `json:"message"`
	Sha      string `json:"sha"`
	URL      string `json:"url"`
	PatchURL string `json:"patch_url"`
}

type CommitInfo

type CommitInfo struct {
	SHA      string `json:"sha"`
	Message  string `json:"message"`
	URL      string `json:"url"`
	PatchURL string `json:"patch_url"`
	Author   struct {
		Name  string `json:"name"`
		Email string `json:"email"`
	} `json:"author"`
}

type Event

type Event struct {
	Domains   []string  `json:"domains"`
	Actor     Actor     `json:"actor"`
	CreatedAt time.Time `json:"created_at"`
	ID        string    `json:"id"`
	Payload   Payload   `json:"payload"`
	Public    bool      `json:"public"`
	Repo      Repo      `json:"repo"`
	Type      string    `json:"type"`
}

func ParseGitHubCommits

func ParseGitHubCommits(filename string) ([]Event, error)

Function to read and parse the JSON file

Input:

filenames ([]string): A list of filenames containing gharchive files. You can use filepath.Walk() here

Output ([]Event): A parsed slice containing all push events

type EventInfo

type EventInfo struct {
	Type    string    `json:"type"`
	Payload PushEvent `json:"payload"`
	Actor   struct {
		Login string `json:"login"` // Username of the person who triggered the event
	} `json:"actor"`
}

Event represents a GitHub event, which could be any type of event

func GetLast100Commits

func GetLast100Commits() ([]EventInfo, error)

type File

type File struct {
	Filename string `json:"filename"`
	Type     string `json:"type"`
	Language string `json:"language"`
	RawURL   string `json:"raw_url"`
	Size     int    `json:"size"`
}

type Files

type Files struct {
	JandedobbeleerPyOMPJSON File `json:"jandedobbeleer_py.omp.json"`
}

type Gist

type Gist struct {
	Comments    int                    `json:"comments"`
	CreatedAt   time.Time              `json:"created_at"`
	Description string                 `json:"description"`
	Files       map[string]interface{} `json:"files"` // Map for files, since it's an empty object here
	GitPullURL  string                 `json:"git_pull_url"`
	GitPushURL  string                 `json:"git_push_url"`
	HtmlURL     string                 `json:"html_url"`
	ID          string                 `json:"id"`
	Public      bool                   `json:"public"`
	UpdatedAt   time.Time              `json:"updated_at"`
	URL         string                 `json:"url"`
	User        User                   `json:"user"`
}

type GistData

type GistData struct {
	URL         string  `json:"url"`
	ForksURL    string  `json:"forks_url"`
	CommitsURL  string  `json:"commits_url"`
	ID          string  `json:"id"`
	NodeID      string  `json:"node_id"`
	GitPullURL  string  `json:"git_pull_url"`
	GitPushURL  string  `json:"git_push_url"`
	HTMLURL     string  `json:"html_url"`
	Files       Files   `json:"files"`
	Public      bool    `json:"public"`
	CreatedAt   string  `json:"created_at"`
	UpdatedAt   string  `json:"updated_at"`
	Description string  `json:"description"`
	Comments    int     `json:"comments"`
	User        *string `json:"user"`
	Owner       Owner   `json:"owner"`
	Truncated   bool    `json:"truncated"`
	RawURL      string  `json:"raw_url"`
}

func ParseGistData

func ParseGistData(data string) ([]GistData, error)

type Owner

type Owner struct {
	Login             string `json:"login"`
	ID                int    `json:"id"`
	NodeID            string `json:"node_id"`
	AvatarURL         string `json:"avatar_url"`
	URL               string `json:"url"`
	HTMLURL           string `json:"html_url"`
	FollowersURL      string `json:"followers_url"`
	FollowingURL      string `json:"following_url"`
	GistsURL          string `json:"gists_url"`
	StarredURL        string `json:"starred_url"`
	SubscriptionsURL  string `json:"subscriptions_url"`
	OrganizationsURL  string `json:"organizations_url"`
	ReposURL          string `json:"repos_url"`
	EventsURL         string `json:"events_url"`
	ReceivedEventsURL string `json:"received_events_url"`
	Type              string `json:"type"`
	UserViewType      string `json:"user_view_type"`
	SiteAdmin         bool   `json:"site_admin"`
}

type Payload

type Payload struct {
	Before       string   `json:"before"`
	Commits      []Commit `json:"commits"`
	DistinctSize int      `json:"distinct_size"`
	Head         string   `json:"head"`
	PushID       int      `json:"push_id"`
	Ref          string   `json:"ref"`
	Size         int      `json:"size"`
	Action       string   `json:"action"`
	Gist         Gist     `json:"gist"`
}

type PushEvent

type PushEvent struct {
	Commits []CommitInfo `json:"commits"`
}

PushEvent represents a push event in GitHub

type Repo

type Repo struct {
	ID   int    `json:"id"`
	Name string `json:"name"`
	URL  string `json:"url"`
}

type User

type User struct {
	AvatarURL  string `json:"avatar_url"`
	GravatarID string `json:"gravatar_id"`
	ID         int    `json:"id"`
	Login      string `json:"login"`
	URL        string `json:"url"`
}