gochrome

README

go-chrome

This package aims to be a complete Chrome DevTools Protocol implementation. The primary use-case behind this project is interacting with headless Google Chrome in a container environment, but it should be appropriate for developing server side and desktop applications for any browser that supports the devtools protocol.

The API is fairly settled and basic code-coverage tests have been implemented but real-world testing is needed. Page.captureScreenshot and related calls are working well and are regularly used for validating the viability of code changes.

This implementation is based on the Tip-of-Tree documentation and may be prone to change. At some point stable versions will be implemented as well, hopefully beginning with v1.3.

Announcements

v1.0.0-rc3 released

v1.0.0-rc3 has been released.

• Fixes an issue with an upstream dependency

Please open an issue to report any problems or suggest any changes.

v1.0.0-rc2 released

v1.0.0-rc2 has been released.

• Fixes an issue with zombie listen process
• Fixes an issue with zombie stop process
• Adds test coverage for socket timeouts

Please open an issue to report any problems or suggest any changes.

[[constraint]]
  name = "github.com/mkenney/go-chrome"
  version = "1.0.0-rc2"

v1.0.0-rc1 released

v1.0.0-rc1 has been released. Please open an issue to report any problems or suggest any changes.

[[constraint]]
  name = "github.com/mkenney/go-chrome"
  version = "1.0.0-rc1"

Examples

There are a few small examples of how to use the framework API on the wiki and in the /test directory.

TODO

Contributions of any kind are very welcome!

• Add framework API examples to the /test directory and wiki.

  Any example scripts showing various ways people are using the framework would be outstanding!

• Refactoring to implement standard interfaces where applicable and review current use of interfaces in the API. Some aren't needed at all and others are used to support test mocks.

• Add more tests, especially for error cases. If you would like to contribute but aren't sure how, take a look at codecov for any tests that could be written. There are many examples of tests in the repo.

• Add integration test scripts to the test/ directory to exercise various functions. The screenshot script is already setup there.

Change Log

2018-05-30

I have refactored the package layout, deleting the cdtp package and moving all the packages within it to the tot package. This cleans up import statements and make the layout a bit more natural to use. The only refactoring this requires is removing that folder from your import path, changing them from (for example):

import (
	"github.com/mkenney/go-chrome/tot/cdtp/accessibility"
)

to simply:

import (
	"github.com/mkenney/go-chrome/tot/accessibility"
)

The v1.0.0-rc1 pre-release candidate coincides with this change. Please open an issue to report any problems or suggest any additional changes to further the v1 milestone.

2018-03-20

Removed the unnecessary array notation for the flags passed to the Chrome process. This is a breaking change but it's simple to update client code and easier to use in general.

When defining your Chrome instance, remove any []interface{}{} declarations in the Flags parameter. To demonstrate, the example code has changed from:

	// Define a chrome instance with remote debugging enabled.
	browser := chrome.New(
		&chrome.Flags{
			"addr":               []interface{}{"localhost"},
			"disable-extensions": nil,
			"disable-gpu":        nil,
			"headless":           nil,
			"hide-scrollbars":    nil,
			"no-first-run":       nil,
			"no-sandbox":         nil,
			"port":               []interface{}{9222},
			"remote-debugging-address": []interface{}{"0.0.0.0"},
			"remote-debugging-port":    []interface{}{9222},
		},
		"/usr/bin/google-chrome",
		"",
		"",
		"",
	)

to simply:

	// Define a chrome instance with remote debugging enabled.
	browser := chrome.New(
		&chrome.Flags{
			"addr":               "localhost",
			"disable-extensions": nil,
			"disable-gpu":        nil,
			"headless":           nil,
			"hide-scrollbars":    nil,
			"no-first-run":       nil,
			"no-sandbox":         nil,
			"port":               9222,
			"remote-debugging-address": "0.0.0.0",
			"remote-debugging-port":    9222,
		},
		"/usr/bin/google-chrome",
		"",
		"",
		"",
	)

It's easier to deal with, simpler to understand, and supporting multiple values wasn't ever useful. An argument could be made for supporting something like pkg/flag I suppose but I don't generally need a CLI interface.

2018-02-06

To provide for more idiomatic package naming the following packages have been renamed:

• tot/cdtp/application_cache -> tot/cdtp/application/cache
• tot/cdtp/cache_storage -> tot/cdtp/cache/storage
• tot/cdtp/device_orientation -> tot/cdtp/device/orientation
• tot/cdtp/dom_debugger -> tot/cdtp/dom/debugger
• tot/cdtp/dom_snapshot -> tot/cdtp/dom/snapshot
• tot/cdtp/dom_storage -> tot/cdtp/dom/storage
• tot/cdtp/headless_experimental -> tot/cdtp/headless/experimental
• tot/cdtp/heap_profiler -> tot/cdtp/heap/profiler
• tot/cdtp/indexed_db -> tot/cdtp/indexed/db
• tot/cdtp/layer_tree -> tot/cdtp/layer/tree
• tot/cdtp/service_worker -> tot/cdtp/service/worker
• tot/cdtp/system_info -> tot/cdtp/system/info

2018-01-21

The core data types have been updated with proper enum support for string properties that only allow specific values.

2018-01-12

In preparation for having stable versions of the library, I updated the directory structure of the repo adding a tot package for the current code (Tip-of-Tree).

2018-01-05

I merged some changes that did change the API a bit. Mainly, all the protocol methods now return a channel instead of blocking until they get a result to better handle the nature of socket data streams. This makes unit testing easier and cleaner and the API more useful, but the race detector still finds false positives due to writing test data to a stack that mocks a socket data stream, which is being drained by an independent goroutine...

I'm not sure what to do with that, or if I care at this point. You can see it by running go test -race ./....

Documentation

Overview

Package gochrome aims to be a complete Chrome DevTools Protocol Viewer implementation.

Versions

Versioned packages are available. Curently the only version is `tot` or Tip-of-Tree. Stable versions will be made available in the future.

import "github.com/mkenney/go-chrome/tot"

Work in progress

This is beta software and hasn't been well exercised in real-world applications.

Official documentation

See https://chromedevtools.github.io/devtools-protocol/

The Chrome DevTools Protocol allows for tools to instrument, inspect, debug and profile Chromium, Chrome and other Blink-based browsers. Many existing projects currently use the protocol. The Chrome DevTools uses this protocol and the team maintains its API.

Instrumentation is divided into a number of domains (DOM, Debugger, Network etc.). Each domain defines a number of commands it supports and events it generates. Both commands and events are serialized JSON objects of a fixed structure. You can either debug over the wire using the raw messages as they are described in the corresponding domain documentation, or use extension JavaScript API.

Protocol API Docs

The latest (tip-of-tree) protocol (tot)

It changes frequently and can break at any time. However it captures the full capabilities of the Protocol, whereas the stable release is a subset. There is no backwards compatibility support guaranteed for the capabilities it introduces.

Resources

• chrome-debugging-protocol mailing list https://groups.google.com/d/forum/chrome-debugging-protocol
• devtools-protocol repo issue tracker https://github.com/chromedevtools/devtools-protocol can also be used for concerns with the protocol. It also hosts the canonical copy of the json files.
• Getting Started with Headless Chrome https://developers.google.com/web/updates/2017/04/headless-chrome
• Headless Chromium readme https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md
• chrome-remote-interface node module https://github.com/cyrus-and/chrome-remote-interface/ Wiki https://github.com/cyrus-and/chrome-remote-interface/wiki Issue tracker https://github.com/cyrus-and/chrome-remote-interface/issues?utf8=%E2%9C%93&q=is%3Aissue%20
• awesome-chrome-devtools page https://github.com/ChromeDevTools/awesome-chrome-devtools#chrome-devtools-protocol links to many of the tools in the protocol ecosystem, including protocol API libraries in JavaScript, TypeScript, Python, Java, and Go. Many applications and libraries already use the protocol.

Basics: Using DevTools as protocol client

The Developer Tools front-end can attach to a remotely running Chrome instance for debugging. For this scenario to work, you should start your host Chrome instance with the remote-debugging-port command line switch:

chrome.exe --remote-debugging-port=9222

Then you can start a separate client Chrome instance, using a distinct user profile:

chrome.exe --user-data-dir=<some directory>

Now you can navigate to the given port from your client and attach to any of the discovered tabs for debugging: http://localhost:9222

You will find the Developer Tools interface identical to the embedded one and here is why:

• When you navigate your client browser to the remote's Chrome port, Developer Tools front-end is being served from the host Chrome as a Web Application from the Web Server.
• It fetches HTML, JavaScript and CSS files over HTTP
• Once loaded, Developer Tools establishes a Web Socket connection to its host and starts exchanging JSON messages with it.

In this scenario, you can substitute Developer Tools front-end with your own implementation. Instead of navigating to the HTML page at http://localhost:9222, your application can discover available pages by requesting: http://localhost:9222/json and getting a JSON object with information about inspectable pages along with the WebSocket addresses that you could use in order to start instrumenting them.

Remote debugging is especially useful when debugging remote instances of the browser or attaching to the embedded devices. Blink port owners are responsible for exposing debugging connections to the external users.

Sniffing the protocol

This is especially handy to understand how the DevTools frontend makes use of the protocol. First, run Chrome with the debugging port open:

alias canary="/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary" canary --remote-debugging-port=9222 http://localhost:9222 http://chromium.org

Then, select the Chromium Projects item in the Inspectable Pages list. Now that DevTools is up and fullscreen, open DevTools to inspect it. Cmd-R in the new inspector to make the first restart. Now head to Network Panel, filter by Websocket, select the connection and click the Frames tab. Now you can easily see the frames of WebSocket activity as you use the first instance of the DevTools.

DevTools protocol via Chrome extension

To allow chrome extensions to interact with the protocol, we introduced chrome.debugger extension API that exposes this JSON message transport interface. As a result, you can not only attach to the remotely running Chrome instance, but also instrument it from its own extension.

Chrome Debugger Extension API provides a higher level API where command domain, name and body are provided explicitly in the `sendCommand` call. This API hides request ids and handles binding of the request with its response, hence allowing `sendCommand` to report result in the callback function call. One can also use this API in combination with the other Extension APIs.

If you are developing a Web-based IDE, you should implement an extension that exposes debugging capabilities to your page and your IDE will be able to open pages with the target application, set breakpoints there, evaluate expressions in console, live edit JavaScript and CSS, display live DOM, network interaction and any other aspect that Developer Tools is instrumenting today.

Opening embedded Developer Tools will terminate the remote connection and thus detach the extension. https://chromedevtools.github.io/devtools-protocol/#simultaneous

Frequently Asked Questions

How is the protocol defined

The canonical protocol definitions live in the Chromium source tree: (browser_protocol.json and js_protocol.json). They are maintained manually by the DevTools engineering team. These files are mirrored (hourly) on GitHub in the devtools-protocol repo.

The declarative protocol definitions are used across tools. Within Chromium, a binding layer is created for the Chrome DevTools to interact with, and separately the protocol is used for Chrome Headless’s C++ interface.

What’s the protocol_externs file

It’s created via generate_protocol_externs.py and useful for tools using closure compiler. The TypeScript story is here.

Are the HTTP endpoints documented

Not yet. See bugger-daemon’s third-party docs. See also the endpoints implementation in Chromium. /json/protocol was added in Chrome 60.

https://github.com/buggerjs/bugger-daemon/blob/master/README.md#api
https://cs.chromium.org/search/?q=f:devtools_http_handler.cc+%22command+%3D%3D+%22&sq=package:chromium&type=cs

How do I access the browser target

The endpoint is exposed as webSocketDebuggerUrl in /json/version. Note the browser in the URL, rather than page. If Chrome was launched with --remote-debugging-port=0 and chose an open port, the browser endpoint is written to both stderr and the DevToolsActivePort file in browser profile folder.

Does the protocol support multiple simultaneous clients

Yes, as of Chrome 63! See Multi-client remote debugging support.

Upon disconnnection, the outgoing client will receive a detached event. For example:

{"method":"Inspector.detached","params":{"reason":"replaced_with_devtools"}}.

View the enum of possible reasons. (For reference: the original patch). After disconnection, some apps have chosen to pause their state and offer a reconnect button.