ja3

README

JA3

JA3 is a technique developed by Salesforce, to fingerprint the TLS client and server hellos.

The official python implementation can be found here. More details can be found in their blog post: https://engineering.salesforce.com/open-sourcing-ja3-92c9e53c3c41

This package provides a pure golang implementation of both the client and server hash functions, with unit tests to ensure correct behavior and performance. This implementation uses dreadl0ck/gopacket for packet decoding.

The bradleyfalzon/tlsx fork used for parsing the TLS handshakes (dreadl0ck/tlsx), has been extended to support extracting the server hello message, unit tests, benchmarks and a faster decoding interface for use with Ja3 fingerprinting.

Assembly of the ja3 bare was previously implemented with the golang 1.10 strings.Builder type, however using byte slices for this turned out to be faster and causes less allocations. (Thanks for the tips @lukechampine) Thanks to @guigzzz for his pull request on further reducing allocations!

Package Usage

This package exports the following API:

Live Capture from interface, outputs CSV with configurable separator or JSON:

func ReadInterface(iface string, out io.Writer, separator string, ja3s bool, asJSON bool, snaplen int, promisc bool, timeout time.Duration) {

Files:

func ReadFileJSON(file string, out io.Writer, doJA3s bool)

func ReadFileCSV(file string, out io.Writer, separator string, doJA3s bool)

Read file and only print Ja3s values, mimics the python implementation from salesforce:

func ReadFileJa3s(file string, out io.Writer)

Using gopacket.Packet:

func BarePacket(p gopacket.Packet) []byte

func BarePacketJa3s(p gopacket.Packet) []byte

func DigestPacket(p gopacket.Packet) [md5.Size]byte

func DigestPacketJa3s(p gopacket.Packet) [md5.Size]byte 

func DigestHexPacket(p gopacket.Packet) string

func DigestHexPacketJa3s(p gopacket.Packet) string 

Using tlsx.ClientHello:

func Bare(hello *tlsx.ClientHello) []byte

func BareJa3s(hello *tlsx.ServerHello) []byte

func Digest(hello *tlsx.ClientHello) [md5.Size]byte

func DigestJa3s(hello *tlsx.ServerHello) [md5.Size]byte

func DigestHex(hello *tlsx.ClientHello) string

func DigestHexJa3s(hello *tlsx.ServerHello) string

Commandline Tool

The commandline program mimics the python reference implementation by default. It reads the pcap file and prints the all extracted client hellos as JSON array to stdout. Printing output as CSV, tab separated values or with a custom separator is also supported.

$ goja3 -h
Usage of goja3:
  -csv
    	print as CSV
  -debug
    	toggle debug mode
  -iface string
    	specify network interface to read packets from
  -ja3s
    	dump ja3s only
  -json
    	print as JSON array (default true)
  -read string
    	read PCAP file
  -separator string
    	set a custom separator (default ",")
  -tsv
    	print as TAB separated values

Benchmark of the python reference implementation VS this one, on a 109 MB PCAP dumpfile (DEF CON 23 ICS Village.pcap). This dump file is interesting for comparison because it contains handshakes without extensions set, as well as TLS traffic over non standard ports.

Python implementation:

# https://github.com/salesforce/ja3
$ time ja3 -a DEF\ CON\ 23\ ICS\ Village.pcap &> /dev/null

real	0m29.031s
user	0m28.803s
sys	0m0.141s

Go implementation:

# github.com/dreadl0ck/ja3
# disable server fingerprint generation to mimic the behavior of the reference implementation
$ time goja3 -ja3s=false -json -read DEF\ CON\ 23\ ICS\ Village.pcap &> /dev/null

real	0m0.996s
user	0m2.245s
sys	0m0.166s

Output:

[
    ...
    {
        "destination_ip": "192.168.0.60",
        "destination_port": 443,
        "ja3": "771,5-4-2-8-20-3-1-21-6-22-23-51-57-25-58-26-24-53-9-10-27-47-52-49168-49158-49173-49163-49153-59-49200-49196-49192-49188-49172-49162-163-159-107-106-56-136-135-49177-167-109-137-49202-49198-49194-49190-49167-49157-157-61-132-49170-49160-19-49175-49165-49155-49199-49195-49191-49187-49171-49161-162-158-103-64-50-154-153-69-68-49176-166-108-155-70-49201-49197-49193-49189-49166-49156-156-60-150-65-49169-49159-49174-49164-49154-18-17-255,11-10-35-13-15,14-13-25-11-12-24-9-10-22-23-8-6-7-20-21-4-5-18-19-1-2-3-15-16-17,0-1-2",
        "ja3_digest": "e3bb8f1cd407701c585e7a84e96c24e1",
        "ja3s": "",
        "ja3s_digest": "",
        "source_ip": "192.168.0.112",
        "source_port": 59935,
        "timestamp": "1438978270.855224"
    },
    {
        "destination_ip": "192.168.0.30",
        "destination_port": 443,
        "ja3": "771,5-4-2-8-20-3-1-21-6-22-23-51-57-25-58-26-24-53-9-10-27-47-52-49168-49158-49173-49163-49153-59-49200-49196-49192-49188-49172-49162-163-159-107-106-56-136-135-49177-167-109-137-49202-49198-49194-49190-49167-49157-157-61-132-49170-49160-19-49175-49165-49155-49199-49195-49191-49187-49171-49161-162-158-103-64-50-154-153-69-68-49176-166-108-155-70-49201-49197-49193-49189-49166-49156-156-60-150-65-49169-49159-49174-49164-49154-18-17-255,11-10-35-13-15,14-13-25-11-12-24-9-10-22-23-8-6-7-20-21-4-5-18-19-1-2-3-15-16-17,0-1-2",
        "ja3_digest": "e3bb8f1cd407701c585e7a84e96c24e1",
        "ja3s": "",
        "ja3s_digest": "",
        "source_ip": "192.168.0.112",
        "source_port": 42455,
        "timestamp": "1438978270.855228"
    }
]

JA3 Details

JA3 gathers the decimal values of the bytes for the following fields: SSL Version, Accepted Ciphers, List of Extensions, Elliptic Curves, and Elliptic Curve Formats.

It then concatenates those values together in order, using a “,” to delimit each field and a “-” to delimit each value in each field.

The field order is as follows:

SSLVersion,Ciphers,Extensions,EllipticCurves,EllipticCurvePointFormats

Example:

769,47–53–5–10–49161–49162–49171–49172–50–56–19–4,0–10–11,23–24–25,0

If there are no SSL Extensions in the Client Hello, the fields are left empty.

Example:

769,4–5–10–9–100–98–3–6–19–18–99,,,

These strings are then MD5 hashed to produce an easily consumable and shareable 32 character fingerprint.

This is the JA3 SSL Client Fingerprint.

JA3 is a much more effective way to detect malicious activity over SSL than IP or domain based IOCs. Since JA3 detects the client application, it doesn’t matter if malware uses DGA (Domain Generation Algorithms), or different IPs for each C2 host, or even if the malware uses Twitter for C2, JA3 can detect the malware itself based on how it communicates rather than what it communicates to.

JA3 is also an excellent detection mechanism in locked-down environments where only a few specific applications are allowed to be installed. In these types of environments one could build a whitelist of expected applications and then alert on any other JA3 hits.

For more details on what you can see and do with JA3 and JA3S, please see this Shmoocon 2018 talk: https://youtu.be/oprPu7UIEuk?t=6m44s

JA3S Details

JA3S is JA3 for the Server side of the SSL/TLS communication and fingerprints how servers respond to particular clients.

JA3S uses the following field order:

SSLVersion,Cipher,SSLExtension

With JA3S it is possible to fingerprint the entire cryptographic negotiation between client and it's server by combining JA3 + JA3S. That is because servers will respond to different clients differently but will always respond to the same client the same.

For the Trickbot example:

JA3 = 6734f37431670b3ab4292b8f60f29984 ( Fingerprint of Trickbot )
JA3S = 623de93db17d313345d7ea481e7443cf ( Fingerprint of Command and Control Server Response )

For the Emotet example:

JA3 = 4d7a28d6f2263ed61de88ca66eb011e3 ( Fingerprint of Emotet )
JA3S = 80b3a14bccc8598a1f3bbe83e71f735f ( Fingerprint of Command and Control Server Response )

In these malware examples, the command and control server always responds to the malware client in exactly the same way, it does not deviate. So even though the traffic is encrypted and one may not know the command and control server's IPs or domains as they are constantly changing, we can still identify, with reasonable confidence, the malicious communication by fingerprinting the TLS negotiation between client and server. Again, please be aware that these are examples, not indicative of all versions ever, and are intended to illustrate what is possible.

Benchmarks & Tests

Run the benchmarks and the correctness test with:

$ go test -v -bench=.
=== RUN   TestDigestHexCorrect
--- PASS: TestDigestHexCorrect (0.00s)
=== RUN   TestDigestHexComparePcap
--- PASS: TestDigestHexComparePcap (0.10s)
=== RUN   TestDigestHexJa3sCorrect
--- PASS: TestDigestHexJa3sCorrect (0.00s)
=== RUN   TestDigestHexJa3sComparePcap
--- PASS: TestDigestHexJa3sComparePcap (0.10s)
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/ja3
BenchmarkDigestHexPacket
BenchmarkDigestHexPacket-12        	  781578	      1499 ns/op	     352 B/op	       9 allocs/op
BenchmarkDigestPacket
BenchmarkDigestPacket-12           	  827286	      1406 ns/op	     288 B/op	       7 allocs/op
BenchmarkBarePacket
BenchmarkBarePacket-12             	 1000000	      1212 ns/op	     288 B/op	       7 allocs/op
BenchmarkDigestHex
BenchmarkDigestHex-12              	  895094	      1278 ns/op	     256 B/op	       3 allocs/op
BenchmarkDigest
BenchmarkDigest-12                 	  996800	      1318 ns/op	     192 B/op	       1 allocs/op
BenchmarkBare
BenchmarkBare-12                   	 1101476	      1188 ns/op	     192 B/op	       1 allocs/op
BenchmarkDigestHexPacketJa3s
BenchmarkDigestHexPacketJa3s-12    	 1654360	       619 ns/op	     120 B/op	       4 allocs/op
BenchmarkDigestPacketJa3s
BenchmarkDigestPacketJa3s-12       	 2328241	       513 ns/op	      56 B/op	       2 allocs/op
BenchmarkBarePacketJa3s
BenchmarkBarePacketJa3s-12         	 3294260	       387 ns/op	      56 B/op	       2 allocs/op
BenchmarkDigestHexJa3s
BenchmarkDigestHexJa3s-12          	 2476090	       464 ns/op	     112 B/op	       3 allocs/op
BenchmarkDigestJa3s
BenchmarkDigestJa3s-12             	 3321294	       357 ns/op	      48 B/op	       1 allocs/op
BenchmarkBareJa3s
BenchmarkBareJa3s-12               	 4964508	       238 ns/op	      48 B/op	       1 allocs/op
PASS
ok  	github.com/dreadl0ck/ja3	18.414s

Performance comparison with other implementations:

# https://github.com/open-ch/ja3
$ time ja3exporter -pcap DEF\ CON\ 23\ ICS\ Village.pcap &> /dev/null

real	0m0.912s
user	0m1.979s
sys	0m0.142s

# github.com/dreadl0ck/ja3
# disable server fingerprint generation to mimic the behavior of the other implementations
$ time goja3 -ja3s=false -json -read DEF\ CON\ 23\ ICS\ Village.pcap &> /dev/null

real	0m0.996s
user	0m2.245s
sys	0m0.166s

# https://github.com/salesforce/ja3
$ time ja3 -a DEF\ CON\ 23\ ICS\ Village.pcap &> /dev/null

real	0m29.031s
user	0m28.803s
sys	0m0.141s

Number of records detected:

# https://github.com/open-ch/ja3
$ time ja3exporter -pcap DEF\ CON\ 23\ ICS\ Village.pcap | wc -l
138

# github.com/dreadl0ck/ja3
$ goja3 -ja3s=false -json -read DEF\ CON\ 23\ ICS\ Village.pcap | jq '. | length'
138

# https://github.com/salesforce/ja3
$ time ja3 -a DEF\ CON\ 23\ ICS\ Village.pcap | jq '. | length'
138

# https://github.com/salesforce/ja3
# without -a flag, only tls traffic over port 443 will be extracted
$ time ja3 DEF\ CON\ 23\ ICS\ Village.pcap | jq '. | length'
96

Notes

The ja3_output_testpcap.json file for output comparison in the unit tests was generated with the python reference implementation using:

ja3 -a test.pcap > ja3_output_testpcap.json

Documentation

Index

• Variables
• func Bare(hello *tlsx.ClientHelloBasic) []byte
• func BareJa3s(hello *tlsx.ServerHelloBasic) []byte
• func BarePacket(p gopacket.Packet) []byte
• func BarePacketJa3s(p gopacket.Packet) []byte
• func BareToDigestHex(bare []byte) string
• func BareToDigestHexJa3s(bare []byte) string
• func Digest(hello *tlsx.ClientHelloBasic) [md5.Size]byte
• func DigestHex(hello *tlsx.ClientHelloBasic) string
• func DigestHexJa3s(hello *tlsx.ServerHelloBasic) string
• func DigestHexPacket(p gopacket.Packet) string
• func DigestHexPacketJa3s(p gopacket.Packet) string
• func DigestJa3s(hello *tlsx.ServerHelloBasic) [md5.Size]byte
• func DigestPacket(p gopacket.Packet) [md5.Size]byte
• func DigestPacketJa3s(p gopacket.Packet) [md5.Size]byte
• func ReadFileCSV(file string, out io.Writer, separator string, doJA3s bool)
• func ReadFileJSON(file string, out io.Writer, doJA3s bool)
• func ReadFileJa3s(file string, out io.Writer)
• func ReadInterface(iface, bpfFilter, dumpPkg string, out io.Writer, separator string, ja3s bool, ...)
• type PacketSource
• type Record

Variables

var (
	// Debug indicates whether we run in debug mode.
	Debug = false
)

Functions

func Bare

func Bare(hello *tlsx.ClientHelloBasic) []byte

Bare returns the JA3 bare string for a given tlsx.ClientHelloBasic instance JA3 is a technique developed by Salesforce, to fingerprint TLS Client Hellos. the official python implementation can be found here: https://github.com/salesforce/ja3 JA3 gathers the decimal values of the bytes for the following fields; SSL Version, Accepted Ciphers, List of Extensions, Elliptic Curves, and Elliptic Curve Formats. It then concatenates those values together in order, using a “,” to delimit each field and a “-” to delimit each value in each field. The field order is as follows: SSLVersion,Ciphers,Extensions,EllipticCurves,EllipticCurvePointFormats Example: 769,47–53–5–10–49161–49162–49171–49172–50–56–19–4,0–10–11,23–24–25,0 If there are no SSL Extensions in the Client Hello, the fields are left empty. Example: 769,4–5–10–9–100–98–3–6–19–18–99,,, These strings are then MD5 hashed to produce an easily consumable and shareable 32 character fingerprint. This is the JA3 SSL Client Fingerprint returned by this function.

func BareJa3s

func BareJa3s(hello *tlsx.ServerHelloBasic) []byte

BareJa3s returns the JA3S bare string for a given tlsx.ServerHelloBasic instance JA3S is JA3 for the Server side of the SSL/TLS communication and fingerprints how servers respond to particular clients. JA3S uses the following field order: SSLVersion,Cipher,SSLExtension

func BarePacket

func BarePacket(p gopacket.Packet) []byte

BarePacket returns the Ja3 digest if the supplied packet contains a TLS client hello otherwise returns an empty string

func BarePacketJa3s

func BarePacketJa3s(p gopacket.Packet) []byte

BarePacket returns the Ja3 digest if the supplied packet contains a TLS client hello otherwise returns an empty string

func BareToDigestHex

func BareToDigestHex(bare []byte) string

BareToDigestHex converts a bare []byte to a hex string.

func BareToDigestHexJa3s

func BareToDigestHexJa3s(bare []byte) string

BareToDigestHex converts a bare []byte to a hex string.

func Digest

func Digest(hello *tlsx.ClientHelloBasic) [md5.Size]byte

Digest returns only the digest md5.

func DigestHex

func DigestHex(hello *tlsx.ClientHelloBasic) string

DigestHex produce md5 hash from bare string.

func DigestHexJa3s

func DigestHexJa3s(hello *tlsx.ServerHelloBasic) string

DigestHex produce md5 hash from bare string.

func DigestHexPacket

func DigestHexPacket(p gopacket.Packet) string

DigestHexPacket returns the hex string for the packet for a packet carrying a TLS Client Hello

func DigestHexPacketJa3s

func DigestHexPacketJa3s(p gopacket.Packet) string

DigestHexPacket returns the hex string for the packet for a packet carrying a TLS Client Hello

func DigestJa3s

func DigestJa3s(hello *tlsx.ServerHelloBasic) [md5.Size]byte

Digest returns only the digest md5.

func DigestPacket

func DigestPacket(p gopacket.Packet) [md5.Size]byte

DigestPacket returns the Ja3 digest for a packet carrying a TLS Client Hello or an empty byte slice

func DigestPacketJa3s

func DigestPacketJa3s(p gopacket.Packet) [md5.Size]byte

DigestPacket returns the Ja3s digest for a packet carrying a TLS Server Hello or an empty byte slice

func ReadFileCSV

func ReadFileCSV(file string, out io.Writer, separator string, doJA3s bool)

ReadFileCSV reads the PCAP file at the given path and prints out all packets containing JA3 digests to the supplied io.Writer

func ReadFileJSON

func ReadFileJSON(file string, out io.Writer, doJA3s bool)

ReadFileJSON reads the PCAP file at the given path and prints out all packets containing JA3 digests formatted as JSON to the supplied io.Writer

func ReadFileJa3s

func ReadFileJa3s(file string, out io.Writer)

ReadFileJa3s reads the PCAP file at the given path and prints out all packets containing JA3S digests to the supplied io.Writer

func ReadInterface

func ReadInterface(iface, bpfFilter, dumpPkg string, out io.Writer, separator string, ja3s bool, asJSON bool, snaplen int, promisc bool, timeout time.Duration)

ReadInterface reads packets from the named interface if asJSON is true the results will be dumped as newline separated JSON objects otherwise CSV will be printed to the supplied io.Writer.

Types

type PacketSource

type PacketSource interface {
	ReadPacketData() ([]byte, gopacket.CaptureInfo, error)
}

PacketSource means we can read Packets.

type Record

type Record struct {
	DestinationIP   string  `json:"destination_ip"`
	DestinationPort int     `json:"destination_port"`
	JA3             string  `json:"ja3"`
	JA3Digest       string  `json:"ja3_digest"`
	JA3S            string  `json:"ja3s"`
	JA3SDigest      string  `json:"ja3s_digest"`
	SourceIP        string  `json:"source_ip"`
	SourcePort      int     `json:"source_port"`
	Timestamp       float64 `json:"timestamp"`
}

Record contains all information for a calculated JA3