The highest tagged major version is v3.

dremio-diagnostic-collector

command module

v0.9.1 Latest Latest Go to latest Published: Jan 24, 2024 License: Apache-2.0 Imports: 5 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/dremio/dremio-diagnostic-collector

Links

Open Source Insights

README ¶

collect logs of Dremio for analysis

IMPORTANT LINKS

Read the FAQ.md for questions on making DDC work the way you want
Read the ddc.yaml as it is well documented and contains all the custom functionality you may want to customer

Install

Download the latest release binaries:

unzip the binary
open a terminal
change to the directory where you unzip your binary
run the command ./ddc -h if you get the help for the command you are good to go.

User Docs

Read the Dremio Diagnostic Collector KB Article

dremio on k8s

ddc connects via ssh or kubectl and collects a series of logs and files for dremio, then puts those collected files in an archive

for kubernetes deployments:

coordinator only

ddc --k8s --namespace mynamespace --coordinator app=dremio-coordinator

coordinator and executors

ddc --k8s --namespace mynamespace --coordinator app=dremio-coordinator --executors app=dremio-executor

to collect job profiles, system tables, kv reports and wlm

ddc --k8s -n mynamespace -c app=dremio-coordinator -e app=dremio-executor --dremio-pat-prompt

dremio on prem

specific executors that you want to collect from with the -e flag and coordinators with the -c flag. Specify ssh user, and ssh key to use.

for ssh based communication to VMs or Bare metal hardware:

coordinator only

ddc --coordinator 10.0.0.19 --ssh-user myuser

coordinator and executors

ddc --coordinator 10.0.0.19 --executors 10.0.0.20,10.0.0.21,10.0.0.22 --ssh-user myuser

to collect job profiles, system tables, kv reports and wlm

ddc --coordinator 10.0.0.19 --executors 10.0.0.20,10.0.0.21,10.0.0.22 --ssh-user myuser  --dremio-pat-prompt

to avoid using the /tmp folder on nodes

ddc --coordinator 10.0.0.19 --executors 10.0.0.20,10.0.0.21,10.0.0.22 --ssh-user myuser --transfer-dir /mnt/lots_of_storage/

dremio on AWSE

If you want to do a log only collection of AWSE say from the coordinator the following command will produce a tarball with all the logs from each node

./ddc awselogs

dremio cloud (Preview)

Specify the following parameters in ddc.yaml

is-dremio-cloud: true
dremio-endpoint: "[eu.]dremio.cloud"    # Specify whether EU Dremio Cloud or not
dremio-cloud-project-id: "<PROJECT_ID>"
dremio-pat-token: "<DREMIO_PAT>"
tmp-output-dir: /full/path/to/dir       # Specify local target directory

and run

./ddc local-collect

Windows Users

If you are running ddc from windows, always run in a shell from the C: drive prompt. This is because of a limitation of kubectl ( see https://github.com/kubernetes/kubernetes/issues/77310 )

ddc.yaml

The ddc.yaml file is located next to your ddc binary, it is well documented and you should edit it to fit your environment.

Flags

Available Commands:
  awselogs      Log only collect of AWSE from the coordinator node
  completion    Generate the autocompletion script for the specified shell
  help          Help about any command
  local-collect retrieves all the dremio logs and diagnostics for the local node and saves the results in a compatible format for Dremio support
  version       Print the version number of DDC

Flags:
  -c, --coordinator string             coordinator to connect to for collection. With ssh set a list of ip addresses separated by commas. In K8s use a label that matches to the pod(s).
      --coordinator-container string   for use with -k8s flag: sets the container name to use to retrieve logs in the coordinators (default "dremio-master-coordinator,dremio-coordinator")
      --ddc-yaml string                location of ddc.yaml that will be transferred to remote nodes for collection configuration (default "/Users/ryan.svihla/Documents/GitHub/dremio-diagnostic-collector/bin/ddc.yaml")
  -t, --dremio-pat-prompt              Prompt for Dremio Personal Access Token (PAT)
  -e, --executors string               either a common separated list or a ip range of executors nodes to connect to. With ssh set a list of ip addresses separated by commas. In K8s use a label that matches to the pod(s).
      --executors-container string     for use with -k8s flag: sets the container name to use to retrieve logs in the executors (default "dremio-executor")
  -h, --help                           help for ddc
  -k, --k8s                            use kubernetes to retrieve the diagnostics instead of ssh, instead of hosts pass in labels to the --coordinator and --executors flags
  -p, --kubectl-path string            where to find kubectl (default "kubectl")
  -n, --namespace string               namespace to use for kubernetes pods (default "default")
      --output-file string             name of tgz file to save the diagnostic collection to (default "diag.tgz")
  -s, --ssh-key string                 location of ssh key to use to login
  -u, --ssh-user string                user to use during ssh operations to login
  -b, --sudo-user string               if any diagnostics commands need a sudo user (i.e. for jcmd)
      --transfer-dir string            directory to use for communication between the local-collect command and this one (default "/tmp/ddc-20240112141658")

Documentation ¶

Overview ¶

main is the standard go entry point for the application

Source Files ¶

View all Source files

main.go

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd cmd package contains all the command line flag and initialization logic for commands	cmd package contains all the command line flag and initialization logic for commands
awselogs
local cmd package contains all the command line flag and initialization logic for commands	cmd package contains all the command line flag and initialization logic for commands
local/apicollect apicollect provides all the methods that collect via the API, this is a substantial part of the activities of DDC so it gets it's own package	apicollect provides all the methods that collect via the API, this is a substantial part of the activities of DDC so it gets it's own package
local/conf package conf provides configuration for the local-collect command	package conf provides configuration for the local-collect command
local/conf/autodetect package autodetect looks at the system configuration and file names and tries to guess at the correct configuration	package autodetect looks at the system configuration and file names and tries to guess at the correct configuration
local/configcollect
local/consent package consent contains the logic for showing what files are collected as well the boilerplate text	package consent contains the logic for showing what files are collected as well the boilerplate text
local/ddcio ddcio include helper code for io operations common to ddc	ddcio include helper code for io operations common to ddc
local/jvmcollect package jvmcollect handles parsing of the jvm information	package jvmcollect handles parsing of the jvm information
local/logcollect package logcollect contains the logic for log collection in the local-collect sub command	package logcollect contains the logic for log collection in the local-collect sub command
local/nodeinfocollect package nodeinfocollect has all the methods for collecting the information for nodeinfo	package nodeinfocollect has all the methods for collecting the information for nodeinfo
local/queriesjson queriesjson package contains the logic for collecting queries.json information	queriesjson package contains the logic for collecting queries.json information
local/restclient
local/threading threading package provides support for simple concurrency and threading	threading package provides support for simple concurrency and threading
root/cli package cli provides wrapper support for executing commands, this is so we can test the rest of the implementations quickly.	package cli provides wrapper support for executing commands, this is so we can test the rest of the implementations quickly.
root/collection collection package provides the interface for collection implementation and the actual collection execution	collection package provides the interface for collection implementation and the actual collection execution
root/ddcbinary
root/helpers helpers package provides some general functions that do not have a good home	helpers package provides some general functions that do not have a good home
root/kubernetes kubernetes package provides access to log collections on k8s	kubernetes package provides access to log collections on k8s
root/ssh ssh package uses ssh and scp binaries to execute commands remotely and translate the results back to the calling node	ssh package uses ssh and scp binaries to execute commands remotely and translate the results back to the calling node
version cmd package contains all the command line flag and initialization logic for commands	cmd package contains all the command line flag and initialization logic for commands
pkg
archive
clusterstats package clusterstats provides a placeholder for summary information found inside of a tarball, used by local and remote collect	package clusterstats provides a placeholder for summary information found inside of a tarball, used by local and remote collect
consoleprint package consoleprint contains the logic to update the console UI	package consoleprint contains the logic to update the console UI
dirs dirs provides helpers for working with directories on the filesystem	dirs provides helpers for working with directories on the filesystem
jps jps package provides logic for extracting values from jps	jps package provides logic for extracting values from jps
masking masking hides secrets in files and replaces them with redacted text	masking hides secrets in files and replaces them with redacted text
output output provides functinos around capturing output	output provides functinos around capturing output
simplelog simplelog package provides a simple logger	simplelog package provides a simple logger
strutils
tests package tests provides helper functions and mocks for running tests	package tests provides helper functions and mocks for running tests
versions