README
¶
The erdle tool suite can be used for just anything that can be done on VCDU (aka CADU) packets.
Started as small in-house scripts, they have first evolved to a simple tool able to dissect VCDU packets and then continue to grow to a full-featured tool able to reproduce features of the HRD-FE and some of the HRDP (with some small related tools).
Features of erdle can be divided into two main areas:
- reporting and debugging packets
- receiving VCUDS and forwarding HRDL packets
erdle comes with a large set of commands that ranges from forwarding reassembled HRDL packets to counting cadus present in a set of files, eg:
- relay
- store
- list
- index
- and many others...
At the core of erdle comes the re-assembling of HRDL packets from a bunch of VCDU packets. To perform this task, erdle can use a set of files or UDP packets stream. the procedure is as follows:
- find a synchronization word (0xf2e83553)
- fill a buffer
- stop filling the buffer when another synchronization word is found
- perform bytes unstuffing if needed
- compare the length of the buffer with the length given in the HRDL header
- if there are not enough bytes, discard the packet
- if there are too many bytes, discard the trailing zeros (that should come from 1-4 VCDU fillers)
- otherwise do nothing and go to the next step
- optionally, verify that the checksum found in the trailer of the HRDL packet matches the calculated one. If the checksums mismatch discard the packet if requested
- repeat the process until end of stream
erdle relay
The relay command can be used as a proxy between a source sending a stream of
VCDU and a destination that is only expecting a stream of HRDL packets.
The role of this command will then be to reassemble HRDL packets from incoming VCDU packets and then send all complete HRDL packets (valid or not - depending of the configuration) to the destination. However, reassembling of one HRDL packet can be aborted when one of the following condition is met:
- missing VCDU and/or unordered VCDU packets
- invalid VCDU CRC
If one of these conditions is met, only the current HRDL buffer is discarded by
the relay. After having discarded the current buffer, it starts to search for
the synchronization word of the next HRDL packet.
The following options can be given to the relay command:
-c use given configuration file to load options
-b BUFFER size of buffer between incoming cadus and reassembler
-q SIZE size of the queue to store reassembled HRDL packets
-i INSTANCE hadock instance
-r RATE outgoing bandwidth rate
-c CONN number of connections to open to remote host
-k don't relay invalid HRDL packets
A configuration file (using toml) can also be use instead of the command line options if multiple instance of this command should runned simulatenously (eg when they have to be managed by systemd):
# sample configuration file for relay command
# incoming cadus
local = "udp://0.0.0.0:11001" # unicast and multicast address are supported
buffer = 67108864
queue = 1024
keep = false
# outgoing hrdl
remote = "tcp://127.0.0.1:10015"
instance = 255
rate = 4194304
connections = 16
Note that configured options will overwrite options given on the command line.
erdle store
the store command of erdle can be used, as its name implies, to store under a
dedicated directory, the two kind of packets that erdle can deal with (but not
both at the same time):
- HRDL packets
- VCDU packets
According to the type of packets to be written on disk, this command will adapt the format used to store these in files. The file format is mainly a binary flat file with some extra headers (specific to the configured type) added before each packets.
HRDL packets will be store as the HRDP does for this kind of packets in rt files. This way existing tools able to decode rt files can also be used to decode and/or manipulate files created by the store command.
VCDU packets will be store as the HRD-FE does when using its built-in dump capabilities.
Files written by this command are automatically rotated according to one of the following conditions (depending also of the options set):
- number of packets written reached threshold set
- number of bytes written reached threshold set
- time interval elapsed between two rotations
- timeout since last write
The following options can be given to the store command:
-c use given configuration file to load options
-i INTERVAL time between automatic file rotation
-t TIMEOUT timeout before forcing file rotation
-s SIZE max size (in bytes) of a file before triggering a rotation
-c COUNT max number of packets in a file before triggering a rotation
-b BUFFER size of buffer between incoming cadus and reassembler
-p PAYLOAD identifier of source payload
-q SIZE size of the queue to store reassemble packets
-k store HRDL packets even if they are corrupted
A configuration file (using toml) can also be use instead of the command line options if multiple instance of this command should runned simulatenously (eg when they have to be managed by systemd):
# sample configuration file for store command
address = "udp://:10015" # unicast and multicast address are supported
datadir = "var/hrdp/vmu"
[hrdl]
# to store VCDU instead of HRDL packets, set the value to the payload to 0 or comment it
payload = 2
buffer = 67108864
queue = 1024
keep = false
[storage]
interval = 300
timeout = 10
maxsize = 0 # only timeout or interval rotation
maxcount = 0 # only timeout or interval rotation
Note that configured options will overwrite options given on the command line.
erdle inspect, index, list, count
this group of commands can be used to get various information about the status of a dataset of VCDU and how HRDL packets will or has been received from a dataset.
the inspect command can give the number of HRDL packets (and their total size)
that will be reassembled at a specific transmission rate and some stats about the
VCDU packets used to reassembled the HRDL packets (missing cadus, fillers,...).
4096 cadus (4032KB), 0 missing, 0 invalid, 0 filler, 75 packets (avg: 53KB, sum: 4020KB)
4096 cadus (4032KB), 0 missing, 0 invalid, 0 filler, 76 packets (avg: 53KB, sum: 4034KB)
4096 cadus (4032KB), 0 missing, 0 invalid, 0 filler, 76 packets (avg: 52KB, sum: 4024KB)
4096 cadus (4026KB), 0 missing, 0 invalid, 6 filler, 75 packets (avg: 53KB, sum: 4016KB)
4096 cadus (4026KB), 0 missing, 0 invalid, 6 filler, 78 packets (avg: 51KB, sum: 3998KB)
the index command creates a link between each HRDL packets and the VCDU packets
needed to reassembled.
763 || 9.92966208s | 40672 | 3558573 | 0 || 14152 | 03 | 3904246 | 2018-09-27 17:25:29.012
764 || 9.93308004s | 40686 | 3558587 | 0 || 70892 | 03 | 3904247 | 2018-09-27 17:25:29.012
765 || 9.95016984s | 40756 | 3558657 | 0 || 78734 | 03 | 3904248 | 2018-09-27 17:25:29.013
766 || 9.96921276s | 40834 | 3558735 | 0 || 14152 | 03 | 3904249 | 2018-09-27 17:25:29.013
767 || 9.97263072s | 40848 | 3558749 | 0 || 78712 | 03 | 3904250 | 2018-09-27 17:25:29.013
768 || 9.99167364s | 40926 | 3558827 | 0 || 70848 | 03 | 3904251 | 2018-09-27 17:25:29.013
769 || 10.00900758s | 40997 | 3558898 | 0 || 14152 | 03 | 3904252 | 2018-09-27 17:25:29.014
770 || 10.01242554s | 41011 | 3558912 | 0 || 70870 | 03 | 3904253 | 2018-09-27 17:25:29.014
771 || 10.02951534s | 41081 | 3558982 | 0 || 78712 | 03 | 3904254 | 2018-09-27 17:25:29.014
the list command prints for each HRDL packets reassembled some of their headers and if they are not corrupted according to the HRDL checksum.
the count command gives the number of VCDU or HRDL packets found in a dataset.
additional standalone commands
in addition to providing the erdle command (and its set of own commands), the
erdle package provides also additional standalone commands that allow to manipulate
VCDU packets but in different ways.
the cacat command allows to take a set of files containing VCDU packets and
concatenate them to form a larger "predictable" set of VCDU packets for, eg, later
replay sessions.
the calist command is usefull to troubleshoot a stream of VCDU captured with
a tcpdump directly on the server where the capture has been made without having
to copy this to another server/workstation that has a, eg, wireshark (GUI) installed
$ calist -l /tmp/capture.pcap
# rows removed
270 | 230.681ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1368 | 3
271 | 230.942ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1368 | 3
272 | 231.203ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1368 | 2
273 | 231.374ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1368 | 2
274 | 231.569ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1024 | 3
275 | 239.264ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1024 | 0
276 | 239.414ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1368 | 0
277 | 239.616ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1368 | 2
278 | 239.798ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1368 | 4
279 | 240.15ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1368 | 3
280 | 240.42ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1024 | 4
281 | 240.743ms | 2018-10-10T09:33:41Z | 127.0.0.1:54322 | 127.0.0.1:10001 | TCP | 1368 | 0
281 cadus (expected: 836 cadus), 192 gaps (165.555ms), 555 missing (66.39%), 345KB
$ calist -g /tmp/capture.pcap
# rows removed
176 | 220.873ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128957 | 4128960 | 3
177 | 221.131ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128960 | 4128964 | 4
178 | 229.321ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128967 | 4128969 | 2
179 | 229.498ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128969 | 4128972 | 3
180 | 229.863ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128973 | 4128975 | 2
181 | 230.044ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128975 | 4128977 | 2
182 | 230.241ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128977 | 4128980 | 3
183 | 230.485ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128980 | 4128982 | 2
184 | 230.681ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128982 | 4128985 | 3
185 | 230.942ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128985 | 4128988 | 3
186 | 231.203ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128988 | 4128990 | 2
187 | 231.374ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128990 | 4128992 | 2
188 | 231.569ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128992 | 4128995 | 3
189 | 239.616ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128997 | 4128999 | 2
190 | 239.798ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4128999 | 4129003 | 4
191 | 240.15ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4129003 | 4129006 | 3
192 | 240.42ms | 2018-10-10T09:33:41Z | 2018-10-10T09:33:41Z | 4129006 | 4129010 | 4
281 cadus (expected: 836 cadus), 192 gaps (165.555ms), 555 missing (66.39%), 345KB
Documentation
¶
Index ¶
- Constants
- Variables
- func CaduReader(r io.Reader, skip int) io.Reader
- func IsCRCError(err error) bool
- func IsCaduError(err error) bool
- func IsMissingCadu(err error) (int, bool)
- func StuffBytes(bs []byte) []byte
- func Sum(bs []byte) uint16
- func SumVCDU() hash.Hash32
- func Unstuff(bs []byte) (int, []byte)
- func UnstuffBytes(src, dst []byte) int
- func VCDUReader(r io.Reader, skip int) io.Reader
- type CRCError
- type MissingCaduError
Constants ¶
const ( WordLen = 4 MagicLen = 4 )
const ( CaduBodyLen = 1008 CaduLen = 1024 CaduHeaderLen = 14 CaduTrailerLen = 2 CaduTrailerIndex = CaduHeaderLen + CaduBodyLen CaduCounterMask = 0xFFFFFF CaduCounterMax = CaduCounterMask )
Variables ¶
var ( Word = []byte{0xf8, 0x2e, 0x35, 0x53} Stuff = []byte{0xf8, 0x2e, 0x35, 0xaa} Magic = []byte{0x1a, 0xcf, 0xfc, 0x1d} )
var ErrMagic = errors.New("cadu: invalid magic")
Functions ¶
func IsCRCError ¶
func IsCaduError ¶
func IsMissingCadu ¶
func StuffBytes ¶
func UnstuffBytes ¶
Types ¶
type MissingCaduError ¶
type MissingCaduError struct {
From, To uint32
}
func (MissingCaduError) Error ¶
func (e MissingCaduError) Error() string
Source Files
Directories