Log files

Everything this package produces is a file in one directory. There is no database, no index and no metadata anywhere else: the filename carries all of it. This chapter is how to read a directory listing and know what you are looking at.

Where they live

/var/lib/muxen/datalogger/

The directory is created by the package. Both recorders write into it and nothing else does.

It is not configurable from the systemd units, which pass only --interface. Run a recorder by hand with --logFolder <path> to put captures somewhere else — a USB stick during a diagnostic session, for instance. The folder must already exist and be readable and writable; the recorder does not create it, and exits at startup if either test fails.

What a listing looks like

ls -l /var/lib/muxen/datalogger/

can0.1.log.gz
can0.2.log.gz
can0.3.log
can0_device_5_0_1.1.log.gz
can0_device_5_0_1.2.log
can0_device_7_0_1.1.log

Three things are readable at a glance:

• .log is the file being written now. .log.gz is finished. A directory has at most one open .log per series.
• The name starts with the interface, so two buses never collide in one folder.
• _device_ in the middle means a decoded CSV; without it, a raw BUSMASTER capture.

Raw captures — muxen-can-datalogger

<logFolder>/<interface>.<index>.log

Part	Example	From
<interface>	can0	the --interface argument, which is the unit instance
<index>	3	a counter, one per file, never reused

One file at a time. Every frame the interface delivers goes into it, in the order it arrived.

Decoded captures — muxen-csv-datalogger

<logFolder>/<interface>_device_<function>_<instance>_<broadcastId>.<index>.log

Part	Example	From
<function>	5	the source function code in the CAN identifier
<instance>	0	the source instance in the CAN identifier
<broadcastId>	1	the broadcast frame identifier
<index>	2	a counter, per file series

The three numbers are extracted from every received identifier, and each distinct combination gets its own file. So can0_device_5_0_1 is "function 5, instance 0, broadcast frame 1" — one device, one kind of frame it sends. A battery that sends three different frames produces three files; two batteries produce six.

Files are created lazily, on the first frame of that combination, and stay open for the life of the process. That has two consequences worth knowing before enabling this recorder on a busy boat:

• The number of open files equals the number of distinct broadcast identifiers on the bus. On a real MUXEN installation that is routinely several dozen.
• Combinations the recorder cannot decode still get a file. An unrecognised frame produces a header line of datetime, and rows containing only a timestamp and an empty field. The file is small but it is real, and it rotates and compresses like any other.

Note the extension: these are CSV files named .log.

Working out which device a file is

The numbers come from the CAN identifier, so the fastest way to map a file back to hardware is to watch the bus and compare:

candump can0 | head -40

A frame's identifier decomposes into the same three fields the filename uses. If you already know the device — from the boat's configuration or from the MQTT device/<function>/<instance>/… topics published by muxen-boat — the first two numbers in the filename are that same function and instance. The third distinguishes the different frames one device sends.

The list of frame kinds this recorder decodes, by name, is in File formats.

Choosing the index

At startup each recorder reads the log folder, matches every file against its own naming pattern, takes the highest index it finds and adds one. It reports the result:

busmaster: next log file index is 4
csv: next log file index is 2

Two properties follow, and both matter operationally:

• A restart never overwrites an existing capture. The scan counts compressed files too, so can0.3.log.gz still reserves index 3.
• Deleting old files lowers the next index. Remove everything and the counter starts again at 1. A retention job that deletes by age therefore recycles indices; the file's modification time, not its index, is what orders a directory across a long deployment.

The counter is a 16-bit value, so indices run to 65535.

Rotation

A file is closed and a new one started when either limit is reached:

Limit	Option	Default
Age of the file	--maxRecordTime <seconds>	10800 (3 hours)
Size of the file	--maxFileSizeBeforeCompression <bytes>	10485760 (10 MiB)

Both are per file, not per series. For the CSV recorder that means per device-and-frame file: each one ages independently from its own first frame.

Three details decide what actually happens on a boat:

Rotation is only evaluated when a frame arrives. The check runs at the top of the write path. A bus that goes quiet leaves the current file open, uncompressed and past its age limit for as long as the silence lasts — the rotation happens on the first frame after it, not at the moment the limit passes.

The age is measured against the frame's own timestamp. The recorder compares the arriving frame's time against the time the file was opened, so a Brain whose clock jumps forward — an NTP correction after boot, typically — rotates immediately, and one whose clock jumps back stops rotating on age until real time catches up.

Whichever limit is reached first wins. On a bus carrying 30 to 40 frames per second the 10 MiB default is reached in under two hours, so the size limit is the one doing the work and the 3 hour limit never fires. On a nearly idle bus it is the other way round. Disk budget turns that into numbers.

The CAN recorder also reserves 79 bytes for the closing lines, so its effective size limit is 79 bytes below the configured one.

Compression

Immediately after closing a file, the recorder starts /usr/bin/gzip --best on it as a separate process. can0.3.log becomes can0.3.log.gz; the recorder does not wait for it and carries on writing the next file.

Two things this means in practice:

• Compression is unconditional. The --gzip and --no-gzip options exist in the command line and are parsed, but the resulting setting is never consulted. There is no way to turn compression off.
• /usr/bin/gzip must exist. If the exec fails the child process does not stop; it continues running as a duplicate recorder. See gzip is not optional in Getting started.

MUXEN bus traffic compresses well. Measured on a real capture, gzip --best takes a BUSMASTER log to about 13 % of its size, roughly 7:1.

The life of one file

For the raw recorder, start to finish:

1. Open. fopen(path, "w+"). If it fails the recorder prints busmaster: failed to create file: <path> and exits.
2. Header. Eleven fixed lines identifying the format, the start date and time in UTC, and the column legend.
3. Rows. One line per frame, flushed as it is written.
4. Footer. Two lines: the end date and time, and a stop marker.
5. Close, then compress.

Steps 4 and 5 happen on rotation and on a clean shutdown. They do not happen on SIGKILL or on a power cut: that file keeps its .log name, has no closing lines, and stays uncompressed. It is still perfectly readable — it is just truncated at whatever the last flushed frame was.

The CSV recorder is the same without the footer: header row, data rows, close, compress.

Durability

Every row is written with fflush() as soon as it is formatted. That pushes it out of the process and into the kernel, so:

• Reading the open file gives you everything up to the last frame.tail -f on a live capture works.
• Killing the process loses nothing. The kernel still has the data and writes it out.
• A power cut loses whatever the kernel had not yet written to storage. There is no fsync() anywhere; the code is explicit that it flushes the library buffer and not the filesystem. On a boat that loses power without a shutdown, expect the last seconds of the open capture to be missing.

Rotated and compressed files are not at risk — they were closed and rewritten by gzip long before.

Running both recorders in one folder

The two naming patterns cannot match each other, so their index scans stay independent and their files never collide. Running both on the same interface is therefore safe as far as the files are concerned — it simply costs twice the disk.

Running two instances of the same recorder on the same interface into the same folder is not safe: both compute the same next index and open the same path, and the capture is corrupted for as long as they overlap. That is easy to do by accident with the shipped units; see the warning in Getting started.