Troubleshooting

This chapter is organised by what you see. The daemon has no control socket and no status command: everything it knows it prints at startup or publishes on MQTT, so those two are the whole toolbox.

journalctl -u muxen-sensors -n 100                    # the startup log and any errors
journalctl -u muxen-sensors -f                        # follow it
systemctl status muxen-sensors
mosquitto_sub -h 127.0.0.1 -t 'app/sensor/#' -v       # what it publishes
mosquitto_sub -h 127.0.0.1 -t 'app/sensor/info' -v   # version, online, data source
mosquitto_sub -h 127.0.0.1 -t 'device/1/+/io' -v      # what a Bloc 8 reports
candump can0                                          # the bus itself

No app/sensor/… topic at all

Nothing is published for any sensor.

Likely cause	What to check	Fix
It publishes under another prefix	config: topic prefix = … in the journal	subscribe to that prefix, or set MUXEN_TOPIC_PREFIX in a drop-in
The daemon is not running	systemctl status muxen-sensors	see the service restarts every 30 seconds below
The broker refused the connection	no main: mqtt connected line	the daemon exits when it cannot connect at startup; check the broker
No entry survived the configuration file	sensor: no usable sensor out of N	read the per-entry reasons printed above that line
Nothing has ever been received	sensors: N variables is right, and the journal is otherwise quiet	see a variable never appears below

A sensor that has never received a frame publishes nothing at all. It does not publish a zero, and it does not publish an error. Silence on app/sensor/# with a healthy daemon means no data is reaching it — or that the daemon was started with a different --prefix. config: topic prefix = … in the startup log says which one; check that first, because everything below assumes you are subscribed to the topic it actually publishes.

The service restarts every 30 seconds

Restart=always with RestartSec=30, so a daemon that fails at startup loops on that period. The journal names the reason on each attempt:

Message	Cause	Fix
config: --sensors is required	MUXEN_DEPLOY expanded to nothing	check a drop-in has not overridden it to an empty value
config: --interface is required unless --mqtt is used	ExecStart was replaced by a drop-in that dropped --interface	restore it, or pass --mqtt
sensor: failed to open '<path>'	the file does not exist, or user muxen cannot read it	check the path and its permissions
sensor: failed to parse the JSON	the file is not valid JSON	validate it
sensor: '<path>' has no 'sensors' key	the right file, the wrong shape — or the wrong file	the array must be under the root key sensors
sensor: no usable sensor out of N	every entry was refused	the reasons are printed one per entry above
init: Failed to load the sensors configuration file	follows any of the four lines above	as above
init: failed to start the CAN monitor	the netlink socket could not be created	rare; report it
nothing after the config: lines	the MQTT broker is not reachable on 127.0.0.1:1883	systemctl status mosquitto

The first two exit with status 1 and print the usage block. The rest exit with status 255.

A missing or down CAN interface is not one of these. Since the daemon supervises the interface rather than requiring it at startup, it stays up, prints can: can0 is DOWN, and starts publishing as soon as the interface appears.

One variable is missing, the rest are fine

The entry was refused when the file was read. Look for its reason above the sensor: failed to load sensors[N], ignoring... line:

Message	Cause
sensor: output name missing	the entry has no name
sensor: output name '<n>' contains invalid MQTT characters	the name contains /, #, + or a space
sensor: input missing / sensor: output missing	the input or output object is absent
sensor: input.deviceType (N) is not supported	not 1, 3 or 17 — including the 0 you get by omitting the key
sensor: input.deviceType must be 0-63 / sensor: input.deviceInstance must be 0-63	the id does not fit its 6 bits on the bus
sensor: input.deviceChannel is unknown (<name>)	the channel name does not exist for that device type
sensor: scaleType '<s>' is unknown	not identity, polynomial, linear or step
sensor: polynomial key is missing (or step, linear)	a scaleType was declared without its companion key
polynomial: order too high (N), max = 5	more than five coefficients
points: count are too high (N), max = 32	more than 32 points
points: point (N) must be an array of two double	a point is not a [reading, result] pair
points: point (N) input (x) must be less than point (M) input (y)	the points are not in increasing order of reading
points: array is empty	an empty linear or step table

sensor: 6 of 7 sensors loaded at the end confirms how many survived.

A variable never appears, but the daemon is healthy

The entry loaded — it is in sensors: N variables — and nothing is ever published on its topic. Nothing has arrived for it. Work through this in order:

1. Is the daemon reading the source you think? config: data source = says can or mqtt. In --mqtt mode the CAN table is empty by design, and the daemon depends entirely on another service publishing the device/… topics.

2. Is the interface up? can: can0 is UP (state = ERROR_ACTIVE, …) is the healthy line. DOWN means the interface is absent or down; BUS_OFF means the controller has taken itself off the bus. The daemon retries forever and rebuilds its socket on each recovery.

3. Does the frame it is waiting for exist? The dispatcher table printed at startup gives the exact CAN identifier, mask and length per variable:

can: canId   	canMask 	canDlc  	name
can: 0x002041	0xFFFFFF	7       	FreshWaterPortside

candump can0,002041:FFFFFF

Nothing there means the device is not sending that frame, or is sending it at a different instance.

4. Is the instance right? This is the most common cause. A device configured at instance 1 while the unit on the bus is at instance 0 produces a subscription that never matches, and looks exactly like a silent device. The instance is the low bits of the identifier in the table above.

5. On a Bloc 8, was the request issued? A Bloc 8 does not broadcast its input frame spontaneously — the daemon asks for it with an RTR request each time the CAN interface comes up. If the interface never came up, the request was never sent. In --mqtt mode this daemon sends no CAN traffic at all, so the frame has to be solicited elsewhere.

The value is wrong, but it is moving

The variable publishes and reacts, and the figure is not the right one. Compare the two numbers in the payload before touching anything:

mosquitto_sub -h 127.0.0.1 -t 'app/sensor/FreshWaterPortside' -v

• input.value is wrong too — the problem is upstream of this daemon: the sender, the wiring, or the wrong channel. Check input.channel and input.functionCode against the installation.
• input.value is right, data.value is not — the problem is the calibration. See Calibration and filtering.

A calibration is usually wrong in one of four ways:

Symptom	Cause
right at one end, wrong in the middle	a straight line fitted to a tank that is not a box — use linear with measured points
off by a constant everywhere	the sender's offset is missing from the curve
proportionally off	the gain is wrong, or the tank's real capacity is not what the drawing says
right on a level boat, wrong under way	the points were measured heeled, or the tank needs a filter

The value is pinned at 0, or at the maximum

A clamp is doing its job on a reading that is outside the declared range.

1. Compare input.value against input.min and input.max, all three of which are in the payload. input.value is published unclamped, so a reading sitting outside those limits is one the input clamp is flattening: the sender is producing something the configuration says it cannot. A disconnected probe reading zero, or a shorted one reading full scale, both land here.
2. If data.value sits exactly on output.min or output.max, the curve is producing a result outside the declared limits. That is the output clamp protecting the display, and the curve is what needs fixing.

A gauge stuck at full is worth treating as a fault rather than a reading: both clamps are designed to make a broken sender obvious rather than plausible.

The value is frozen

The topic is there, the number never changes.

The daemon republishes every second whether or not new data arrived, so a live topic does not mean a live sensor. metadata.rxdate is the discriminator: it carries the time the last frame was received, not the time of publication.

mosquitto_sub -h 127.0.0.1 -t 'app/sensor/FreshWaterPortside' -v

• rxdate advances — the sensor is alive and the reading really is constant.
• rxdate is stuck in the past — nothing has been received since then. The payload also carries expireAfterSec: 30, which is how a screen is meant to age the reading out.
• Nothing arrives at all — the daemon is not publishing; go back to no app/sensor/… topic at all.

If a variable you removed from the configuration is still visible after a restart, a copy of its last message is being kept by the broker. Clear it:

mosquitto_pub -h 127.0.0.1 -t app/sensor/OldName -r -n

Two variables read the same number

Both entries point at the same device, instance and channel. Check input.functionCode, input.instance and input.channel in the two payloads — they are published for exactly this.

The usual cause is an omitted deviceChannel: an entry without it reads channel 0 of the device, whatever that is. Two entries that both left it out read the same channel.

Two entries sharing a name produce the same effect from the other direction: nothing checks names for uniqueness, so both publish to one topic, alternately, once a second each.

The reading flickers

A jittery number on the screen, moving with the boat rather than with the level.

Cause	Fix
The tank is moving	add liquid.filterPeriod — see Calibration and filtering
The sender is electrically noisy	a filter helps; so does the wiring
A step curve near a threshold	the value flips between two steps; either add intermediate points or use linear

value is not finite

mqtt: <name> value is not finite (raw = nan, value = nan), not publishing

The scaled value came out as infinity or not-a-number, and the daemon drops the sample rather than publish JSON that no consumer can parse. The line is printed once, on the transition, not once per second.

Causes: a device reporting a non-numeric value, or a curve that overflows — a high-order polynomial with a large coefficient and a large reading. Bound the input with input.min / input.max and check the curve.

--mqtt mode publishes nothing

In MQTT mode the daemon subscribes to one topic per variable, listed in the table it prints at startup:

mqtt: dispatcher table size: 1
mqtt: regex                 	pdata   	name
mqtt: ^device/1/1/io$       	0x…     	FreshWaterPortside

Check the topic actually exists, with the same instance:

mosquitto_sub -h 127.0.0.1 -t 'device/#' -v

Nothing on device/… means the service that publishes them is not running — the package Recommends: muxen-boat for that reason. Note that in this mode muxen-sensors transmits nothing on CAN, so a Bloc 8 that is waiting to be asked for its input frame will not be asked by this daemon.

A calibration session shows nothing

The live session in raken stays empty, or the reading never moves. The daemon answers every request, including the ones it refuses, so the first thing to do is read the answer:

mosquitto_sub -h 127.0.0.1 -t 'app/sensor/calibration/response' -v

What the answer says	Cause	Fix
nothing at all	the daemon predates the feature, or is not running	needs muxen-sensors 5.0.0 or later; check systemctl status
"state": "no-data"	nothing has been received on that channel for five seconds	wrong deviceType / deviceInstance / deviceChannel, or muxen-boat is not publishing device/…
"state": "rejected"	the candidate curve is not usable	error carries the daemon's own words — points out of order, an unsupported device type, too many points
"state": "expired"	no refresh arrived before the ttl ran out	the page stopped answering; reopen the session
"error": "too many active calibration sessions"	four sessions are already open	close one, or wait for its ttl — the limit is four

A session reads the device/… topics rather than this daemon's own input, so it works on a channel no variable is configured for — that is the point of it. It also means muxen-boat has to be publishing those topics even when this daemon runs in CAN mode, and that a session on a Bloc 8 nobody is reading will be quiet until the daemon starts asking for it, which takes a second.

Neither calibration topic is retained: subscribing after the fact shows nothing, and there is no state to clear. --prefix does not move them — they are app/sensor/calibration/… on every install.

To drive one by hand, see Watching one by hand in Calibration and filtering.

Getting more detail

journalctl -u muxen-sensors -n 200                 # the whole startup, including both tables
sudo -u muxen /usr/bin/muxen-sensors -v -i can0 -s /etc/muxen/deploy.json

-v prints the full interpretation of every entry — channel, limits, curve, every coefficient and every point. That tree is the fastest way to confirm that the file says what its author meant. Stop the service first, or the two instances compete for the same MQTT client identifier.

-vv adds a trace of every CAN frame and every MQTT message. On a live bus it is a firehose; use it for a few seconds at a time.

FAQ

Why is my tank level wrong? Look at the message the daemon publishes for that tank: it carries both the reading it got from the sender (input.value) and the litres it made of it (data.value). If the reading is wrong, the problem is the sender or its wiring. If the reading is right and the litres are not, the tank's calibration is wrong and needs an installer — the conversion lives in the boat's configuration file, not in the sensor.

Why does my tank read a little low all the time? Deliberately. A tank fitted with a filter averages the lower part of the recent readings, so the figure errs under rather than over. A gauge that promises more water than the boat has is worse than one that promises less.

I filled the tank and the gauge took half a minute to move. Is something broken? No. A filtered tank ignores short-lived changes so that waves and heel do not make the gauge dance, and the arithmetic makes it slower to rise than to fall. With the usual one-minute setting a fill takes about 30 seconds to start showing and about 45 to settle. Draining shows up in about half that.

The gauge is right when the tank is full and wrong when it is half empty. The calibration was made with two points and a straight line, and the tank is not a rectangular box. Hull tanks are wider at the top or narrower at the bottom, so the same centimetre of level is not the same number of litres everywhere. The fix is to measure the tank in steps and record the real points — an installer job, done once.

The tank is empty but the gauge still shows something. Either the sender does not go all the way down — many do not — or the lowest calibration point was taken with liquid still in the tank. Both are fixed by re-measuring the empty point.

My tank shows 100 % and never moves. Treat that as a fault, not as a reading. It is what a disconnected or shorted sender looks like: the value has hit the limit declared for it and is being held there.

A gauge disappeared from the screen after somebody edited the configuration. Two possibilities. The variable was renamed — the name is the address the screens subscribe to, so renaming it makes the old one vanish. Or its entry was refused when the file was read, in which case the journal names the reason and every other sensor is unaffected.

Two tanks show the same level. They are configured to read the same input. It is usually a missing channel setting: an entry that does not say which channel to read falls back to the first one.

Why does the number keep updating after I unplugged the sensor? The daemon republishes its last value once a second regardless. Each message carries the time the reading was actually taken, and a validity of 30 seconds, so a screen that honours it stops trusting the figure. A screen that ignores it shows a plausible value forever.

Do I have to restart anything after changing the configuration? The daemon reads the file only at startup, so it does have to restart — but that normally happens by itself. muxen-systemd, which this package depends on, watches /etc/muxen/deploy.json and restarts muxen-deploy.target when the file changes, and this daemon is part of that target. systemctl restart muxen-sensors does the same thing directly.

Can it work without a CAN bus? Yes — --mqtt takes the same readings from the device/… MQTT topics instead of the bus. It needs another service to publish them, and in that mode this daemon transmits nothing on the bus at all.

Does it ever command anything? No. The one thing it transmits is a request asking a Bloc 8 to start reporting its inputs. It never switches an output, never writes a configuration, never commands a device.

What unit will the screen show? Whatever string the configuration puts in output.unit. The daemon does not interpret it and does not convert anything — a curve that produces litres must be paired with a unit that says litres.

Why does one sensor have a raw field and another does not? Only filtered variables publish both: data.value is the smoothed figure and data.raw is the same figure before smoothing. Unfiltered variables have one value and no raw.

Tips

Run the daemon by hand with -v at commissioning and read the tree it prints. Every sensor should be there, with the channel, the limits and the curve you intended. It is faster than diagnosing a published payload afterwards.

Always declare deviceChannel explicitly. Omitting it does not disable the input — it reads channel 0, which is a digital input on a Bloc 8 and a relay on a Generic I/O. That is rarely what anyone meant.

Set input.min and input.max to what the sender really produces. They are the boat's only protection against a broken sender being pushed through a curve and published as a believable number.

Set output.min and output.max to the tank's real capacity. They bound the display, and they are what a screen uses to draw a percentage. Both are only carried in the payload when they are set.

Calibrate with the boat level and settled, and measure in steps across the whole range. Points measured heeled put the heel into every future reading.

Prefer linear with measured points over a fitted polynomial. A high-order polynomial passes exactly through the points you gave it and is wrong everywhere else, credibly enough that nobody notices for a season.

Choose variable names once. The name is the topic. Renaming one at a later refit silently disconnects every screen, alarm and logger that subscribed to the old name.

Do not put /, #, + or a space in a name. They are refused outright, and the sensor is skipped.

Filter tanks, not currents. A filter costs response time. It is right for a liquid level that moves with the boat, and wrong for anything an alarm has to react to quickly.

Record the calibration points at handover. They are the only record of how the gauge was made; re-deriving them later means emptying the tank again.

Compare rxdate against the clock when in doubt. A published value proves the daemon is alive. Only rxdate proves the sensor is.

Watch the count line after every configuration change.sensors: N variables, and sensor: X of Y sensors loaded when they differ, is the only place a silently dropped entry is announced.