Troubleshooting

This chapter is about the tools themselves — what to do when the instrument, rather than the boat, is the thing not working. It is organised by what you see.

These tools have no service, no journal and no log file. Everything they know is on the screen, and everything they cannot do they say on standard error before exiting.

muxen-diag-io --version                 # is it installed, and which build
muxen-diag-io -h 127.0.0.1 2>&1 | head  # the error, if it will not start
ip link show can0                       # is the CAN interface up

One thing to hold on to throughout: four of the five tools cannot change anything. muxen-diag-io, muxen-diag-voltage, muxen-diag-sfsp and muxen-diag-data-explorer only subscribe and draw. Nothing you press in them can make a boat worse. Only muxen-diag-devices acts, and only on its action keys.

error: cannot connect to MQTT broker at <host>:<port>

The tool exited immediately instead of showing an empty table. That is deliberate: a failed first connection leaves nothing to retry on, so sitting on "Disconnected" forever would be a lie.

Likely cause	What to check	Fix
Wrong address	the -h you typed	the Brain's address on that network
The broker is not running	systemctl status mosquitto on the Brain	start it
The broker is not listening off-host	connect from the Brain itself with no -h	if that works, the broker is bound to localhost only
A firewall between laptop and boat	port 1883 to the Brain	open it, or run the tool on the Brain over SSH

muxen-diag-sfsp in MQTT mode reports it differently, because it stays running: mqtt: connection failed (rc=…) on standard error, and then no events.

Error: bus scan failed (is muxen-uds installed?)

muxen-diag-devices could not complete its initial scan and refused to start. It runs muxen-uds as a child process for every bus operation, so this is a failure of that call.

1. Is muxen-uds on PATH? muxen-uds --version. The package Depends: muxen-uds (>= 9.8.2), so this should not happen on a normally installed Brain, but a manually installed binary or an unusual PATH can produce it.
2. Does the CAN interface exist? ip link show can0, or whatever -i named.
3. Does a scan work by hand? Run muxen-uds directly on the same interface. If it fails there, the problem is not in this tool.
4. Is /tmp writable? The scan is passed between the two programs through a temporary file, created with mkstemp under /tmp and removed immediately afterwards. A full or read-only /tmp fails here.

The table is empty — "Waiting for devices…"

The tool connected to the broker and nothing has arrived. The title bar says Connected, so the broker is reachable; it simply has no topics this tool cares about.

Likely cause	What to check
Nothing is decoding the bus onto MQTT	run muxen-diag-data-explorer — three empty branches means the broker is genuinely empty
The devices are not on the bus	run muxen-diag-devices
The devices are on the bus but of families this tool ignores	muxen-diag-io shows only Bloc 8 and Generic I/O; muxen-diag-voltage only the twelve electrical families
The broker is the wrong one	check the address in the title bar

The order that eliminates most: data-explorer first (is anything there at all), then devices (is the hardware there at all).

"No matching devices" instead of "Waiting for devices…"

Devices were received; the filter excluded all of them. Press Esc.

The filter matches the displayed name, and the displayed instance is counted from 1. Searching 5 matches Battery #5, which is the device at instance 4 on the bus. To search by topic, include a / or the word device: device/5/4 matches the topic and nothing else.

[!] need 123 cols in the title bar

The terminal is narrower than the layout, and the columns past the right edge are not being drawn. Nothing is wrong with the data; you are not seeing all of it.

Tool	Needs
muxen-diag-io	123 columns
muxen-diag-devices	105
muxen-diag-voltage	87

Widen the terminal, reduce the font, or turn off the extra columns with D in muxen-diag-voltage. Over SSH from a narrow window, resizing the local terminal is enough — the tools re-measure on every frame.

muxen-diag-data-explorer never warns; it reflows instead.

Every line is yellow

Yellow is the stale colour: nothing live has arrived for that device in thirty seconds. A whole screen of yellow means the tool is connected to a broker that has stopped being fed.

Two shapes, and they mean different things:

• The lines appeared and then went yellow. Data was flowing and stopped. The decoder or the bus stopped, not the broker.
• The lines were yellow from the start, with AGE showing -. These are retained messages sitting on the broker from a previous run. The broker is holding the last thing it was told, indefinitely, and every screen that subscribes sees the same plausible-looking history.

The second case is the one that catches people out, on the boat's own screens as much as here. muxen-diag-data-explorer settles it: select the topic and watch whether Updated advances.

A device shows in one tool and not another

This is normal and it is informative. Each tool cuts the chain at a different point:

Seen in devices	Seen in io / voltage	Conclusion
yes	yes	the whole chain to the broker is intact
yes	no	the board is on the bus and nothing is decoding it onto MQTT
no	yes	you are looking at retained or stale MQTT data for a board that has left the bus
no	no	the board is not on the bus

The third row deserves care: MQTT topics persist. A board removed from the boat can go on appearing in muxen-diag-voltage from a retained message for as long as nobody clears it.

Instance numbers disagree between tools

They do, and it is a fixed convention rather than a fault:

Where	The battery at bus instance 0 appears as
muxen-diag-voltage, muxen-diag-io	Battery #1
muxen-diag-devices	Inst column 0
muxen-diag-data-explorer	… (0), topic device/5/0/…
the boat's screens	counted from 1

The rule: the two table tools that mirror the screens count from 1; the two tools that talk about the bus count from 0. When you cross-check a device between them, subtract or add one.

read failed in muxen-diag-devices

The board answered the bus scan but its configuration read did not come back. It is present and it cannot be addressed individually.

Likely cause	How to confirm	Fix
An address collision	two rows with the same function and the same instance	see below
A parked board	the Inst column shows P	give it an instance with [I], which activates it by UID first
A firmware too old to answer	the board is old, and every other board reads fine	update it, which needs it individually addressable first
A transient bus error	press [E] — a failed read is retried on a rescan	none needed if it clears

A failed read is never sticky: [E] clears the flag and tries again. Press it once before diagnosing anything.

Two boards cannot be read — an address collision

Two devices presenting the same function and the same instance answer at the same address. While that lasts neither can be read or written individually, and both appear as read failed. It looks exactly like a dead segment.

It is the first thing to rule out, because nothing else in these tools can see the difference.

Confirming it. Take one of the pair off the bus, or power it down, and press [E]. If the remaining one starts reading normally, that was it.

Clearing it. [I] cannot repair the collision it is trying to fix, because setting an instance requires addressing the board individually. The ways out, in order of preference:

1. Isolate. Remove one board from the bus, give the other its instance with [I], put the first back and address it in turn. This is the reliable procedure.
2. Park one. [P] moves a board to instance 63 and out of the contended address.

A board that has never been given an instance sits at 63 by default. If several such boards are on the bus at once, the parking address itself is contended, and they have to be addressed one at a time.

[U] says No firmware available for <product>

There is no compatible firmware file on this Brain for that product. Compatible means the same major version — a board on 5.x is only ever offered 5.x.

1. Look under /usr/lib/muxen/firmware/<productCode>/. Files are named <productCode>-v<version>-<name>.srec.
2. If the directory holds only 6.x files and the board runs 5.x, the Latest column reads - and [U] refuses. That is the guard working, not a fault.
3. Neither the directory nor products.json is shipped by this package. Install or update whatever provides them on that Brain.

The Latest column shows (Not released firmware, it's CE !)

The board runs a firmware newer than anything on this Brain. That is a pre-release build on the board, usually from a bench session, and the yellow is a flag to check whether it was meant to stay there.

[U] refuses with Already at latest version, since it will not downgrade.

Firmware update FAILED on device 0x…

The muxen-uds upload returned a non-zero status. The tool ran it with the table suspended, so its full output is on screen above the Press ENTER line — read that first, it is the actual diagnosis.

Then check, in order:

1. Is the board still there? Press [E]. A board that answers normally afterwards did not take the firmware and is unharmed.
2. Is the file readable? The path is printed in the echoed command line.
3. Was the bus busy? An upload competing with heavy traffic can fail; retry on a quiet bus.

Instance written but reset FAILED — power-cycle the device

The instance was written to the board, but the reset that applies it did not go through. The board is still running at its old address with its new instance stored.

Power-cycle the board. Do not write the instance again — it is already there, and the second write against the old address only adds confusion.

After the power cycle, press [E] and confirm the board appears at the new instance.

Parked device 0x… — not seen parked after rescan

The park command succeeded but the board did not turn up at instance 63 in the rescan that followed. The tool waits five seconds for the reboot, which is usually enough and occasionally is not.

1. Press [E] again. A slow board often appears on the second rescan.
2. If it is still absent, check that it has power — a park is a reboot, and a board that failed to come back has a different problem.
3. If it reappears at its old instance, the park did not take. Retry.

[!] table full (N max) — some devices not shown

The tool's device table is full and the list you are looking at is incomplete. It says so rather than hiding devices silently.

Tool	Limit
muxen-diag-devices	128 device rows
muxen-diag-io	64 devices
muxen-diag-voltage	64 devices

In muxen-diag-io and muxen-diag-voltage the limit counts every device seen since the tool started, including any that have since gone quiet, so a long-running session on a busy boat can reach it. Restart the tool.

muxen-diag-sfsp prints nothing

Silence is its normal state — it prints only on change. Establish that it is working before blaming a switch:

1. MQTT mode: it prints mqtt: connected to <host>:<port> on standard error at startup. No such line means it never connected.
2. CAN mode: check ip link show can0, and that other traffic is on the bus.
3. Press a button. A pressed line and a released line should follow.
4. Try the other source. A press that appears in CAN mode and not in MQTT mode places the fault between the bus and the broker; one that appears in neither never reached the bus.

Only the first four buttons of a switch are reported. A fifth button producing nothing is expected behaviour here, not a fault.

muxen-diag-data-explorer shows instance=<n> instead of names

The instance level of the tree falls back to instance=<n> when it has no configured name for that device.

Likely cause	Check
--no-deploy was given	the command line
/etc/muxen/deploy.json is absent or unreadable	ls -l /etc/muxen/deploy.json
The device is not in the deployment file	look for its function and instance there
The device has no Name parameter	the device's parameters in that file

Running from a laptop, the deployment file is read from the laptop, not from the boat — so remote sessions normally show instance=<n> for everything. That is expected.

muxen-diag-data-explorer shows three empty branches

device, app and nmea are all present and all empty. The tool is connected and nothing at all is being published.

That is a broker-level finding, and it is the strongest one these tools produce: no MUXEN service on that Brain is publishing anything. Check that the decoding daemons are running before looking at any device.

muxen-diag-data-explorer shows an empty app > sensor branch

Everything else is publishing, but no tank or temperature shows up.

The tool subscribes to app/sensor/#, which is where muxen-sensors publishes since 5.0.0. An older daemon — or one pinned back with MUXEN_TOPIC_PREFIX=variable in a drop-in — publishes on variable/… instead, and nothing subscribes there any more.

Check	How
the daemon version	dpkg -l muxen-sensors
the pin	systemctl show muxen-sensors -p Environment
what is actually on the broker	mosquitto_sub -t 'variable/#' -v

Upgrade the daemon and drop the pin. The two have to move together: a reader on app/sensor/ and a publisher on variable/ see nothing of each other.

The payload pane is blank, or scrolled past its end

j scrolls the payload freely; the pane clamps so the last line cannot be pushed off the top. If it looks blank, the selected node is a branch rather than a leaf — branches have no payload and show (branch) as their topic.

Two of these tools disconnect each other from the broker

They do not. Each tool builds its MQTT client identifier from its own name, the host name and its process id, so several tools — in several terminals, on several laptops — connect to the same broker simultaneously without evicting one another.

If you are seeing a disconnect loop, something else is connecting with a colliding identifier.

There are no colours

The tools check whether the terminal supports colour and fall back to reverse video for the selected row if it does not. Everything still works; the state information carried by colour is then carried only by the text (ON, SC, NC, FAULT, ALARM).

Check TERM, and prefer a TERM your terminal actually is over TERM=dumb.

--help prints nothing when piped

Four of the five tools print their usage on standard error: muxen-diag-io, muxen-diag-voltage, muxen-diag-data-explorer and muxen-diag-devices. Only muxen-diag-sfsp prints it on standard output.

muxen-diag-io --help 2>&1 | less

All of them exit 0 for --help.

FAQ

Which tool do I open first?muxen-diag-devices if you suspect the hardware, because it answers a yes-or-no question about the board being there. muxen-diag-data-explorer if you suspect the data, because it shows everything without deciding for you what matters.

Can I break anything with these? Not with four of the five. muxen-diag-io, muxen-diag-voltage, muxen-diag-sfsp and muxen-diag-data-explorer only read. muxen-diag-devices can change a board — its firmware, its configuration, its address — and its [R] Reset key acts with no confirmation at all.

Can I run them while the boat is in use? The four read-only tools, yes; they add one MQTT subscriber, nothing more. muxen-diag-devices performs a bus scan and a configuration read per board, which puts real traffic on the bus, and its action keys reset and reprogram boards. Treat that one as a maintenance activity.

Can I run several at once? Yes. Each connects to the broker under its own identifier, so two terminals or two laptops do not interfere. Running muxen-diag-sfsp and muxen-diag-io side by side is the standard way to watch a switch press turn into an output.

Why does the same device have a different number in different tools? Because two of them count instances the way the screens do, from 1, and two count them the way the bus does, from 0. See Instance numbers disagree between tools above.

Why does a column show - and not 0.0?- means the value has never arrived; 0.0 means the device measured zero. Distinguishing them is the point — an inverter that is not reporting its AC side would otherwise look exactly like an inverter producing nothing.

Why is a line yellow when the numbers look fine? Yellow means nothing live has arrived for thirty seconds. The numbers on the line are the last ones received, and they may be old.

The tool shows the right value and the screen shows the wrong one. Which do I believe? The tool, and then muxen-diag-data-explorer to see the raw payload. If the payload holds the right value, the fault is above the broker and nothing in this package will fix it.

Can muxen-diag-devices run from my laptop? No. It needs a CAN interface on the machine it runs on, so it runs on the Brain. The MQTT tools are the ones that work remotely.

Do I need to be root? Nothing in this package asks for privilege of its own. Whether you need it depends on the access your account has to the CAN interface and, for muxen-diag-devices, on what muxen-uds needs on that Brain.

Is there a log file? No. These are interactive tools with no service and no journal. Pipe muxen-diag-sfsp to a file if you want a record of switch activity; the ncurses tools cannot usefully be redirected.

Does the version number match the package?-V prints a git describe string from the build, not the Debian package version. Use dpkg -l muxen-diagnostic-tools for that.

Tips

Work down the chain, not around it. Board on the bus → topic on the broker → value in the payload → value on the screen. Four checks, four tools, and each one eliminates everything below it. Guessing costs more than checking.

Run muxen-diag-devices once at the start of any commissioning, and keep the output. The UID list is the boat's hardware inventory, and it is the only place the mapping from UID to installed position is visible. A board swapped later is obvious against it.

Note the firmware versions at handover. The Version and Latest columns are a one-screen fleet-consistency check, and a board that is alone on an old version usually became so for a reason worth remembering.

Treat SC and NC as measurements, not errors. They are the two readings in muxen-diag-io that no screen shows and that no amount of software work can produce. A short circuit and an open circuit look identical from above the board.

Watch Updated in the data explorer before believing any payload. A retained topic on a broker looks exactly like a live one everywhere else, including on the boat's screens.

Pause the data explorer before reading a payload. On a busy boat the pane repaints faster than you can read it, and the paused view is exactly the message you selected.

Widen the terminal before you start, not after. [!] need N cols is easy to miss in a title bar, and the missing columns are the ones on the right, which in muxen-diag-io are the analogue inputs and in muxen-diag-voltage the entire EXTRA field.

Do not leave muxen-diag-devices open on a boat in service. Its action keys are single keystrokes, [R] has no confirmation, and the selected row moves with the arrow keys. Quit it when you are done looking.

Before [C] Clear configuration, know what was configured. There is no export and no undo in this tool, and the board's parameters are gone the moment it returns.

After [I] Set instance, check the boat's deployment file. The board moved; /etc/muxen/deploy.json still describes it at its old address, and the services that read that file at startup still hold the old one until they restart.

Ask for the CAN mode of muxen-diag-sfsp when a switch is suspected. MQTT mode is convenient but it cannot distinguish "the receiver heard nothing" from "the decoder stopped".

On a boat with several unpaired switches, expect noise. The unknown-switch line is printed on every frame, not only on change, so one repeatedly pressed unpaired switch fills the screen. In a marina, some of those ids belong to other boats.

Keep two terminals. Almost every question in this manual is answered faster by two tools side by side than by one tool and a memory of what the other one said.