Troubleshooting

This chapter is organised by what you see. The service has no control socket, no status command and no configuration: everything it knows it says in one line on the journal, and everything it decided is visible in its exit status. Those two, plus netbird status, are the whole toolbox.

systemctl status muxen-eagle                # state, exit status, condition
journalctl -u muxen-eagle -n 50             # the one line it printed
netbird status                              # what the client thinks
netbird status --json | jq '.management'    # the fields the script reads
sudo ls -l /etc/muxen/eagle.key             # absent on an enrolled boat

Before anything else, two facts that explain most confusion:

• An enrolled boat has no setup key on it. The file is deleted on every success path. Its absence is health, not a fault.
• The service is oneshot. active (exited) is the healthy end state. It is not supposed to be running.

The unit is inactive (dead) and nothing ever ran

Installing the package starts nothing — the packaging passes dh_installsystemd --no-start on purpose. The unit is WantedBy=muxen-deploy.target, so it first runs when that target comes up.

Likely cause	What to check	Fix
The Brain has not been deployed yet	systemctl status muxen-deploy.target	commission the boat; enrolment is meant to follow it
The MUXEN targets are missing	systemctl cat muxen-deploy.target	they come from another MUXEN package, which this one does not depend on — install it
The unit was never enabled	systemctl is-enabled muxen-eagle	sudo systemctl enable muxen-eagle

To run it now regardless: sudo systemctl start muxen-eagle.

systemctl status reports a failed start condition

Condition: start condition unmet

The unit carries ConditionHost=!bali-58-999. On a Brain whose host name is bali-58-999 the unit is skipped: systemd reports the start job as successful and the script never executes. Check with hostnamectl. This is not a fault and there is nothing to repair — on any other host the condition passes.

The unit is failed, exit status 78

Main PID: … (code=exited, status=78)

No setup key at /etc/muxen/eagle.key and not enrolled: cannot register with eagle

78 is EX_CONFIG, and it is the only failure the unit refuses to retry. It means both of these at once: this Brain has never been enrolled, and there is no usable setup key on it.

Likely cause	What to check	Fix
The package came from an unprotected branch pipeline	sudo wc -c /etc/muxen/eagle.key — a zero-length file	install a release build; see The setup key
Someone deleted the key before enrolment finished	the file is absent and netbird status shows no eagle URL	reinstall muxen-eagle, which restores the key file, then systemctl restart muxen-eagle
The boat is enrolled and the diagnosis is wrong	netbird status --json | jq .management.url	if it reads https://eagle.muxen.fr:443, the script would have exited 0 — re-run it

Reinstalling the package is the normal repair: the key file is not a conffile, so apt install --reinstall muxen-eagle puts it back unconditionally.

The unit retries every five minutes

Restart=on-failure with RestartSec=300, and the start rate limiter switched off, so a failing run loops on that period forever by design. A boat that is offline for a week is expected to sit here. Read the journal line to find out which failure it is:

Message	Cause	Action
netbird daemon not answering yet; will retry	the netbird daemon is down, or has not finished starting	systemctl status netbird — see below
(a netbird command's own error output)	set -e aborted the run on a failing netbird call	run the script by hand and read the failing command
(nothing at all in the journal)	the script exited before printing	check the unit's recorded exit status

netbird daemon not answering yet; will retry

netbird status --json returned nothing usable. The script treats this as unknown-and-therefore-retryable and exits 1 — never 78 — because a daemon that has not come up yet says exactly the same thing as a boat that was never enrolled.

1. Is the daemon running? systemctl status netbird.
2. Did a netbird service reconfigure just restart it? The script runs one when NB_DISABLE_SSH_CONFIG=true is missing from the netbird service environment, and that regenerates the unit and restarts the daemon. The next five-minute retry normally finds it up.
3. Is jq installed? It is a hard Depends, but a broken installation shows up here first, since every JSON read goes through it.

Enrolled with eagle but not connected yet — and it stays that way

The boat holds NetBird credentials for eagle.muxen.fr but the link is not up. muxen-eagle has correctly done nothing: keeping the tunnel alive is the netbird daemon's job, not this service's, and it will reconnect on its own.

So this is not an enrolment problem. Work down the connectivity:

1. Does the boat have internet at all? No route, no VPN.
2. Is the clock right? timedatectl. TLS to eagle.muxen.fr fails with a certificate error on a Brain whose clock is far out — which is common on a board whose backup battery is flat.
3. Is 443 reachable? Enrolment and management both run over HTTPS to eagle.muxen.fr:443. A marina network that filters outbound traffic is a real cause.
4. What does the client say? netbird status reports its own view of the management connection.

The service is active (exited) but MUXEN cannot reach the boat

Enrolment succeeded — that is all active (exited) claims. Being reachable additionally needs the netbird daemon connected now, and the boat's peer to be permitted on the management server.

netbird status --json | jq '{url: .management.url, connected: .management.connected}'

• connected: false — a link problem; see the section above.
• connected: true and still unreachable — the boat is on the network and the problem is on the far side: peer permissions, group or routing on the management server. Nothing on the Brain will show it.

Duplicate eagle profiles, or profile select fails as ambiguous

Newer NetBird clients key profiles by ID and allow duplicate names, so netbird profile select eagle fails once two profiles share the name. Older releases of this package could create that situation: they added a profile on every run, and a failing run retried.

The current script cannot: it short-circuits on netbird status before touching any profile, and when it does touch them it works by ID — keeping the eagle whose ID prefixes the one in /var/lib/netbird/active_profile.json, and removing the rest.

If a boat is already in that state, running the service once repairs it:

sudo systemctl restart muxen-eagle
netbird profile list --show-id

Removals are best-effort (|| true), so a duplicate that refuses to be removed leaves the run to continue rather than fail. If one survives repeated runs, remove it by ID by hand.

Outbound ssh from the Brain stopped working

NetBird installs /etc/ssh/ssh_config.d/99-netbird.conf, which rewrites the behaviour of every outbound ssh on the host. muxen-eagle-init disables that by setting NB_DISABLE_SSH_CONFIG=true on the netbird service — but a netbird package upgrade wipes the service environment, and the setting has to be re-applied.

What to check	Expected
systemctl show netbird --property=Environment	contains NB_DISABLE_SSH_CONFIG=true
ls /etc/ssh/ssh_config.d/	no 99-netbird.conf

Running sudo systemctl restart muxen-eagle re-applies it — except on a legacy install, where the script exits at step 1 before reaching that code. On such a boat, apply it by hand:

sudo netbird service reconfigure --service-env NB_DISABLE_SSH_CONFIG=true

The netbird daemon restarts whenever muxen-eagle runs

netbird service reconfigure regenerates the netbird unit and restarts the daemon. The script only calls it when the setting is missing, precisely so that this is a one-off. Seeing it on every run means the setting is not surviving — something is regenerating or overwriting the netbird unit between runs. Check what owns /etc/systemd/system/netbird.service on that image.

apt install muxen-eagle did not put the boat on the network

Correct behaviour. The package is installed with --no-start, so installation enrols nothing. The service runs when muxen-deploy.target comes up — at boot, or when the deployment is restarted — or when you start it by hand.

The key file reappeared after an upgrade

Also correct. /etc/muxen/eagle.key is not a conffile, so every install and upgrade writes it back. The next run of the service short-circuits on netbird status before reading it, and deletes it again. On an enrolled boat the file's brief reappearance changes nothing.

FAQ

What does this actually do for me? It lets MUXEN help you without coming to the boat. Once the Brain has joined the support network, an engineer can look at the system, read logs and apply fixes remotely.

Is MUXEN watching my boat all the time? This service does one thing: it registers the Brain on the network. It opens no session by itself, sends no data anywhere, and stops as soon as registration is done. What can be reached, by whom, is controlled on MUXEN's side.

Do I need to do anything for it to work? No. It runs by itself once the boat is commissioned, provided the Brain has an internet connection.

The service says "active (exited)". Has it crashed? No. It is a one-shot task, not a background service; finishing is what success looks like. failed would be the bad word.

There is no key file in /etc/muxen. Has something deleted it? Yes, and on purpose. The key is used once and removed from the boat immediately afterwards, so it cannot be copied off a Brain later.

Nothing happened when I installed the package. Is it broken? No. Installing does not enrol. Enrolment happens when the MUXEN deployment starts, which on a new boat is after commissioning.

Do I need to reinstall it after changing the boat's configuration? No. It reads no configuration at all.

The boat has been without internet for three days. Do I need to intervene? No. It keeps trying every five minutes indefinitely and enrols itself the moment a connection appears.

Can I move a Brain to a different network, or take it off the fleet? Not with this package — it has no un-enrol path. That is done on the NetBird management server.

Does it work if the boat is behind 4G or marina Wi-Fi? It needs outbound HTTPS to eagle.muxen.fr on port 443. Networks that block outbound traffic will stop it.

Tips

Check enrolment at commissioning, not later. One command settles it:

netbird status --json | jq '{url: .management.url, connected: .management.connected}'

A boat that leaves the yard unenrolled is a boat that has to be visited.

Set the clock before you deploy. Enrolment is an HTTPS call, and a Brain whose clock is far out fails certificate validation with an error that looks nothing like a clock problem.

Treat exit 78 as a packaging alarm, not a boat fault. It almost always means the .deb on that Brain was built from an unprotected branch and carries an empty key. Check wc -c /etc/muxen/eagle.key before touching anything on the boat.

Do not hand-edit /etc/muxen/eagle.key. It is not a conffile and it is rewritten by every install; anything you put there is transient, and anything you delete comes back on the next upgrade.

Never pass a setup key on a command line. The script uses --setup-key-file because argv is world-readable through /proc while the process runs. If you are reproducing a step by hand, keep the file.

Verify NB_DISABLE_SSH_CONFIG after every netbird upgrade. Netbird package upgrades wipe the service environment, and the symptom — every outbound ssh on the Brain misbehaving — points nowhere near NetBird.

Do not "tidy" a legacy install onto the eagle profile. A boat enrolled on the default profile is fully working; the script leaves it alone deliberately. Migrating it risks a remote link you cannot restore remotely.

Watch it once by hand on a bench. sudo /usr/bin/muxen-eagle-init ; echo $? tells you in one line which of the four outcomes a given Brain lands in. It is safe to repeat.