Calibration and filtering

A sender is not a gauge. The number it puts on the bus is an electrical reading, and the relationship between that reading and litres, degrees or amperes belongs to the tank, the probe and the installation — not to the device. This chapter is about writing that relationship down.

For an owner, the two things worth knowing are that the gauge on the screen is only as good as the points somebody measured at commissioning, and that a tank fitted with a filter reacts slowly on purpose — a level that lags a fill by half a minute is behaving normally.

The pipeline

Every reading goes through the same three steps, in this order:

device reading ──► clamp to input.min/max ──► curve ──► clamp to output.min/max ──► value

1. Input clamp. The reading is brought inside input.min … input.max. Both default to infinity, so nothing is clamped until they are set.
2. Curve. identity, polynomial, linear or step.
3. Output clamp. The result is brought inside output.min … output.max, again unbounded by default.

The two clamps do different jobs, and both matter. The input clamp keeps a broken sender out of the curve. The output clamp keeps the answer inside the physical limits of the thing being measured.

A result that is not a finite number — infinity, or not-a-number — is not published at all. The daemon logs it once, on the transition:

mqtt: FreshWaterPortside value is not finite (raw = inf, value = inf), not publishing

and the variable ages out of the boat's screens through its metadata rather than showing nonsense.

identity

No conversion. The reading is clamped and published as it stands. This is the default when scaleType is absent.

Use it when the device already reports the quantity you want — a source current on an interconnection, for instance:

{
  "name": "Solar",
  "input": { "deviceType": 3, "deviceInstance": 1, "deviceChannel": "source0" },
  "output": { "unit": "obix:units/ampere" }
}

polynomial

"scaleType": "polynomial",
"polynomial": [0, 50]

The array holds up to 5 coefficients, lowest order first:

value = c0 + c1·x + c2·x² + c3·x³ + c4·x⁴

[0, 50] is the straight line 50·x: a sender producing 0…5 gives 0…250 litres. [-25, 100] is 100·x − 25, which is how a sender with a dead band at the bottom is handled. Coefficients that are not given are zero.

More than five entries is refused with polynomial: order too high. Every entry must be a number.

A polynomial is the right tool for a linear or near-linear sender in a prismatic tank. It is the wrong tool for the two cases that break most gauges: a tank that is not a rectangular box, and a sender whose curve nobody derived. Fitting a cubic to four measurements produces a curve that is exact at those four points and wrong everywhere else, and worse, it goes wrong smoothly enough to look credible.

linear — the one to use for a real tank

"scaleType": "linear",
"linear": [
  [0.35,  0],
  [1.10,  50],
  [1.95,  100],
  [2.80,  150],
  [3.60,  200],
  [4.40,  250]
]

Each point is [reading, result]. Between two points the value is interpolated on the straight line joining them; the curve as a whole is whatever shape the points describe.

Rules:

• Up to 32 points. More is refused.
• Points must be strictly increasing in reading. An out-of-order pair is refused with the two offending points named, which is the usual way a transcription error announces itself.
• Below the first point the first result is returned; above the last point, the last one. The curve is flat outside its measured range, never extrapolated.

This is the honest way to calibrate a tank, because it needs no theory about the sender or the hull: it needs measurements.

Measuring the points

At commissioning, with the tank empty:

1. Read the raw value. mosquitto_sub -h 127.0.0.1 -t 'app/sensor/<name>' -v and take input.value — the device reading, published before any clamp or curve is applied to it. On an install still pinned to the old prefix the topic is variable/<name>; config: topic prefix = … in the startup log says which.
2. Add a known quantity — a metered 50 litres, or a full jerrycan.
3. Record the pair.
4. Repeat to full.

Six to eight points over the range is plenty for a normal tank, and more points help exactly where the tank changes shape. Take the readings with the boat level and settled: a tank measured on a heeled boat calibrates the heel in.

The pairs go into linear in the order they were measured, lowest reading first.

step

"scaleType": "step",
"step": [
  [0.0, 0],
  [1.0, 25],
  [2.0, 50],
  [3.0, 75],
  [4.0, 100]
]

Same table, no interpolation: the result is the one attached to the highest point whose reading has been reached, and it holds until the next one. Below the first point, the first result is used.

Use it for a sender that is genuinely discrete — a set of float switches, a level probe with fixed contacts — or where the boat wants a quarter-tank display rather than a litre count. Do not use it to smooth a continuous sender; that is what the filter is for.

The same limits apply as for linear: 32 points, strictly increasing.

The filter

Setting liquid.filterPeriod attaches a smoothing filter to a variable:

"liquid": { "filterPeriod": 60 }

The value is a number of one-second samples, clamped to 12…900 without a message. There is no filter unless the key is present.

What it does

The filter keeps the last filterPeriod published values in a window. Once a second it sorts that window, discards the lowest quarter and the highest half, and averages what is left — the samples sitting between the 25th and 50th percentile of the window.

Three consequences follow from that arithmetic, and they explain most of what an owner notices about a filtered tank:

• It rejects splashing. A tank sloshing at anchor produces a wide spread of readings; the extremes at both ends are thrown away before the average is taken.
• It reads on the low side. The retained band sits below the middle of the window, so the published level is under the median of the recent readings. A tank shows slightly less than it holds rather than slightly more, which is the useful direction to be wrong in.
• It falls faster than it rises. Because the retained band is in the lower half of the window, a level going down enters it sooner than a level going up. With filterPeriod: 60, a drain starts moving the gauge after about 15 seconds and has settled by 30; a fill does not move it at all for about 30 seconds and settles around 45.

Choose the period accordingly: long enough to cover the boat's motion, short enough that filling a tank produces visible feedback. 60 seconds is a reasonable starting point for a water or fuel tank in a seaway.

What it does not do

• It does not ramp up after a restart. The window is filled with the first value it sees, so the first published figure is the instantaneous one, not a slow climb from zero.
• It does not stop when the sensor does. The window is fed once a second by the publishing timer, not by the arrival of a frame. A sensor that goes silent keeps pushing its last value into the window, which then converges on it. Use metadata.rxdate to tell a live reading from a frozen one.
• It does not apply to the raw reading. The filter sits at the very end of the pipeline, on the value that came out of the curve and the output clamp.

What it changes in the payload

A filtered variable publishes two figures instead of one:

Field	Unfiltered variable	Filtered variable
data.value	the scaled value	the filtered scaled value
data.raw	absent	the scaled value before filtering
input.value	the device reading	the device reading

data.raw is a confusing name for what it holds: it is not the raw sensor reading — that is input.value — but the unfiltered result of the curve. The wire contract predates the distinction. A screen wanting a live, unsmoothed number reads data.raw; a screen wanting the settled one reads data.value.

Choosing a curve

Situation	Use
the device already reports the quantity	identity
a linear sender in a prismatic tank	polynomial, two coefficients
a sender with a known offset and gain	polynomial
any tank whose shape is not a box	linear, measured
any sender whose curve is not documented	linear, measured
float switches, discrete contacts, quarter-tank display	step
a reading that moves with the boat rather than with the level	add liquid.filterPeriod

Calibrating from raken, live

Measuring points with mosquitto_sub works, but it means reading a number in a terminal, writing the curve into the configuration, and deploying before finding out whether the curve was right. A calibration session closes that loop: raken sends a candidate curve, the daemon evaluates it against the live reading, and the answer comes back several times a second while the point is being dragged.

Two topics carry it, neither of them retained:

Topic	Direction
app/sensor/calibration/request	raken → daemon
app/sensor/calibration/response	daemon → raken

What a session is, and is not

A session is a preview. The candidate curve is evaluated in a shadow copy of the sensor, and nothing about it reaches the rest of the boat: app/sensor/<name> keeps publishing the deployed configuration throughout, so the datalogger, the alarms and the screens see no trace of somebody experimenting. Nothing is written to the configuration file. A curve is only adopted by saving and deploying it, exactly as before.

A session expires on its own, after ttl seconds — 120 by default, clamped to 10…600 — and each request refreshes the deadline. A tablet that goes to sleep on the calibration page therefore leaves nothing behind. So does a daemon restart: sessions live in memory only.

The channel does not have to be configured. The reading is taken from the device/… topics muxen-boat publishes for every channel on the bus, not from this daemon's own inputs, so a sender wired to a channel no sensor has ever been written for can be calibrated before its first deployment. For a BLOC8, which only reports in answer to an RTR request, the session asks on the channel's behalf.

The request

The sensor object is a configuration entry, in exactly the form chapter 2 documents — the same object that would be written to the configuration file:

{
  "sessionId": "b3f1c8e2",
  "name": "FreshWaterPortside",
  "ttl": 120,
  "sensor": {
    "input":  { "deviceType": 1, "deviceInstance": 1,
                "deviceChannel": "analogInput0", "min": 0.0, "max": 5.0 },
    "output": { "unit": "obix:units/liter", "min": 0, "max": 250 },
    "scaleType": "linear",
    "linear": [[0.35, 0], [1.10, 50]]
  }
}

sessionId is any string that identifies this conversation; every answer carries it back, so a late reply to a request that has been superseded can be recognised and dropped. name is cosmetic here — a channel being calibrated before it is named needs none.

Ending early:

{ "sessionId": "b3f1c8e2", "action": "stop" }

The response

{
  "sessionId": "b3f1c8e2",
  "state": "active",
  "name": "FreshWaterPortside",
  "input":     { "value": 2.31, "min": 0.0, "max": 5.0 },
  "candidate": { "value": 118.4, "unit": "obix:units/liter" },
  "saved":     { "value": 115.5, "name": "FreshWaterPortside" },
  "metadata":  { "rxdate": "…", "expireAfterSec": 5 }
}

• input.value is the device reading, before any clamp or curve — the same figure input.value carries on app/sensor/<name>, and the one to record when measuring points.
• candidate.value is what the curve in the request makes of it. Unfiltered, always: the filter window is 12…900 one-second samples, so a filtered figure would not move at the speed anybody works at. liquid.filterPeriod is accepted in the request and ignored for this purpose.
• saved.value is what the deployed configuration is producing on the same channel right now, so the candidate can be read against it. It is absent when no deployed sensor reads that channel, which is the normal case for a channel being calibrated for the first time. The match is on the input routing, not the name, so a sensor renamed in raken still finds itself.

state is one of:

state	Meaning
active	The session is running. With a reading, it carries input and candidate; the first answer to a request arrives before any reading and carries neither
no-data	Subscribed and asking, but nothing has arrived on that channel for five seconds. Wrong instance, device offline, or a bus that is down
rejected	The candidate was refused. error carries the reason, in the same words the configuration file would be refused with
stopped	Ended by an explicit action: "stop"
expired	The ttl ran out without a refresh

A rejection is worth reading rather than retrying:

{
  "sessionId": "b3f1c8e2",
  "state": "rejected",
  "error": "points: point (2) input (5.000000) must be less than point (3) input (2.500000)"
}

Watching one by hand

mosquitto_sub -h 127.0.0.1 -t 'app/sensor/calibration/response' -v

and, in another terminal, a session that reads a BLOC8 analogue input straight through:

mosquitto_pub -h 127.0.0.1 -t app/sensor/calibration/request -m '{
  "sessionId": "bench", "ttl": 60,
  "sensor": {
    "input":  { "deviceType": 1, "deviceInstance": 1, "deviceChannel": "analogInput0" },
    "output": { "unit": "obix:units/volt" }
  }}'

Nothing arriving at all means no-data after five seconds; see chapter 4.