Cleaning incoming data

Your incoming data may come directly from a sensor, but sensors can become faulty or give intermittently bad readings.

How should you clean up this data, so that only in-range values are shown?

Calculated fields, not incoming fields

The answer is to show calculated field values within your portal, not the raw incoming field values.

Incoming values are always stored with as little cleaning-up as possible, for future auditing. But if they need to be cleansed, then you should have a corresponding calculated field for each incoming field. 

Example

Let's say your incoming data has a field called sensor.temperature. If you edit your schema, you should see a field in your incoming data looking like this:

Create a field in the calculated fields section like this:

Now you need some Phi script that calculates the clean.temp field every time some data arrives from the sensor.

Enter your Phi script at the bottom of the page, like this:

upperLimit = 99
lowerLimit = -20
if (thisRow.value['sensor.temperature'] >= lowerLimit
 && thisRow.value['sensor.temperature'] <= upperLimit) {
    setValue('clean.temp', thisRow.value['sensor.temperature'])
}

The first two lines define your limits, and the if() statement checks if the raw sensor reading is in range, and if so sets clean.temp to the incoming value. If out of range, clean.temp will be null for that reading.