## Applied Nate Silver – the noise is getting louder and overwhelming the signal

So here I am again on another Sunday with yet another anomalous reading on weight. After monitoring it closely, on daily basis, after last Sunday’s shock, I now get another shock. Simply put the variation is getting so huge it is beginning to completely swamp the long-term trends plus negate all the effort I’m making. So my results are getting to be very slim compared to effort which, of course, has a strongly negative effect on my motivation. So let’s look at the “money graph” (the one graph I’m using for long term progress):

After the additional measurement, analysis and discussion during this past week I eventually decided there was no replacement value for week 19 since: a) the measured value was clearly anomalous, but, b) no other value was based on sound enough logic to substitute an imputed value for a measured value. It turns out that a similar problem, but not as severe also exists today.

According to my food logging in the BodyMedia app and calories burned as measured by the BodyFit I should have a calorie deficit of 19,682 for two weeks which translates to 5.62lbs loss, but the measured loss is only 2.5lbs or 44% of predicted. This is where actual measured weight is seriously deviating from any underlying basis. But the measured value for today (week 20) is itself a 3.9lb increase since yesterday! Now that translates to 13,650 calories accumulated since yesterday! Now I did pig out (although with some restraint) at the family party yesterday but I can’t consume that much in one day. So this is another crazy point (actually the measured Saturday value was mostly the crazy part). I’ll show another graph on this but the long-story-short is now that short-term (daily) “noisy” fluctuations are now much larger than week-to-week real losses.

The magnitude of change and duration is getting too large to see details so let’s scale the money graph as shown below:

In this case the polynomial trendline seems to more correctly reflect the trend than does the linear. This makes some intuitive sense in that after 50lbs of loss it’s reasonable I might be beginning to “plateau”, both due to the fact that I probably only have about 20lbs more I can lose (that would reverse nearly 40 years of gains) and there are probably other factors, more than adipose tissue, now affecting my weight measurements.

So the polynomial predicts 201lbs next week whereas the linear predicts more like 198.7. Isn’t this reasonable? Maybe not, as shown below:

This graph shows seven daily weigh-ins: the blue dots are raw data (scale variability) and the red dots are average of raw data. Note that Saturday’s weight already achieved the 201 level (well, 201.1). So today’s weight is another hugely anomalous reading (just like last Sunday) where it is higher than both Thursday and Friday despite running a 2974 calorie deficit for those two days (possibly as much as 15% less, but no more error than that). So, in short, yesterday’s weight was highly anomalous and today’s weight was somewhat anomalous. Or, IOW, these data are getting more and more noisy and thus harder to etierh use for measuring progress or even worse trying to predict. So, today’s weight should have been something more like what the trendline shows, i.e. about 203.5. Let’s look at this in bigger picture:

The trendline is based on the average of the Sunday weigh-ins (excluding week 19, last week’s outlier), but look at the scatter in the daily readings from last week seen against this larger scale view. There is all that noise I’m talking about, so vast it’s almost completely obscuring the signal.

Now while I’ve had some daily variation in calorie deficit that variation is nowhere nearly as much as this weight variation. While I can’t claim a consistent plan (exercise and fasting) over all the past 140 days it is fairly consistent and thus should produce something like a second order polynomial (plateau effect due to requiring less calories/day as weight drops and exercise burning less per session as weight drops). But this relatively constant adherence to plan is getting completely obscured by the noise.

And that’s a real problem for me. Not only am I now having trouble knowing what is “real” (like am I deviating from plan more than I subjectively believe) but I’m losing that all important feedback of making progress. To starve every day and grind around for hours on exercise equipment requires some attaboys and I’m getting less and less of that. But it’s also a problem for prediction which will be critical when I try to establish maintenance (I’m now within 20lbs of my best case targeted loss plan so maintenance will soon be the primary goal). But interestingly the “big data” view (gross resolution) is more predictive than any of this detailed stuff, as shown below:

Here I discard no weekly “raw” data and the trendline actually looks like it predicts the weekly fairly well despite all the fairly significant deviations (on specific weeks).  And so therein lies my projection for next week. Despite the fact I’ve already reached 201.1 (average of Saturday measured) and declared today to be 204.0 (and thus my modest 1.2lbs/week loss (lately) would extrapolate to 202.8) I’m going to go for a bit more optimistic (hopefully no deviations from plan this week and I can seriously fast like I did last week and the heel pain that started Saturday doesn’t impede exercise – we’ll see what excuses I have next week):

# 201.9

(This is somewhat based on idea that I’m really at 203.4 now and then assuming 1.5lbs/week is new weekly loss rate). If I hadn’t already seen the Saturday value this prediction would be too optimistic. So let’s see what the dailies show and maybe make a mid-course adjustment about Wednesday.