Applied Nate Silver – Unexpected/Unexplained pattern

Lately I’ve been having an unpleasant feeling in the morning when I do my weigh-ins. The first number I see looks “good” (meaning what I want, i.e. lower than previous day) but then the numbers bounce up. By the time I’ve got enough for an average I’m feeling low because once again I’m stuck on a plateau, starving myself and accomplishing nothing.

So I wondered if there really was a pattern. The issue of scale variability continues to bug me. The variations are huge, now bigger swings in a single set of weigh-ins than my weekly change. That means the daily noise is now greater than the weekly signal. So I spend all the previous day starving and I can’t tell, in the weigh-in, whether it did any good, thus denying myself any sense of accomplishment and starting out the day in a bad mood.

So is there a pattern?

In order to study the variability more I decided rather than normalizing daily weights by just subtracting, I should actually adjust by the average. That is, after getting a set of weights from the scale, get the average of that set, then go back and divide each individual number by the average, thus generating a distribution centered on and normalized to 1.0. This has created a somewhat lumpy looking histogram but something approximately looking like a normal distribution.


So I’ve been patiently accumulating more data so that law of large numbers will kick in and I can see (better) what the actual distribution is.

But that’s the long-term goal, what about the idea that weights increase as I accumulate more samples each day. So, just to make things simpler I just took the first seven samples of each day as the dependent variables and the ordinal number of which weigh-in the sample is as the independent variable (i.e. first sample x=1, last sample x=7) and here’s the result:


Sure enough, there is a gradual rise (0.05lb) for each consecutive weigh-in. Not only does the trendline show that but also the min values and max values show the same effect (I’ve already been noticing that pattern too, that using min or max or ave are all correlated).

So the distress I’ve been feeling, initially seeing a good value, then seeing worse ones later is a real effect, not just one I imagine since it’s easy to hypothesize that any value worse than the current minimum is “disappointing”.

But once again what is the truth here? Is the first value really the best value? Or are the later values better?

The only thing I can think might be happening in the scale itself is that I’m warming it up by use. The scale is made of glass and I get on it bare-footed. I do the weigh-ins in the the morning when the scale has been sitting in a cool place all night. I’m also moving the scale from where I store it to where I can stand on it (using the pattern in the tile floor to almost exactly position the scale in same place each time).

You might think I could do a better job of collecting data if I just left the scale in one place and got consecutive readings, BUT, I’ve found that somehow the scale seems to have short-term memory and that if I don’t move it + delay before getting another sample I always (where ‘always’ is a conclusion not carefully tested, but it’s the only result I’ve ever seen in my few experiments) get the same reading (I don’t know if it’s the moving or the amount of delay between readings).

But this scale is frustrating. It’s a good scale and my continuing attempts to measure its statistical properties indicated its standard deviation is about 0.3%, which is pretty good for a measuring device. But despite that apparent accuracy I’m trying to measure a “signal” (daily change) that is (now) about 0.1% so you can see how the scale variability so significantly overcomes the signal.

I don’t have enough data yet (getting more data means days of hunger) to see what my new weight loss trajectory will be (given I’m now at a 30 year low and already have do about 65lbs) but I have the sense that at best my current weekly drops are about 1/2 what I achieved months ago. That is the ‘plateau’ effect and it means that with this very slow drops now in terms of elapsed time my goal is now potentially 3 months in the future, which is a lot of days of starving just to get another 15% drop.

So the scale variability is freaking me out. All the “advice” says to ignore daily weights and just do weekly. Fine, I concur. BUT, at random the single weekly weight may be anomalously high (which happened to me and now that’s why I’m doing the daily detail). So again that advice is WRONG – you should actually do multiple samples daily and this kind of statistical analysis to see the pattern for the week, such as:


Now maybe most people won’t bother with this much data and using Excel to analyze it, but look at all that scatter in the blue dots (single measurement) and tell me the pattern without having the larger amount of data and regression analysis to show the trend.

The results from weigh-ins have a psychological effect, including freaking out and over-reacting, but failing to spot patterns and then reacting (a bad day of dieting must be made up for in future days, so tell yourself the inconvenient truth rather than what you want to hear). Sure, stick to a consistent eating/exercise plan and let the weigh-ins fall where they may, BUT, “consistent” isn’t really possible in real life (and my life is simpler and more controllable than most people will face). So feedback and response is critical and that’s why the variability in the scale is so frustrating.



About dmill96

old fat (but now getting trim and fit) guy, who used to create software in Silicon Valley (almost before it was called that), who used to go backpacking and bicycling and cross-country skiing and now geodashes, drives AWD in Wyoming, takes pictures, and writes long blog posts and does xizquvjyk.
This entry was posted in comment, musing, whine and tagged , . Bookmark the permalink.

6 Responses to Applied Nate Silver – Unexpected/Unexplained pattern

  1. dmill96 says:

    btw: I didn’t really analyze it, but it is clear that this pattern can only be really spotted with a fairly substantial set of data. Generally that is true about anything in statistics, more data, better answers, but this also demonstrates something about today’s world of “big data”. Patterns that no one really expected can be detected (assuming they exist) with enough data.

    In the past getting data is expensive (and it is for me since data represents one more day of starving, but also one more day off the calendar – I cannot accelerate my data gathering). So we’ve probably done a lot of our work with relatively small amounts of data. But one consequence of the Net (and interactivity, i.e. participation) on it is that accumulation of big data. The idea that good analyzers run against raw Twitter data can predict trends faster than any other method is surprising at first glance, but really, on second thought, just one more example of what massive real-time data collection (of very noisy and uncontrolled data) can do.

  2. Nice post!!!! Just a little encouragement: I’ve spent the past 1 1/2 years with a digital scale, and simply eating less of what I eat, using 5″ x 5″ plate. I weigh every morning and if I see I’m loosing a few ounces each day, I know I’m on track. I’ve lost 25 lbs (can now wear size 6 womans trousers rather than size 8, so that is great!!!). The less rapid, the healthier. Once I lost fist 5 lbs,, I focused on next five lbs., no matter if it took 2 months to lose, I kept focusing and adjusting when I needed to treat myself. I’ve got 10-15 more to lose, I know I will, but it will be slower because I refuse to starve and feel hungry. Husband is huntsman, so I eat wild game a.m., chicken or fish usually for noon, I take lots of herbs and vitamins, in afternoon I eat whole fruit, 2 small bowls bran, use several stalks of celery or other fiber for regularity. I found that regularity is essential, and weighing after elimination makes one feel better about oneself. Elimination and weighing after elimination in a.m. and timing fiber intake the day before really assists in loosing the lbs. With just the 25 lb loss I have purchased a whole new wardrobe, making me feel better about self, and when I lose the rest, I can go from a size 8 suit jacket to a size 6. Taking things in increment, and changing wardrobe really assists in one’s better feeling about self!!!!!

    • dmill96 says:

      Congratulations. It sounds like you’ve found the plan that works for you. Plus the whole key part is how you feel about yourself, not the need to satisfy the expectations of others.

      I think each person has to find what works for them and none of the “formula” can be directly applied. Now that I’m down by 65lbs I’m finding it hard to lose the last 15lbs of fat as part of my target, plus I’ve lost too much muscle. So I’m now in “Phase 2” with emphasis on strength training, high intensity (rather than calorie burning) exercise, and increased protein consumption. So another part of any plan may involve a change from what originally worked.

  3. Wonderful accomplishment!!!!!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s