How to properly test modifications (AKA: don't believe every MPG claim you read)

[oldbeaver]

Originally Posted by jakobnev

Your professor was

(Dammit, that was supposed to be linked but not embedded!)

No, my professor wasn´t the same as yours. However, I like him.

My professor in Physics was Oscar García, Ph.D., Scientist and International Referee.

However, I wasn´t that good student. Don´t command Entropy.

[SecondWind]

to MetroMPG: I've read and re-read your post which starts this thread. I have no training in statistics, and although I can do the "A-B-A" testing, I don't know how to tell whether differences are significant. Can you describe "statistical significance" in layman's terms? For instance, if I do 5 two-way "A" runs with an average of 59.8 MPG, and standard deviation of 1.4, then 5 "B" runs of 60.6 MPG and s.d. of 1.4, does this mean anything?

[Grant-53]

Yes, it does - sort of. The term 'significant' by convention means that 5% of the values sampled fall outside of an acceptable range of deviation (1.95 x s.d.) in a normal distribution. Whether the 0.8 mpg is due to a modification or the randomness of the samples is another question. Since the difference of the two averages is barely half of the s.d. of each sample, most samples fall within a range common to both A and B.

[SecondWind]

Thanks for your reply. It sounds as though 5 runs is not enough to determine whether some mpg figures are "outliers", or whether the set of mpg figures just vary a lot. I'll try a set of 10 runs next time, and post results here.

[Grant-53]

Confidence level in testing is somewhat personal. The question is how certain do we want to be that our tests give are truly representative of what is actually going on. As a quality inspector I would be comfortable with five samples and a standard deviation of 1.4 to accept a batch of product from a proven process. If I was going prove to a major customer that a process line was operating within tolerance 99.9% of the time I would need to take 40 samples or more.
What we are doing is called 'hypothesis testing' in the statistics books. We are testing the claim that a certain product or technique will provide a detectable (and economically feasible) improvement in fuel economy. We may decide that we want to be 99% sure that on average there is a 5% improvement in mpg. Given that test, there are means to calculate the number of trials to run. Most libraries will have books on statistics or quality and you can pick one that matches your reading level.

[Grant-53]

There is a great deal of debate over Intelligent Design, Evolution, faith, and science with most of it being as acrimonious as it narrow minded on all sides. The reality of the situation is that Science is a means to describe what we can observe. Faith, in the Biblical sense, is confidence based on relationship with God. As in science, we gain confidence in a relationship by repeated trials and tests. If we are honest with ourselves we can learn a great deal. This I have learned: a) God is very scientific, b) the easiest person to outsmart is yourself.

[SecondWind]

Grant-53... I found a copy of "Statistics for Dummies" at the library. Thought I'd start at the VERY beginning!

[wmjinman]

Guys, I find this very interesting and appreciate the knowledge available in this thread. I've had my questions about "when do you have results that mean anything?", and this discussion gives me some much appreciated guidance. Thank you!!
Bill

[RiderofBikes]

So, maybe its just me, but i hate statistics, too "general".
But i am a lazy human, that will eventually get around to my own testing ...eventually.

If you were to say 100=(100%) people gathered for a "study", revealing that 90% = 90 people had mental disorders, 100% of the time. You still whined up at 90% = 90 people. Thats great, but there's nothing that Specifically determines this result. Just a overall Broad Conclusion... IE: What type of disorders do they classify, and or test for individually?! if at all for the overall result. One test might result with flying colors, while the same "participant" failing the next test miserably.
^^^Example only^^^

Now for this type of testing, I'd "prefer" to have individual "Lab" testing for certain things (funds pending) to eliminate variables like mentioned before..
But Backyard Testing, I would much rather include a little more detail into my Results if I am certainly going to be testing in them. While Caost down test (Aero), or Instant feedback instrumentation good for (MPG) runs, are great, they include all the Variables, and come out with a more or less consistant result if done right enough times for a satisfactory conclusion (for that specific time frame).
I would run the test as "normal", multiple times, but just testing specifically at a certain Weather/temperature, Atitude, traffic vs. no traffic, "individual variable" basis instead.

Temps are one of the biggest variables geographically(anywhere), next would be Altitude.
(X)Temp x (X)Altitude = (X)mpg, OR (X)Aero efficiency
Hence forth I think that if I so happen to live geographically(not at nor near sea level) for both of these variables then I would just be conducting this test multiple times. Then at a later time, when the temp has changed, rerun the test at the same altitude to see how much(if any) change has accured.
Later on if you have the time or ability, Run a similar test at a different altitude/temp combo, etc.

Continue this testing pattern for the other variables where possible.
While Each graph wont be exactly the same, result wise, you will be able to see the corrolation between the variables specific to your vehicle

This would make for a battering amount of Tests and Combinations, Time and Effort used would be astronamical. But once its all said and done you have all variables on their own, that way you could maybe provide yourself info for YOUR "Optimum" curcumstance to test at for an Overall Result based on your previous research.

The Ideal curcumstances would be to unload the car, inflate tires(specific PSI), Cruise control, Instruments calibrated, 0 wind, 0 traffic, "real world" 70-80 avg tepms, humidity, Sea Level, etc, etc... But not all those factors align perfectly for everyone...

[wmjinman]

RiderofBikes,

During my past life as a land surveyor, I picked up a few concepts in my 27 year career. While maybe not "scientifically rigorous" statistics-wise, we had many methods for ensuring our results were good. We hardly did anything without checking it. If we measured a distance, we'd measure it twice. If we'd turn an angle, we'd turn it twice (or more). Same thing with elevations. The concept of "closing the traverse" is to make sure that when you come back to where you started from, your measurements in your notes ALSO say you came back to that same point. Same thing with "closing a level loop"

When I read Metro MPGs article on "proper testing" with the "A-B-A" method, it made perfect sense to me as a surveyor. So while maybe not statistically rigorous, I think three "sets" of data can tell you something IF they're consistant. The first one gives you a number. The second one gives you a comparison. Then the third one confirms (or disputes) that your set is consistant.

For example, you make 3 "A" runs (two way, of course) and get 40.2, 40.1 and another 40.1. Actually, that would be 6 runs, since these are all the averages of the "out and back". But I would personally be very satisfied to go away thinking the results were 40.133 or whatever it comes out to. (actually, 40.1 taking "significant figures" into account). However, if the runs were 40.2, 41.8, and 33.4, I'd want to do some more, because the spread was so great.

Doing two-way runs is EXTREMELY important because it automatically averages-out so many error factors. Grade, wind, etc. If one-way is bad because it's not level, doing it the other way will compensate because it's not level, but in the opposite direction. Same thing with wind. (within reason, of course - doing it on a hill so steep you have to go up it in low gear and brake on the way back down obviously won't give you good results.)

But back to the surveying, my wise old bosses introduced a concept called "does it LOOK right?" A newer member on the crew once tried to stake a new property line out into the street. The boss said, "It doesn't matter how powerful your calculator is, we KNOW that property line doesn't go out into the street". So if a set of three runs "looks good" (tight grouping), then I'm happy with it. But if it's "all over the place", then we need to do more until we figure out where it should be.

Anyway, sorry if this is a rant, but for me, I don't (usually) need more than a few runs to be able to determine what I want. For example, yesterday I did 3 out & back runs in the "A" configuration. Two-way averages were: 30.95, 30.95, and 30.80. Overall average of 30.90. I considered this "good", since the maximum spread was only 0.15 mpg. Then I folded the mirrors back to see what the result would be. Two way averages were: 30.95, 31.05, and 31.05. Overall average of 31.02. When you round off for significant figures (ScanGauge only reads to a tenth of a mpg), you get 30.9 vs. 31.0, or one tenth of a mpg gain by folding the mirrors.

(I did do a final "A" run with the mirrors back out, that came out at 30.65. Not sure why it was so much less than the first 3, but at least I knew the "B" test going up wasn't because "conditions" went up). Including that with the 3 original "A" tests still only shows the mirror fold as less than 0.2 mpg gain.

In my mind, I got the answer I was looking for: "It helps, but precious little". Considering the lack of rearward visibility and potential attention from the cops, I think I'm leaning towards "not worth it". To me, that's all I really wanted to learn from that test. I also learned that same day that adding about 10 more psi of air to my tires seems to be good for 1.3 mpg!!! Now there's something that IS worth it!!!!