How to properly test modifications (AKA: don't believe every MPG claim you read)
 

Here on EcoModder we see a lot of claims made about the effectiveness (or lack of effectiveness) of specific modifications. It's an understatement to say not all claims are made equal.

Just because someone says a modification worked for them, that does not make it "proof".

When looking at the claims a person or company makes about a modification, we need to understand that the the likelihood their claim is true or not is directly related to whether they know how to conduct a proper test.

So we should be respectfully skeptical of fuel economy claims made until we know the details of how the modification was tested. We should remain skeptical if the testing was weak or the test details aren't given.

(Note that "skepticism" doesn't equal "disbelief".)

-------------------
Lab testing is king
-------------------

Testing done on a dynamometer or in a wind tunnel is ideal, since it reduces to a minium the number of variables that can affect the outcome.

(We should be especially skeptical of companies selling supposed fuel saving products that don't do lab testing or offer the testing details to consumers. After all, if they're convinced their product works, they should feel confident about investing in the cost of high quality testing. Wouldn't it improve the product's sales potential?)

Of course, it's still entirely possible to screw up lab testing, but it's the best starting point.


------------------------------------------------
On-road testing: the poor cousin of lab testing
------------------------------------------------

Unfortunately, not too many EcoModder members have access to chassis dynamometers or wind tunnels. This leaves on-road testing as the next option, and so right off the bat the potential quality of the results goes down.

Why? Because outside of a lab, the number of variables that can distort a test goes waaaaaay up.

That said, there are different degrees of on-road testing. It can range from truly useless "junk science" to reasonably acceptable experimentation. The difference boils down to the amount of effort made to reduce the number of variables that can influence the outcome.


---------------------------------------
What's the big deal about variability?
---------------------------------------

In short: we need to be confident that the results we are seeing are from the modification being tested, and not from some outside factor(s).

Fuel consumption is remarkably sensitive to all kinds of variables:

• ambient temperature
• wind
• humidity
• barometric pressure
• vehicle temperature (not just engine, but the entire drivetrain including transmission, joints, bearings, tires)
• fuel grade/quality
• payload
• elevation/road grade
• road surface conditions / type
• speed
• other traffic
• driving style / driver psychology

All of these factors have an impact on fuel use through changing engine load, aerodynamic load, rolling resistance, chemical reactivity and thermodynamic efficiency.

Most individual vehicle mods have a very small potential impact on overall fuel consumption - typically in the range of a couple of percent. So you can see how trying to conduct a test while all kinds of other variables are changing would make the results meaningless.

Not only could you end up seeing an improvement that isn't actually caused by the modification, the reverse is also true: Uncontrolled variables could also prevent an actual improvement from showing up in the data.

The reason lab testing is king is because it can eliminate the highest number of these variables from a test.


--------------------------------------------------------
How to do "as controlled as possible" on-road testing
--------------------------------------------------------

So we've established that we should be automatically skeptical of on-road testing. Just how skeptical depends on how many variables the tester has managed to eliminate.

The further you get from this following example, the less confidence we should have in the results.

1) The vehicle should be fully warmed up, including drivetrain / tires.

2) Remove the driver's foot from the test, meaning cruise control should be used (set once, and cancelled with the brake between runs to ensure the same speed in multiple runs).

Any testing that can be affected by driver input - eg. city driving is the worst - is dramatically less scientific. A driver may unconsciously change driving style to get the desired result (experimenter bias). An ideal experiment would be double blind.

Failing to "remove the driver" as much as possible from the test is a huge red flag and we should be very skeptical of conclusions made.

3) The route should be devoid of other traffic, to avoid significant aerodynamic impacts of vehicles ahead or overtaking.

4) Weather conditions should be as calm and stable as possible (wind gusts/changes in wind speed & temperature changes will affect results). Evening or night time testing can be preferable where atmospheric conditions are typically calmer, and there may be less traffic.

5) The route should be as flat & straight as possible.

6) Bi-directional runs should be done to average out effects of grade/wind, if present. Meaning, test on the route in both directions.

7) Use A-B-A comparisons. That means establishing a baseline (the first "A" set of runs), more test runs after making a change (the "B" set), and then additional runs after undoing the change (the last "A" set).

Why do A-B-A testing? Undoing the change and immediately re-testing the final "A" set increases confidence that any difference seen in the "B" runs was caused by the modification, and not by other uncontrolled factors.

A-B-A illustration: (from Drive-cycle economy and emissions measurement - Fuel saving gadgets - a professional engineer's view)

http://ecomodder.com/imgs/testing-aba-1.gif http://ecomodder.com/imgs/testing-aba-2.gif

8) A-B-A comparison runs should be done immediately one after the other to minimize the effects of changing weather conditions, vehicle temperature, weight, fuel quality, etc.

9) Be prepared to abandon runs with unexpected changes (eg. a car overtaking you, affecting aerodynamic drag)

10) The more runs the better (larger data set = higher confidence)

11) Share your raw data.


---------------------------------
Drat! Quality testing isn't easy
---------------------------------

It can actually be a big pain in the butt, and a time-eater too. (I'll often wait a week or more for calm, stable weather, and then spend 2-3 hours doing a simple A-B-A test.)

(Note, I'm not suggesting by this post that my own testing is perfect. I've done some stinker tests myself. And if I refer to the less than ideal ones, I try to point out their flaws.)

In the end, if we're not willing to do testing that's "as controlled as possible", we should have the intellectual honesty to admit that the results are questionable, and we should be wary of drawing conclusions from the data.


------
Links
------

• Empirical skepticism (Wikipedia)

• Experimenter's bias (Wiki)

• The dangers of testimonial evidence - Fuel saving gadgets - a professional engineer's view

• Drive-cycle economy and emissions measurement - Fuel saving gadgets - a professional engineer's view

---------------------------------------------
Topics that may be added to this post later
---------------------------------------------

• figuring out the "quality" of data gathered in a test

• examples of bad on-road testing

• explanation of why on-road testing fuel/oil additives is especialy problematic

• "Invoking Carl Sagan" - or why extraordinary or controversial claims (or commercial claims) should be held to a higher standard of proof - see: http://ecomodder.com/forum/showthrea...uel-15251.html

• Other testing methods: eg. coastdown, for A-B-A of aero mods

Very nice compilation Darin.

Very cool info man.

some additions and notes

* ambient temperature
* wind
* humidity
* barometric pressure
* vehicle temperature (not just engine, but the entire drivetrain including tires)
* fuel grade/quality
* payload
* elevation/road grade
* road surface conditions / type
* speed
* other traffic
* driving style / driver psychology

almost all of this stuff with reasonable caution is MINOR in the bigger picture.

Example look at my fuel economy history for ryo-ohki it really does not deviate a whole lot from day to day. in fact is usually does not deviate more than 1mpg overall.

what is needed is LOTS AND LOTS AND LOTS AND LOTS AND LOTS AND LOTS and ... just a few more Runs again and again.

the more data points you have the more irrelevant small variables are.

even a 40' change in air temperature is NOT going to alter your fuel economy at all. Yes technically it will but not enough that ANY of us likely has the equipment to measure off a dyno :-)

so just make lots and lots of runs. through out the higher and low points and average the data.

lets say you make 20 runs and most are 47mpg 2 are 49 and 50 and 3 are 45 44

well toss out the 45 44 runs and the 49 50 runs average the rest.

for the purposes of deciding if more energy into a project is WORTHWHILE or not this is more than accurate enough data to come to some sort of conclusion.

While this info is VERY important for us when BUYING improvements for the community its just not needed

99% of us WANT to see improvement in our MPG and we will work to that goal 100% of the time a b or a. IE when your offering up info USUALLY the person has no reason to decieve or burn people and they have nothing to sell you so they have little to gain (there is of course always that 1% who just like being dicks :-)

when you make your A B A runs just make sure you do 20+ runs for A and 20+ runs for B and then just enough for the last A to confirm the results.

that should be more than enough data points to average out outside variables if your reasonably cautious and careful.

now there is one thing that IS a huge variable that used to NOT be a huge variable.

FUEL. in over a million driven miles I saw ZERO difference that could not be argued away as noise between gas at one station and gas at another station.

lately though with the introduction of ethanol I have seen MASSIVE huge swings in results from fuel. as much as 8mpg from one station to another. since I started ONLY buying gasoline at Wawa Gas stations (the lowest ethanol content and consistent so far) my fuel economy has settled down and gotten a LOT more consistent again.

Thanks MetroMPG:)
I'm glad this thread is sticky.

True. Big samples help.

"NOT .. at all," eh? Strongly disagree on this point. I can post a number of credible sources that show the relationship of ambient temperature on fuel economy. But I'll give you a chance to find some supporting evidence for your claim first. ;)

Actually, that's pretty surprising. I'm betting there's much more stability in your log than the average member's.

Have to wonder how much climate plays into your trend, considering your location. Does your Metro have air conditioning? Many people in hot areas see fuel economy fall in the summer, from A/C use, vs. people in the north who see a dip in winter from the larger efficiency hit of colder ambient temps.

No its an Xfi no air ALTHOUGH the AC in my cherokee had very little impact on fuel economy (less than half a mpg overall) same in the van but even less I could not measure a difference in FE ac on and ac off tank to tank.

as for evidence to support me only anecdotal evidence sadly. IE when it was 75-80 degrees out I got 47mpg when it was 25' out I got 47mpg. this tells me that air temps have very little impact on fuel economy at least for my little metro.

I guess its my commute. very consistant daily day in day out (its how I got 493,000 miles on the cherokee)

in fact until ethanol I saw almost NO change in my FE unless I did something drastic such as extreme highway driving or going up to the mountains etc..

My clubwagon got a SOLID and I mean SOLID 19mpg pretty much no matter what I did to it. Now if I hit the mountains or I towed the camper ok it went down but overall I got a very solid 19mpg. hence why I was so surprised when I started to get 13mpg out of the blue :-) thats another story about ethanol.

Even summer to winter I saw almost no change in FE maybe 1mpg at most.

I am kind of hoping that changes now since I am a wee bummed at 47mpg (I am still amazed I can say that with a straight face) and am hoping for 50mpg :-)

Digging out my graphs... give me a minute... ;)

Here it is. 2 full years of daily data (1500+ data points), averaged for each degree level. I discarded the whole first year of data, because my driving changed so much the data was useless.

Of course, this only applies to ME driving MY CAR in MY LOCATION - standard disclaimer about YMMV.
http://mcguckin.us/files/car/mpg_temp_2009_12.png

?? you see a TWENTY mpg difference with a 75 degree change in temperature?? WOW.

There's someone whose sample size overshadows other variables!

Thanks for posting that, Andrew.

Yeah, that's what this thread is about, really.

---

As for why you don't see as much of a swing in MPG vs. temperature, Pale's chart suggests a good reason: Note the temps where his fuel economy curve plateaus from roughly 65-70F and on up. Pretty darn close to the range of daily high temps in New Orleans.

(*edit* Correction on my part: I mistook "NO" in his user info to mean that Nerys lived in New Orleans.)

If you had air in your Metro, you'd notice a significant fuel economy hit.