How to properly test modifications (AKA: don't believe every MPG claim you read)
 

Here on EcoModder we see a lot of claims made about the effectiveness (or lack of effectiveness) of specific modifications. It's an understatement to say not all claims are made equal.

Just because someone says a modification worked for them, that does not make it "proof".

When looking at the claims a person or company makes about a modification, we need to understand that the the likelihood their claim is true or not is directly related to whether they know how to conduct a proper test.

So we should be respectfully skeptical of fuel economy claims made until we know the details of how the modification was tested. We should remain skeptical if the testing was weak or the test details aren't given.

(Note that "skepticism" doesn't equal "disbelief".)

-------------------
Lab testing is king
-------------------

Testing done on a dynamometer or in a wind tunnel is ideal, since it reduces to a minium the number of variables that can affect the outcome.

(We should be especially skeptical of companies selling supposed fuel saving products that don't do lab testing or offer the testing details to consumers. After all, if they're convinced their product works, they should feel confident about investing in the cost of high quality testing. Wouldn't it improve the product's sales potential?)

Of course, it's still entirely possible to screw up lab testing, but it's the best starting point.


------------------------------------------------
On-road testing: the poor cousin of lab testing
------------------------------------------------

Unfortunately, not too many EcoModder members have access to chassis dynamometers or wind tunnels. This leaves on-road testing as the next option, and so right off the bat the potential quality of the results goes down.

Why? Because outside of a lab, the number of variables that can distort a test goes waaaaaay up.

That said, there are different degrees of on-road testing. It can range from truly useless "junk science" to reasonably acceptable experimentation. The difference boils down to the amount of effort made to reduce the number of variables that can influence the outcome.


---------------------------------------
What's the big deal about variability?
---------------------------------------

In short: we need to be confident that the results we are seeing are from the modification being tested, and not from some outside factor(s).

Fuel consumption is remarkably sensitive to all kinds of variables:

• ambient temperature
• wind
• humidity
• barometric pressure
• vehicle temperature (not just engine, but the entire drivetrain including transmission, joints, bearings, tires)
• fuel grade/quality
• payload
• elevation/road grade
• road surface conditions / type
• speed
• other traffic
• driving style / driver psychology

All of these factors have an impact on fuel use through changing engine load, aerodynamic load, rolling resistance, chemical reactivity and thermodynamic efficiency.

Most individual vehicle mods have a very small potential impact on overall fuel consumption - typically in the range of a couple of percent. So you can see how trying to conduct a test while all kinds of other variables are changing would make the results meaningless.

Not only could you end up seeing an improvement that isn't actually caused by the modification, the reverse is also true: Uncontrolled variables could also prevent an actual improvement from showing up in the data.

The reason lab testing is king is because it can eliminate the highest number of these variables from a test.


--------------------------------------------------------
How to do "as controlled as possible" on-road testing
--------------------------------------------------------

So we've established that we should be automatically skeptical of on-road testing. Just how skeptical depends on how many variables the tester has managed to eliminate.

The further you get from this following example, the less confidence we should have in the results.

1) The vehicle should be fully warmed up, including drivetrain / tires.

2) Remove the driver's foot from the test, meaning cruise control should be used (set once, and cancelled with the brake between runs to ensure the same speed in multiple runs).

Any testing that can be affected by driver input - eg. city driving is the worst - is dramatically less scientific. A driver may unconsciously change driving style to get the desired result (experimenter bias). An ideal experiment would be double blind.

Failing to "remove the driver" as much as possible from the test is a huge red flag and we should be very skeptical of conclusions made.

3) The route should be devoid of other traffic, to avoid significant aerodynamic impacts of vehicles ahead or overtaking.

4) Weather conditions should be as calm and stable as possible (wind gusts/changes in wind speed & temperature changes will affect results). Evening or night time testing can be preferable where atmospheric conditions are typically calmer, and there may be less traffic.

5) The route should be as flat & straight as possible.

6) Bi-directional runs should be done to average out effects of grade/wind, if present. Meaning, test on the route in both directions.

7) Use A-B-A comparisons. That means establishing a baseline (the first "A" set of runs), more test runs after making a change (the "B" set), and then additional runs after undoing the change (the last "A" set).

Why do A-B-A testing? Undoing the change and immediately re-testing the final "A" set increases confidence that any difference seen in the "B" runs was caused by the modification, and not by other uncontrolled factors.

A-B-A illustration: (from Drive-cycle economy and emissions measurement - Fuel saving gadgets - a professional engineer's view)

http://ecomodder.com/imgs/testing-aba-1.gif http://ecomodder.com/imgs/testing-aba-2.gif

8) A-B-A comparison runs should be done immediately one after the other to minimize the effects of changing weather conditions, vehicle temperature, weight, fuel quality, etc.

9) Be prepared to abandon runs with unexpected changes (eg. a car overtaking you, affecting aerodynamic drag)

10) The more runs the better (larger data set = higher confidence)

11) Share your raw data.


---------------------------------
Drat! Quality testing isn't easy
---------------------------------

It can actually be a big pain in the butt, and a time-eater too. (I'll often wait a week or more for calm, stable weather, and then spend 2-3 hours doing a simple A-B-A test.)

(Note, I'm not suggesting by this post that my own testing is perfect. I've done some stinker tests myself. And if I refer to the less than ideal ones, I try to point out their flaws.)

In the end, if we're not willing to do testing that's "as controlled as possible", we should have the intellectual honesty to admit that the results are questionable, and we should be wary of drawing conclusions from the data.


------
Links
------

• Empirical skepticism (Wikipedia)

• Experimenter's bias (Wiki)

• The dangers of testimonial evidence - Fuel saving gadgets - a professional engineer's view

• Drive-cycle economy and emissions measurement - Fuel saving gadgets - a professional engineer's view

---------------------------------------------
Topics that may be added to this post later
---------------------------------------------

• figuring out the "quality" of data gathered in a test

• examples of bad on-road testing

• explanation of why on-road testing fuel/oil additives is especialy problematic

• "Invoking Carl Sagan" - or why extraordinary or controversial claims (or commercial claims) should be held to a higher standard of proof - see: http://ecomodder.com/forum/showthrea...uel-15251.html

• Other testing methods: eg. coastdown, for A-B-A of aero mods

Very nice compilation Darin.

Very cool info man.

some additions and notes

* ambient temperature
* wind
* humidity
* barometric pressure
* vehicle temperature (not just engine, but the entire drivetrain including tires)
* fuel grade/quality
* payload
* elevation/road grade
* road surface conditions / type
* speed
* other traffic
* driving style / driver psychology

almost all of this stuff with reasonable caution is MINOR in the bigger picture.

Example look at my fuel economy history for ryo-ohki it really does not deviate a whole lot from day to day. in fact is usually does not deviate more than 1mpg overall.

what is needed is LOTS AND LOTS AND LOTS AND LOTS AND LOTS AND LOTS and ... just a few more Runs again and again.

the more data points you have the more irrelevant small variables are.

even a 40' change in air temperature is NOT going to alter your fuel economy at all. Yes technically it will but not enough that ANY of us likely has the equipment to measure off a dyno :-)

so just make lots and lots of runs. through out the higher and low points and average the data.

lets say you make 20 runs and most are 47mpg 2 are 49 and 50 and 3 are 45 44

well toss out the 45 44 runs and the 49 50 runs average the rest.

for the purposes of deciding if more energy into a project is WORTHWHILE or not this is more than accurate enough data to come to some sort of conclusion.

While this info is VERY important for us when BUYING improvements for the community its just not needed

99% of us WANT to see improvement in our MPG and we will work to that goal 100% of the time a b or a. IE when your offering up info USUALLY the person has no reason to decieve or burn people and they have nothing to sell you so they have little to gain (there is of course always that 1% who just like being dicks :-)

when you make your A B A runs just make sure you do 20+ runs for A and 20+ runs for B and then just enough for the last A to confirm the results.

that should be more than enough data points to average out outside variables if your reasonably cautious and careful.

now there is one thing that IS a huge variable that used to NOT be a huge variable.

FUEL. in over a million driven miles I saw ZERO difference that could not be argued away as noise between gas at one station and gas at another station.

lately though with the introduction of ethanol I have seen MASSIVE huge swings in results from fuel. as much as 8mpg from one station to another. since I started ONLY buying gasoline at Wawa Gas stations (the lowest ethanol content and consistent so far) my fuel economy has settled down and gotten a LOT more consistent again.

Thanks MetroMPG:)
I'm glad this thread is sticky.

True. Big samples help.

"NOT .. at all," eh? Strongly disagree on this point. I can post a number of credible sources that show the relationship of ambient temperature on fuel economy. But I'll give you a chance to find some supporting evidence for your claim first. ;)

Actually, that's pretty surprising. I'm betting there's much more stability in your log than the average member's.

Have to wonder how much climate plays into your trend, considering your location. Does your Metro have air conditioning? Many people in hot areas see fuel economy fall in the summer, from A/C use, vs. people in the north who see a dip in winter from the larger efficiency hit of colder ambient temps.

No its an Xfi no air ALTHOUGH the AC in my cherokee had very little impact on fuel economy (less than half a mpg overall) same in the van but even less I could not measure a difference in FE ac on and ac off tank to tank.

as for evidence to support me only anecdotal evidence sadly. IE when it was 75-80 degrees out I got 47mpg when it was 25' out I got 47mpg. this tells me that air temps have very little impact on fuel economy at least for my little metro.

I guess its my commute. very consistant daily day in day out (its how I got 493,000 miles on the cherokee)

in fact until ethanol I saw almost NO change in my FE unless I did something drastic such as extreme highway driving or going up to the mountains etc..

My clubwagon got a SOLID and I mean SOLID 19mpg pretty much no matter what I did to it. Now if I hit the mountains or I towed the camper ok it went down but overall I got a very solid 19mpg. hence why I was so surprised when I started to get 13mpg out of the blue :-) thats another story about ethanol.

Even summer to winter I saw almost no change in FE maybe 1mpg at most.

I am kind of hoping that changes now since I am a wee bummed at 47mpg (I am still amazed I can say that with a straight face) and am hoping for 50mpg :-)

Digging out my graphs... give me a minute... ;)

Here it is. 2 full years of daily data (1500+ data points), averaged for each degree level. I discarded the whole first year of data, because my driving changed so much the data was useless.

Of course, this only applies to ME driving MY CAR in MY LOCATION - standard disclaimer about YMMV.
http://mcguckin.us/files/car/mpg_temp_2009_12.png

?? you see a TWENTY mpg difference with a 75 degree change in temperature?? WOW.

There's someone whose sample size overshadows other variables!

Thanks for posting that, Andrew.

Yeah, that's what this thread is about, really.

---

As for why you don't see as much of a swing in MPG vs. temperature, Pale's chart suggests a good reason: Note the temps where his fuel economy curve plateaus from roughly 65-70F and on up. Pretty darn close to the range of daily high temps in New Orleans.

(*edit* Correction on my part: I mistook "NO" in his user info to mean that Nerys lived in New Orleans.)

If you had air in your Metro, you'd notice a significant fuel economy hit.

but he sees HIGHER mpg the higher the temps. that kills the AC thing.

I am just curious why I see almost NO difference and he sees such a HUGE difference over the SAME temperature changes.

now granted maybe I will also see a change now that I have a tiny car but as noted I have driven this metro in 80' temps and in 20' temps with NO apparent change in mpg beyond noise.

whats different between us? I live in PA (outside philly) and my commute is between 19057 (home) and 08215 (work)

I will have to start logging the temps as I drive.

how do you factor in the temp changes over the period of a tank? ie I fill upo every 3 or 4 days usually (multiple cars) do I just average the temps over the 4 days?

Correction on my part: I mistook "NO" in your user info to mean you lived in New Orleans.

Trip length has a lot to do with the efficiency hit of ambient temperature. (The longer your trip, the smaller its impact.) How far is your daily drive?

Edit: Google Maps says 56.7 miles between those two postal codes. That would explain a lot.

yeah 54miles actual each way I end up about 110 miles each day mostly semi city/highway

Here is a route plot that I have found the most efficient (the least number of stops all 50mph)

My Commute

I take precisely the same route in reverse to go back home. the start point is within a mile of home. yeah the no is when I was annoyed at sites asking for details so until I trust a site I fill in random stuff if it forces me to put something in IE no is "no" you don't need that :-) hehehe

what does it explain?

Glad to see you trust us now... :P

...and, there's also "engine break-in"...slowly but surely increasing MPG for no other (apparent) reason than accumulated age/miles...on both brand new engine and tires!

...also, I use "Olympic Scoring" where "highest" and "lowest" MPG values are discarded (to help reduce the "too good to be true" number flukes).

I don't use the AC in the summer.
I also don't have "abnormal" trips in there, such as a fully warmed-up start.

Absolutely. My average fuel economy with my new Mustang was steadily increasing over time even before I found ecomodder.com and made adjustments to my driving technique and commuting route. I can only attribute that increase to engine and tire break-in. I'm not looking forward to the FE hit I'll take when I have to re-tire it.:p
http://ecomodder.com/forum/fe-graphs/graph2691.gif

Good points about engine & tire break-in.

It explains why you are less affected by falling ambient temperature than someone who lives in the same climate but has a more typical (shorter) commute.

54 miles is a long commute, and fuel economy is directly related to trip length. Google "fuel consumption" as a "function of trip length" and look through the results.

i'm selling all my chassis dynamometers and wind tunnels on ebay.

...uh, was it something we said?

ahh I understand. warming up gets you a lot lower fuel economy DURING the warm up. so a shorter trip means that warm up period is much larger percentage wise so the colder temps will hit this "warm up" period more and since its a larger percentage of your total trip the effect is visible in the numbers. So I won't see as much a gain if any from a engine heater on my FE as for example he might?

I think that's a big factor here. With 10 miles, a cool-running engine, and lots of low-speed and EOC driving, my entire drive is the warmup period.

Yes. I drove my wife's car to Las Vegas from the Bay Area last month, and gassed up when we got there. That night, we drove to another part of town to see a show. The next day, I drove home. The night I drove to the casino, I put 13 miles on the car at 17 mpg. It took me the next 450 miles to bring my mileage up over 40 mpg the next day, because I'd used nearly a full gallon to go 13 miles on a cold engine in city conditions.

A GALLON IN THIRTEEN MILES!! what kind of car uses a gallon in 13 miles warming up but gets WELL over 40mpg normally?

even my 5 liter 5500 pound van does not use a gallon in 13 miles (unless I get 10-11% ethanol :-) hehe

You name me a car capable of getting 40 mpg, and I'll show you how to get 13 (actually 17) mpg out of it.;)

The car I did it with is my wife's '06 AT Hyundai Elantra. We drove at night, had to run the heater and lights, and didn't hit many stoplights on green. We also had to drive the Strip on a weekend, with its stop and go traffic. Even FASing at lights, I could only get 17 mpg that night.

When I lived in Alaska, I could only expect 5 mpg in the winter with my 4X4 Scout. Winter driving kills fuel economy.

I actually averaged >41 mpg over ~1000 miles on that trip. It just took all day to overcome the effects of that first 13 miles at 17 mpg. When you're doing short commutes every day, you never overcome the cold starts.

...sometimes it's NOT your milage that's so bad, but rather all the unwelcomed HELP you get from other drivers, stop lights and grid-lock traffic.

wow - even with bad parts clogged egr etc... I never got under 42mpg in my metro. My van never saw less than 13mpg even towing a 7000 pound trailer on Ethanol with the AC on!!

I just did not think it was possible under normal conditions to have such a massive swing in FE in a vehicle short of massive issues with the car.

I was just shocked is all :-)

well actually I don't think you CAN get 13mpg in that car. maybe 13mpg INSTANT reading maybe but average even over a few miles I don't think you can get 13mpg without REALLY hammering it or something.

I forgot you have a computer read out of your FE as you drive :-)

That was trip and tank average fuel economy for the 13 miles. It's not uncommon for an AT econocar to stay under 20 mpg for the first few miles when it's started up. My wife's car doesn't have a fe readout. I had a Scangauge plugged into it.

yeah I know Scangauge was your FE read out :-)

Darrell's point is that he is telling you that 13 mpg was an average reading, not just some random "instant" MPG value, as suggested.

Nerys: careful about insisting so strongly that people must be wrong. Your experience isn't everyone's experience. Lots to learn from the members here.

I know and I did not mean to come off like that.

under normal conditions I would think it would be pretty hard to get 13mpg. I mean you would have to drive a whole tank of gas at WOT and up a hill etc..

it hit me afterwards that he said first 17 miles which he can not possibly know "NORMALLY" then I realized he was using an FE read out (scangauge) so he was getting instant readings or real time readings and feedback etc..

instant as in he knows before he adds more gas about what his FE is which also explains how he can know he got 13mpg his first 17miles.

Once all of that "clicked" in my head it made sense :-) I am not always the greatest at expressing thought to words :-)

I apologize for any mis communication errors on my part :-)

Don't beat yourself up. If you learned something and I taught something, we both end up winners.

too late - hehehe - ouch that hurt :-)

nah just wanted to clear it up. I can come off very "strong" sometimes so I wanted to clear it up :-)

cut my MPG in half!
 

I totally understand you man, but I had a pretty bad experience with my 2006 Kia Sedona (4700lb curb weight, EPA 25 HWY). It's pretty regular at getting anywhere from 22-26 mpg on the HWY at 70mph (my wife likes AC in the summer). But.... I made a trip from eastern Texas to western Washington and my average MPG was 14.6 with a midsized Uhaul trailer with a total weight of 2100 on one axle. Long story, 3000lbs of people and stuff + aero-deficiant trailer + running a day behind and going 75mph+ (went 85 for a solid day)= 13.2 as my lowest fillup and 18.8 as my highest actual tank mpg(mountains VS plains).

On your ambient temperature effect on milage point. I have a Nissan Altima and it has a air intake temperature sensor which when the temperature is low the ecu will add fuel to help engine run. This is in addition to the normal choke like action when starting the car. When I am driving as the car warms up I can actully watch the speed increase as the the rpm decrease.

wow man !!! 3000 pounds AC and 85mph. THATS HAMMERING IT :-) hehe not bad considering :-)

I just drove my wife's Elantra 400 miles to my mom's, averaging 40.8 mpg. Not bad, considering it's Winter, & half my drive was in the dark with lights on. I tanked up near my mom's this morning & drove down to San Diego to my son's. Got 44 mpg through LA. Love the warm weather & light holiday traffic.

Let's try to keep this thread on topic. Thanks!

I thought this might be a good place to jump in . I read on another thread as so as I find it I again will post a link. I guy or someone who ever it is ? Claims you cannot acheive over 50 MPG out of a Geo Metro ( What and idiot )

Here are my specs

1990 metro stock
Unknown High Milage
Burnt exaust valve # 3
Very low compresion
Overload car
Burns a quart of oil per tank
Blowby blowing out of everything
I have to carry a spare plug with me so when it fouls

I dont or cant have scangauge so with last consecutive 3 full tank runs

42.5
43.7
43.2

Not bad for a totally shot car - 50 is an easy grab. I am going for 60. Got alot of parts on the way. Just thought you guys would get a chuckle out of it :) Thanks John

A galon in 15 miles
 

I you had a onboard computer (as I have) you would be surprised.

My car has a VW 1.9 TDI engine and can show fuel yield in real time, factory installed.
I can get up to 70 MPG while at 70km/h (about 45 miles/h) on a flat road, after warming up, of course.

However, when I am driving on the streets of my town, warming up and between traffic boxes, my same car starts yielding 10 mpg ! Not sport accelerating ! Just normal getting there.

It is only after the car increases its speed (not possible in high traffic between traffic boxes) and warms up, that fuel yield improves. A normal short round trip in town, even trying to get most of it, yields me about 30 mpg.

Oldbeaver