Go Back   EcoModder Forum > EcoModding > EcoModding Central
Register Now
 Register Now
 

Reply  Post New Thread
 
Submit Tools LinkBack Thread Tools
Old 12-14-2009, 02:21 PM   #1 (permalink)
Batman Junior
 
MetroMPG's Avatar
 
Join Date: Nov 2007
Location: 1000 Islands, Ontario, Canada
Posts: 19,688

Blackfly - '98 Geo Metro
Team Metro
90 day: 70.72 mpg (US)

ForkenSwift - '92 Geo Metro EV
Last 3: 95.68 mpg (US)

MPGiata - '90 Mazda Miata
90 day: 47.99 mpg (US)
Thanks: 2,145
Thanked 4,920 Times in 2,474 Posts
Arrow How to properly test modifications (AKA: don't believe every MPG claim you read)

Here on EcoModder we see a lot of claims made about the effectiveness (or lack of effectiveness) of specific modifications. It's an understatement to say not all claims are made equal.

Just because someone says a modification worked for them, that does not make it "proof".

When looking at the claims a person or company makes about a modification, we need to understand that the the likelihood their claim is true or not is directly related to whether they know how to conduct a proper test.

So we should be respectfully skeptical of fuel economy claims made until we know the details of how the modification was tested. We should remain skeptical if the testing was weak or the test details aren't given.

(Note that "skepticism" doesn't equal "disbelief".)

-------------------
Lab testing is king

-------------------

Testing done on a dynamometer or in a wind tunnel is ideal, since it reduces to a minium the number of variables that can affect the outcome.

(We should be especially skeptical of companies selling supposed fuel saving products that don't do lab testing or offer the testing details to consumers. After all, if they're convinced their product works, they should feel confident about investing in the cost of high quality testing. Wouldn't it improve the product's sales potential?)

Of course, it's still entirely possible to screw up lab testing, but it's the best starting point.


------------------------------------------------
On-road testing: the poor cousin of lab testing
------------------------------------------------

Unfortunately, not too many EcoModder members have access to chassis dynamometers or wind tunnels. This leaves on-road testing as the next option, and so right off the bat the potential quality of the results goes down.

Why? Because outside of a lab, the number of variables that can distort a test goes waaaaaay up.

That said, there are different degrees of on-road testing. It can range from truly useless "junk science" to reasonably acceptable experimentation. The difference boils down to the amount of effort made to reduce the number of variables that can influence the outcome.


---------------------------------------
What's the big deal about variability?
---------------------------------------

In short: we need to be confident that the results we are seeing are from the modification being tested, and not from some outside factor(s).

Fuel consumption is remarkably sensitive to all kinds of variables:
  • ambient temperature
  • wind
  • humidity
  • barometric pressure
  • vehicle temperature (not just engine, but the entire drivetrain including transmission, joints, bearings, tires)
  • fuel grade/quality
  • payload
  • elevation/road grade
  • road surface conditions / type
  • speed
  • other traffic
  • driving style / driver psychology
All of these factors have an impact on fuel use through changing engine load, aerodynamic load, rolling resistance, chemical reactivity and thermodynamic efficiency.

Most individual vehicle mods have a very small potential impact on overall fuel consumption - typically in the range of a couple of percent. So you can see how trying to conduct a test while all kinds of other variables are changing would make the results meaningless.

Not only could you end up seeing an improvement that isn't actually caused by the modification, the reverse is also true: Uncontrolled variables could also prevent an actual improvement from showing up in the data.

The reason lab testing is king is because it can eliminate the highest number of these variables from a test.


--------------------------------------------------------
How to do "as controlled as possible" on-road testing
--------------------------------------------------------

So we've established that we should be automatically skeptical of on-road testing. Just how skeptical depends on how many variables the tester has managed to eliminate.

The further you get from this following example, the less confidence we should have in the results.

1) The vehicle should be fully warmed up, including drivetrain / tires.

2) Remove the driver's foot from the test, meaning cruise control should be used (set once, and cancelled with the brake between runs to ensure the same speed in multiple runs).

Any testing that can be affected by driver input - eg. city driving is the worst - is dramatically less scientific. A driver may unconsciously change driving style to get the desired result (experimenter bias). An ideal experiment would be double blind.

Failing to "remove the driver" as much as possible from the test is a huge red flag and we should be very skeptical of conclusions made.

3) The route should be devoid of other traffic, to avoid significant aerodynamic impacts of vehicles ahead or overtaking.

4) Weather conditions should be as calm and stable as possible (wind gusts/changes in wind speed & temperature changes will affect results). Evening or night time testing can be preferable where atmospheric conditions are typically calmer, and there may be less traffic.

5) The route should be as flat & straight as possible.

6) Bi-directional runs should be done to average out effects of grade/wind, if present. Meaning, test on the route in both directions.

7) Use A-B-A comparisons. That means establishing a baseline (the first "A" set of runs), more test runs after making a change (the "B" set), and then additional runs after undoing the change (the last "A" set).

Why do A-B-A testing? Undoing the change and immediately re-testing the final "A" set increases confidence that any difference seen in the "B" runs was caused by the modification, and not by other uncontrolled factors.

A-B-A illustration: (from Drive-cycle economy and emissions measurement - Fuel saving gadgets - a professional engineer's view)



Quote:
The data set on the left shows what would be expected from a genuine fuel saving device - economy improves when it is fitted, then worsens again when it is removed. The data set on the right shows economy improving, but then staying at this higher level when the device is removed again. The logical conclusion is that the improved economy is not due to the device, but to some other factor
8) A-B-A comparison runs should be done immediately one after the other to minimize the effects of changing weather conditions, vehicle temperature, weight, fuel quality, etc.

9) Be prepared to abandon runs with unexpected changes (eg. a car overtaking you, affecting aerodynamic drag)

10) The more runs the better (larger data set = higher confidence)

11) Share your raw data.


---------------------------------
Drat! Quality testing isn't easy
---------------------------------

It can actually be a big pain in the butt, and a time-eater too. (I'll often wait a week or more for calm, stable weather, and then spend 2-3 hours doing a simple A-B-A test.)

(Note, I'm not suggesting by this post that my own testing is perfect. I've done some stinker tests myself. And if I refer to the less than ideal ones, I try to point out their flaws.)

In the end, if we're not willing to do testing that's "as controlled as possible", we should have the intellectual honesty to admit that the results are questionable, and we should be wary of drawing conclusions from the data.


------
Links
------


---------------------------------------------
Topics that may be added to this post later
---------------------------------------------

  • figuring out the "quality" of data gathered in a test
  • examples of bad on-road testing
  • explanation of why on-road testing fuel/oil additives is especialy problematic
  • Other testing methods: eg. coastdown, for A-B-A of aero mods

__________________
Latest mods test: 15 mods = 15% MPG improvement: A-B test, 2007 Honda Civic 1.8L, 5-speed
Ecodriving test:
Manual vs. automatic transmission MPG showdown: Nissan Micra 1.6L



EcoModder
has launched a forum for the efficient new Mitsubishi Mirage
www.MetroMPG.com - fuel efficiency info for Geo Metro owners
www.ForkenSwift.com - electric car conversion on a beer budget
  Reply With Quote
The Following 52 Users Say Thank You to MetroMPG For This Useful Post:
ausias (07-04-2012), ballo (05-08-2012), Blue Angel (03-11-2013), Cd (12-29-2010), Christ (12-15-2009), christofoo (12-21-2012), CigaR007 (08-11-2011), Daox (12-14-2009), dcb (06-21-2010), dirtydave (05-13-2014), doviatt (09-07-2013), dr200 (05-20-2011), E.Roy (08-24-2014), echo-francis (02-04-2012), F8L (10-13-2011), Frank Lee (12-15-2009), GaryTDI (11-23-2012), herbivor (12-12-2015), HyperMileQC (10-06-2012), IamIan (10-13-2012), Istas (05-15-2014), jamesshaw89 (02-17-2012), Jean-Seb (05-20-2011), Joggernot (10-05-2013), JRMichler (09-11-2012), Jyden (09-07-2013), katiep (07-06-2011), LeanBurn (12-03-2012), MadisonMPG (12-14-2009), mcrews (07-07-2011), mikeyjd (03-19-2015), MPGeo (09-06-2016), MrMiata (09-22-2010), NachtRitter (03-15-2013), oldbeaver (12-21-2013), petergo (09-05-2017), Piwoslaw (12-14-2009), pletby (08-10-2011), pRoy (12-03-2012), robbcoll (06-19-2015), slowmover (09-30-2012), tasdrouille (12-14-2009), thatguitarguy (12-14-2009), The Toecutter (05-08-2010), UltArc (07-15-2015), undeRGRound (11-11-2014), viio (04-07-2012), Weather Spotter (12-14-2009), wesman1299 (03-06-2011), wmjinman (10-13-2012), Wonderboy (12-15-2009), ZAX9000 (04-23-2013)
Alt Today
Popular topics

Other popular topics in this forum...

   
Old 12-14-2009, 02:33 PM   #2 (permalink)
Administrator
 
Daox's Avatar
 
Join Date: Dec 2007
Location: Germantown, WI
Posts: 10,332

CM400E - '81 Honda CM400E
90 day: 39.62 mpg (US)

Daox's Grey Prius - '04 Toyota Prius
Team Toyota
90 day: 49.53 mpg (US)

Daox's 04 Civic - '04 Honda Civic DX
90 day: 42.25 mpg (US)

Daox's Insight - '00 Honda Insight
90 day: 64.33 mpg (US)

Swarthy - '14 Mitsubishi Mirage DE
Thanks: 1,844
Thanked 2,012 Times in 1,235 Posts
Very nice compilation Darin.
__________________
Current project: Heating the manual trans with engine coolant
  Reply With Quote
Old 12-14-2009, 03:54 PM   #3 (permalink)
Grrr :-)
 
Nerys's Avatar
 
Join Date: May 2008
Location: Levittown PA
Posts: 800

Cherokee - '88 Jeep Cherokee
90 day: 19.44 mpg (US)

Ryo-Ohki - '94 Geo Metro Xfi
90 day: 50.15 mpg (US)

Vger 2 - '00 Plymouth Grand Voyager SE

Ninja - '89 Geo Tracker
90 day: 30.27 mpg (US)
Thanks: 12
Thanked 31 Times in 25 Posts
Very cool info man.

some additions and notes

* ambient temperature
* wind
* humidity
* barometric pressure
* vehicle temperature (not just engine, but the entire drivetrain including tires)
* fuel grade/quality
* payload
* elevation/road grade
* road surface conditions / type
* speed
* other traffic
* driving style / driver psychology

almost all of this stuff with reasonable caution is MINOR in the bigger picture.

Example look at my fuel economy history for ryo-ohki it really does not deviate a whole lot from day to day. in fact is usually does not deviate more than 1mpg overall.

what is needed is LOTS AND LOTS AND LOTS AND LOTS AND LOTS AND LOTS and ... just a few more Runs again and again.

the more data points you have the more irrelevant small variables are.

even a 40' change in air temperature is NOT going to alter your fuel economy at all. Yes technically it will but not enough that ANY of us likely has the equipment to measure off a dyno :-)

so just make lots and lots of runs. through out the higher and low points and average the data.

lets say you make 20 runs and most are 47mpg 2 are 49 and 50 and 3 are 45 44

well toss out the 45 44 runs and the 49 50 runs average the rest.

for the purposes of deciding if more energy into a project is WORTHWHILE or not this is more than accurate enough data to come to some sort of conclusion.

While this info is VERY important for us when BUYING improvements for the community its just not needed

99% of us WANT to see improvement in our MPG and we will work to that goal 100% of the time a b or a. IE when your offering up info USUALLY the person has no reason to decieve or burn people and they have nothing to sell you so they have little to gain (there is of course always that 1% who just like being dicks :-)

when you make your A B A runs just make sure you do 20+ runs for A and 20+ runs for B and then just enough for the last A to confirm the results.

that should be more than enough data points to average out outside variables if your reasonably cautious and careful.

now there is one thing that IS a huge variable that used to NOT be a huge variable.

FUEL. in over a million driven miles I saw ZERO difference that could not be argued away as noise between gas at one station and gas at another station.

lately though with the introduction of ethanol I have seen MASSIVE huge swings in results from fuel. as much as 8mpg from one station to another. since I started ONLY buying gasoline at Wawa Gas stations (the lowest ethanol content and consistent so far) my fuel economy has settled down and gotten a LOT more consistent again.
  Reply With Quote
The Following 4 Users Say Thank You to Nerys For This Useful Post:
Joggernot (10-05-2013), MobilOne (12-13-2015), MrMiata (09-22-2010), wmjinman (10-13-2012)
Old 12-14-2009, 04:29 PM   #4 (permalink)
aero guerrilla
 
Piwoslaw's Avatar
 
Join Date: Oct 2008
Location: Warsaw, Poland
Posts: 3,495

Svietlana - '05 Peugeot 307 SW
Diesel
90 day: 54.1 mpg (US)

Theodor - '01 Peugeot 106
90 day: 46.17 mpg (US)
Thanks: 1,005
Thanked 600 Times in 376 Posts
Thanks MetroMPG
I'm glad this thread is sticky.
__________________
e·co·mod·ding: the art of turning vehicles into what they should be

What matters is where you're going, not how fast.

"... we humans tend to screw up everything that's good enough as it is...or everything that we're attracted to, we love to go and defile it." - Chris Cornell

Piwoslaw's Peugeot 307sw modding thread


  Reply With Quote
Old 12-14-2009, 06:32 PM   #5 (permalink)
Batman Junior
 
MetroMPG's Avatar
 
Join Date: Nov 2007
Location: 1000 Islands, Ontario, Canada
Posts: 19,688

Blackfly - '98 Geo Metro
Team Metro
90 day: 70.72 mpg (US)

ForkenSwift - '92 Geo Metro EV
Last 3: 95.68 mpg (US)

MPGiata - '90 Mazda Miata
90 day: 47.99 mpg (US)
Thanks: 2,145
Thanked 4,920 Times in 2,474 Posts
Quote:
Originally Posted by Nerys View Post
the more data points you have the more irrelevant small variables are.
True. Big samples help.

Quote:
even a 40' change in air temperature is NOT going to alter your fuel economy at all.
"NOT .. at all," eh? Strongly disagree on this point. I can post a number of credible sources that show the relationship of ambient temperature on fuel economy. But I'll give you a chance to find some supporting evidence for your claim first.

Quote:
fuel economy history for ryo-ohki it really does not deviate a whole lot from day to day. in fact is usually does not deviate more than 1mpg overall.
Actually, that's pretty surprising. I'm betting there's much more stability in your log than the average member's.

Have to wonder how much climate plays into your trend, considering your location. Does your Metro have air conditioning? Many people in hot areas see fuel economy fall in the summer, from A/C use, vs. people in the north who see a dip in winter from the larger efficiency hit of colder ambient temps.
__________________
Latest mods test: 15 mods = 15% MPG improvement: A-B test, 2007 Honda Civic 1.8L, 5-speed
Ecodriving test:
Manual vs. automatic transmission MPG showdown: Nissan Micra 1.6L



EcoModder
has launched a forum for the efficient new Mitsubishi Mirage
www.MetroMPG.com - fuel efficiency info for Geo Metro owners
www.ForkenSwift.com - electric car conversion on a beer budget
  Reply With Quote
Old 12-14-2009, 06:42 PM   #6 (permalink)
Grrr :-)
 
Nerys's Avatar
 
Join Date: May 2008
Location: Levittown PA
Posts: 800

Cherokee - '88 Jeep Cherokee
90 day: 19.44 mpg (US)

Ryo-Ohki - '94 Geo Metro Xfi
90 day: 50.15 mpg (US)

Vger 2 - '00 Plymouth Grand Voyager SE

Ninja - '89 Geo Tracker
90 day: 30.27 mpg (US)
Thanks: 12
Thanked 31 Times in 25 Posts
No its an Xfi no air ALTHOUGH the AC in my cherokee had very little impact on fuel economy (less than half a mpg overall) same in the van but even less I could not measure a difference in FE ac on and ac off tank to tank.

as for evidence to support me only anecdotal evidence sadly. IE when it was 75-80 degrees out I got 47mpg when it was 25' out I got 47mpg. this tells me that air temps have very little impact on fuel economy at least for my little metro.

I guess its my commute. very consistant daily day in day out (its how I got 493,000 miles on the cherokee)

in fact until ethanol I saw almost NO change in my FE unless I did something drastic such as extreme highway driving or going up to the mountains etc..

My clubwagon got a SOLID and I mean SOLID 19mpg pretty much no matter what I did to it. Now if I hit the mountains or I towed the camper ok it went down but overall I got a very solid 19mpg. hence why I was so surprised when I started to get 13mpg out of the blue :-) thats another story about ethanol.

Even summer to winter I saw almost no change in FE maybe 1mpg at most.

I am kind of hoping that changes now since I am a wee bummed at 47mpg (I am still amazed I can say that with a straight face) and am hoping for 50mpg :-)
  Reply With Quote
Old 12-14-2009, 06:46 PM   #7 (permalink)
Hypermiler
 
PaleMelanesian's Avatar
 
Join Date: Dec 2007
Location: Texas
Posts: 2,272

PaleCivic (retired) - '96 Honda Civic DX Sedan
90 day: 69.2 mpg (US)

PaleFit - '09 Honda Fit Sport
Team Honda
Wagons
90 day: 54.75 mpg (US)
Thanks: 534
Thanked 396 Times in 258 Posts
Digging out my graphs... give me a minute...

Here it is. 2 full years of daily data (1500+ data points), averaged for each degree level. I discarded the whole first year of data, because my driving changed so much the data was useless.

Of course, this only applies to ME driving MY CAR in MY LOCATION - standard disclaimer about YMMV.
__________________



11-mile commute: 100 mpg - - - Tank: 90.2 mpg / 1191 miles

Last edited by PaleMelanesian; 12-14-2009 at 06:54 PM..
  Reply With Quote
The Following 8 Users Say Thank You to PaleMelanesian For This Useful Post:
Christ (12-15-2009), Frank Lee (12-15-2009), HyperMileQC (10-06-2012), MetroMPG (12-14-2009), mikeyjd (03-19-2015), MobilOne (12-13-2015), Piwoslaw (12-15-2009), pletby (11-23-2014)
Old 12-14-2009, 07:06 PM   #8 (permalink)
Grrr :-)
 
Nerys's Avatar
 
Join Date: May 2008
Location: Levittown PA
Posts: 800

Cherokee - '88 Jeep Cherokee
90 day: 19.44 mpg (US)

Ryo-Ohki - '94 Geo Metro Xfi
90 day: 50.15 mpg (US)

Vger 2 - '00 Plymouth Grand Voyager SE

Ninja - '89 Geo Tracker
90 day: 30.27 mpg (US)
Thanks: 12
Thanked 31 Times in 25 Posts
?? you see a TWENTY mpg difference with a 75 degree change in temperature?? WOW.
  Reply With Quote
The Following User Says Thank You to Nerys For This Useful Post:
Cobb (10-15-2012)
Old 12-14-2009, 07:21 PM   #9 (permalink)
Batman Junior
 
MetroMPG's Avatar
 
Join Date: Nov 2007
Location: 1000 Islands, Ontario, Canada
Posts: 19,688

Blackfly - '98 Geo Metro
Team Metro
90 day: 70.72 mpg (US)

ForkenSwift - '92 Geo Metro EV
Last 3: 95.68 mpg (US)

MPGiata - '90 Mazda Miata
90 day: 47.99 mpg (US)
Thanks: 2,145
Thanked 4,920 Times in 2,474 Posts
There's someone whose sample size overshadows other variables!

Thanks for posting that, Andrew.
__________________
Latest mods test: 15 mods = 15% MPG improvement: A-B test, 2007 Honda Civic 1.8L, 5-speed
Ecodriving test:
Manual vs. automatic transmission MPG showdown: Nissan Micra 1.6L



EcoModder
has launched a forum for the efficient new Mitsubishi Mirage
www.MetroMPG.com - fuel efficiency info for Geo Metro owners
www.ForkenSwift.com - electric car conversion on a beer budget
  Reply With Quote
Old 12-14-2009, 07:36 PM   #10 (permalink)
Batman Junior
 
MetroMPG's Avatar
 
Join Date: Nov 2007
Location: 1000 Islands, Ontario, Canada
Posts: 19,688

Blackfly - '98 Geo Metro
Team Metro
90 day: 70.72 mpg (US)

ForkenSwift - '92 Geo Metro EV
Last 3: 95.68 mpg (US)

MPGiata - '90 Mazda Miata
90 day: 47.99 mpg (US)
Thanks: 2,145
Thanked 4,920 Times in 2,474 Posts
Quote:
Originally Posted by Nerys View Post
as for evidence to support me only anecdotal evidence sadly.
Yeah, that's what this thread is about, really.

---

As for why you don't see as much of a swing in MPG vs. temperature, Pale's chart suggests a good reason: Note the temps where his fuel economy curve plateaus from roughly 65-70F and on up. Pretty darn close to the range of daily high temps in New Orleans.

(*edit* Correction on my part: I mistook "NO" in his user info to mean that Nerys lived in New Orleans.)

If you had air in your Metro, you'd notice a significant fuel economy hit.

__________________
Latest mods test: 15 mods = 15% MPG improvement: A-B test, 2007 Honda Civic 1.8L, 5-speed
Ecodriving test:
Manual vs. automatic transmission MPG showdown: Nissan Micra 1.6L



EcoModder
has launched a forum for the efficient new Mitsubishi Mirage
www.MetroMPG.com - fuel efficiency info for Geo Metro owners
www.ForkenSwift.com - electric car conversion on a beer budget
  Reply With Quote
Reply  Post New Thread


Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Test: Alternator vs. no alternator = 10% MPG gain @ 70 km/h MetroMPG EcoModding Central 363 12-06-2017 12:27 AM
Project: Rebuilding an '01 Honda Insight as a nonhybrid Fabio Hybrids 158 01-12-2013 12:59 PM
Test: 200w electric bicycle efficiency = 1512 MPG equivalent MetroMPG Fossil Fuel Free 30 07-03-2008 01:25 PM



Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Content Relevant URLs by vBSEO 3.5.2
All content copyright EcoModder.com