How to properly test modifications (AKA: don't believe every MPG claim you read)

[RiderofBikes]

wmjinman,

No i get where your coming from, and it's as simple as what were doing these days already.
Albeit, You don't need crazy individual "precise" statistics to find something out. The fact the Current Basis way of testing includes all the variables. Which convenient, could still have a greater margin.

What i'm simply referring too, is to the point where, at what detail circumstance can you inform yourself that this is optimum? Pure EV's would have a greater significant +/- range issue based on Temps, not so much elevation tho(unless your climbing).

With your previous experience as an Surveyor, If it LOOKS right, is a tool for all of us used here and on our projects anyways. You probably weren't paid to just lolligag around either, requiring you to not be so "precise" but within the constraints of whatever your task was. Not everyone is lucky enough to have such consistant results, so your obviously doing something right overall.

Lets say,(Pre warm up of all parts both tests) You result 1.3mpg increase with higher tire pressure at Ambient 80-85* without any other extreme variables mixed in.. Then Test the same psi during Ambient 25-30*(without any other "stuff on the road ways" to compromise RR), would Estimate only a 0.9mpg increase?

Of course, I specifically don't know that for sure... It's just a hypothesis to me(to anyone who doesn't know, especially, when types/brands/tire sizes are compared), given nature being much more forgiving in higher temps than in cold, excluding Warm-up times. Who knows, maybe the temp doesn't make a difference?
Maybe comparable "LRR" tires, same size, used on same vehicle under silmilar circumstances, X Brand has .3mpg loss given temps, compared to Y Brand having .8mpg loss due to temp.

There may be a trend in results when it covers more detail than just a 1 day testing including (X variables).

[wmjinman]

RiderofBikes,

Yeah, I think I "get" what you're saying, too. And you're absolutely right that we had "tolerances" in our surveys. Oftentimes we'd turn 2 "sets" of angles (direct and "reverse" with the instrument), and if they were within 8 seconds of angle, we were "good". If the spread was more, we'd do another one. Now, in the winter, especially, when the frozen ground might be melting under the tripod foot (or feet) as we're turning the angles, the results might "climb". Like, the first set might be 21.2, the second one 29.9. That's over 8 seconds, so we'd do another. But what if it was 31.3? That's well within the 8 second "window" of the second angle, but why was the first one so low? In other words, it kept climbing. What we would like to see would be a third one in between the first two. So in the winter time, what we'd do is go check the sights and re-set the "gun" (the survey instrument). In the summertime, if we were set up on the "oil" (asphalt pavement), the feet could actually sink in if it was soft enough. And watch out if one foot was in the sun and the other two were in the shade (or any combination like that!!)

So I'm very aware of outside influences. And like you say, just because raising the air pressure in the tires 10 psi in the summer gives you 1.3 more mpg, you can't automatically assume it will work exactly the same on a different car, or with different tires, or at a different time of the year. Although, I think it's safe to assume from this that it will probably yield an improvement of some sort. Of course, another variable is the pressure you're working with. Chances are, raising it from 15psi to 25, will probably yield even better results than from 45 to 55. And I would suspect (assuming it's even possible) that raising it from 90 psi to 100 would probably benefit LESS.

Even now, I'm wondering if folding the mirrors in is more "worthwhile" at higher speeds? Because the wind drag probably isn't as severe at 50. If I did it at 70, I wonder if the savings would go way up? So I see your point that to REALLY KNOW, it takes a lot of data.

Also, I re-read part of this whole thread, and see MetroMPG was primarily talking about the kind of testing that should be done before trying to "separate people's money from their wallets". And, yes, I absolutely agree!! back to my mirror test, just becasue on one day, at 50 mph, my Jimmy "only" got 0.1 (or 0.2) mpg better and I'm thinking maybe it isn't "worth it", certainly wouldn't take that as "sufficient evidence" to start selling rearview TV cameras and trying to convince people to delete their mirrors. Uh, wait - that doesn't make sense. What I mean is if the mirror folding had looked like it helped a LOT on that one test, I wouldn't decide that it helps a LOT on ALL cars at all speeds.

These tests of mine were also in Washoe Valley, elevation 5200 ft. The air is a lot thinner up there than where 90% of the population lives nearer sea level. So my suspicion is that the aero forces are LESS significant in Washoe Valley. But it's near where I live and where I drive, so whaddam I gonna do, right?

[RiderofBikes]

This has been productive

I think a good "base" to test any modification is with non-hypermilling driving habits 1st. (obviously) In the most controlled manor possible.

Do your 1st A-B-A test at posted speeds,
Then do the A-B-A test over with your own driving technique.

[darrylrobida]

Originally Posted by MetroMPG

Here on EcoModder we see a lot of claims made about the effectiveness (or lack of effectiveness) of specific modifications. It's an understatement to say not all claims are made equal.

Just because someone says a modification worked for them, that does not make it "proof".

When looking at the claims a person or company makes about a modification, we need to understand that the the likelihood their claim is true or not is directly related to whether they know how to conduct a proper test.

So we should be respectfully skeptical of fuel economy claims made until we know the details of how the modification was tested. We should remain skeptical if the testing was weak or the test details aren't given.

(Note that "skepticism" doesn't equal "disbelief".)

-------------------
Lab testing is king
-------------------

Testing done on a dynamometer or in a wind tunnel is ideal, since it reduces to a minium the number of variables that can affect the outcome.

(We should be especially skeptical of companies selling supposed fuel saving products that don't do lab testing or offer the testing details to consumers. After all, if they're convinced their product works, they should feel confident about investing in the cost of high quality testing. Wouldn't it improve the product's sales potential?)

Of course, it's still entirely possible to screw up lab testing, but it's the best starting point.


------------------------------------------------
On-road testing: the poor cousin of lab testing
------------------------------------------------

Unfortunately, not too many EcoModder members have access to chassis dynamometers or wind tunnels. This leaves on-road testing as the next option, and so right off the bat the potential quality of the results goes down.

Why? Because outside of a lab, the number of variables that can distort a test goes waaaaaay up.

That said, there are different degrees of on-road testing. It can range from truly useless "junk science" to reasonably acceptable experimentation. The difference boils down to the amount of effort made to reduce the number of variables that can influence the outcome.


---------------------------------------
What's the big deal about variability?
---------------------------------------

In short: we need to be confident that the results we are seeing are from the modification being tested, and not from some outside factor(s).

Fuel consumption is remarkably sensitive to all kinds of variables:

• ambient temperature
• wind
• humidity
• barometric pressure
• vehicle temperature (not just engine, but the entire drivetrain including transmission, joints, bearings, tires)
• fuel grade/quality
• payload
• elevation/road grade
• road surface conditions / type
• speed
• other traffic
• driving style / driver psychology

All of these factors have an impact on fuel use through changing engine load, aerodynamic load, rolling resistance, chemical reactivity and thermodynamic efficiency.

Most individual vehicle mods have a very small potential impact on overall fuel consumption - typically in the range of a couple of percent. So you can see how trying to conduct a test while all kinds of other variables are changing would make the results meaningless.

Not only could you end up seeing an improvement that isn't actually caused by the modification, the reverse is also true: Uncontrolled variables could also prevent an actual improvement from showing up in the data.

The reason lab testing is king is because it can eliminate the highest number of these variables from a test.


--------------------------------------------------------
How to do "as controlled as possible" on-road testing
--------------------------------------------------------

So we've established that we should be automatically skeptical of on-road testing. Just how skeptical depends on how many variables the tester has managed to eliminate.

The further you get from this following example, the less confidence we should have in the results.

1) The vehicle should be fully warmed up, including drivetrain / tires.

2) Remove the driver's foot from the test, meaning cruise control should be used (set once, and cancelled with the brake between runs to ensure the same speed in multiple runs).

Any testing that can be affected by driver input - eg. city driving is the worst - is dramatically less scientific. A driver may unconsciously change driving style to get the desired result (experimenter bias). An ideal experiment would be double blind.

Failing to "remove the driver" as much as possible from the test is a huge red flag and we should be very skeptical of conclusions made.

3) The route should be devoid of other traffic, to avoid significant aerodynamic impacts of vehicles ahead or overtaking.

4) Weather conditions should be as calm and stable as possible (wind gusts/changes in wind speed & temperature changes will affect results). Evening or night time testing can be preferable where atmospheric conditions are typically calmer, and there may be less traffic.

5) The route should be as flat & straight as possible.

6) Bi-directional runs should be done to average out effects of grade/wind, if present. Meaning, test on the route in both directions.

7) Use A-B-A comparisons. That means establishing a baseline (the first "A" set of runs), more test runs after making a change (the "B" set), and then additional runs after undoing the change (the last "A" set).

Why do A-B-A testing? Undoing the change and immediately re-testing the final "A" set increases confidence that any difference seen in the "B" runs was caused by the modification, and not by other uncontrolled factors.

A-B-A illustration: (from Drive-cycle economy and emissions measurement - Fuel saving gadgets - a professional engineer's view)



8) A-B-A comparison runs should be done immediately one after the other to minimize the effects of changing weather conditions, vehicle temperature, weight, fuel quality, etc.

9) Be prepared to abandon runs with unexpected changes (eg. a car overtaking you, affecting aerodynamic drag)

10) The more runs the better (larger data set = higher confidence)

11) Share your raw data.


---------------------------------
Drat! Quality testing isn't easy
---------------------------------

It can actually be a big pain in the butt, and a time-eater too. (I'll often wait a week or more for calm, stable weather, and then spend 2-3 hours doing a simple A-B-A test.)

(Note, I'm not suggesting by this post that my own testing is perfect. I've done some stinker tests myself. And if I refer to the less than ideal ones, I try to point out their flaws.)

In the end, if we're not willing to do testing that's "as controlled as possible", we should have the intellectual honesty to admit that the results are questionable, and we should be wary of drawing conclusions from the data.


------
Links
------

• Empirical skepticism (Wikipedia)

• Experimenter's bias (Wiki)

• The dangers of testimonial evidence - Fuel saving gadgets - a professional engineer's view

• Drive-cycle economy and emissions measurement - Fuel saving gadgets - a professional engineer's view

---------------------------------------------
Topics that may be added to this post later
---------------------------------------------

• figuring out the "quality" of data gathered in a test

• examples of bad on-road testing

• explanation of why on-road testing fuel/oil additives is especialy problematic

• "Invoking Carl Sagan" - or why extraordinary or controversial claims (or commercial claims) should be held to a higher standard of proof - see: http://ecomodder.com/forum/showthrea...uel-15251.html

• Other testing methods: eg. coastdown, for A-B-A of aero mods

Thanks for a very well written post on the benefits of lab testing.
Inaccurate MPG claims can confuse the ecomodder community.
I have found computer based monitors in vivo to be needing better
calibration. In particular, the VSS (Vehicle Speed Sensor) can vary
with effective tire radius which varies with inflation pressure and
actual weight load and tire temperature. And, the fuel injector
open time is not an exact measure of the actual fuel consumption.
Therefore, I'm planning to do my MPG calculations with a graduated cylinder
mounted directly on top of my fuel tank and my distance calculations with
an unloaded trailing bicycle wheel style odometer.

[RustyLugNut]

I have found that a small secondary tank that I can quickly attach and remove for weighing on an accurate scale is the best way to verify fuel use. Of course it depends on your application.

I also use an early model Garmin Hiking GPS that is accurate to 1/100th of a mile for distances traveled.

This setup is much easier on my old Mercedes and vehicles with inline fuel pumps versus in-tank pumps. But, it yields accurate results.

[rattroddrebel]

Originally Posted by Daox

Very nice compilation Darin.

I support the idea of what you are stating, however there are alot of posters that have personally crapped on anything I said, some are in your click, but i am going back to the drawing board and I will come back to this site when I have data that will NOT be argued. I have had numorous negative responses to my posts just because I am new here, or these guys are not objective. So I will follow your guidelines unlike several members and bring my data to the table. If i would have known I would get so much crap for giving information that I thought could help people, instead nothing but headaches. Thanks for the data suggestions.

[wmjinman]

Originally Posted by rattroddrebel

I support the idea of what you are stating, however there are alot of posters that have personally crapped on anything I said, some are in your click, but i am going back to the drawing board and I will come back to this site when I have data that will NOT be argued. I have had numorous negative responses to my posts just because I am new here, or these guys are not objective. So I will follow your guidelines unlike several members and bring my data to the table. If i would have known I would get so much crap for giving information that I thought could help people, instead nothing but headaches. Thanks for the data suggestions.

Hey rattroddrebel, let me try to apologize a little. I read Old Mechanic's responses and am very impressed by the clearness and coolness with which he makes his case(s). I, on the other hand, will get all worked up, then get sarcastic in my comments, and come off as an insulting a$$hole. Maybe I can try to be more like him....

A couple of the other guys made the point that your "delivery" of info/facts/opinions can easily be read (mis-read?) to possibly give the wrong idea. An example in this particular post, (..."just because I'm new here"...) Although your number of posts was mentioned by some, to me, that's not what it is. I only joined in October, myself. Rather, it's the - how can I say this - "barrage" of supposed "documentation" and "facts", etc. And then when questioned, an even bigger barrage of so-called facts, etc. along with your constant insistance of what an expert you are. Maybe you are, but to me, it was getting ridiculous, you were sounding (to me) just like a high-pressure, pushy salesman, and I started responing in my a$$hole way.

So again, I apologize for that. My main thing is this: Lots of "claims" are made in the field of fuel economy, and a lot of them simply don't hold up when put to the test. I was just offering to put this fuel additive to the test. If it really works, I'll see it in my test. If it doesn't (or the improvement is too small to show), then I'd think (in my own mind) that it's not worth buying. I've heard that keeping your car clean and waxed helps MPG too (better aerodynamics), but I don't see it in actual tests.

If that AmsOil stuff shows an improvement, I'll happily report it - and quite possibly start using it. And I realize this is ONLY as it applies to my car with all those factors you keep listing. It would be good if someone else here tested it, too, to either confirm or dispute to my results. Then we can take it from there.

But I guess my other thing is, if it "really works", it should work regardless of the ambient outside temperature, size of my engine, my tires, brand of oil, make & model of my car, etc. In other words, if it only works for cars meeting certain of these criteria, then maybe advertize it as working for only those criteria. I think that's what triggered all my sarcasm, etc.

Oh, and I don't think I attacked "everything you said" - just the stuff I thought was ridiculous (highly exaggerated). Too bad your car is too old for a ScanGauge - those things are worth their weight in gold. You might look into the other one they talk about here - the "MPGuino", or whatever it's called. Anyway, I'd better shut up now. - peace.

[kanoshee]

[QUOTE=PaleMelanesian;147590]Digging out my graphs... give me a minute...

Here it is. 2 full years of daily data (1500+ data points), averaged for each degree level. I discarded the whole first year of data, because my driving changed so much the data was useless.

Of course, this only applies to ME driving MY CAR in MY LOCATION - standard disclaimer about YMMV.

i have to get the feeling you crank it and let it warm up during the cold and the two spikes at the coldest were way down any note on that or is that just the data? and may i ask how long your commute is? and what does ymmv stand for?

[RedDevil]

Originally Posted by kanoshee

i have to get the feeling you crank it and let it warm up during the cold and the two spikes at the coldest were way down any note on that or is that just the data? and may i ask how long your commute is? and what does ymmv stand for?

I want to know too, as I find I get good mileage in the cold once my car is heated up, generally after 10-15 miles when it is freezing.
When it is warm it obviously needs less time and miles to reach that point.
A short commute would lead to large variation by ambient temperature, a long commute much less so.
Using a block heater may straighten this out a bit.

YMMV stands for Your Mileage May Vary BTW.

[PaleMelanesian]

It's a 10 mile commute. I don't warm up the car, just crank and go.

Here's the graph in question. The blip at low temperature is an artifact of having only a few data points down there. It's smoother in the middle where I have more data. This graph covers 2 years of driving.


Here's an updated one that covers 4 years of driving. The data points are smoother, and the linear temperature/mpg trend is clearer.