Fuel Economy, Hypermiling, EcoModding News and Forum - EcoModder.com

EcoModder.com (https://ecomodder.com/forum/)

- Instrumentation (https://ecomodder.com/forum/instrumentation.html)

- - Sharing Garmin code (https://ecomodder.com/forum/showthread.php/sharing-garmin-code-8089.html)

Sharing Garmin code

Hi,

This is a quick Perl hack used to convert the Garmin data file into text records that can be loaded into excel. Once in excel, macros can convert and reduce the data to useable representations:

#!/usr/bin/perl
#
# trace <file>
#
# This program decodes a Garmin nuvi trace data file.
# Found in the GARMIN\GPS\Current.gpx file.
#
open(FH, "<Current.gpx"); # Get the file
#
while ($line = <FH>) # Read all lines
{
printf("%d\tlength\n", length($line)); # Report size
#
# Convert raw block into records
#
$cc = index($line, ">"); # Find end of first record

while ($cc >= 0) # Pass through until all records found
{
$cc+=1; # Add ending, delimiter character
printf("%d\t%s\n", $cc, substr($line, 0, $cc)); # Each field
$line = substr($line, $cc, length($line)); # Grab rest
$cc = index($line, ">"); # Find end of next record
}

if ( length($line) > 0 ) # Check for any left-over text
{
printf("%d\t%s\n", length($line), $line); # Finish
}
printf("\n"); # Extra space to improve readibility
}
close(FH); # Done with file
# end source

There appears to be two legacy XML constructs that include a binary "NULL" character. This causes the read loop to see three blocks of text, not one huge block. Regardless, these legacy text are written as text records even though I don't have any use for them.

Macintosh and Linux systems have Perl but a Windows user may need to find a Perl system. The typical scenario is:

Mount the Garmin on the computer with the USB cable
Copy the 'GARMIN\GPS\Current.gpx' file to a directory with the Perl program
Run the Perl program and redirect the output into a text file
Use excel (or equivalent) to load the records

My plan is to add subroutines to assemble the trip data, <trkseq>, into files of tab delimited, data points, <trkpt>, with the format:

time_stamp - day and fraction
X axis miles - relative to the left or western most longitude
y axis miles - relative to the bottom or sourthern most latitude
mph - calculated from the preceding and following points and times
altitude - meters from GPS data

Another routine will extract the way-points with the names and X and Y axis miles. Eventually, I'll combine the trace data with the checkpoint so we can identify the start of each "coast down" segment, the ultimate goal.

Bob Wilson

Thanks Bob, I assume you are talking about a GPS here :)

Just a suggestion, if the destination is Excel, that perl code would probably translate into an excel macro (read the raw garmin file straight from excel) and might be a bit more user friendly as you can avoid copy/convert/install perl steps.

It looks like gpx is a standard XML schema based format, can be read directly by most anything these days without writing your own parser or resorting to perl ;)

for this excel example I had a gpx file at c:\tmp.gpx (actually I just saved http://www.topografix.com/fells_loop.gpx since I don't have a gps)

I went into Excel
then hit:
Data->Import External Data->New Web Query
Then for address I put in the location of the gpx file:
file:///C:/tmp.gpx

And then I was looking at tons of data :) I had to scroll right to see the parts that change since it is expanded to rows.

fyi, using excel 2002

I appreciate the heads up on excel. With my older excel, I get just three rows and a fraction of the data.

The problem with most spreadsheets is a limit on the number of rows, typically 64k rows. The browser Camino performs the xml parsing perfectly and I could probably cut-and-paste into excel but that doesn't solve the scaling problem.

The Perl program reduces the number of rows (aka, records) from 7,122 to 1,158 and this was just one day's worth of data, four trips. A full tank or more of data could easily exceed the analysis capacity of any spreadsheet. But I think you've suggested an excellent, parallel approach.

It sounds as if the more recent spreadsheets with xml parsing could import Garmin xml data directly. With a well crafted set of macros, the spreadsheet could identify the 'coast down' data and solve the data recording problem of 'coast down' testing. Sad to say, my old excel won't be able to help.

Bob Wilson

Hi,

Here is the latest source code for my Perl script. It now takes a P1 argument to open a specific file.

GOOD LUCK!
Bob Wilson

I'm glad to see the interest in things computer, but perl is not the best choice for xml schema handling, or arguably much else. Here is a python example of reading gpx and making csv files out of them.

Python and XML -- by Mike Hostetler ,

The program is trivial in python (this is the whole thing, I just use php tags cuz it looks pretty, it is python), worked as-is for me:

PHP Code:


		
			
import cElementTree as ET
import string

if __name__ == '__main__':

    mainNS=string.Template("{http://www.topografix.com/GPX/1/0}$tag")

    wptTag=mainNS.substitute(tag="wpt")
    nameTag=mainNS.substitute(tag="name")

    et=ET.parse(open("everything.gpx"))

    for wpt in et.findall("//"+wptTag):
        wptinfo=[]
        wptinfo.append(wpt.get("lat"))
        wptinfo.append(wpt.get("lon"))
        wptinfo.append(wpt.findtext(nameTag))

        print ",".join(wptinfo)

and here was the output ( after I saved http://www.topografix.com/fells_loop.gpx to everything.gpx )

Code:

42.438878,-71.119277,5066

42.439227,-71.119689,5067

42.438917,-71.116146,5096

42.443904,-71.122044,5142

42.447298,-71.121447,5156

42.454873,-71.125094,5224

42.459079,-71.124988,5229

42.456979,-71.124474,5237

42.454401,-71.120990,5254

42.451442,-71.121746,5258

42.454404,-71.120660,5264

42.457761,-71.121045,526708

42.457089,-71.120313,526750

42.456592,-71.119676,527614

42.456252,-71.119356,527631

42.458148,-71.119135,5278

42.459377,-71.117693,5289

42.464183,-71.119828,5374FIRE

42.465650,-71.119399,5376

42.439018,-71.114456,6006

42.438594,-71.114803,6006BLUE

42.436757,-71.113223,6014MEADOW

42.441754,-71.113220,6029

42.436243,-71.109075,6053

42.439250,-71.107500,6066

42.439764,-71.107582,6067

42.434766,-71.105874,6071

42.433304,-71.106599,6073

42.437338,-71.104772,6084

42.442196,-71.110975,6130

42.442981,-71.111441,6131

42.444773,-71.108882,6153

42.443592,-71.106301,6171

42.447804,-71.106624,6176

42.448448,-71.106158,6177

42.453415,-71.106783,6272

42.453434,-71.107253,6272

42.458298,-71.106771,6278

42.451430,-71.105413,6280

42.453845,-71.105206,6283

42.459986,-71.106170,6289

42.457616,-71.105116,6297

42.467110,-71.113574,6328

42.464202,-71.109863,6354

42.466459,-71.110067,635722

42.466557,-71.109410,635783

42.463495,-71.107117,6373

42.401051,-71.110241,6634

42.432621,-71.106532,6979

42.431033,-71.107883,6997

42.465687,-71.107360,BEAR HILL

42.430950,-71.107628,BELLEVUE

42.438666,-71.114079,6016

42.456469,-71.124651,5236BRIDGE

42.465759,-71.119815,5376BRIDGE

42.442993,-71.105878,6181CROSS

42.435472,-71.109664,6042CROSS

42.458516,-71.103646,DARKHOLLPO

42.443109,-71.112675,6121DEAD

42.449866,-71.119298,5179DEAD

42.459629,-71.116524,5299DEAD

42.465485,-71.119148,5376DEAD

42.462776,-71.109986,6353DEAD

42.446793,-71.108784,6155DEAD

42.451204,-71.126602,GATE14

42.458499,-71.122078,GATE16

42.459376,-71.119238,GATE17

42.466353,-71.119240,GATE19

42.468655,-71.107697,GATE21

42.456718,-71.102973,GATE24

42.430847,-71.107690,GATE5

42.431240,-71.109236,GATE6

42.439502,-71.106556,6077LOGS

42.449765,-71.122320,5148NANEPA

42.457388,-71.119845,5267OBSTAC

42.434980,-71.109942,PANTHRCAVE

42.453256,-71.121211,5252PURPLE

42.457734,-71.117481,5287WATER

42.459278,-71.124574,5239ROAD

42.458782,-71.118991,5278ROAD

42.439993,-71.120925,5058ROAD

42.453415,-71.106782,SHEEPFOLD

42.455956,-71.107483,SOAPBOX

42.465913,-71.119328,5376STREAM

42.445359,-71.122845,5144SUMMIT

42.441727,-71.121676,5150TANK

Thanks!

I'm not one to worry too much about which language is used but I do have a couple of suggestion-questions:

Are those 'way points' in the output file?
I didn't see 'trips' in the data - every time the Garmin is powered up, it starts a new 'trip' in the data trace. I didn't see separate traces.
Altitude? - I didn't see anything that relates to altitude. When doing coast down tests or hill climb or specific trip analysis, altitude is a very nice thing to have.

I'll try to post some sample data later this evening.

Bob Wilson

I would love to see a real data file. I looked at the gpx schema data definiton(s) at http://www.topografix.com/GPX/1/0/gpx.xsd but did not see any elements named "trip", maybe "trip" is a data value?

here it is with ele added, (and updated for python 2.5+) had to make the ele append conditional because one entry didn't have it, which is legal because ele is defined as minOccurs="0" in the xsd (as is "name" BTW).

These are data from waypoints, identified by wpt tags in the xml.

PHP Code:


		
			
import xml.etree.cElementTree as ET
import string

if __name__ == '__main__':

    mainNS=string.Template("{http://www.topografix.com/GPX/1/0}$tag")

    wptTag=mainNS.substitute(tag="wpt")
    nameTag=mainNS.substitute(tag="name")
    eleTag=mainNS.substitute(tag="ele")

    et=ET.parse(open("everything.gpx"))

    for wpt in et.findall("//"+wptTag):
        wptinfo=[]
        wptinfo.append(wpt.get("lat"))
        wptinfo.append(wpt.get("lon"))
        wptinfo.append(wpt.findtext(nameTag))
        if (wpt.findtext(eleTag)):
            wptinfo.append(wpt.findtext(eleTag))

        print ",".join(wptinfo)

getting started (windows), if anyone is interested

installed python 2.6.2 (msi)
Download Python Software

created a working directory

saved the code to gpx.py in the working dir

saved the .gpx data file to the same directory as everything.gpx

ran: gpx.py from the working dir
output:
C:\Python26\wip>gpx.py
42.438878,-71.119277,5066,44.586548
42.439227,-71.119689,5067,57.607200
42.438917,-71.116146,5096,44.826904
42.443904,-71.122044,5142,50.594727
42.447298,-71.121447,5156,127.711200
42.454873,-71.125094,5224,96.926400
42.459079,-71.124988,5229,82.600800
...

If you want to move to the dark side and get a TomTom, TripMaster is an easy to use addon program that does all of this and can store into in a variety of formats.

Here is an example of my output:
$ ./trace Current.gpx
-1 Bob Work 34.726111 -86.664279 -0.110000
-1 Coast 01 34.667243 -86.588927 168.840000
-1 Coast 02 34.662488 -86.597305 167.870000
-1 Coast 03 34.638889 -86.628989 175.080000
-1 Coast 04 34.609462 -86.630221 178.450000
-1 Coast 05 34.601898 -86.630350 177.250000
-1 Coast 06 34.628383 -86.629168 172.920000
-1 Dr Haynes 36.155062 -86.801553 114.040000
-1 Duckworth 34.691953 -86.572217 193.350000
-1 Hair Care Lifestyles 34.661346 -86.540009 192.630000
-1 Home 34.652834 -86.571580 184.220000
-1 I-565 W 34.717527 -86.637090 -0.110000
-1 Propst 34.738917 -86.575751 -0.110000
-1 Saigon 34.680198 -86.747715 189.980000
0 2009-04-12 17:09:08 34.653001 -86.572144 182.700000
0 2009-04-12 17:09:34 34.652971 -86.571829 177.000000
0 2009-04-12 17:09:46 34.652704 -86.571851 176.500000
0 2009-04-12 17:09:49 34.652696 -86.571843 177.000000
0 2009-04-12 17:09:57 34.652682 -86.572242 173.100000
0 2009-04-12 17:10:12 34.652682 -86.573432 165.900000

The first entries tagged "-1" are waypoints. The ones with "-0.11" altitude are manual entries. Then comes the first data track. In the XML, this is what I look for:

<trk>
<name>ACTIVE LOG: 12 APR 2009 12:09</name>
−
<trkseg>
−
<trkpt lat="34.653001" lon="-86.572144">
<ele>182.79</ele>
<time>2009-04-12T17:09:08Z</time>
</trkpt>
−
<trkpt lat="34.652971" lon="-86.571829">
<ele>177.02</ele>
<time>2009-04-12T17:09:34Z</time>
</trkpt>
−
<trkpt lat="34.652704" lon="-86.571851">
<ele>176.54</ele>
<time>2009-04-12T17:09:46Z</time>
</trkpt>
−
<trkpt lat="34.652696" lon="-86.571843">
<ele>177.02</ele>
<time>2009-04-12T17:09:49Z</time>
</trkpt>
−
<trkpt lat="34.652682" lon="-86.572242">
<ele>173.17</ele>
. . .

Bob Wilson