Sharing Garmin code

[bwilson4web]

Hi,

This is a quick Perl hack used to convert the Garmin data file into text records that can be loaded into excel. Once in excel, macros can convert and reduce the data to useable representations:

#!/usr/bin/perl
#
# trace <file>
#
# This program decodes a Garmin nuvi trace data file.
# Found in the GARMIN\GPS\Current.gpx file.
#
open(FH, "<Current.gpx"); # Get the file
#
while ($line = <FH>) # Read all lines
{
printf("%d\tlength\n", length($line)); # Report size
#
# Convert raw block into records
#
$cc = index($line, ">"); # Find end of first record

while ($cc >= 0) # Pass through until all records found
{
$cc+=1; # Add ending, delimiter character
printf("%d\t%s\n", $cc, substr($line, 0, $cc)); # Each field
$line = substr($line, $cc, length($line)); # Grab rest
$cc = index($line, ">"); # Find end of next record
}

if ( length($line) > 0 ) # Check for any left-over text
{
printf("%d\t%s\n", length($line), $line); # Finish
}
printf("\n"); # Extra space to improve readibility
}
close(FH); # Done with file
# end source

There appears to be two legacy XML constructs that include a binary "NULL" character. This causes the read loop to see three blocks of text, not one huge block. Regardless, these legacy text are written as text records even though I don't have any use for them.

Macintosh and Linux systems have Perl but a Windows user may need to find a Perl system. The typical scenario is:

1. Mount the Garmin on the computer with the USB cable
2. Copy the 'GARMIN\GPS\Current.gpx' file to a directory with the Perl program
3. Run the Perl program and redirect the output into a text file
4. Use excel (or equivalent) to load the records

My plan is to add subroutines to assemble the trip data, <trkseq>, into files of tab delimited, data points, <trkpt>, with the format:

time_stamp - day and fraction
X axis miles - relative to the left or western most longitude
y axis miles - relative to the bottom or sourthern most latitude
mph - calculated from the preceding and following points and times
altitude - meters from GPS data

Another routine will extract the way-points with the names and X and Y axis miles. Eventually, I'll combine the trace data with the checkpoint so we can identify the start of each "coast down" segment, the ultimate goal.

Bob Wilson

[dcb]

Thanks Bob, I assume you are talking about a GPS here

Just a suggestion, if the destination is Excel, that perl code would probably translate into an excel macro (read the raw garmin file straight from excel) and might be a bit more user friendly as you can avoid copy/convert/install perl steps.

[dcb]

It looks like gpx is a standard XML schema based format, can be read directly by most anything these days without writing your own parser or resorting to perl

for this excel example I had a gpx file at c:\tmp.gpx (actually I just saved http://www.topografix.com/fells_loop.gpx since I don't have a gps)

I went into Excel
then hit:
Data->Import External Data->New Web Query
Then for address I put in the location of the gpx file:
file:///C:/tmp.gpx

And then I was looking at tons of data I had to scroll right to see the parts that change since it is expanded to rows.

fyi, using excel 2002

[bwilson4web]

I appreciate the heads up on excel. With my older excel, I get just three rows and a fraction of the data.

The problem with most spreadsheets is a limit on the number of rows, typically 64k rows. The browser Camino performs the xml parsing perfectly and I could probably cut-and-paste into excel but that doesn't solve the scaling problem.

The Perl program reduces the number of rows (aka, records) from 7,122 to 1,158 and this was just one day's worth of data, four trips. A full tank or more of data could easily exceed the analysis capacity of any spreadsheet. But I think you've suggested an excellent, parallel approach.

It sounds as if the more recent spreadsheets with xml parsing could import Garmin xml data directly. With a well crafted set of macros, the spreadsheet could identify the 'coast down' data and solve the data recording problem of 'coast down' testing. Sad to say, my old excel won't be able to help.

Bob Wilson

[bwilson4web]

Hi,

Here is the latest source code for my Perl script. It now takes a P1 argument to open a specific file.

GOOD LUCK!
Bob Wilson

[dcb]

I'm glad to see the interest in things computer, but perl is not the best choice for xml schema handling, or arguably much else. Here is a python example of reading gpx and making csv files out of them.

Python and XML -- by Mike Hostetler ,

The program is trivial in python (this is the whole thing, I just use php tags cuz it looks pretty, it is python), worked as-is for me:

PHP Code:

import cElementTree as ET
import string

if __name__ == '__main__':

    mainNS=string.Template("{http://www.topografix.com/GPX/1/0}$tag")

    wptTag=mainNS.substitute(tag="wpt")
    nameTag=mainNS.substitute(tag="name")

    et=ET.parse(open("everything.gpx"))

    for wpt in et.findall("//"+wptTag):
        wptinfo=[]
        wptinfo.append(wpt.get("lat"))
        wptinfo.append(wpt.get("lon"))
        wptinfo.append(wpt.findtext(nameTag))

        print ",".join(wptinfo) 


and here was the output ( after I saved http://www.topografix.com/fells_loop.gpx to everything.gpx )

42.438878,-71.119277,5066
42.439227,-71.119689,5067
42.438917,-71.116146,5096
42.443904,-71.122044,5142
42.447298,-71.121447,5156
42.454873,-71.125094,5224
42.459079,-71.124988,5229
42.456979,-71.124474,5237
42.454401,-71.120990,5254
42.451442,-71.121746,5258
42.454404,-71.120660,5264
42.457761,-71.121045,526708
42.457089,-71.120313,526750
42.456592,-71.119676,527614
42.456252,-71.119356,527631
42.458148,-71.119135,5278
42.459377,-71.117693,5289
42.464183,-71.119828,5374FIRE
42.465650,-71.119399,5376
42.439018,-71.114456,6006
42.438594,-71.114803,6006BLUE
42.436757,-71.113223,6014MEADOW
42.441754,-71.113220,6029
42.436243,-71.109075,6053
42.439250,-71.107500,6066
42.439764,-71.107582,6067
42.434766,-71.105874,6071
42.433304,-71.106599,6073
42.437338,-71.104772,6084
42.442196,-71.110975,6130
42.442981,-71.111441,6131
42.444773,-71.108882,6153
42.443592,-71.106301,6171
42.447804,-71.106624,6176
42.448448,-71.106158,6177
42.453415,-71.106783,6272
42.453434,-71.107253,6272
42.458298,-71.106771,6278
42.451430,-71.105413,6280
42.453845,-71.105206,6283
42.459986,-71.106170,6289
42.457616,-71.105116,6297
42.467110,-71.113574,6328
42.464202,-71.109863,6354
42.466459,-71.110067,635722
42.466557,-71.109410,635783
42.463495,-71.107117,6373
42.401051,-71.110241,6634
42.432621,-71.106532,6979
42.431033,-71.107883,6997
42.465687,-71.107360,BEAR HILL
42.430950,-71.107628,BELLEVUE
42.438666,-71.114079,6016
42.456469,-71.124651,5236BRIDGE
42.465759,-71.119815,5376BRIDGE
42.442993,-71.105878,6181CROSS
42.435472,-71.109664,6042CROSS
42.458516,-71.103646,DARKHOLLPO
42.443109,-71.112675,6121DEAD
42.449866,-71.119298,5179DEAD
42.459629,-71.116524,5299DEAD
42.465485,-71.119148,5376DEAD
42.462776,-71.109986,6353DEAD
42.446793,-71.108784,6155DEAD
42.451204,-71.126602,GATE14
42.458499,-71.122078,GATE16
42.459376,-71.119238,GATE17
42.466353,-71.119240,GATE19
42.468655,-71.107697,GATE21
42.456718,-71.102973,GATE24
42.430847,-71.107690,GATE5
42.431240,-71.109236,GATE6
42.439502,-71.106556,6077LOGS
42.449765,-71.122320,5148NANEPA
42.457388,-71.119845,5267OBSTAC
42.434980,-71.109942,PANTHRCAVE
42.453256,-71.121211,5252PURPLE
42.457734,-71.117481,5287WATER
42.459278,-71.124574,5239ROAD
42.458782,-71.118991,5278ROAD
42.439993,-71.120925,5058ROAD
42.453415,-71.106782,SHEEPFOLD
42.455956,-71.107483,SOAPBOX
42.465913,-71.119328,5376STREAM
42.445359,-71.122845,5144SUMMIT
42.441727,-71.121676,5150TANK

[bwilson4web]

Thanks!

I'm not one to worry too much about which language is used but I do have a couple of suggestion-questions:

• Are those 'way points' in the output file?
• I didn't see 'trips' in the data - every time the Garmin is powered up, it starts a new 'trip' in the data trace. I didn't see separate traces.
• Altitude? - I didn't see anything that relates to altitude. When doing coast down tests or hill climb or specific trip analysis, altitude is a very nice thing to have.

I'll try to post some sample data later this evening.

Bob Wilson

[dcb]

I would love to see a real data file. I looked at the gpx schema data definiton(s) at http://www.topografix.com/GPX/1/0/gpx.xsd but did not see any elements named "trip", maybe "trip" is a data value?

here it is with ele added, (and updated for python 2.5+) had to make the ele append conditional because one entry didn't have it, which is legal because ele is defined as minOccurs="0" in the xsd (as is "name" BTW).

These are data from waypoints, identified by wpt tags in the xml.

PHP Code:

import xml.etree.cElementTree as ET
import string

if __name__ == '__main__':

    mainNS=string.Template("{http://www.topografix.com/GPX/1/0}$tag")

    wptTag=mainNS.substitute(tag="wpt")
    nameTag=mainNS.substitute(tag="name")
    eleTag=mainNS.substitute(tag="ele")

    et=ET.parse(open("everything.gpx"))

    for wpt in et.findall("//"+wptTag):
        wptinfo=[]
        wptinfo.append(wpt.get("lat"))
        wptinfo.append(wpt.get("lon"))
        wptinfo.append(wpt.findtext(nameTag))
        if (wpt.findtext(eleTag)):
            wptinfo.append(wpt.findtext(eleTag))

        print ",".join(wptinfo) 


getting started (windows), if anyone is interested

installed python 2.6.2 (msi)
Download Python Software

created a working directory

saved the code to gpx.py in the working dir

saved the .gpx data file to the same directory as everything.gpx

ran: gpx.py from the working dir
output:
C:\Python26\wip>gpx.py
42.438878,-71.119277,5066,44.586548
42.439227,-71.119689,5067,57.607200
42.438917,-71.116146,5096,44.826904
42.443904,-71.122044,5142,50.594727
42.447298,-71.121447,5156,127.711200
42.454873,-71.125094,5224,96.926400
42.459079,-71.124988,5229,82.600800
...

[Who]

If you want to move to the dark side and get a TomTom, TripMaster is an easy to use addon program that does all of this and can store into in a variety of formats.

[bwilson4web]

Here is an example of my output:
$ ./trace Current.gpx
-1 Bob Work 34.726111 -86.664279 -0.110000
-1 Coast 01 34.667243 -86.588927 168.840000
-1 Coast 02 34.662488 -86.597305 167.870000
-1 Coast 03 34.638889 -86.628989 175.080000
-1 Coast 04 34.609462 -86.630221 178.450000
-1 Coast 05 34.601898 -86.630350 177.250000
-1 Coast 06 34.628383 -86.629168 172.920000
-1 Dr Haynes 36.155062 -86.801553 114.040000
-1 Duckworth 34.691953 -86.572217 193.350000
-1 Hair Care Lifestyles 34.661346 -86.540009 192.630000
-1 Home 34.652834 -86.571580 184.220000
-1 I-565 W 34.717527 -86.637090 -0.110000
-1 Propst 34.738917 -86.575751 -0.110000
-1 Saigon 34.680198 -86.747715 189.980000
0 2009-04-12 17:09:08 34.653001 -86.572144 182.700000
0 2009-04-12 17:09:34 34.652971 -86.571829 177.000000
0 2009-04-12 17:09:46 34.652704 -86.571851 176.500000
0 2009-04-12 17:09:49 34.652696 -86.571843 177.000000
0 2009-04-12 17:09:57 34.652682 -86.572242 173.100000
0 2009-04-12 17:10:12 34.652682 -86.573432 165.900000

The first entries tagged "-1" are waypoints. The ones with "-0.11" altitude are manual entries. Then comes the first data track. In the XML, this is what I look for:

<trk>
<name>ACTIVE LOG: 12 APR 2009 12:09</name>
−
<trkseg>
−
<trkpt lat="34.653001" lon="-86.572144">
<ele>182.79</ele>
<time>2009-04-12T17:09:08Z</time>
</trkpt>
−
<trkpt lat="34.652971" lon="-86.571829">
<ele>177.02</ele>
<time>2009-04-12T17:09:34Z</time>
</trkpt>
−
<trkpt lat="34.652704" lon="-86.571851">
<ele>176.54</ele>
<time>2009-04-12T17:09:46Z</time>
</trkpt>
−
<trkpt lat="34.652696" lon="-86.571843">
<ele>177.02</ele>
<time>2009-04-12T17:09:49Z</time>
</trkpt>
−
<trkpt lat="34.652682" lon="-86.572242">
<ele>173.17</ele>
. . .

Bob Wilson