Go Back   EcoModder Forum > EcoModding > Instrumentation > OpenGauge / MPGuino FE computer
Register Now
 Register Now
 

Reply  Post New Thread
 
Submit Tools LinkBack Thread Tools
Old 09-26-2009, 12:32 AM   #1 (permalink)
EcoModding Lurker
 
Join Date: Sep 2009
Location: Melbourne, Australia
Posts: 15
Thanks: 1
Thanked 10 Times in 8 Posts
Format for storing values on a memory stick

As discussed in another thread (http://ecomodder.com/forum/showthrea...ega-10300.html) I now have a version of OBDuino32K that writes GPS and OBD values to a CSV file on a USB memory stick. It's still very preliminary but it is working, so before I get too stuck on a particular on-disk format I thought it would be worth having a quick discussion about what that format should be.

I expect that other related projects such as MPGuino may well add datalogging support in future, and it makes sense to keep the projects using the same file format so that if there are any tools developed to read / process that data they will work identically with data from MPGuino, OBDuino, or whatever else. For example, I'm intending to create some kind of parser to convert the data into a format that will display a trip on Google Earth with things like vehicle speed and RPM superimposed, and if that works out OK I'm sure people with other devices would be interested in trying it out.

My critical #1 requirement is that the log has to be on a standard filesystem and use a plain text file format. Supporting filesystems such as VFAT on a microcontroller is really hard and many people resort to writing raw binary data to something like a MicroSD card using SPI, but that then locks the data into the device and you can only extract it using routines in the microcontroller itself. I want to be able to unplug a standard USB memory stick, put it in a laptop, mount it, and open the log in a plain text editor or spreadsheet. I already have it doing that using a VDIP1 module but it does set a fairly high base requirement for other people wanting to implement something similar, and pretty much mandates the use of something like a VDIP1 to handle the filesystem. Perhaps people taking a minimalist approach won't like that and would prefer to use a raw binary format but I think it's necessary to make the system simple enough for anyone to use.

So other relevant questions are:

1. Should the parameters that are logged relate to the parameters that have been configured on the screens, or should they be independently configurable? Or just hard-coded? Or just every single parameter supported?

2. Should a header be inserted either once or periodically to show the column names?

3. What sample rate? Once for every time through the main loop (ie: multiple records per second), or just every, say, 5 seconds?

4. Should the system periodically start a new file? When? Daily, or every time the car starts?

5. What should the file be named? Because we're limited to DOS 8.3 filenames we can't include a timestamp in the filename unfortunately, so if the system creates a new file regularly it will need to use something like an incrementing serial number to distinguish them.

Right now for debugging I have it logging in a format that looks like this:

-36.84131,146.21652,119.40,18520000.00,COOLANT_TEMP =40,VEHICLE_SPEED=100

(No, my car is not underwater off the coast of Japan - the GPS values have been fudged for privacy)

That's obviously a very inefficient format because it bloats the rows with parameter names and makes it harder to parse, but I'm just doing that so I can see what's going on while working on the logging code. I don't plan to include the parameter names in the final format.

Comments? Suggestions?

Cheers :-)

Jonathan Oxer
jon.oxer.com.au / twitter.com/jonoxer / www.geekmyride.org

  Reply With Quote
Alt Today
Popular topics

Other popular topics in this forum...

   
Old 09-26-2009, 02:28 PM   #2 (permalink)
EcoModding Apprentice
 
Join Date: Aug 2009
Location: terra firma
Posts: 138
Thanks: 4
Thanked 24 Times in 22 Posts
Since storage is cheap:

-Hardcode the data fields;
-Single header line at beginning of file;
-Output in readable text, not hex;
-Sample rate of 0.5 sec (the main loop length in MPGuino... i've never looked at OBGuino);
-Start a new file every car start (ie, activity detected after at least 1 minute of no activity)

If you have a system clock, keep in mind you could code the timestamp into the filename using base36 instead of base10. Worst case, you'd need 4 characters [0-9A-Z] for yymmdd, and another 4 for SecondsSinceMidnight (36^3 = 46,656, 36^4 = 1,679,616). Then ".obd" for your .ext. Your data viewer program would then decode the base36 into "yymmdd hhmmss"
  Reply With Quote
Old 09-26-2009, 02:46 PM   #3 (permalink)
Pokémoderator
 
cfg83's Avatar
 
Join Date: Dec 2007
Location: Southern California
Posts: 5,864

1999 Saturn SW2 - '99 Saturn SW2 Wagon
Team Saturn
90 day: 40.49 mpg (US)
Thanks: 439
Thanked 529 Times in 356 Posts
jonoxer -

If you can make folders on the usb stick, then you can make "long" filenames. For instance, for today at 11:44:36 AM :

20090926\1144\run36.csv

You could then post-process the "long" name into a date stamp filename :

run.20090926.114436.csv

CarloSW2
__________________

What's your EPA MPG? Go Here and find out!
American Solar Energy Society
  Reply With Quote
Old 09-26-2009, 03:48 PM   #4 (permalink)
Pokémoderator
 
cfg83's Avatar
 
Join Date: Dec 2007
Location: Southern California
Posts: 5,864

1999 Saturn SW2 - '99 Saturn SW2 Wagon
Team Saturn
90 day: 40.49 mpg (US)
Thanks: 439
Thanked 529 Times in 356 Posts
jonoxer -

Another way to fit the datestamp into the filename is store it as hex. Next year will be 2010, so you need at least 2 year digits. That makes this time maximum for the rest of the century :

2099/12/31 , 23:59:59 => 991231235959 => E6C9FC5777 Hex => E6C9FC5.777

That's a 7.3 sized filename.

Ha ha, you can actually go for 999 years and still fit in 8.3 format :

2999/12/31 , 23:59:59 => 9991231235959 => 91643C9E777 Hex => 91643C9E.777
The beginning of the file could have the desired filename for easy post-process translation :

r29991231.235959.csv


CarloSW2
__________________

What's your EPA MPG? Go Here and find out!
American Solar Energy Society
  Reply With Quote
Old 09-26-2009, 11:28 PM   #5 (permalink)
EcoModding Lurker
 
Join Date: Sep 2009
Location: Melbourne, Australia
Posts: 15
Thanks: 1
Thanked 10 Times in 8 Posts
nickdigger - Good suggestions. I'd definitely go for decimal values rather than hex, as you suggest. Putting the column names in the first row is a good idea too. For sample rate it may need to be slightly less so it doesn't slow everything else down: storage is cheap, but the bandwidth / CPU time to write to it isn't. I have a friend who designed a motorcycle datalogging system (Dyna Pro) who put me onto the VDIP module, and he's logging several dozen parameters 100 times / second but to do that he's cranking the module along at a much higher baud rate and running a really optimized main loop.

Timestamps may or may not be a problem. Right now I'm getting timestamp data from the GPS module but for some reason they're all offset by 2 hours, and I can't figure out why. People who don't have GPS would need some alternative way to get the current time but that could be done with an RTC chip. I've done some coding for a DS1307 in a rocket telemetry system and it's a piece of cake to work with so that's a possibility.

cfg83 - I like the idea of using directory structures to indicate the date. That's how things like iPhoto work too. I'd prefer not to use hex values in the filenames though, that just makes it too obscure for human processing.

Cheers :-)

Jonathan Oxer

  Reply With Quote
Reply  Post New Thread


Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
OBDuino Mega jonoxer OpenGauge / MPGuino FE computer 23 03-22-2015 11:55 PM



Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.
Content Relevant URLs by vBSEO 3.5.2
All content copyright EcoModder.com