View Single Post
Old 09-26-2009, 01:32 AM   #1 (permalink)
EcoModding Lurker
Join Date: Sep 2009
Location: Melbourne, Australia
Posts: 15
Thanks: 1
Thanked 10 Times in 8 Posts
Format for storing values on a memory stick

As discussed in another thread ( I now have a version of OBDuino32K that writes GPS and OBD values to a CSV file on a USB memory stick. It's still very preliminary but it is working, so before I get too stuck on a particular on-disk format I thought it would be worth having a quick discussion about what that format should be.

I expect that other related projects such as MPGuino may well add datalogging support in future, and it makes sense to keep the projects using the same file format so that if there are any tools developed to read / process that data they will work identically with data from MPGuino, OBDuino, or whatever else. For example, I'm intending to create some kind of parser to convert the data into a format that will display a trip on Google Earth with things like vehicle speed and RPM superimposed, and if that works out OK I'm sure people with other devices would be interested in trying it out.

My critical #1 requirement is that the log has to be on a standard filesystem and use a plain text file format. Supporting filesystems such as VFAT on a microcontroller is really hard and many people resort to writing raw binary data to something like a MicroSD card using SPI, but that then locks the data into the device and you can only extract it using routines in the microcontroller itself. I want to be able to unplug a standard USB memory stick, put it in a laptop, mount it, and open the log in a plain text editor or spreadsheet. I already have it doing that using a VDIP1 module but it does set a fairly high base requirement for other people wanting to implement something similar, and pretty much mandates the use of something like a VDIP1 to handle the filesystem. Perhaps people taking a minimalist approach won't like that and would prefer to use a raw binary format but I think it's necessary to make the system simple enough for anyone to use.

So other relevant questions are:

1. Should the parameters that are logged relate to the parameters that have been configured on the screens, or should they be independently configurable? Or just hard-coded? Or just every single parameter supported?

2. Should a header be inserted either once or periodically to show the column names?

3. What sample rate? Once for every time through the main loop (ie: multiple records per second), or just every, say, 5 seconds?

4. Should the system periodically start a new file? When? Daily, or every time the car starts?

5. What should the file be named? Because we're limited to DOS 8.3 filenames we can't include a timestamp in the filename unfortunately, so if the system creates a new file regularly it will need to use something like an incrementing serial number to distinguish them.

Right now for debugging I have it logging in a format that looks like this:

-36.84131,146.21652,119.40,18520000.00,COOLANT_TEMP =40,VEHICLE_SPEED=100

(No, my car is not underwater off the coast of Japan - the GPS values have been fudged for privacy)

That's obviously a very inefficient format because it bloats the rows with parameter names and makes it harder to parse, but I'm just doing that so I can see what's going on while working on the logging code. I don't plan to include the parameter names in the final format.

Comments? Suggestions?

Cheers :-)

Jonathan Oxer / /

  Reply With Quote