Format for storing values on a memory stick

[jonoxer]

As discussed in another thread (http://ecomodder.com/forum/showthrea...ega-10300.html) I now have a version of OBDuino32K that writes GPS and OBD values to a CSV file on a USB memory stick. It's still very preliminary but it is working, so before I get too stuck on a particular on-disk format I thought it would be worth having a quick discussion about what that format should be.

I expect that other related projects such as MPGuino may well add datalogging support in future, and it makes sense to keep the projects using the same file format so that if there are any tools developed to read / process that data they will work identically with data from MPGuino, OBDuino, or whatever else. For example, I'm intending to create some kind of parser to convert the data into a format that will display a trip on Google Earth with things like vehicle speed and RPM superimposed, and if that works out OK I'm sure people with other devices would be interested in trying it out.

My critical #1 requirement is that the log has to be on a standard filesystem and use a plain text file format. Supporting filesystems such as VFAT on a microcontroller is really hard and many people resort to writing raw binary data to something like a MicroSD card using SPI, but that then locks the data into the device and you can only extract it using routines in the microcontroller itself. I want to be able to unplug a standard USB memory stick, put it in a laptop, mount it, and open the log in a plain text editor or spreadsheet. I already have it doing that using a VDIP1 module but it does set a fairly high base requirement for other people wanting to implement something similar, and pretty much mandates the use of something like a VDIP1 to handle the filesystem. Perhaps people taking a minimalist approach won't like that and would prefer to use a raw binary format but I think it's necessary to make the system simple enough for anyone to use.

So other relevant questions are:

1. Should the parameters that are logged relate to the parameters that have been configured on the screens, or should they be independently configurable? Or just hard-coded? Or just every single parameter supported?

2. Should a header be inserted either once or periodically to show the column names?

3. What sample rate? Once for every time through the main loop (ie: multiple records per second), or just every, say, 5 seconds?

4. Should the system periodically start a new file? When? Daily, or every time the car starts?

5. What should the file be named? Because we're limited to DOS 8.3 filenames we can't include a timestamp in the filename unfortunately, so if the system creates a new file regularly it will need to use something like an incrementing serial number to distinguish them.

Right now for debugging I have it logging in a format that looks like this:

-36.84131,146.21652,119.40,18520000.00,COOLANT_TEMP =40,VEHICLE_SPEED=100

(No, my car is not underwater off the coast of Japan - the GPS values have been fudged for privacy)

That's obviously a very inefficient format because it bloats the rows with parameter names and makes it harder to parse, but I'm just doing that so I can see what's going on while working on the logging code. I don't plan to include the parameter names in the final format.

Comments? Suggestions?

Cheers :-)

Jonathan Oxer
jon.oxer.com.au / twitter.com/jonoxer / www.geekmyride.org

[nickdigger]

Since storage is cheap:

-Hardcode the data fields;
-Single header line at beginning of file;
-Output in readable text, not hex;
-Sample rate of 0.5 sec (the main loop length in MPGuino... i've never looked at OBGuino);
-Start a new file every car start (ie, activity detected after at least 1 minute of no activity)

If you have a system clock, keep in mind you could code the timestamp into the filename using base36 instead of base10. Worst case, you'd need 4 characters [0-9A-Z] for yymmdd, and another 4 for SecondsSinceMidnight (36^3 = 46,656, 36^4 = 1,679,616). Then ".obd" for your .ext. Your data viewer program would then decode the base36 into "yymmdd hhmmss"

[cfg83]

jonoxer -

If you can make folders on the usb stick, then you can make "long" filenames. For instance, for today at 11:44:36 AM :

20090926\1144\run36.csv

You could then post-process the "long" name into a date stamp filename :

run.20090926.114436.csv

CarloSW2

[cfg83]

jonoxer -

Another way to fit the datestamp into the filename is store it as hex. Next year will be 2010, so you need at least 2 year digits. That makes this time maximum for the rest of the century :

2099/12/31 , 23:59:59 => 991231235959 => E6C9FC5777 Hex => E6C9FC5.777

That's a 7.3 sized filename.

Ha ha, you can actually go for 999 years and still fit in 8.3 format :

2999/12/31 , 23:59:59 => 9991231235959 => 91643C9E777 Hex => 91643C9E.777
The beginning of the file could have the desired filename for easy post-process translation :

r29991231.235959.csv


CarloSW2

[jonoxer]

nickdigger - Good suggestions. I'd definitely go for decimal values rather than hex, as you suggest. Putting the column names in the first row is a good idea too. For sample rate it may need to be slightly less so it doesn't slow everything else down: storage is cheap, but the bandwidth / CPU time to write to it isn't. I have a friend who designed a motorcycle datalogging system (Dyna Pro) who put me onto the VDIP module, and he's logging several dozen parameters 100 times / second but to do that he's cranking the module along at a much higher baud rate and running a really optimized main loop.

Timestamps may or may not be a problem. Right now I'm getting timestamp data from the GPS module but for some reason they're all offset by 2 hours, and I can't figure out why. People who don't have GPS would need some alternative way to get the current time but that could be done with an RTC chip. I've done some coding for a DS1307 in a rocket telemetry system and it's a piece of cake to work with so that's a possibility.

cfg83 - I like the idea of using directory structures to indicate the date. That's how things like iPhoto work too. I'd prefer not to use hex values in the filenames though, that just makes it too obscure for human processing.

Cheers :-)

Jonathan Oxer