Re: Multiple databases or Redis/Stats/Graphite/Loggly

On Wed, 2 May 2012 00:51:44 -0700 (PDT), Jean-Mark
<jeanmark.wright@gmail.com> declaimed the following in
gmane.comp.python.django.user:

> I'm working on an app that will store gps coordinates from devices. That
> information will come in every 5 or so minutes of each hour. So it's alot

12 data points per device per hour? That's miniscule... Most
hand-held GPS units update 30 times a minute.

> of data. Any recommendations on how I should store this? Do I use multiple
> databases and use say MongoDB or Couch for the storage of the points or do
> I use a solution like Redis / Graphite/Stats or Loggly?

What, exactly, is the data to be stored? NMEA sentences? Just the
timestamp, lat, long? (or UTM grid coordinates).

ID, Device, Timestamp, UTM Easting, UTM Northing

is NOT a lot of data -- about 64 bytes per record (if double-precision).
Raw NMEA sentences are only around 80-100 bytes each. SQLite3 could
probably handle lots of them (Device-Timestamp would be a unique index,
though ID would be an autonumber and hence unique also). The apparent
overhead of the storage format of MongoDB/CouchDB (text, with
field-name:field-value pairs) would tend to double the size of the data,
and slow down the processing as numeric data has to be converted from
string to binary and back.

It is not clear just what you intend to do with these data points? I
suspect Numpy/Scipy and MatPlotLib could do most of the basic analysis
operations, and they stay within Python realms. The R system could maybe
do more complex analysis (it is mostly a statistics package; a
work-alike for S, and something more modern than SPSS -- whereas Octave
competes for the MatLab/Maple-V numeric world).


--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.