database vs filesystem performance

[Andrew Hammond]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> I know my question is quite vague. Let me try to throw a little more details. 
> I am working on an application which pulls data from a cache at a very fast 
> rate (configurable but it could be up to 3000 times by second) therefore 
> performance -in terms of speed- is an issue. The data is in itself binary. 
> Some people suggested to convert this into strings and to store it in the 
> database (MS-SQL). I heard that databases are not that good with binary data 
> and also that the filesystem has been designed to handle files efficiently. I 
> wonder what is best between storing this data in the database (eventually as 
> string or as a BLOB) or dumping it into a big file (or breaking it in a 
> sequence of files eventually). The journaling filesystem would make this 
> reliant to a system crash, right?

If your cache will need to handle concurrency issues beyond simple
expiration, or it will require multiple or non-trivial indexing, or it
will manage relationships between data then it might make sense to use a
database (Postgres is pretty good. Avoid MySQL at all costs).

Otherwise, just the filesystem should be sufficient. In terms of
performance, I've seen impressive results from reiser4. You'd want to
test it before you put it into production, obviously. You might be able
to tie into the reiser attributes system and get some significant
performance improvements.

Later in the thread you asked about NTFS. It's quite possibly the most
dog-slow filesystem I've ever had the dubious pleasure of encountering.
Absent political or marketing reasons which mandate using it, the
technical drawbacks of Windows are so severe that I can't imagine anyone
seriously recommending it as a development platform.

- --
Andrew Hammond    416-673-4138    ahammond-swQf4SbcV9C7WVzo/KQ3Mw at public.gmane.org
Database Administrator, Afilias Canada Corp.
CB83 2838 4B67 D40F D086 3568 81FC E7E5 27AF 4A9A
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFC92gTgfzn5SevSpoRAsJgAJ9nkinXG3fb0yTQ8LRWshKJIEapHgCfaOdo
3pY97dmagfoKO7FmYuU5Hf4=
=LWJw
-----END PGP SIGNATURE-----
--
The Toronto Linux Users Group.      Meetings: http://tlug.ss.org
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://tlug.ss.org/subscribe.shtml