database vs filesystem performance

[Walter Dnes]

On Mon, Aug 08, 2005 at 12:37:21AM -0400, Marc Lijour wrote
> On August 8, 2005 00:32, Ansar Mohammed wrote:
> > It all comes down to the nature of your application and data.
> > Is your application read only? Are you modifying data? How large are the
> > files and what kind of files are they?
> 
> I am just getting a very fast stream of binary data which I have to store 
> (fast) with the idea of retrieving later to process it. Hence it must be 
> indexed in some way, but a coarse-grained indexing should work (many files 
> may be).

  See http://www.unitedlinux.com/pdfs/whitepaper4.pdf for a discussion
on file system limits.  3000 files/second adds up really quickly.

[m1800][waltdnes][~] echo $(( 3000 * 3600 * 24 * 365 ))
94608000000

[m1800][waltdnes][~] echo $(( 3000 * 3600 * 24 * 366 ))
94867200000

  That's over 94.5 *BILLION FILES* in a regular year, and pushing close
to 95 billion in a leap year.  That's over a *TRILLION FILES* in 11
years.  Would it be considered "insider information" if I ran out and
invested my life-savings in disk-drive manufacturers, based on your
question<g>.

  I really have my doubts about a regular file system handling this.
According to the above paper, reiserfs 3 "only" allows 1 TB partition,
and 4 billion files.  Fuggedaboutit.  Ext2 and 3 are limited to 4 TB
partitions, with however many files you can cram in.  JFS is a 64-bit
system, and might be able to hold the files and data you need, but I
don't really have a clue about speed.

-- 
Walter Dnes <waltdnes-SLHPyeZ9y/tg9hUCZPvPmw at public.gmane.org>
My musings on technology and security at http://tech_sec.blog.ca
--
The Toronto Linux Users Group.      Meetings: http://tlug.ss.org
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://tlug.ss.org/subscribe.shtml