database vs filesystem performance

[Marc Lijour]

On August 8, 2005 06:47, Walter Dnes wrote:
> On Mon, Aug 08, 2005 at 12:37:21AM -0400, Marc Lijour wrote
>
> > On August 8, 2005 00:32, Ansar Mohammed wrote:
> > > It all comes down to the nature of your application and data.
> > > Is your application read only? Are you modifying data? How large are
> > > the files and what kind of files are they?
> >
> > I am just getting a very fast stream of binary data which I have to store
> > (fast) with the idea of retrieving later to process it. Hence it must be
> > indexed in some way, but a coarse-grained indexing should work (many
> > files may be).
>
>   See http://www.unitedlinux.com/pdfs/whitepaper4.pdf for a discussion
> on file system limits.  3000 files/second adds up really quickly.
>
> [m1800][waltdnes][~] echo $(( 3000 * 3600 * 24 * 365 ))
> 94608000000
>
> [m1800][waltdnes][~] echo $(( 3000 * 3600 * 24 * 366 ))
> 94867200000

In that case it would make sense to concatenate some of the info...

>   That's over 94.5 *BILLION FILES* in a regular year, and pushing close
> to 95 billion in a leap year.  That's over a *TRILLION FILES* in 11
> years.  Would it be considered "insider information" if I ran out and
> invested my life-savings in disk-drive manufacturers, based on your
> question<g>.
>
>   I really have my doubts about a regular file system handling this.
> According to the above paper, reiserfs 3 "only" allows 1 TB partition,
> and 4 billion files.  Fuggedaboutit.  Ext2 and 3 are limited to 4 TB
> partitions, with however many files you can cram in.  JFS is a 64-bit
> system, and might be able to hold the files and data you need, but I
> don't really have a clue about speed.
--
The Toronto Linux Users Group.      Meetings: http://tlug.ss.org
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://tlug.ss.org/subscribe.shtml