[GTALUG] File larger than partition? (Docker-related)

[Giles Orr]

On 20 November 2017 at 18:58, Lennart Sorensen <lsorense at csclub.uwaterloo.ca
> wrote:

> On Mon, Nov 20, 2017 at 05:25:16PM +0000, Giles Orr via talk wrote:
> > I ran a backup today and noticed one file because it took so long to back
> > up.  I'm using FC25 on this machine, and used the OS packages of
> Docker.  I
> > seem to have a 100G file on a 12G partition:
> >
> > root at toshi7:/var/lib/docker/devicemapper/devicemapper# ls -lh
> > total 34M
> > -rw------- 1 root root 100G Oct 25 22:02 data
> > -rw------- 1 root root 2.0G Oct 25 22:02 metadata
>
> Try ls -lhs
>
> For example:
>
> ~> dd if=/dev/zero bs=1M count=1 seek=1000 of=testfile
> 1+0 records in
> 1+0 records out
> 1048576 bytes (1.0 MB) copied, 0.00203937 s, 514 MB/s
> ~> ls -lhs testfile
> 1.0M -rw-r--r-- 1 lsorensen users 1001M Nov 20 13:56 testfile
>
> So 1MB allocated out of 1001MB file size.  The first 1000MB are a 'hole'
> in the file that isn't allocated yet.
>
> The -s option to ls makes it show the allocated space in the first column.


Wow.  I did not know that, thank you.  And I see there's a specific switch
to rsync for better handling of sparse files.

But ... then what exactly does straight-up 'ls' (without the '-s') report?
The man page says '-s' "print[s] the allocated size of each file, in
blocks."  I was under the mistaken impression (for 23 years now) that that
was more or less what 'ls' was already doing.

Here are some answers: https://en.wikipedia.org/wiki/Sparse_file ...  I get
the utility of the idea, but it seems to come with some fairly significant
hazards.

'ls' can be made to indicate directories (with '/') and links (with '@')
and a couple other things with '-F'/'--classify': sparse files would seem
to be staggeringly misleading and thus a good target for this kind of
marking as well ...  Is that possible?

Where else am I likely to run into sparse files?  Sounds like mostly things
that create file systems, like VirtualBox and friends, Docker (obviously)
... anywhere else?

Sorry to ask so many questions, but 'ls' seems like one of the most basic
commands of Linux and I thought I knew what it did: I'm suddenly feeling
like a newbie again and would like to get a handle on this ...

-- 
Giles
https://www.gilesorr.com/
gilesorr at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gtalug.org/pipermail/talk/attachments/20171120/4511ebb6/attachment.html>