[GTALUG] File larger than partition? (Docker-related)

[Giles Orr]

On 20 November 2017 at 23:31, Giles Orr <gilesorr at gmail.com> wrote:

> On 20 November 2017 at 18:58, Lennart Sorensen <
> lsorense at csclub.uwaterloo.ca> wrote:
>
>> On Mon, Nov 20, 2017 at 05:25:16PM +0000, Giles Orr via talk wrote:
>> > I ran a backup today and noticed one file because it took so long to
>> back
>> > up.  I'm using FC25 on this machine, and used the OS packages of
>> Docker.  I
>> > seem to have a 100G file on a 12G partition:
>> >
>> > root at toshi7:/var/lib/docker/devicemapper/devicemapper# ls -lh
>> > total 34M
>> > -rw------- 1 root root 100G Oct 25 22:02 data
>> > -rw------- 1 root root 2.0G Oct 25 22:02 metadata
>>
>> Try ls -lhs
>>
>> For example:
>>
>> ~> dd if=/dev/zero bs=1M count=1 seek=1000 of=testfile
>> 1+0 records in
>> 1+0 records out
>> 1048576 bytes (1.0 MB) copied, 0.00203937 s, 514 MB/s
>> ~> ls -lhs testfile
>> 1.0M -rw-r--r-- 1 lsorensen users 1001M Nov 20 13:56 testfile
>>
>> So 1MB allocated out of 1001MB file size.  The first 1000MB are a 'hole'
>> in the file that isn't allocated yet.
>>
>> The -s option to ls makes it show the allocated space in the first column.
>
>
> Wow.  I did not know that, thank you.  And I see there's a specific switch
> to rsync for better handling of sparse files.
>
> But ... then what exactly does straight-up 'ls' (without the '-s')
> report?  The man page says '-s' "print[s] the allocated size of each file,
> in blocks."  I was under the mistaken impression (for 23 years now) that
> that was more or less what 'ls' was already doing.
>
> Here are some answers: https://en.wikipedia.org/wiki/Sparse_file ...  I
> get the utility of the idea, but it seems to come with some fairly
> significant hazards.
>
> 'ls' can be made to indicate directories (with '/') and links (with '@')
> and a couple other things with '-F'/'--classify': sparse files would seem
> to be staggeringly misleading and thus a good target for this kind of
> marking as well ...  Is that possible?
>
> Where else am I likely to run into sparse files?  Sounds like mostly
> things that create file systems, like VirtualBox and friends, Docker
> (obviously) ... anywhere else?
>
> Sorry to ask so many questions, but 'ls' seems like one of the most basic
> commands of Linux and I thought I knew what it did: I'm suddenly feeling
> like a newbie again and would like to get a handle on this ...
>

As a short follow-up: current VirtualBox does use some form of expandable
file system for its disk images (.vdi files), but if they're sparse files,
they're hidden inside the image file somehow (I could have researched the
VDI file format ... but I didn't.  Not the point right now).  I have quite
a number of VDI images that are 8GiB in use, but on disk they're 1-3Gib and
use the same amount of disk space as their file size.

-- 
Giles
https://www.gilesorr.com/
gilesorr at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gtalug.org/pipermail/talk/attachments/20171121/bb40f2f1/attachment.html>