[GTALUG] Watching a network folder: is there a smart way of doing this?

[Giles Orr]

On 17 February 2017 at 15:41, Stewart C. Russell via talk
<talk at gtalug.org> wrote:
> I need to watch a folder on a network share (a scanner) and see when new
> files are created. There are a couple of special things about this location:
>
> * It's over CIFS. There's nothing I can do about that. I think that
> means I can't use Inotify.
>
> * It doesn't have an accurate clock for timestamping. There's nothing I
> can do about that, either.
>
> * If the share is being written to by the device, it effectively
> disappears: stat() complains loudly.
>
> * It reuses the lowest available file name of the form
> “EPSON%03d.(PDF|JPG)”, so synching the share to a sensible filesystem
> might overwrite files accidentally.
>
> * I don't need to watch it very often; a few times per hour would be fine.
>
> I tried using Perl's File::ChangeNotify::Watcher, which is supposed to
> pick the smartest method available. Unfortunately, this seems to be
> calling stat() every two seconds, resulting in:
>
> 1. Messages (one per existing file on the share) saying that the scanner
> has deleted them. This happens when someone starts scanning
>
> 2. An error message from stat() every two seconds while the device is
> scanning
>
> 3. Messages saying that all of the old files (plus the new one) have
> been created.
>
> I was naïvely hoping for a mechanism that would report “Hey, you have a
> new file called $filename” but that seems to have been a pipedream. What
> do People Who Actually Know What They're Doing use, please?

How's your coding?  I'm visualizing this in Bash on a cron job.

This is purely theory, and I apologize if it's not useful.  It seems
like it would work if I've understood the circumstances correctly ...

- every fifteen minutes get a checksum on all the remote files
- get checksums on the files in the local copy of the folder
- if a checksum exists on the remote but not the local ('uniq -c ...'
may help), copy the file and datestamp it (ie. 'cp -vi //CIFS/a.tif
/localcopy/a.2017-02-17.1610.tif').
- hmm - might work to embed the checksum in the local filename for
ease of access ...

There's a bit of juggling to be done on comparing checksums, but it
shouldn't be too hard.  Would this work?

-- 
Giles
http://www.gilesorr.com/
gilesorr at gmail.com