[GTALUG] Anybody using rclone?

[Alvin Starr]

On 2023-06-05 11:16, Scott Sullivan via talk wrote:
> On 2023-06-05 09:14, Alvin Starr via talk wrote:
>> Does anybody know of an volume based backup solution that can work in 
>> an incremental manner?
>
> This questions has big unstated conditional. Are you looking for
>
> A) 'volume based backup that agnostic of the volumes it is backing up'
>
> For which I have no answers.
That is kind of my preference.
>
>
> B) 'An alternative filesystem / volume solution that can support 
> incremental backup'
>
> ZFS snapshots fit the bill here. Most folks will jump to the 
> conclusion that you have to stream from one ZFS to another. But in 
> reality zfs send is just standard out, that is only connected to a zfs 
> receive by convention. You can just dump the incremental stream as a 
> file/object that  doesn't need to be applied to a receiving ZFS 
> immediately. You are then into the same know problem set of full vs 
> incremental offline database backups.
>
Changing the filesystem would be a major lift and likely take months to 
copy the data over.
This is also the underlying storage for a Gluster volume so I am not 
sure of the ZFS support for Gluster.
But it is food for thought.

We are in this situation because some "wizard" said "don't store PDF 
data in a data base keep it on a file system and just link using the paths".

Well that works well till you get tens of millions of files.

Backing up a database in the TB size is orders of magnitude faster than 
trying to back up that same data in a filesystem

If I had my way I would convert the data to something like an Elastic 
database were the numbers of files are more manageable.



-- 
Alvin Starr                   ||   land:  (647)478-6285
Netvel Inc.                   ||   Cell:  (416)806-0133
alvin at netvel.net              ||