OSX rsync, chflags, Spotlight

[AndyD]

A couple of questions...

I've been using rsync for almost a year to backup our main server drives. We use three identical drives, each with three identical partitions. Rsync backs up each partition on the main drive to its corresponding partition on the other two drives. It's been working flawlessly - until recently.

I noticed a discrepancy in the data sizes on the backup drives compared to the main drive and discovered some files had been locked (I assume one of those random things that happens). Rsync has problems with them and appears to possibly skip some other files when it encounters them (not sure about this though). It definitely won't delete them from the target drives.

So I used a short "chflags" script I wrote a while ago to globally unlock all files in a target directory and I unlocked all files on the first backup drive (all three partitions) - no problems. But when I pointed it at the second backup drive it threw up errors related to Spotlight (I run the chflags from an Applescript and it hangs once the error window is full). There is something different between the two drives but I don't know what. As far as I can recall, they were treated identically when the system was set up.

I've also noticed there can still be significant differences in total file size (>1GB) between this backup and the other. Flicking through folders on each dive, I can see lots of copied files that have small size differences (KBs).

Basically, I'm a bit confused.

 

[BenBurch]

What are the relative sizes of the drives? Because the size of the allocation block depends on the size of the drive. Also depends on the exact type of formatting of the drive.

Also I hope you are telling rsync to copy extended attributes?

-Ben

 

[AndyD]

The drives are all identical make, model and size (640GB) all purchased and partitioned at the same time with identical volume sizes.

One difference, which may have caused corruption, was that I did not set rsync to "--delete" on the second drive and it eventually ran out of space (The setting was intentional but I obviously wasn't paying attention to just how full it had become). It was when I saw the errors listed in the report that I started investigating. I deleted a few obviously outdated files manually then changed the script to include the "--delete" setting and let it rip to clean up the rest.

It's been a while since I wrote the script (and I was learning as I went) but I'm pretty sure it copies extended attributes. Either way, the scripts are now (after fixing "--delete") identical other than the target volume address.

Yesterday I renamed the backup folder on one volume (rsync backs up into a folder with the source volume name) then set the backup to run, creating a fresh backup which was so close to the same size as to be comforting. Then I deleted the original folder and ran the chflags script again. This time it only reported one error related to ".trashes" (I think). I'll run it again in a few days, when I'm back at work, and see if that error goes away or gets worse.

I did do a "ls -a" from Terminal on the volumes and there were differences in the hidden files (can't tell you what at the moment though). I was wondering if it would hurt anything to delete the hidden files - like ".Spotlight-V100/"(?) and ".temporary items"(?) - assuming it's possible to delete them from Terminal or some other way.

I do recall one volume had a folder called "Spotlight-V100 without the "dot" in front yet it is invisible. I suspect I made this when I tried to make the hidden folder visible by removing the dot in the Get Info window. It let me take the dot off but wouldn't let me put it back - and it remained hidden anyway.