Backing Up Local USB Drive to Network Drive Using Windows and Stinky Batch Files
Well, after much arsing around, I finally found a half-way useful method to backup my plethora of rubbish to/from the Lacie NetworkSpace.Â This really bugged me, what with the none-too-nippy performance of the NetworkSpace, and its abysmal backup functions.Â Also, using Windows around the house can be a real pain; Unices make this stuff so much easier but I really wanted a quick Windows method to perform the file-by-file backup.
So back to the setup: 1TB Lacie NetworkSpace drive, and 1TB Toshiba USB number, one drive (Toshiba USB) to be used as a primary lump of storage, and one (NetworkSpace) as a backup.Â Initially I had the Lacie mounted as a Windows network drive, with the Toshiba plugged into the Lacie`s USB port, with the intention of kicking off a backup every week or so, depending on how much extra rubbish I plonked on the Lacie.Â The Lacie`s backup utility is simply awful – read the other post if you`re interested, and with the extra drop in performance from running the drive over the network, I had a change of plan.
Now I run the Toshiba over the laptop`s USB connection, something I`m not over the moon about as it limits portability, but I get the performance and means iTunes doesn`t lock up everytime I try to access a file.Â That just leaves syncing up the data so I still have a useful backup.Â Having taken a look online for a useful tool and coming up blank, I came up with a half-sensible, half-hacky solution.
The setup consists of:
Drive U: usb connected Toshiba 1GB drive
Drive M: network mounted Lacie NetworkSpace drive
For trivia, I chose “U” for “USB drive”, and M for “media drive” as the Lacie was originally to be just for music and the like; it kinda stuck when I plonked all of my other bits and bobs on there.Â The purpose of the mission was to keep M: (the backup) up to date with U: the 1TB goody bag of my photos, work, and other rubbish.
Had I been running Linux all the time, a simple rsync install on the Linux box would have done the job nicely.Â For anyone who`s never used rsync, it`s a really nice way to synchronise files between a source and a target, whether they be drives, directories (“folders”), or just individual files.Â However, over in Windows land, everything tends to be clicky-click button driven, so I was looking for something of that ilk, more out of laziness than lack of technical nouse.
I came across one tool that sort of did the job for me, DeltaCopy, available from http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
Using the DeltaCopy GUI, which to be honest, was… a bit of a pain, I ran a full sync from the Lacie drive to the Toshiba drive to get the Toshiba drive (which had started out empty) up to date.Â This took blummin` ages.Â For anyone who`s never used rsync to sync a large amount of data (~600GB in this case), this can take blummin` ages.Â When this succeeded (after a few days), I tried to set up a reverse sync from the Toshiba to the Lacie, i.e. from the fast, locally connected drive, to the network drive.Â This is where problems crept in including lots of “code 5″ “chdir failed” errors.Â So back to [insert search engine of choice here] I went to see if I could just sort out DeltaCopy.Â I gave up after a little hacking – seems a lot of people encountered the same problem, “nads to that”, thought I.
DeltaCopy is an interesting tool, and is essentially a build of rsync for Windows, along with some necessary libraries from cygwin (a really nice UNIX-compatible environment for Windows), and a GUI on top.Â It`s also usable from the command line – I wanted to avoid this just out of avoiding hassle, but eventually submitted.
Sample rsync Batch Files
So instead of making use of the GUI, I opened Notepad, that trusty Windows text editor (…) and hacked together a bunch of quick scripts to rsync specific directories over to the network drive.Â I won`t always want to backup everything, and may just want to backup a bunch of photos I`ve recently copied onto the USB drive.
Backing up the whole drive, backupall.bat
“C:Program FilesSynametrics TechnologiesDeltaCopyrsync.exe” –log-file=”/cygdrive/c/rsync-out.txt” -h –progress –delete-during -v -v -rlt “/cygdrive/u/” “/cygdrive/m/”
So, what`s going on here?
“C:Program FilesSynametrics TechnologiesDeltaCopyrsync.exe” is the name of the program we`re running, including the full path on disk (i.e. where it`s been installed); it needs double quotes as there are some spaces.Â That entire mess is just to run one command.Â Everything that appears after it are options.
I wanted a log file to check when the job`s been done, to check for errors, etc.Â You`ll see the /cgydrive/c/ stuff, which is interesting.Â Under the hood, cygwin can see your local and network drives if prefixed with /cygwin.Â In this case, we are referencing the C: drive, which cygwin can access as /cygdrive/c -Â so the log file will be plonked onto C:rsync-out.txt
Human-readable output – I want to see how many GB and MB were copied/deleted, not bytes.. I can`t think that quickly when I`m bored senseless by a backup job
Show the progress of each transfer – even if extra information slows jobs down, I prefer to see that things are still trundling away
This is the more interesting one.Â This will delete files from the TARGET if they don`t exist on the SOURCE.Â So if I was to delete some files on the USB drive then run the sync, they`d be deleted from the network drive.Â delete-during says “do it during the run, not before the run, and not after the run” – check out the rsync.html link below for the before/during/after settings.
Super-duper verbose.Â Again, I really like to see things are happening, even if it slows things down.
-r Copy directories (folders) recursively, i.e. if we see any directories within whatever we`re syncing, copy those as well
-t keep modification times on files; there`s nothing worse than files in the backup having a completely unrelated date/time!
The source to copy FROM is /cygdrive/u which is the top level of the U: drive
The destination to copy TO is /cygdrive/m which is the top level of the M: drive
In other words, this job will copy everything from U: to M: or from the USB drive to the network drive.Â This is the catch-all backup-all script that I`ll run once in a while to make sure any other backup batch files I create don`t miss files.
Something a little more useful, and faster:
Backing up a specific directory (folder) backupphotos.bat
So, I don`t necessarily want to backup everything, but instead I want to backup something specific.Â Perhaps my super-sexy photos:
“C:Program FilesSynametrics TechnologiesDeltaCopyrsync.exe” –log-file=”/cygdrive/c/rsync-out.txt” -h –progress –delete-during -v -v -rt “/cygdrive/u/photos” “/cygdrive/m”
This runs a lot faster for me as it will only sync the photos directory to the network drive.Â If I only wanted to backup the “christmas2009″ directory, I could run the following, which will be faster again:
“C:Program FilesSynametrics TechnologiesDeltaCopyrsync.exe” –log-file=”/cygdrive/c/rsync-out.txt” -h –progress –delete-during -v -v -rt “/cygdrive/u/photos/christmas2009″ “/cygdrive/m/photos/”
Obviously it would be possible to create a batch (.bat) file for every source directory on the USB drive, but that wouldn`t make a lot of sense; it would take too long to keep those batch files up-to-date, and they`d more than likely not be generic enough to be very useful.Â Instead, although taking longer to execute, I have 3 or 4 batch files, 1 to rsync absolutely everything (on the off chance), 1 to rsync photos, 1 to rsync work files, and 1 batch file to rsync just media files (or more specifically the M:media directory, specified as /cygdrive/m/media).Â Each batch file is identical to those above, just with a slightly different source and target directory.Â If I know I`ve changed a lot of photos, I`ll run the batch file to rsync the photos, same with the work files directory.Â Takes a lot less time and keeps me happy that I have an up-to-date backup.
The above is not using an rsync server, which means you lose one of rsync`s selling points, in that it can work out if only sections of files have changed, and send only those changes; instead, rsync will see there`s been a changed file and re-copy the whole file to the target.Â It however is a more simple way to get the rsync working without fiddling around with DeltaCopy apart from installation.
You can see all sorts of other rsync goodies on this page: http://www.samba.org/ftp/rsync/rsync.html If you`re playing around with rsync, I`d recommend not using a “delete” or “prune” flag.Â Mmmm prunes.
In other uninteresting news, I`m down with a nasty bug that`s kept me off work for 2 days.Â Not nice.