Category Archives: necessary evils

That r-syncing Feeling – Part II

Using the DeltaCopy Server to Keep The Drives In Sync – Using Deltas!

Once again, being not very happy with the previous not-quite-right way of doing things, I had a little poke, a nudge, and a further poke at DeltaCopy and its configuration to try to get the rsync working as rsync should work.  Ideally, you`ll want to use rsync through a server, and in this case the rsync server is DeltaCopy, or specifically, the DeltaCopy server installed with the application.  The previous post provides a method to implement a simple “if the file or directory exists, check it`s up-to-date, and if not, re-copy it; if it doesn`t exist, copy it”, but one of rsync`s biggest features is that of copying deltas, hence the name of the DeltaCopy tool.  The deltas are differences in files, and rsync will attempt to copy only changes in files to bring them up to date, thus saving in the amount of time required to copy.  Now, copying deltas can be oversold (depending on the type of file, it`s not always possible just to apply the changes, so a full re-copy is necessary) and it is more resource intensive, but for a large number of files, particularly text files (HTML, log files, code, etc.) can produce a massive speed up in the sync time.

rsync-ing the Photos Directory

Here`s the scenario, if you`ve already forgotten, or can`t be arsed reading the previous post:

USB drive mounted on U:
Lacie NetworkSpace network drive mounted on M:

U: is my primary work drive with all my gubbins on it, and M: is a network-attached drive for keeping a backup of everything.  I don`t always want to re-sync everything, as even with rsync, just finding the files and directories in order to test whether or not they`re up to date can take ages.  What we`ll do here is just rsync the photos directory, U:photos to M:photos.

The file we`re interested in here is deltacd.conf, located if you`ve installed using defaults, to

C:Program FilesSynametrics TechnologiesDeltaCopydeltacd.conf

Open this file in Notepad and take a look.  What we see here are a number of “modules” which can be source or destination locations.  We can`t use a source module and a destination module if they`re both going through the server.  So we need to specify a module pointing to the U drive, and hack our little batch file to point to the M drive.

Here`s my module representing the photos directory on the U: (USB-attached) drive:


path = /cygdrive/u/photos/
comment = photos on usb drive
read only = false
auth users =
secrets file =

You can see this just points to U:photos using the cygwin notation.  In order to make use of this, I then edit my little batch file, backup-photos-to-network.bat:

“C:Program FilesSynametrics TechnologiesDeltaCopyrsync.exe” –log-file=”/cygdrive/c/rsync-photos-out.txt” -h –progress –delete-during -v -v -rlt localhost::usbphotos// “/cygdrive/m/photos”

So, same parameters as before, but different source and destination.  The localhost::usbphotos// tells rsync to talk to a rsync server (“daemon”) on localhost and use the usbphotos module to grab its actual location, which of course is /cygdrive/u/photos AKA U:photos  There`s also a pause in there to keep the DOS prompt open when the job completes – always nice to be able to see the status, else the DOS prompt will vanish after completion, which won`t be helpful if you`ve an error somewhere in the batch file.

One thing you`ll notice when using rsync deltas is that you`ll see a lot more CPU usage on your machine.  However, assuming you see uplift from the delta copying, this can make it all worthwhile.  Also, if you`ve a lot of files, it can be worth specifying a different log file for each job as the files can grow very large very quickly.  Finally, if you`re that way inclined, you can use the nasty Windows task scheduler to schedule regular runs of these batch files to keep everything up to date on the network drive.

That concludes my quick and nasty DeltaCopy rambling; basically I found the GUI too awkward for something I wanted to do quickly, and then ended up bypassing it completely to do what I needed to do.  I now however have a lovely mirror of my files.  Now to find somewhere to hide the drive!  I`ve been thinking about online file storage, but currently trying to keep 600GB and counting online would take too long to sync, and it`s just easier to have another drive, although that`s of little comfort if the house burns down.

Output From The Batch File

C:>”C:Program FilesSynametrics TechnologiesDeltaCopyrsync.exe” –log-file=”/cygdrive/c/rsync-work-out.txt” -h –progress –delete-during -v -v -rlt localhost::usbwork// “/cygdrive/m/work”
opening tcp connection to localhost port 873
sending daemon args: –server –sender -vvltre.iLs . usbwork//
receiving incremental file list
delta-transmission enabled
DSCF6251.JPG is uptodate
DSCF6261.JPG is uptodate
DSCF6277.JPG is uptodate is uptodate
Thumbs.db is uptodate
databases/headstaggers.mdb is uptodate
databases/headstaggers_gallery.txt is uptodate
databases/hscv.sql is uptodate
databases/hscv.sql_original_test is uptodate
databases/mydyson.txt is uptodate
databases/ is uptodate
databases/v.jsp is uptodate

I obviously need to do a bit of clean up!  Again, if you`re playing around and testing, avoid using –delete-during until you`re sure things are working!  “YMMV” as they seem to say.

That r-syncing Feeling

Backing Up Local USB Drive to Network Drive Using Windows and Stinky Batch Files

Well, after much arsing around, I finally found a half-way useful method to backup my plethora of rubbish to/from the Lacie NetworkSpace.  This really bugged me, what with the none-too-nippy performance of the NetworkSpace, and its abysmal backup functions.  Also, using Windows around the house can be a real pain; Unices make this stuff so much easier but I really wanted a quick Windows method to perform the file-by-file backup.

So back to the setup: 1TB Lacie NetworkSpace drive, and 1TB Toshiba USB number, one drive (Toshiba USB) to be used as a primary lump of storage, and one (NetworkSpace) as a backup.  Initially I had the Lacie mounted as a Windows network drive, with the Toshiba plugged into the Lacie`s USB port, with the intention of kicking off a backup every week or so, depending on how much extra rubbish I plonked on the Lacie.  The Lacie`s backup utility is simply awful – read the other post if you`re interested, and with the extra drop in performance from running the drive over the network, I had a change of plan.

Now I run the Toshiba over the laptop`s USB connection, something I`m not over the moon about as it limits portability, but I get the performance and means iTunes doesn`t lock up everytime I try to access a file.  That just leaves syncing up the data so I still have a useful backup.  Having taken a look online for a useful tool and coming up blank, I came up with a half-sensible, half-hacky solution.

The setup consists of:
Drive U: usb connected Toshiba 1GB drive
Drive M: network mounted Lacie NetworkSpace drive

For trivia, I chose “U” for “USB drive”, and M for “media drive” as the Lacie was originally to be just for music and the like; it kinda stuck when I plonked all of my other bits and bobs on there.  The purpose of the mission was to keep M: (the backup) up to date with U: the 1TB goody bag of my photos, work, and other rubbish.

Had I been running Linux all the time, a simple rsync install on the Linux box would have done the job nicely.  For anyone who`s never used rsync, it`s a really nice way to synchronise files between a source and a target, whether they be drives, directories (“folders”), or just individual files.  However, over in Windows land, everything tends to be clicky-click button driven, so I was looking for something of that ilk, more out of laziness than lack of technical nouse.

I came across one tool that sort of did the job for me, DeltaCopy, available from

Using the DeltaCopy GUI, which to be honest, was… a bit of a pain, I ran a full sync from the Lacie drive to the Toshiba drive to get the Toshiba drive (which had started out empty) up to date.  This took blummin` ages.  For anyone who`s never used rsync to sync a large amount of data (~600GB in this case), this can take blummin` ages.  When this succeeded (after a few days), I tried to set up a reverse sync from the Toshiba to the Lacie, i.e. from the fast, locally connected drive, to the network drive.  This is where problems crept in including lots of “code 5″ “chdir failed” errors.  So back to [insert search engine of choice here] I went to see if I could just sort out DeltaCopy.  I gave up after a little hacking – seems a lot of people encountered the same problem, “nads to that”, thought I.

DeltaCopy is an interesting tool, and is essentially a build of rsync for Windows, along with some necessary libraries from cygwin (a really nice UNIX-compatible environment for Windows), and a GUI on top.  It`s also usable from the command line – I wanted to avoid this just out of avoiding hassle, but eventually submitted.

Sample rsync Batch Files

So instead of making use of the GUI, I opened Notepad, that trusty Windows text editor (…) and hacked together a bunch of quick scripts to rsync specific directories over to the network drive.  I won`t always want to backup everything, and may just want to backup a bunch of photos I`ve recently copied onto the USB drive.

Backing up the whole drive, backupall.bat

“C:Program FilesSynametrics TechnologiesDeltaCopyrsync.exe” –log-file=”/cygdrive/c/rsync-out.txt” -h –progress –delete-during -v -v -rlt “/cygdrive/u/” “/cygdrive/m/”

So, what`s going on here?
“C:Program FilesSynametrics TechnologiesDeltaCopyrsync.exe” is the name of the program we`re running, including the full path on disk (i.e. where it`s been installed); it needs double quotes as there are some spaces.  That entire mess is just to run one command.  Everything that appears after it are options.

I wanted a log file to check when the job`s been done, to check for errors, etc.  You`ll see the /cgydrive/c/ stuff, which is interesting.  Under the hood, cygwin can see your local and network drives if prefixed with /cygwin.  In this case, we are referencing the C: drive, which cygwin can access as /cygdrive/c -  so the log file will be plonked onto C:rsync-out.txt

Human-readable output – I want to see how many GB and MB were copied/deleted, not bytes.. I can`t think that quickly when I`m bored senseless by a backup job

Show the progress of each transfer – even if extra information slows jobs down, I prefer to see that things are still trundling away

This is the more interesting one.  This will delete files from the TARGET if they don`t exist on the SOURCE.  So if I was to delete some files on the USB drive then run the sync, they`d be deleted from the network drive.  delete-during says “do it during the run, not before the run, and not after the run” – check out the rsync.html link below for the before/during/after settings.

-v -v
Super-duper verbose.  Again, I really like to see things are happening, even if it slows things down.

-r Copy directories (folders) recursively, i.e. if we see any directories within whatever we`re syncing, copy those as well
-t keep modification times on files; there`s nothing worse than files in the backup having a completely unrelated date/time!

“/cygdrive/u/” “/cygdrive/m/”
The source to copy FROM is /cygdrive/u which is the top level of the U: drive
The destination to copy TO is /cygdrive/m which is the top level of the M: drive

In other words, this job will copy everything from U: to M: or from the USB drive to the network drive.  This is the catch-all backup-all script that I`ll run once in a while to make sure any other backup batch files I create don`t miss files.

Something a little more useful, and faster:

Backing up a specific directory (folder) backupphotos.bat

So, I don`t necessarily want to backup everything, but instead I want to backup something specific.  Perhaps my super-sexy photos:

“C:Program FilesSynametrics TechnologiesDeltaCopyrsync.exe” –log-file=”/cygdrive/c/rsync-out.txt” -h –progress –delete-during -v -v -rt “/cygdrive/u/photos” “/cygdrive/m”

This runs a lot faster for me as it will only sync the photos directory to the network drive.  If I only wanted to backup the “christmas2009″ directory, I could run the following, which will be faster again:

“C:Program FilesSynametrics TechnologiesDeltaCopyrsync.exe” –log-file=”/cygdrive/c/rsync-out.txt” -h –progress –delete-during -v -v -rt “/cygdrive/u/photos/christmas2009″ “/cygdrive/m/photos/”

Obviously it would be possible to create a batch (.bat) file for every source directory on the USB drive, but that wouldn`t make a lot of sense; it would take too long to keep those batch files up-to-date, and they`d more than likely not be generic enough to be very useful.  Instead, although taking longer to execute, I have 3 or 4 batch files, 1 to rsync absolutely everything (on the off chance), 1 to rsync photos, 1 to rsync work files, and 1 batch file to rsync just media files (or more specifically the M:media directory, specified as /cygdrive/m/media).  Each batch file is identical to those above, just with a slightly different source and target directory.  If I know I`ve changed a lot of photos, I`ll run the batch file to rsync the photos, same with the work files directory.  Takes a lot less time and keeps me happy that I have an up-to-date backup.

The above is not using an rsync server, which means you lose one of rsync`s selling points, in that it can work out if only sections of files have changed, and send only those changes; instead, rsync will see there`s been a changed file and re-copy the whole file to the target.  It however is a more simple way to get the rsync working without fiddling around with DeltaCopy apart from installation.

You can see all sorts of other rsync goodies on this page: If you`re playing around with rsync, I`d recommend not using a “delete” or “prune” flag.  Mmmm prunes.

In other uninteresting news, I`m down with a nasty bug that`s kept me off work for 2 days.  Not nice.

Lacie NetworkSpace Backup is a Waste of Space

Around September 2008 I purchased a Lacie NetworkSpace 1TB hard disk, a nice little number with built in Ethernet so it could be tucked away where the sun don`t shine and keep all my files nice and safe whilst I flung the laptop into my bag for the day-to-day.  Since then, I`ve been filling the drive up will all manner of crap.  Having become more interested in photography, there`s now several GB of photos, RAW files, etc. and lots of my CDs on there.

A "successful" backup...

A "successful" backup...

The drive itself was “designed by Neil Poulton”, and if it`s anything to go by, Neil Poulton is something of a manic depressive, unimaginative lazybones.  The casing is shiny black, rectangular, and features a very bright, blue LED – flavour of the month.  So, it`s essentially a black, shiny brick.  The little ventilation holes look OK – little squares – and the device can run pretty hot.  Round the back we have the power, a switch (handy – and the drive waits until it shuts down before fully powering off), and the ethernet port.  At the front is the LED, and a USB connector.  And here lies the problem…

Being a middle-aged worrying sort, I didn`t want to risk losing all that stuff – absolute shite to most people, but a lot of stuff I`ve made, taken photos of, etc. and bought a 1TB USB external drive to use as a backup for the Lacie.  Through the Lacie`s web interface, I saw it was possible to set up one drive as the source (Lacie) and one as the destination (new Toshiba drive) and thought “easy peasy – link them over USB, click the buttons, and off it goes”.  And off nothing went.

After around 2 hours, the only backups I have apparently made have been 8 empty directories on the Toshiba, from a total of several thousand files.  Not a great ratio.  I thought I must have cocked something up in the plethora of 4 or 5 options, so tried again, with the same result.  Then I took a look at the logs from the Lacie.

The Lacie uses some form of embedded Linux – a few blogs mention it may be Yellow Dog – and does its biz using BusyBox, so I figured there would be some useful, sensible, really nice logging.  Here`s what I saw:

Apr 26 00:37:40  httpd:                     pam_unix(httpd:session): session closed for user admin
Apr 26 00:37:59  httpd:                     pam_unix(httpd:session): session opened for user admin by (uid=0)
Apr 26 00:37:59  httpd:                     pam_unix(httpd:session): session closed for user admin
Apr 26 00:38:13  httpd:                     pam_unix(httpd:session): session opened for user admin by (uid=0)

Now, pardon my French, but what the fuck use is that to anyone? No mention of backup starting, backup finishing, backup failing, backup progress. Not a sausage. There`s only one thing worse than no logging, it`s useless logging. And here we see a combination of both, almost bugger all logging, and what`s there is completely useless. I know I was logged into the web interface – I just want to check my supposed backup!
So it`s likely to be a case of the Toshiba drive becoming a work drive, and having to run rsync to backup to the Lacie as it`s just a little too slow to use as a primary drive over Ethernet. Very disappointed. The drive has been great for a number of months – not spectacularly fast over the home network (wireless G / 100Mb Ethernet – it has Gb Ethernet I believe), so I can`t do the device itself down. But come on guys – what`s the point in using a completely shite backup function as a selling point?  There also doesn`t seem too be a way of upgrading the firmware or operating system short of unofficial hacks, ruling out fixing this awful functionality.  It may work – somehow – under Windows, but if it`s a case of linking two drives and pressing a few buttons, using the advertised method, it`s surprising this simple operation can`t be carried out.  If there had been anything in the logs to suggest too-long filenames, extra-long paths – whatever – I could have done something to rectify.  But zilch.

So it`s off to rsync I go… farting in the general direction of Lacie.  Several times.  Poo poo to you.