Automating external storage backups with udisks-glue

Tuesday, February 17, 2015

The amount of administration time saved with effective automation can be quite staggering when calculated over several months. A well developed automation script/program can greatly reduce errors (typically introduced by human interaction), be ran at specified times throughout the day, perform reporting operations, and much more.

Whenever I find myself repeating a "static" task more than two or three times on a regular basis I consider the possibility of automating said task. With automation I can accelerate my personal workflow whilst giving me an opportunity to improve my scripting skills in Bash or Python.

Intro

The process of backing up data is one of the prime cases where automation offers a wealth of benefits over manual interaction. Common uses of the cron daemon are for data backup purposes, e.g. /home partition data.
Note: Be careful not to overload a cron backup job! Make sure the time between backup operations is sufficient before initiating another one.

Automating backups for external storage such as SD cards, memory sticks, or portable HDDs is a little trickier than the configuration of a typical cron job.
Unlike cron the vast amount of people don't (usually) have an exact, specified time in which they connect their external storage - they tend to do it when they wish to access the stored media.
Fortunately other utilities exist for handling this uncertainty.

This guide covers the necessary preparation to (1) automount a specified partition of a generic external storage device and (2) execute command(s)/script(s) post mount. This post is dedicated to my dad who could benefit from an automated process for "intelligently" filing and backing up photos from his camera's SD card.

Packages Used

dbus: 1.6.8-1+deb7u5
policykit-1: 0.105-3
udisks: 1.0.4-7wheezy1
udisks-glue: 1.3.4-1
wget: 1.13.4-3+deb7u2

Automounting

Udisks operates as both a D-Bus daemon (udisks-daemon) for polling, enumerating, and querying disk devices as well as a commandline utility (udisks). Unlike traditional daemons which are commonly invoked through init scripts at startup udisks-daemon is dynamically loaded by D-Bus when its services are requested.
All queries and actions through either the daemon or commandline utility are done so with PolicyKit configurable user privileges.
The relationship between all these utilities can be roughly identified as:

D-Bus <--- PolicyKit <--- Udisks <--- Udisks-glue

Udisks-glue builds upon Udisks by providing a simple mechanism for running certain matches from preconfigured filters. This enables the execution of commands/scripts post mount and unmount of specified filesystems.

Note: This guide covers udisks1 configuration/operation not the Gnome DE constrained udisks2!

Install the necessary packages:
sudo aptitude install dbus policykit-1 udisks udisks-glue
With your favourite text editor create a PolicyKit rule file: /etc/polkit-1/localauthority/50-local.d/10-storage.pkla that allows users of the floppy group to automount devices:

[automount]
    Identity=user-group:floppy
    Action=org.freedesktop.udisks.filesystem-mount
    ResultAny=yes

Debian uses the floppy group for all removable external storage (i.e. SD card) which, upon initial installation of the Debian system, the user will be a member of.

3. Restart the D-Bus service in order to safely restart the PolicyKit daemon:
sudo service dbus restart
In graphical desktop environments it is most likely safer to restart the display manager by logging out and logging in again ~ see here.

4. Check the D-Bus service is working correctly by listing the services currently available:

dbus-send --system --print-reply --type=method\_call --dest=org.freedesktop.DBus /org/freedestop/DBus org.freedesktop.DBus.ListNames

# Response should be something similar to this...
method return sender=org.freedesktop.DBus -> dest=:1.0 reply_serial=2  
array [  
    string "org.freedesktop.DBus"
    string ":1.0"  
]

5. Now connect the external storage device which is intended to be automounted by the system and identify the targeted partition's UUID:
sudo blkid /dev/sdb1
-- or --
sudo udevadm info --query=all --name=sdb1 | grep FS_UUID=

6. Test that PolicyKit permissions have been correctly applied by attempting to mount the targeted partition:
udisks --mount /dev/disk/by-uuid/$UUID_OF_PARTITION
If PolicyKit permissions have been configured correctly you should now have an accessible mount point at /media/$UUID_OF_PARTITION

7. If udisks has operated in accordance to the PolicyKit rule made in 2. (i.e. mounted without errors as a standard user) we can now create a generic udisks-glue configuration file ~/.udisks-glue.conf:

#
# Filters
#
filter externalStorageDevice  
{
    optical = false
    partition_table = false
    usage = filesystem
    uuid = $UUID_OF_PARTITION
}

#
# Rules to apply
#
match externalStorageDevice  
{
    automount = true
    automount_filesystem = vfat
    automount_options = sync
    post_mount_command = "/path/to/an/executable/./script.sh"
    post_unmount_command = "mount-notify unmounted %device\_file %mount\_point"
}

For more examples see: man udisks-glue.conf

8. Now we can grab a SysVinit script for ensuring udisks-glue starts at boot time (Credits : Andrew Bythell ~ abythell):
wget --no-check-certificate https://gist.githubusercontent.com/abythell/5399914/raw/33ed0e67c05c8aabed043151a25efffc298b86ac/udisks-glue

9. For our user specified configuration we need to alter the downloaded SysVinit script slightly:
sed -i 's/\/etc\/udisks-glue.conf/\/home\/$USER\/.udisks-glue.conf/g' ~/udisks-glue
Substitute $USER with the name of the currently logged in user.

10. Now we need to make the SysVinit script executable. To do this change its ownership to root, place it in the /etc/init.d/ directory, and run update.rc to automatically symlink the script for the default runlevels:
chmod 755 ~/udisks-glue
sudo chown root:root ~/udisks-glue
sudo mv ~/udisks-glue /etc/init.d
sudo update-rc.d udisks-glue defaults

11. Test the udisks-glue script by unmounting the partition, disconnect the external storage device, and finally reconnecting the external storage device:
udisks --unmount /dev/disk/by-uuid/$UUID_OF_PARTITION

12. Start the udisks-glue makeshift daemon:
sudo service udisks-glue start

13. Finally connect the external storage device with the targeted partition to the system. See that the device was located and that the desired partition was actually mounted.
dmesg | tail -n 50
mount
If the command desired isn't executed check first that you can still mount with udisks and that if it is a script that is called it is addressed via a global path and has sufficient correct permissions.

Photo management script

Below is the ~~Bash~~ Python script (**Update 09/14/2015** I have rewritten my backup script in Python as it resulted in a cleaner, more robust implementation) I have been developing for my dad for use with his camera's SD card. The ultimate goal is to have a system whereby insertion of the SD card alone would be sufficient to appropriately organise and sync all media.
SMS e-mail gateways have been considered as a means of alleviating the necessity of internet connectivity to track sync progress. After careful consideration it was decided that my dad would take charge of manually deleting the photos so as to avoid any potential loss of precious data by a misbehaving script! The core components of the final script are:

1. Check that the hardcoded source & destination directories exist. In addition to this we must check that any external binaries depended upon (i.e. rsync) by the script are available. Any failures at this stage must stop the script and result in a failure e-mail containing the reason for failure being sent to my dad.

2. With an appropriate data structure organise all the source media files depending on the year and month they were taken. Create year and subsequent child month directories in the destination directory if they do not exist already.

3. E-mail dad informing him that the sync operation has commenced, this e-mail can serve as a rough timestamp of the sync operation's start time. Now begin syncing (via rsync) the filtered source files to their corresponding year and month destination directory. Keep a count of the number of files that have been synced.

4. Once the sync process has completed the SD card should be unmounted. A final e-mail should be generated containing the count of how many files were synced in total as well as the mounted status of the SD card (so as to guarantee whether or not it is safe to remove the media!).

Note: As of current (09/14/2015) there are still various features missing; Functionality such as e-mail notifications at various points throughout the script's lifetime, adequate error handling for various plausible scenarios (e.g. insufficient disk space), and basic logging (aimlessly printing to stdout currently) still remain outstanding. Any recommendations/improvements/constructive criticisms are encouraged and welcomed for both the guide and the script.

#!/usr/bin/python 
# @author Myles Grindon
# @version 0.1

import os  
import time  
import string  
import calendar  
from datetime import date, datetime  
from subprocess import call

""" Static locations for sync location, destination, and the temporary filter file """
src = "/media/64F3-FACC/DCIM/Sony/"  
dst = "/home/grindon/pictures/"  
tempListFile = "/home/grindon/.temp"

""" Global variables for storing file names and their date of creation """
allFiles = []  
years = ()  
monthFormat = ()

def checkSrc():  
    """ Checking source media is available """
    if not os.path.isdir(src):  
        print "Error! Media not detected - looking for: %s" % src
        exit(1)
    else:
        pass

def initFiles():  
    """ Prepare media files for syncing to the destination """
    global allFiles
    global years
    global monthFormat

    # Create list of strings of media file names (basename) located on the "src" directory
    allFiles = [file for file in os.listdir(src) if os.path.isfile(os.path.join(src,file))]

    # Handles an arbitrary range of years
    years = set((date.fromtimestamp(int(os.path.getmtime(os.path.join(src,year))))).year for year in allFiles)
    monthFormat = sorted(set((string.ascii_uppercase[index-1] + " - " + calendar.month_name[index] for index in range(1,13)))) # Format E.g. A - January, B - February

def checkDst():  
    """ Checking destination media is present and the correct directory structure is present for syncing"""
    if not os.path.isdir(dst):
        print "Error! Destination location not found - looking for: %s" % dst
        exit(1)
    else:
        # Check & Create necessary directory structure for saving photos
        for year in years:
            for month in monthFormat:
                if not os.path.isdir(os.path.join(dst,str(year),month)):
                    os.makedirs(os.path.join(dst,str(year),month), 0755)

def checkBin():  
    """ Checking necessary binary applications are installed """
    if not call(["which", "rsync"]):
        pass
    else:
        print "Missing the 'rsync' application!"
        exit(1)


def sortMedia():  
    """ Sorting source media files based on timestamps and syncing via rsync to their correct directory in destination """
    sortedDict = {year: {month: [] for month in range(1,13)} for year in years}     # {2014: {1: [], 2: [], etc.

    # Filter files for year and month
    for file in allFiles:
        fileDate = date.fromtimestamp(int(os.path.getmtime(os.path.join(src,file))))
        sortedDict[fileDate.year][fileDate.month].append(file)

    # Sync files to appropriate destination directory 
    for year in sortedDict:
        for month in sortedDict[year]:
            with open(tempListFile, 'w') as tempList:
                tempList.write("\n".join(sortedDict[year][month]))      # List of media files in that specifc year & month

            dstDir = os.path.join(dst,str(year),monthFormat[month-1])

            rsyncCommand = "rsync --verbose --ignore-existing " \
                        + "--include-from="+tempListFile + " --exclude='*' " \
                        + src+"* " \
                        + "'"+dstDir+"'"    # _Will_ not work with sources that have spaces in!

            if not call(["bash", "-c", rsyncCommand]):
                print "Month '%s' successfully synced with %s!" % (monthFormat[month-1], dstDir)
            else:
                print "Month '%s' has failed!" % monthFormat

def unmountMedia():  
    if not call(["udisks", "--unmount", "/dev/disk/by-uuid/64F3-FACC"]):    
        print "Unmounted media successfully!"
    else:
        print "Error unmounting media!"

checkSrc()  
initFiles()  
checkDst()  
checkBin()  
sortMedia()  
unmountMedia()