lightroom sync with s3 windows/macos

lightroom sync with s3 windows/macos

- 14 mins

HOWTO Lightroom sync with s3 windows/macos

1. The pitch

As I am an avid photographer & travelling quite a lot, I wanted to sync my Adobe Lightroom library from Macbook to my Microsoft Windows desktop at home with very little/no human intervention. It is also a good thing that both Lightroom versions (mac & windows) share a common file system/structure, this helped alot in synchronizing data from one environment to another.

This HOWTO assumes you already have an AWS account created and running with an IAM user configured.

Put together, the solution would look as such (pardon my poor diagram skills):

                                  _  
macbook pro                     (`  ).                     windows desktop
----------- --> internet -->  ( amazon'`. --> internet --> ---------------
  remote                      (   s3    )                       home
                               ` __.:'-'

For this tutorial/use-case, I am using:

software

hardware

some data/stats

As of today, my small Adobe Lightroom library runs about 140gb using exclusively RAW Nikon format photos (.NEF). A quick check on my S3 Bucket:

~:nico bucketsize my-bucket
total size:
139.298 GB
total count:
37460 objects

You can find the source code for my bucketsize alias on my gist here

2. Getting our hands dirty

1. installing pre-reqs

Open your Terminal App then paste these lines:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install python
sudo pip install --upgrade pip
sudo pip install awscli

2. configuring the cli:

aws configure

Follow the steps of this command to fill in the following information from your already existing IAM user: + Access Key / Secret Key combination + Default region setting + Default output format setting

This will create the following configuration files in ~/.aws/ :

~:nico ls ~/.aws/
config      credentials

config

This file contains the configuration details per profile as configured using step 2 as follows:

~:nico cat ~/.aws/config
[default]
output = json
region = eu-west-1

credentials

This file contains the default and additional profiles you have configured using using step 2, modulo a few switches as explained in the documentation:

~:nico cat ~/.aws/credentials
[default]
aws_access_key_id = YOURACCESSKEY
aws_secret_access_key = YOURSECRETKEY

3. creating our IAM users

By creating separate IAM accounts for the purpose of synchronizing our files from the mac to s3 and from s3 to the Microsoft Windows desktop, we limit the impact on our infrastructure in the event of these accounts being compromised to strictly access s3.

Account Function Additional info
s3sync-mac Synchronizes my photos from Mac to Amazon S3. Secured with Virtual MFA.
s3sync-pc Synchronizes my photos from Amazon S3 to PC. Limited to a single IP using a /32 CIDR within a policy.

To add an extra security layer, we attach a virtual MFA device to the s3sync-mac user account thus allowing secure authentication from any internet-enabled location.

For this section, references are as follows + IAM Create User + IAM Create Access Key + IAM Virtual MFA Device + IAM Enable Virtual MFA Device + IAM List Virtual MFA Device

creating s3sync-mac & s3sync-pc user accounts in IAM

aws iam create-user --user-name s3sync-mac
aws iam create-user --user-name s3sync-pc

For each of these commands you should be getting answers in the following format:

{
    "User": {
        "UserName": "s3sync-mac",
        "Path": "/",
        "CreateDate": "YYYY-MM-DDTHH:MM:SS.XXXZ",
        "UserId": "SOMEUSERID",
        "Arn": "arn:aws:iam::YOURACCOUNTID:user/s3sync-mac"
    }
}

Copy the output of both commands and store it somewhere while we continue with the creation of the accounts.

s3sync-mac user account

We’ll now create then enable the MFA device to the s3sync-mac user account:

aws iam create-virtual-mfa-device --virtual-mfa-device-name s3sync-mac --outfile /Users/nico/Desktop/s3sync-mac-QRCode.png --bootstrap-method QRCodePNG

Which will return:

{
  "VirtualMFADevice": {
      "SerialNumber": "arn:aws:iam::YOURACCOUNTID:mfa/s3sync-mac"
  }
}

Let’s enable this MFA to our s3sync-mac user account. We first need to scan the newly generated QRcode on our desktop with our TOTP security token application on our mobile phone. I use Google Authenticator. Once scanned, we’ll get a first code, then a second one: write both of them down, we’ll use them to validate our virtual MFA from the command line:

aws iam enable-mfa-device --user-name s3sync-mac --serial-number arn:aws:iam::YOURACCOUNTID:mfa/s3sync-mac --authentication-code-1 123456 --authentication-code-2 789012

No output for this one, but you can check your MFA device status using this command and easily check that everything is working accordingly:

{
  "VirtualMFADevices": [
    {
    "SerialNumber": "arn:aws:iam::YOURACCOUNTID:mfa/s3sync-mac",
    "EnableDate": "YYYY-MM-DDTHH:MM:SS.XXXZ",
    "User": {
        "UserName": "s3sync-mac",
        "Path": "/",
        "CreateDate": "YYYY-MM-DDTHH:MM:SS.XXXZ",
        "UserId": "SOMEUSERID",
        "Arn": "arn:aws:iam::YOURACCOUNTID:user/s3sync-mac"
      }
    }
  ]
}

We’ll now generate the Access Key / Secret Key pair, then configure a new profile on the CLI.

aws iam create-access-key --user-name s3sync-mac

Which will return:

{
  "AccessKey": {
      "UserName": "s3sync-mac",
      "Status": "Active",
      "CreateDate": "YYYY-MM-DDTHH:MM:SS.XXXZ",
      "SecretAccessKey": "MYSECRETKEY",
      "AccessKeyId": "MYACCESSKEY"
  }
}

Take the output of this command to create your new CLI profile:

~:nico aws configure --profile s3sync-mac
AWS Access Key ID [None]: MYSECRETKEY
AWS Secret Access Key [None]: MYACCESSKEY
Default region name [None]: eu-west-1
Default output format [None]: json

We now have a functional user account for our photos synchronization from the mac to our Amazon S3 Bucket, a corresponding profile set up with the CLI and added security via the use of a Virtual MFA token.

s3sync-pc user account

Our s3sync-pc user account has been created at the very beginning of this step 3.; we’ll now generate a pair of Access Key/Secret Key and secure this account with an IAM policy.

aws iam create-access-key --user-name s3sync-pc

Which will return:

{
  "AccessKey": {
      "UserName": "s3sync-pc",
      "Status": "Active",
      "CreateDate": "YYYY-MM-DDTHH:MM:SS.XXXZ",
      "SecretAccessKey": "MYSECRETKEY",
      "AccessKeyId": "MYACCESSKEY"
  }
}

Now, let’s setup the CLI on our Microsoft Windows desktop:

Download then run the installation package on your Microsoft Windows desktop: + AWS CLI 32bits + AWS CLI 64bits

Once this is done, we can take the output of the previous command to create your new CLI profile on the Microsoft Windows desktop:

c:\aws configure --profile s3sync-pc
AWS Access Key ID [None]: MYSECRETKEY
AWS Secret Access Key [None]: MYACCESSKEY
Default region name [None]: eu-west-1
Default output format [None]: json

creating your Amazon s3 bucket: my-lightroom-bucket

Now that our users are created and configured using profiles in the CLI, let’s create our s3 bucket to store our photos. A quick reminder about Amazon s3:

The CLI will tell you so:

~:nico aws s3 mb s3://photos
make_bucket failed: s3://photos/ A client error (BucketAlreadyExists) occurred when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

Let’s be creative:

~:nico aws s3 mb s3://my-lightroom-bucket
make_bucket: s3://my-lightroom-bucket/

Be sure to adapt the commands and the following bucket policy to reflect your bucket name.

securing my Amazon s3 bucket: my-lightroom-bucket

Amazon Identity and Access Management (IAM) will help us in limiting the access to s3 for both our IAM users:

For this purpose we will use the following policy:

{
  "Id": "Policy1442860910448",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1442860826662",
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-lightroom-bucket",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "123.123.123.123/32"
        }
      },
      "Principal": {
        "AWS": [
          "arn:aws:iam::MYACCOUNTID:user/s3sync-pc"
        ]
      }
    },
    {
      "Sid": "Stmt1442860864410",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-lightroom-bucket/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "123.123.123.123/32"
        }
      },
      "Principal": {
        "AWS": [
          "arn:aws:iam::MYACCOUNTID:user/s3sync-pc"
        ]
      }
    },
    {
      "Sid": "Stmt1442860885410",
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-lightroom-bucket",
      "Principal": {
        "AWS": [
          "arn:aws:iam::MYACCOUNTID:user/s3sync-mac"
        ]
      }
    },
    {
      "Sid": "Stmt1442860908897",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-lightroom-bucket/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::MYACCOUNTID:user/s3sync-mac"
        ]
      }
    }
  ]
}

Why the policy has 2 parts for each user, with different bucket orthographs ? Well, let’s explain this quickly:

s3:GetObject applies to the objects in the bucket:

"Resource": "arn:aws:s3:::my-lightroom-bucket/*"

while s3:ListBucket applies to the Bucket itself:

"Resource": "arn:aws:s3:::my-lightroom-bucket"

Now that we have that out of our way, we’ll attach the policy to the bucket and test it (attach + put file + list bucket):

~:nico aws s3api put-bucket-policy --bucket my-lightroom-bucket --policy file://my-lightroom-policy.json
~:nico aws s3 cp my-lightroom-policy.json s3://my-lightroom-bucket --profile s3sync-mac
upload: ./my-lightroom-policy.json to s3://my-lightroom-bucket/my-lightroom-policy.json
~:nico aws s3 ls my-lightroom-bucket --profile s3sync-mac
2015-09-21 20:53:17       1500 my-lightroom-policy.json

For this section, references are as follows + S3 Put Bucket Policy

3. Synchronizing our photos

1. From the macbook to Amazon s3

Using the s3sync-*mac* account, we will now sync our Adobe Lightroom photo library to Amazon s3:

aws s3 sync /Volumes/Lightroom/lightroom/ s3://my-lightroom-bucket --profile s3sync-mac

Using Amazon s3’s sync command, we’ll push only the new files that don’t exist on our Amazon s3 bucket. The new files will then be synched by my Microsoft Windows desktop at home.

For this section, references are as follows + S3 Sync

2. From Amazon s3 to the Microsoft Windows desktop at home

To automate things, we’ll use the following bach script to then create a scheduled task that will run every hour and synchronize the Adobe Lightroom library back on its hard drive. As a precaution, we’ll log everything that happens into a unique logfile; so first things first, let’s create our filetree:

c:\> mkdir c:\my-lightroom-sync\logs c:\my-lightroom-sync\bin\

Now that this is done, let’s batch things up !

<a href='https://nuage.ninja/echo' class='user-mention'>@echo</a> off
goto :sync
***********************************************************************
id:       my-lightroom-sync.bat
author:   nicolas@nuage.ninja
version:  1.0

desc:     This batch synchronizes from Amazon s3 downwards to the local disk
usage:    my-lightroom-sync.bat (no arguments needed)
***********************************************************************

:sync
set AWS_CLI_PROFILE=s3sync-pc
set SYNC_REMOTE_BUCKET=my-lightroom-bucket
set SYNC_LOCAL_DIR=c:\Lightroom
set SYNC_LOGS=c:\my-lightroom-sync\logs

set SYNC_TIMESTAMP_DATE=%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%
set SYNC_TIMESTAMP_TIME=%TIME:~0,2%%TIME:~3,2%%TIME:~6,2%
set SYNC_ERRLOG=%SYNC_LOGS%\my-lightroom-sync-%SYNC_TIMESTAMP_DATE%-%SYNC_TIMESTAMP_TIME%.log

<a href='https://nuage.ninja/echo' class='user-mention'>@echo</a> ### %SYNC_TIMESTAMP_TIME% - starting synchronization >> %SYNC_ERRLOG%
aws s3 sync s3://%SYNC_REMOTE_BUCKET% %SYNC_LOCAL_DIR% --profile %AWS_CLI_PROFILE% >> %SYNC_ERRLOG%
set SYNC_TIMESTAMP_END=%TIME:~0,2%%TIME:~3,2%%TIME:~6,2%
<a href='https://nuage.ninja/echo' class='user-mention'>@echo</a> ### %SYNC_TIMESTAMP_END% - ending synchronization >> %SYNC_ERRLOG%

Batch needs to be saved as c:\my-lightroom-sync\bin\my-lightroom-sync.bat using your favourite editor. Once this is done, you will need to run the first occurence manually in order to synchronize the initial library stored in Amazon s3. After that, you should be able to use the scheduled task:

C:\> schtasks /create /tn "my-lightroom-sync" /tr c:\my-lightroom-sync\bin\my-lightroom-sync.bat" /sc HOURLY /np
SUCCESS: The scheduled task "my-lightroom-sync" has successfully been created.

C:\> schtasks /query /TN my-lightroom-sync
Folder: \
TaskName                                 Next Run Time          Status
======================================== ====================== ===============
my-lightroom-sync                        9/22/2015 1:06:00 AM   Ready

3. you’re done.

The only thing you now need to worry about is to run the sync command from your mac when you have new content in your Adobe Lightroom library and let the scheduled task run the synchronization for you. Additionally, you might want to adapt this to your usage.

/Volumes/lightroom/Lightroom:nico aws s3 sync . s3://my-lightroom-bucket --profile s3sync-mac
upload: Photos/2015/2015-07-26/DSC_0517.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0517.NEF
upload: Photos/2015/2015-07-26/DSC_0515.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0515.NEF
upload: Photos/2015/2015-07-26/DSC_0516.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0516.NEF
upload: Photos/2015/2015-07-26/DSC_0514.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0514.NEF
upload: Photos/2015/2015-07-26/DSC_0518.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0518.NEF
upload: Photos/2015/2015-07-26/DSC_0520.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0520.NEF
...
upload: Photos/2015/2015-08-09/DSC_0118.NEF to s3://my-lightroom-bucket/Photos/2015/2015-08-09/DSC_0118.NEF
Completed 493 part(s) with ... file(s) remaining
Nicolas David

Nicolas David

Entrepreneur, Autodidact, OpenSource enthusiast, Amateur photographer & Food aficionado.

rss facebook twitter github youtube mail spotify instagram linkedin google pinterest medium vimeo