Overloading Lightroom

Last night I discovered how easy it is to overload the Lightroom database and cause it to malfunction and crash.

Here’s the story.

I had been adding location information to the IPTC fields of my pictures since I was using ACDSee. IPTC defines five location fields:

  1. Location.
  2. City.
  3. Province.
  4. Country.
  5. ISO Country Code.

(It isn’t defined whether the fields define where a picture is taken, or what the picture is of. I use them for the “of” whereas the geocodes are always the “where”.)

Most of my pictures are taken in Bangkok. I adopted the convention that the location is the district (khet), the city is Bangkok, I left the province blank, the country is Thailand and the ISO Country Code is TH.

However, when I started using Jeff Friedl’s Geocoding plugin for Lightroom it uses a different convention. When Reverse Geocoding (filling in the IPTC location fields from a lookup to Google Earth using the Lat-Long) it generally leaves the Location blank, the City is the Khet, the Province is Bangkok and the Country is Thailand.

(Jeff has a challenge mapping the address information returned by Google into those fields. It can be different for each country. He does the best he can. I could argue with his choices based on my knowledge of living here but it isn’t worth it. I can live with his choices as long as my database is consistent.)

That left me with a big inconsistency in the Ligfhtroom database which made it hard to find, for example, all photos of khet Din Daeng where I live.

“No problem!”, I thought. I’ll do it Jeff’s way. I can select all photos with City = Bangkok and do an update so that State = Bangkok. That’s a good start. Then I can similarly move my Location (Khet) to the City.

Click for a larger image

Click for a larger image

The picture above shows the situation:

  • 33,942 pictures in the database.
  • 33,113 of them in Thailand.
  • BUT 10.533 with State = Bangkok and,
  • 18.756 with City = Bangkok.

It’s trivial in Lightroom to use filters to select all the photos with City = Bangkok. Then in Grid view (only) you can use the metadata panel on the right to edit the displayed fields for ALL the selected images.

However, what’s easy to do in the user interface is an immense task for the database engine.

First it has to accumulate all the metadata for the selected images and show it in the Metadata panel. Where all images have the same values it shows them. Where they differ it shows <mixed>. It also accumulates all the keywords (in my case hundreds of them) in the Keywording panel.

This takes a fair time even on a fast machine. The user must be patient while the display shows “Accumulating Metadata” in the top-left.

Lightroom accumulating metadata

Once that’s complete, you can theoretically type new data in the Metadata panel and the changes will be applied instantly to all the selected pictures.

The key is “instantly”. Rule 1 – don’t change the City field first. The selection is City = Bangkok. If I change that the pictures immediately disappear from grid view as they no longer satisfy the Filter Criteria.

Lightroom wants to keep the display up to date at all times so that makes sense, but the user has to understand that when planning updates. Anything I type in any writable metadata affects all the selected pictures. That’s 18,756 database transactions initiated by a keystroke.

Since Lightroom doesn’t have a “commit” button for the Metadata it has to guess when the user has finished typing. That’s typically when she tabs out of a field or presses Return.

For my first attempt I selected all the 18.756 pictures with City = Bangkok. The State was blank and I typed ‘Bangkok’ in that field.

My fairly powerful PC ground to a halt with a frenzy of database activity. I knew what was happening so I didn’t disturb it.

Here’s the Windows Task Manager when Lightroom was busy:

Windows Task Manager with Lightroom running

Good that it is only using 52% of the CPU. I guess that means it was disk bound. But I was worrked that the Page File Usage was steadily increasing. “You’re going to run out of Virtual Memory!” I thought. And so it turned out…

After a few minutes (I didn’t time it) I got an error dialog box from Windows:

Lightroom has encountered a problem

I love the delicate phrasing “has encountered a problem”. Why not tell it like it is – Lightroom has crashed!

Of course my heart sank. I thought I’d lost my database. I retarted Lightroom and it showed the message:

Lightroom Restoring Catalog

I am not sure what it meant by this. It didn’t ask me to restore from a backup. I think the word ‘restore’ must mean ‘recover’ – i.e. check all the internal database references and fixup the problems.

I wonder if the Lightroom database engine guarantees ACID properties for transactions, even in a single-user system?

Remember that as well as updating the database Lightroom also wants to update the original picture files with the new IPTC information. So with a few keystrokes I initiated the update of over 18,000 files – many of them big DNG files.

I don’t know how Lightroom handles those updates. Does it queue them? If it hasn’t finished before I shut down lightroom, is the queue persistent?

I found the only way to work was to select no more than 1,000 pictures at a time and change the Province and City. That’s two transactions for each picture – I can’t tell Lightroom to defer updating the database while I make two metadata changes.

They try to make it simple for the user and it works well for a few images. But the user can easily get herself into trouble as I did when working with thousands. It’s good that I did this update when I had only 18,000 pictures to change.

Note that before I did any of this I backed up my database, optimised it and even defragged my hard drive. I thought I may speed things up by closing the Keywording panel (so it didn’t have to accumulate al the keywords as I obviously wasn’t going to change them).

I even used Jeff Friedl’s Metadata Preset builder to build a Preset that only showed the location information. I thought that would save the database some more time compared with the verbose view I normally use.

Lightroom Location Only

Unfortunately that didn’t seem to make any difference to the performance. I think the user interface doesn’t simplify the query it sends to the database based on what’s being displayed in the Metadata panel. Too hard!

I finished my editing at 2:15 this morning. That included fixing a lot of other errors that I found in the location metadata and making a final backup. Everything seems fine with the database, but I am concerned because it has crashed and the recovery is only as good as the engineer who wrote that code.

So – the moral of the story is to be very careful with a powerful user interface like Lightroom. As a user you do have to think a bit about what you’re asking Lightroom to do when you make a change.


One Response to “Overloading Lightroom”

  1. GB-in-TH Says:

    To be more accurate: Lightroom reveals the location fields I use. I think IPTC defines more like “Scene”. I don’t know what they are all intended for.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: