Posts Tagged ‘database’

Lightroom Crash Worries Me

December 15, 2009

Perhaps this integrated approach to image management has its downside.

I was using my newly acquired skills with the Lightroom Adjustment Brush to try “dodging” an over-exposed portion of a picture.

From This ...

Without any warning the image portion went completely black and Lightroom hung. It was unresponsive to keyboard or mouse.

To This

I had to terminate the Lightroom process. I rebooted my machine (Windows XP, Lightroom 2.5). Everything seemed fine after I restarted but I backed up and optimized my catalog to be on the safe side. The incident wasted 10 to 15 minutes and increased my blood pressure.

I expect Lightroom ran out of memory: a memory leak. I’d been using it for some time without restarting. I read somewhere that the advanced editing tool use a lot of memory. I had not used them much before.

The problem started me thinking that maybe it is not a good idea to combine my photo management database and most of my photo editing in one application. If the editing component crashes it could easily corrupt the catalog database. For performance reasons database applications often delay writing their buffers to disk. The corruption may not show up for some time.

In the old days when I used Thumbs Plus I would edit a picture in Photoshop. Photoshop was a separate operating system process. So it was very unlikely that a Photoshop crash could affect the Thumbs plus (Microsoft Access) database.

Almost 30 years after the PC was introduced software is still unreliable and always will be. It’s written by humans and it’s fiendishly complex. I wonder if I am putting too much reliance on one piece of software (Lightroom) and it will come back to haunt me one day.

As the Thais say, I “think too much.”

50,000 Pictures in Lightroom

November 7, 2009

This evening I imported my 50,000th picture to my Lightroom 2 Catalog (database). That’s a year’s photography.

Lightroom 50000 Photos

Lightroom 50,000 Photos

Much to my surprise Lightroom didn’t burst into flames or otherwise have a meltdown. There’s an old joke about an undocumented IBM S/360 opcode: HCF – halt and catch fire.

I was heartened by the fact that Jeff Friedl told me he has over 77,000 photos in a single catalog.

I’d love to know what database tweaking Adobe did / plan on Lightroom 3 to improve its scalability.

Browsing the Lightroom Catalog

October 20, 2009

I have not given up on my quest to be able to generate reports on my Lightroom database catalog. I talked about my reasons for it here and here.

The imaginatively named SQLite Database Browser I found at is very slow.

SQLite Database Browser About

SQLite Database Browser About

But yesterday a poster to this blog recommended a better one. It’s called SQLite Spy. It’s free for non-commercial use. I like free! You can download it here.

SQLiteSpy About

SQLiteSpy About

It is very fast – even when examining my 1GB main Lightroom Catalog. (I am living in fear that it will break soon. I’m approaching 50,000 pictures catalogued. See here and here.)

Here’s the SQLiteSpy display of the table “Adobe_images” that has one row for every photo (or virtual copy) catalogued by Lightroom.

SQLiteSpy Adobe_images - click to see full sized

SQLiteSpy Adobe_images - click to see full sized

Of course, this is an extremely powerful tool. I must be very careful, especially with my “production” catalog. I read a cautionary tale here about how a Lightroom user / database geek made Lightroom behave very strangely by making a change to the database using SQL.

He deleted references to some sidecar files by replacing the contents of the field with NULL. But Lightroom started acting strangely and would not display thumbnails consistently. It turned out that he should have used an empty string rather than NULL.

That distinction is a classic database programming error but it is very obscure to most people. Try explaining the difference between NULL and an empty string to a child! Actually, I thought that using NULL was better programming practice as database are optimized for NULL checks. But the Adobe engineer who wrote that part of Lightroom used an empty string and never imagined that anybody would mess with her tables.

I should spend the time to reverse-engineer the Lightroom Catalog schema. That’s the database geek word for the tables and most importantly the relationships between them. But my database knowledge is rusty and it would take me weeks. Adobe will release Lightroom 3.0 before I’m satisfied.

Also …

The ability to execute arbitrary SQL on the database is useful and it may save me from some problems that cannot be fixed using the Lightroom client. Maybe there’s something wrong with the database that causes my “Ever Changing Status” problem. Every time Lightroom crashes I worry the catalog will be corrupted.

But – it isn’t what I want. I have yet to find a fully functional ODBC driver that will allow me to attach SQLite tables to Microsoft Access. Then I can do the database reports that I crave. There’s so much good information in my metadata but I cannot present it as I would like.

The Truth About This Whole “Cataloging” Thing

August 16, 2009

Adobe Photoshop Lightroom Killer Tips » The Truth About This Whole “Cataloging” Thing.

Here’s a thought provoking post from the “Lightroom Killer Tips” site.

A few thoughts of my own:

  • I am not “normal”! I actually enjoy keywording my pictures. I enjoy both the process – knowing that I have a keywording scheme that works for me. I get a kick out of Lightroom’s attempts – 75% successful – to predict what keywords I’ll want to add based on past behaviour (the Keyword Suggestions panel).
  • But when it gets it wrong I want to scream at it: “No! That photo already has a Place tag. Now it needs a Type tag. Don’t show me any more Places.”.
  • Keywording is like knitting, it can be therapeutic if your keywords are well organized and you have a plan.
  • I hope the engineers who worked on Lightroom keywording read this. Yes, as Matt says in his post

Nobody keywords. I know that lots of people commented on it here last week but trust me – you’re in the extreme minority (and I mean serious keywording, not just casual “I do it once in a while” keywording).

But I use them to the extent that I am stretching the Lightroom SQLLite database and the user interface to its limits.

  • I enjoy the knowledge that I can find any picture I have ever taken based on the vaguest of criteria because my keywording is “perfect”. Thus when I was trying to work out the system of license plates for Thai buses and trucks I could locate all candidates very quickly and see if I could extend my list.
  • That’s why I wish I could keep every photo I have ever taken in one Lightroom database. See here and here.
  • I use operating system folders to organize photos by date taken. It is easy to set up during the Import function and occasionally I want to look at pictures I took on a certain date. Yes, I could use Smart Collections for that task, but I’d have to set them up.
  • I liked ACDSee‘s Calendar View for the few months I used it. I wish Lightroom had the same. I like to analyse statistics on my photography and numbers of photos taken by day / month / year is useful.
My Lightroom Statistics

My Lightroom Statistics

  • When I used Thumbs Plus I had a completely different folder structure: Camera Identity and then a folder for each block of 1,000 pictures.
Folder Structure for ThumbsPlus

Folder Structure for ThumbsPlus

Here’s the folder of pictures taken with the Canon EOS-30D that I still have catalogued in ThumbsPlus. Each contains approximately 1,000 pictures. Each folder has a sub-folder called “Edited”. That’s because ThumbsPlus does not support the non-destructive editing or stacking like Lightroom so any edits I made went into a separate folder.

  • I accept that some time, maybe by the end of the year, I will have to split my Lightroom database. Doing it by date is the easiest way and trivial if I have the pictures organized by date in an operating system folder.

I am happy with the way I have things set up now. It is great that Lightroom supports my working style (workflow ugh!). But I’ll never criticise anybody for using a different scheme.

And I know I am not a mainstream photographer because of my interest in information management / databases that’s as strong as my interest in photography.

Checking Integrity

August 13, 2009

Lightroom - Checking Integrity

When I do a catalog (database) backup in Lightroom I accept the option to check the integrity of the database. It proceeds in two phases.

I’d like to know what happens in each of these phases. What happens if the check reveals an error? Does it attempt to fix it? What does the cache check entail? Does it delete old thumbnails? Lightroom’s thumbnail cache is enormous.

I read somewhere about a Lightroom user who diligently backed up her database yet it was corrupt and she wasn’t informed until something went seriously wrong.

I wish I could get the ODBC driver for the SQLLite database to work. I discussed it here. Then I could make my own checks by attaching it to a MS Access client.

As it is i have an uneasy feeling that all may not be well with the catalog but I won’t know until it is too late. Am I “thinking too much”?

Image Reporter

July 4, 2009

One of the things I miss about Thumbs Plus when using Lightroom is the fact that Thumbs Plus uses a Microsoft Access database. In a previous life I did a lot of Access programming and I was able to attach the Thumbs Plus database to the Access 2003 client. the schema was self-explanatory and I wrote lots of reports on my photos.

Thumbs Plus has the concept of “user fields” where I could add as many typed fields as I liked to describe each image. I collect information on land, air and sea vehicles and I used user fields to store information like make, model, owner, fleet number and so on. I think a simpler method of defining user fields is a high priority enhancement for Lightroom.

I’ll talk more in another post about why user fields are much more useful than keywords or collections for categorizing my photos.

Anyway, the database geek in me was interested in this tool – Image Reporter. It comes from the same company that produced Image Ingester and Image Verifier.

Analyzes metadata in a Lightroom catalog and reports on cropping, camera makes and models, lenses, average focal length, ISO, and more. Filters by image type, rating, and capture time. Described in an article by Marc Rochkind at The Online Photographer.

I like free, so I downloaded it and tried it. Here’s a screen shot of my first report:

Image Reporter

Sorry to say its output is not very interesting to me. Cameras and lenses are not as interesting as statistics I can store and analyse on where my pictures were taken and what they are of.

Note it is a stand-alone program, not a Lightroom add-in.

I’d like to be able to write my own reporting program for Lightroom. My ideal would be to get a ODBC driver so I can attach its tables to Microsoft Access. Then I could write SQL queries against the database and use VBA to write tools.

Mark Rochkind in the article quoted about talks about how Lightroom’s catalog is based on a SQLite open source database and he references a tool that can examine it. I will investigate this. Maybe a solution to my reporting problem is closer than I thought.

I should also re-read the article and take note of how he uses its reports. Maybe I have more to learn than I thought in my summary dismissal above.

Lightroom: How Many Photos Can I Have In A Catalog? []

June 24, 2009

Lightroom: How Many Photos Can I Have In A Catalog? [].

I was going to write my own entry on this subject, but the Foto-Biz blog beat me to it.

I think the post has a misunderstanding about the SQLite database that Lightroom uses. SQLLite, to quote their About page is

SQLite is a in-process library that implements a self-contained serverless transactional SQL database engine.

To me that says that you compile the database code into your application but it doesn’t mean that the database itself is all in-memory (virtual or real). The Lightroom catalog is a true database with indexes and tables all in one operating system file. SQLite manages that file.

One analogous PC product is Microsoft Access. Thumbs Plus uses Access for its database. The major drawback with Access is that it isn’t cross-platform and Adobe needed something that works on the Mac.

Enterprise databases like Oracle and SQL Server use a separate database server process. That makes them scalable to many users but it so much more work to administer. Lightroom is simple, but in its current architecture it will not scale to support multiple users of the same catalog.

Interestingly Thumbs Plus is more scalable than Lightroom: you can use any ODBC compliant database with it. I experimented with that using the popular free open source database MySQL. It works but the database administration was too much trouble for my single user environment.

The Foto-Biz blog further says:

Backing up does garbage collection and shrinks the catalog

I don’t think so: I believe a backup is a pure file copy. To shrink the catalog you need to Optimize it using the facility presented in the Lightroom Catalog Settings / General tab.

There’s a brief discussion of this at the Peachpit Photoshop Lightroom Reference guide here.

I don’t want to be too rude but I think the Foto-biz post contains a lot of mis-information.

For myself I want to have all my images in one Lightroom database and damn the disk space! I want to be able to find any photo I have taken at any time without having to think what database I stored it in. But I will never have time to import and index all the photos in my old Thumbs Plus database.

But from here on out I do not intend to use more than one Lightroom database. I want the catalog I have now to index every photo I take for the rest of my life.