File/Database questions

[JPotatoes]

I provide support for both businesses and home users. Recently Photoshop Elements has started to appear on a number of client's systems and of course I get asked all kinds of questions since I am the ‘computer guy’.

I have read a number of forum threads and followed the links provided. The amount of creativity and wonderful work that I have seen is truly amazing. Congratulations to everyone, this is truly inspiring work.

I am a technical guy and have the imagination and creativity of a rock. My questions are technical in nature. Any help would be appreciated.

It seems that the PSE Organizer must have its own database and associated schema. Is that schema available? Are their any third party products that allow access?

A big concern is having the database on a network or shared storage area. Ward Grant, in a recent posting, has indicated that multiple user access is not supported. That is an expected behavior. Does the database have some locking mechanism or can it be corrupted by two users inadvertently accessing it simultaneously.

Stuff happens and often databases get corrupted. Are there any tools for fixing/reconstructing the PSE database? It would seem a shame that a user could possibly loose years worth of cataloging and work to a corrupted database.

Thanks for any help or references you can provide.

[NickLewis]

Hi, and welcome to the forum...:)

The Organiser maintains its catalog in a file called, by default, My Catalog.psa, which is an MS Access database. It can be opened with Access, although I have never played about inside it, for obvious reasons....... At a quick glance, the table design seems to be in relatively plain English! The default location for the catalog is C:\Documents and Settings\All Users\Application Data\Adobe\Catalogs, although this location and the catalog name can be changed by the user.

I have no specific knowledge, but a recent thread implied that, in PSE5, Adobe have now shut off the possibility of multiple concurrent access, presumably because of the risk of file corruption. (A user was rather upset he couldn't do it anymore!)

PSE Organiser contains three facilities for maintaining catalog integrity:

1) A Reconnect facility which allows for the relinking of photo image files to their catalog records, if, for example, a file has been moved outside Orgainser, so that it loses it. Organiser attempts an automatic reconnection based on file similarities and also provides an option for manual reconnection.

2) A Recover facility which repairs internal catalog errors. It's run automatically before a backup, but is also available to be run as often as a user wants. Personally I run it when I've done major updates.

3) The aforementioned Backup facility, which produces backup of both the catalog and all the image files it references. The backup can be written to multiple CD/DVD media or hard disc, and can be full or incremental.

A user that actually made use of alll these facilities would be unlucky to lose a substantial amount of work irrecoverably, I feel.

Hope that helps, post back if you want more information,

Nick

[JPotatoes]

Thanks for taking the time to respond.

You were correct that this is a garden variety Access database. Renaming the file extension from .psa to .mdb allows easy access to the database in both data and design mode.

Generally speaking, this type of database can be accessed simultaneously from different users in a network environment without contamination. Access keeps track of the file locking and access issues via an .ldb file which is created when the database is open if it is not already open. In all databases there is a record locking feature that will lock various pages/segments if they are being accessed for update. Others should be able to access other areas of the database. So goes the theory.

I have not tested simultaneous access yet but depending on the user’s file/directory access privileges, simultaneous access may not be allowed because the .ldb file cannot be created and or accessed by an ‘under privileged’ user. There may be other issues, but in general terms if users have administrative rights to the program and data storage area simultaneous access to the database should work but not to records which are being updated. Adobe would have to gone to additional trouble to disallow simultaneous access so it is probably available for a properly privileged user.

Since it is an Access database there are many third party repair and restoration products that could be applied in case of a disaster situation.

I am much relieved since I now understand how to advise clients and modify their backup scheme to insure that the Catalog itself is backed up. More importantly, the Catalog can be located on central storage area and moved as appropriate with the right changes to the underlying database. This would involve some hacks but it is certainly doable.

Thanks for the information.

[Ward Grant]

Just a quick toss in to some excellently written discussions:

Since my name was mentioned in thread #1, let me clarify my knowledge is that of former "power" user of Access and not a database administrator. In previous posts and some reading, I thought true multi-user use of the Elements PSA file was not officially supported by Adobe and such use has caused users problems in the past. JPotatoes, you have much deeper database knowledge so I'll accept your theory.

The only other file you may want to look at is the proxy files used for offline storage, if any of our clients use this feature. This data is stored by default in

C:\Documents and Settings\All Users\Application Data\Adobe\Catalog Folders.

The default folder will be My Catalog folder, but if you have a different catalog name or multiple catalogs, this name could change.

Also, if a user has multiple catalogs, he will have a PSA file for each catalog . A backup routine you employ may want to check for multiple PSA files.

[JPotatoes]

Thanks for your information.

For anyone interested, here is my current thinking and some of the reasoning behind that approach.

Microsoft developed the user account approach to allow different users to use the same system and yet keep their data separate it is in the %systemroot%:\Documents and Settings\username area of the principal hard drive.

Use of a single PC by multiple users involves selecting some method of allowing multiple users access to common data. Depending on the number of users, ages, technical competency and trust level between the users, data storage schemes can be developed.
• For mature adult households, I typically recommend that there be a common set of data at the root of the hard drive. That area is easily accessed by all users. Email and other personal data is still in the specific user area; only the common data is centrally located.
• For households with younger children, it is important to segregate the storage so they do not inadvertently destroy it.
• Households with teenagers are subject to circumstantial placement depending on a variety of issues which are a result of social engineering and behavior patterns.
• For business users (and any household that has a reliable network) I always put the data on a network storage device and allow access to all appropriately privileged users.

Typically backup strategies focus on how the data is stored since backing up all data on every system would be very expensive, time consuming and over time increasingly problematic.

Each catalog is nothing more than an Access file with a variety of data and pointers to the location of the picture and album items. I can easily envision business clients having multiple catalogs and over time I can see households having many catalogs each focused on some project or life event. This selection is similar to how you organize your closets or garage – very personal and focused on how you think and work.

This is my current thinking/recommendations for clients:
• Consider where you want your catalogs stored.
• Have a similar discipline on the storage of the actual pictures and revised/edited files.
• Consider storing the pictures in separate project/life event directories and having the Catalog in that same directory. This makes backup and movement fairly easy.
• Back up all of the Adobe .psa files just like you do for email and documents.
• A backup should include the .psa files and the corresponding picture/album data.
• Carefully think about whether you want multiple catalogs and correspondingly multiple .psa files versus one giantgudous (sp?) .psa file with every photo and life event in the one catalog. My bias would be multiple catalogs since database problems most often occur as the size and number of records increases. This bias flies in the face of the users desire to have all of the information in one location. I will observe client reaction.

[NickLewis]

JPotatoes,

There are a couple of issues you may also like to consider:

1) If I've understood you correctly, you are envisaging recommending that your clients do not use the PSE internal backup routine, but adopt their own strategy for keeping backup copies using some independent solution. I know that some people do this, but, as far as I know, this option is neither recommended nor supported by Adobe. I am not aware of any specification or release in the public domain that authoritatively confirms that the Organiser catalog can be successfully recreated in all circumstances simply from a copy of the underlying images and the .psa file. As Ward stated, there are other files, especially the proxy file for offline masters, which are held in other locations - notably the Catalog Folders subfolder in the same tree as the Catalogs file. There may or may not be data in other undocumented locations. In the absence of an authoritative statement of that sort, recommending backup by copying isolated files simply amounts to taking a punt on one's understanding of how the application works.

For example, I know from a brief investigation of my own that Organiser stores email contact data in an HTML file elsewhere. (I'm sorry, I can't recall where it is offhand.)

2) Although I understand that how any person chooses to set up his or her catalog(s) is ultimately a personal choice, suggesting that separate catalogs are used for individual projects or life events runs counter to the very philosophy behind the creation of Organiser, and severely limits the usefulness of various facilities built in to it. For example, searches across multiple catalogs are not possible, and various creations (slideshows etc) can only be produced from within a single catalog. Also, photos cannot be directly moved and copied between catalogs, and although the same photo can be imported into more than one catalog, only one of the catalogs can readily track updates and edits.

Indeed, in his book on the use of Organiser, one of the co-authors of the original software specifically recommends against the use of multiple catalogs except to accomodate multiple users of the same PC, where obviously there may be privacy/confidentiality issues involved.

Regards
Nick

[Ward Grant]

Just a quick thought on your comments:

From previous posts, multiple users have some problems in networked catalogs as the full path is part of the image data. As long as each person accessing the catalog and images references the node with the same drive letter, there seems to be fewer problems.

On multiple catalogs-Adobe's official mantra is "One catalog for all, all for one." Tags and collections are used to separate the images into discrete projects and use. Users can ignore this concept or embrace it;I've seen several users with a Personal and Business (or Personal and Hobby) separation of catalogs.

Remember a user can only have one catalog open at a time. If you wish to use images from different catalogs for a slide show or web gallery, combining them is difficult. Normally you have to make a copy of the image and import into the catalog that is missing the image-adds to total amount of space used for image storage.

[JPotatoes]

Thanks to both for your interest, help and time. I really appreciate the effort you have put into your responses since they raise the right issues to be considered.

Concerns were raised about third party backup software. A computer guy, like any technician, is supposed to understand the object being repaired, the appropriate technology and how to accomplish the objectives of his client, i.e. ‘get the job done’. In every sense, his solutions should be brand aware but his decisions should be vendor agnostic – essentially use the right procedure to get the job done to meet the client’s needs.

Backup methods need to consider many issues. Here is a typical list…
• How much time is available to do the restoration activity? For many businesses, 1 hour is the absolute maximum and costs escalate into the hundreds of thousands of dollars for extended outages.
• How much data is being backed up? Most clients have at least 150 Gbytes of data of various kinds. Graphic images are only part of the data, music, accounting information, CAD and design information, business documents, etc. The list goes on and on and will change as the client changes usage patterns.
• Compression methods used – a very big issue.
• Encryption methods used – the current hot button of the security folks.
• Cost of media. The current industry preference is still tape backup because of media cost and the ability to move media off site and then recycle latter.
• Method of backup – we see increasing use of high speed network backups.

Any backup solution that does not consider the entire system and client needs would be remiss.

To answer NickLewwis observation, yes, I have to understand enough about the application space to develop a holistic backup plan. I would like to view it as a little more than ‘taking a punt’ but in the end that is what I get paid for.

My view is that the Adobe backup approach could be called ‘Adobe Back-up Lite’. It gets the job done for small data stores for the PSE application but it has little concept of the other issues I mentioned previously.

I think I understand the design objective of the original Organizer developer. It works for some consumers. However, it may not be practical for more demanding clients or business users.

It is with this background in mind that I was focused on multiple .psa files. Testing has indicated that it is very easy to open multiple .psa files – just open multiple instances of PSE. If you do that there is no limit to the number of .psa files that can be opened. My initial testing indicates that the picture items and modifications, if kept in a commonly accessed storage area, can be available in multiple catalogs if desired.

Until it was mentioned, I did not understand that PSE had a Contact Book. Apparently it is used for mail activities. I will research to determine where it is located. Thank you.

[NickLewis]

You are obviously thinking deeply on the subject, but I'm not sure you've fully grasped the problem that using multiple catalogs creates.

Take myself as an example. Not all, but most, of my photography is associated with family holidays and events of one sort or another. They would be the obvious "life events" into which to segregate my catalogs in your paradigm. It's how I used to - and still do - set up my folder structure on disc.

At the same time, my wife and I have overarching interests - she's an avid birdwatcher and I'm a spaceflight nerd. So, during any family trip, if there's an opportunity to see and photograph either of these interests, we grasp it. As a result, I'll typically return from a trip with a few thousand images, some of which are purely family mementoes, some of which reflect those special interests, and some of which are taken for general photographic interest (landscapes, architecture, etc.). But all of which form part of the family holiday.

Afterward, I usually want to use those photographs in several ways - I may want to produce a slideshow of our holiday, I may use some images in a landscape project, or I may want to illustrate a talk on spaceflight. My wife may want to produce a scrapbook of bird photographs. I may want to submit images to a competition.

The first need can be easily met if my catalogs are split into life events. The others can't - because they will usually draw on images which were taken on many trips, and so would be stored in many different catalogs. Even if I open multiple instances of PSE, I can't do searches across multiple catalogs without extra work, and I can't use Organiser's facilities for creating projects (e'g. slideshows, web galleries, etc) using images from different catalogs in any simple fashion.

Conversely, if I set up my catalogs on a subject or project basis, I would lose the ability to readily produce family holiday records, because I would certainly want to include that beautiful landscape or special museum visit as part of the trip. Effectively, I would have recreated many of the limitations of a one-dimensional, folder-based filing system inside Organiser, which is a tool specifically created to provide multi-dimensional access to my images.

Obviously, whether this matters to any individual depends on the purpose behind their photography, and the use to which it is put. But I suspect that, even in a professional context, there are very few photographers who, at the shooting stage, can divide their entire portfolio into mutually exclusive categories, and never want to work with cross-category selections. In pre-computer days, photographers set up elaborate index card systems to get round this problem, and, even before the introduction of digital imaging, were using databases to catalogue their film images.

You said that the single catalog approach may suit some consumers, but not more demanding (business) users. I'd argue the reverse. It is the more advanced enthusiast and semi-professional user who will benefit most from a seamless use of Organiser's facilities. The more demands you place on how your images are used, the more benefit you reap from being able to access them flexibly.

(I say semi-professional simply because PSE in general is not aimed at the fully professional market - there are other tools available for that.)

Incidentally, I think you'll find that 150Gb is not very much data at all for any serious photographer. I'm keen but not obsessive, I've only been fully digital for about 18 months, and have so far digitised only the tiniest fraction of my film stock. But I already have over 60Gb of image files.......

Nick