Archiving on Flash Media

by David Waelder

After 150 years, the negatives that Timothy O’Sullivan exposed for Mathew Brady at Gettysburg still yield prints. Many of the glass plates were recklessly discarded after the war, but those that survived still hold images and the pictures have become part of our cultural legacy.

There is no format for digital storage that can match that performance, no gold standard for archiving audio information. Reel-to-reel magnetic tape is generally considered fairly stable but it is vulnerable to emulsion shedding and delaminating after only 10 years. With careful storage, tape will usually have a life expectancy of at least 30 years and is often playable after more than 50 years.

Experience with digital files on magnetic or flash media is still so brief that any advice must be flagged with an asterisk. Hard drives all fail eventually but a RAID storage on multiple drives seems to be reasonably dependable. Flash media is very promising as there are no moving parts, no spinning disc and floating head that might collide, no lubricant to dry out and no layers that might lead to the delamination that is a vulnerability of optical discs.

Until someone devises a universally accepted archival format, we will have to be satisfied with pretty good performance. And, by any reasonable standard, long-term storage is the responsibility of the producer, not technicians who may be hired on a day-playing basis. Still, when turning in the day’s work, one wants to be sure that its important content will be more permanent than a drawing on an Etch A Sketch.

​SanDisk first developed the Compact Flash (CF) in 1994. It originally used the Intel NOR flash memory design but later changed over to Toshiba’s NAND memory management protocol. The Toshiba protocol incorporates a “memory leveling” operation to evenly distribute writing among the sectors for even wear.

Flash memory is comprised of two elements: the matrix of memory cells and a separate controller circuit that manages the flow of data. Manufacturing irregularities may produce bad sectors in the memory cells but the controller circuit is programmed to identify and map those sectors and remove them from active use. This provision to permit some compromised cells makes it possible to economically manufacture a reliable product. Since manufacturers are all getting their memory wafers from the same few sources, the design and construction of the controller circuits is the primary distinction between a cheapo and a premium memory card.

In practice, the CF cards have proven to be remarkably resilient, surviving accidental trips through the washing machine, exposure to magnetic fields and extreme environmental conditions. Even the standard cards operate in temperatures ranging from freezing to more than 120ºF and the range for safe storage is even wider.

There are some known issues that can cause memory loss. Most of these are related to errors in writing protocol, like removing the device before properly closing down. Flash memory can also be damaged by static discharge and by writing with a failing power source such as a discharged battery. Although they seem to shrug off airport X-ray machines, the cards are vulnerable to the more powerful scanners used by the post office and should not be mailed unless contained in a shielded pouch.

When a memory card loses data without any identifiable cause, like a static discharge, it is almost always a failure in the controller circuit rather than a failure in the memory cells themselves. This is an important distinction as it means that the data is really still on the card and available for recovery. Sometimes recovery can be accomplished with software programs but, with a cataclysmic controller failure, it may be necessary to send the card to a specialty company that has the equipment to bypass the bad controller and access the data matrix directly. This may not be cheap or convenient but it’s an argument in favor of flash memory storage that it is almost always possible to restore the original files.

Estimates of archival storage capabilities run the gamut from cards that lost data almost immediately to cards that have been stable for many years. General consensus is that cards that accept formatting without incident are likely to be reliable for many years. The market is infested with counterfeit cards and this may explain some of the premature failures. For use as an archival master, it is especially important that cards be purchased from a reliable source. Location Sound, Trew Audio and The Audio Department all take pains to get their flash media from reliable suppliers. Favored brands are Delkin Devices, Transcend and SanDisk.

Nikon and Canon continue to use Compact Flash as the primary storage medium in their high-end cameras and Leica uses it in their new medium format camera. This is significant because photography has been one of the primary applications for Compact Flash; continued use in professional and semi-pro equipment assures that the format will be viable for the foreseeable future.

For very long-term storage, magnetic media in RAID arrays with regular data transfers is preferred to any flash media. SanDisk does offer a Memory Vault they claim to be suitable for storage “up to a century” but the product is too recent to have any track record and, in any event, is not configured for direct connection to recorders. For very long-term storage, users will have to encode their data onto wet-plate collodion emulsions on glass.

CF Cards As Masters

by Thomas Brandau

While I’d like to thank Scott Smith, CAS for starting the conversation about digital asset management, I’d like to revisit that oft-heard saying that “In Hollywood we don’t plan, we just do it, then spend whatever time necessary to fix the problem, then declare ourselves expert at something else.”

Recently, the post supervisor on a network series came down to the set with a list of WAV files to ask if we could get him copies from three episodes ago. They were handed in and went through the dailies process so, if they had been missing on the day, we would have heard about it long before. No reason was given for the request, but I have to assume “something happened to them.” They were either on my 788T or on the other unit’s record machine. No problem really, we just needed to know who shot these files and on what day of the episode. Of course, the timecard question came front and center, that being: when was I to look for these files, at wrap or lunch? And, of course, there was the problem.

In another situation, a feature film this time, the picture had been locked and, as sound editorial was conforming the sound, several days’ work had gone missing from the drives. Again, picture editorial had received the files, dailies discs had been struck and distributed, but at the last stage, now a problem. It’s simply amazing how quickly and how hot it got. “Where are your backups, how did this happen?” etc. The incident went up the chain to studio management in a day.

 In this case it turns out that the DIT guy had the whole show on his drives and, as he hadn’t worked since the shoot, he hadn’t formatted and erased the sound or picture files. They were recovered and it only remains to troubleshoot how and at what point these files disappeared from both a primary and backup drive.

Lesson learned: Compact Flash (CF) cards are relatively cheap. In the old days, we’d turn in four or so 1/4-inch master tapes each workday, easily $300 a week. Why are we reformatting this media? The current workflow is for sound to give the Compact Flash cards to the DIT guy or gal (or to whoever is downloading the picture files) to be included with the picture files on the same shuttle drive going to post. The original files remain on those CF cards and on the internal drive on my record machine.

Again, why would you erase the “master cards”? With good file management, sound will fill four or five 16GB CF cards every week. Or two 32GB cards, assuming you fill those cards and make daily files for editorial to track. Personally, I like 16GB cards. If one is lost or destroyed, there’s less on it, and then there’s the time required to format and load 32GB cards. You can argue however that 32GB cards are a bit more cost-effective and you’d be right.

Which brings us to the subject of the speed and quality of the media itself. Of course, you want the best quality and a reliable product, goes without saying, but it’s also true that a machine running all eight tracks will only need 133X or 166X cards. This comes directly from Sound Devices. Poly WAV files are just not that large or complicated to require faster write speeds. With every reformat you are asking the card to overwrite the data. I know of no one taking the time to lo-level format, so when you “erase” the card you are simply telling the directory to overwrite the sectors. This is where the digital errors come in to play.

There is so much talk about digital permanence that I contacted some manufacturers directly. It’s interesting that the story is the same wherever you go. Flash and SSD memory “wafers,” as they are called, are pretty much like LCD screens: identical, all coming from the same factory. It’s the build and the video amplifiers that make the difference between television brands and the same is true with memory.

The controller on the card or drive determines accuracy and permanence. SanDisk and Delkin make their own controllers, and Transcend, Kingston and others outsource theirs. Also, there are “industrial” and “consumer” controllers. A German company, Hyperstone, makes bulletproof industrial controllers for single-layer Compact Flash cards. These are used in all the mil spec and medical applications where failure or loss of data is not an option. Single layer, write once, cards are about $300 for 16GB. We are all using MLC, multi-layer media, but that doesn’t mean we can’t treat our cards like SLC media.

So, I’m arguing for good-quality cards, written once and held as a master library, for what, $150 a week at retail?

It’s also astounding to me that what is essentially the camera negative, as a common QuickTime file, is being copied drive to drive without any real thought of longevity and security. With a 7200rpm G Drive, it’s not if, but when that media will fail. And just think of the rental Avid workstations or the DIT guy with your camera “negative” and sound masters in his garage after the shoot.

A subject for another day, but how can the studios and producers allow this?