Search form

Digital Storage proves problematic

6 posts / 0 new
Last post
Digital Storage proves problematic

[quote]The National Archives, which holds 900 years of written material, has more than 580 terabytes of data - the equivalent of 580,000 encyclopaedias - in older file formats that are no longer commercially available.

Ms Ceeney said: "If you put paper on shelves, it's pretty certain it is going to be there in a hundred years.

"If you stored something on a floppy disc just three or four years ago, you'd have a hard time finding a modern computer capable of opening it."

"Digital information is in fact inherently far more ephemeral than paper," warned [chief executive Natalie] Ceeney.

She added: "The pace of software and hardware developments means we are living in the world of a ticking time bomb when it comes to digital preservation.

"We cannot afford to let digital assets being created today disappear. We need to make information created in the digital age to be as resilient as paper."

But Ms Ceeney said some digital documents held by the National Archives had already been lost forever because the programs which could read them no longer existed.


And this:

[quote][b]Digital proves problematic ;
Industry lacks method to store footage[/b]

By DAVID S. COHEN -- "Variety"

As far as movies are concerned, digital, like diamonds, was supposed to
be forever.

No more dyes to fade, no more film stocks to decay or catch fire. Just
pristine digital data, preserved for all time, and release prints as
clear and sharp as the images caught by the camera.

Just one problem: For long-term storage, digital is -- so far --
proving to be a time bomb, more permanent than sand painting but not much else.

Simply put, there's no generally accepted way to store digital
"footage" for more than a few months. After that the industry is using a hodgepodge of improvised solutions, some rather costly, others not very reliable.

That looked like a small problem when digital filmmaking was limited to low-budget indies, animation houses and tech pioneers like James Cameron and George Lucas.

Now, though, that small problem is growing geometrically as the major
studios shift away from film to digital capture. Such recent releases
as "300," "Apocalypto," "Zodiac" and "Superman Returns" were shot on
digital. Their digital masters could be seriously degraded if the problem isn't addressed quickly.

In fact, the problem is so severe that the Academy of Motion Picture
Arts & Sciences' Science and Technology Council warned in 2005 that within just a few years films shot with digital cameras could be lost.

Two years on, digital is going mainstream, but "The problem is still
there," says Phil Feiner, chairman of the Acad Sci-Tech Council's archiving committee. And those few years the council warned of are nearly up.

It's not that there's no way to store digital data. On the contrary,
there are dozens of ways to store it, most of which go obsolete in just a few years. Remember 5" floppies and Zip disks?

And the disks that have stuck around? Not so reliable.

Data tapes are balky and can fall apart. Data DVDs and CDs have a history of "rotting" and can't be counted on to last as long as their commercially pressed cousins.

Plus there's no reason to expect that the computers of 20 years from
now -- never mind 100 -- will be able to plug in to today's hard disks.Some private companies are jumping in as awareness of the problem grows, and Feiner's committee will be launching several initiatives over the months to come.

But the amount of digital footage that needs to be archived is growing
faster than ever.

More than one tech expert, including the Academy's Sci-Tech Council
director Andy Maltz, told Variety they had found archival tapes unreadable just 18 months after they were made.

Feiner, the former longtime prexy of Pacific Title, says when he worked
on studio feature films he found missing frames or corrupted data on 40% of the data tapes that came in from digital intermediate houses.

The tapes were only nine months old.

"On certain pictures we had to go into the DI negative and re-scan the
data," he says. "You couldn't retrieve it. Gone."

Milt Shefter, who is a team leader on Feiner's digital archiving
committee, warns that "Long term, it's possible that we're looking going back to the early days of motion pictures, where films are made, put out for a week or two, then thrown away."

With acetate or polyester film, the typical approach to archiving has
been summed up as "store and ignore."

Color film can be turned into black-and-white color separations on
polyester stock. Properly stored in cool vaults at low humidity, such film can last centuries. But there's no way to "store and ignore" digital.

Instead, digital data has to be copied, or "migrated," to new storage
every few years. Migration, however, takes computers, an IT staff, software and a lot of labor. In short: money.

While indies may lack the funds to do regular migration, studios are
plunging in.

Sony's VP of asset management and film restoration, Grover Crisp, says
the studio has put in a program of migrating every two to three years.

"The motion pictures and original material, those are primary assets of
the company," says Crisp. "We all want to do whatever we can to protect those assets."

Disney's VP of production technology Howard Lukk, says as the studios'
digital archives grow, migration becomes a bigger job.

"It's like painting the Golden Gate Bridge and it getting a foot longer
every year."

Not only are more films shot digitally now, but digital filmmaking
encourages directors to shoot more footage.

"The technological issues here are not going to be solved by the
entertainment industry," says Shefter. "It's going to take big
business, big science and maybe big government."

In the meantime, the Academy is stepping in to make the motion picture industry's voice heard in any big business initiative to solve the problem.

The digital archive project is the broadest initiative launched since
the Academy decided in 2003 to fund the current incarnation of the Science & Council.

Maltz expects a report that will pin down what the industry needs to do
to be released in a few months.

Meanwhile, private industry is attacking the problem of digital archiving, too, with at least one announcement in the field planned for NAB.

At NAB, Elektrofilm Digital Studios and Sun Microsystems announced a
service to manage and archive the vast amounts of video from feature film production.

Many tech experts expect the studios to eventually outsource all their
archiving and migration to companies like Elektrofilm rather than try
to do it themselves. Feiner says what is happening is, in effect, the birth of a new business: digital archiving.

He speaks from experience. Earlier this year, three companies received
Science & Technology Awards for their work on archiving. Feiner and his
Pacific Title team were among the winners.

[b]Their solution takes the data from a digital intermediate and turns it
into three-color separation negatives. In other words, they take the digital movie and turn it into good old-fashioned film.[/b][/quote]

EustaceScrubb's picture
"EustaceScrubb" has left the building

"EustaceScrubb" has left the building

This is problematic. I think the bigger question is who is going to be responsible for the conservation. Things aren't progressing so fast that the material is already lost in most cases. I think going back to paper or film for achival is not the right direction.

With hard drives becoming larger and more stable I really see that as the form of archive in the future, not disc, floppy or other medium. It's easy to transfer the data from one drive to another when new technology is developed, and conserves more space and natural resources then taking a step backwards. Not drives that are used on a regular basis, more like stacked servers that receive the data and are wharehoused.

If new technology truly wants to remain productive, then built into the new hardware and software will have to be an awareness of backward compatibility.

New computers can still read the data on old dos discs if they have the necessary hardware to convert. It's the attitude that by ignoring backward compatibility you force the acceptance and market for the new technology. It's time the consumer said whoa.

Pat Hacker, Visit Scooter's World.

But Ms Ceeney said some digital documents held by the National Archives had already been lost forever because the programs which could read them no longer existed.

Somewhere, someone still has that software. It's just a matter of finding it and converting the material.

Pat Hacker, Visit Scooter's World.

I don't imagine it's outside the realm of possibility, either, for someone to say "Well what did that program do? What formats did it use?" and someone to try to rebuild it from the ground up.

I play with old computers and I know there are networks of people out there are maintaining most of the old programs. I really feel that's a lame excuse. And like Scattered said, if the program can't be found, I am sure it can be reverse engineered and read somehow.

Pat Hacker, Visit Scooter's World.

You know up until the last few years microfilm and microfiche were the archival medium of newspapers, somehow they've managed to keep up with the technology.

I am a little suspicious that someone is trying to develop a high priced market for archival, that probably isn't all that valuable.

Certainly the amount needed to be archived mounts each year, that's a given. Not a factor that should lead to the technology being more expensive.

I see the articles posted by Eustace as being spin trying to establish a market.

Pat Hacker, Visit Scooter's World.