Bernie Walsh looks at some of the considerations for designing a central storage system and the consequent technical issues that need to be addressed.
We have now achieved a certain level of maturity in the way we view digital broadcasting technology. First we had individual digital applications, usually at the high end of post-production.
From there, we have grown to a widespread use of digital acquisition, editing and manipulation, albeit often using proprietary formats and interconnecting through a dedicated video-style interface: SDI.
By saying we have reached a certain level of maturity, I mean that we are finally recognising digital video and audio files for what they really are: data. As a succession of 1s and 0s, they are identical to any other data in any other computer application.
We need to treat them as such. Imagine for a moment a world where your spreadsheets needed to run on separate computers, networks and servers from your email; where documents in Microsoft Word needed to be decoded to raw ASCII before they could be passed on to someone else. This is clearly unacceptable.
Once you have a facility-wide system that can exchange digital files as data, with each broadcast application being able to share common content, then the logical step is to provide a central store for that content. But what are the considerations for designing a central storage system, and what are the technical issues that need to be resolved?
The traditional IT approach to centralised storage is to create a hierarchical view of the system, broadly speaking:
Online - expensive spinning disks with very high throughput for immediate delivery of data.
Nearline - less expensive spinning disks, which can move content to the online server quickly when required.
Archive - a tape or optical disk library system which takes content from the nearline disks when capacity is an issue.
Offline - shelved storage of tapes or optical media when the archive system is full.
From top to bottom, those four levels decrease in convenience and speed of access, but also decrease in cost. The art of the system designer is to get the right capacity at each stage to meet the service level requirements at the minimum cost.
A central storage system will normally be spread across multiple layers. The broadcast application itself - editor or playout server, for example - will have its own local storage.
In some products this will be a buffer store: the local disk on a non-linear editor. Other broadcast applications have sufficient capacity and its own network capabilities to provide what is in effect an online server.
The Avid Interplay system for example, provides networked editing via its own ISIS storage sub-system.
But because it does not offer connectivity between broadcast applications, for the purposes of this paper it should be regarded as the buffer store.
It is fed by the central online server. In turn that may be backed up by nearline disks, archive and/or offline shelf storage - whether you have every level will depend on the size and complexity of your requirements.
To the broadcast application, though, it looks like a single system: you ask it for content, or you offer it content, and it deals with it.
Cluster for resilience
In the storage management system, every node is physically identical. Typically they are HP DL380 servers or equivalent, and each runs Windows and Microsoft SQL. A Microsoft-clustered SQL database uses two servers sharing the same RAID storage.
The database is held on the RAID. At the simplest level, should one of the servers fail for any reason the other will take over.
The storage system nodes have identical hardware and the same set of software. They communicate with the database server cluster over Ethernet. They will also be fitted with host bus adaptors (HBA) to communicate with the central storage devices and with the broadcast applications.
To add resilience, dual port HBAs will normally be used, so if one physical port or device driver goes down there is a spare to provide continuous service.
Maintained in the database and stored in RAID is a 'heartbeat' table. Every node writes a timestamp to the heartbeat table, every five seconds. As soon as a timestamp is missed, every other node in the network realises that there is a probable failure.
Once the failed component is brought back into service it commences by sending timestamps to the heartbeat table. The database recognises this and will start allocating it tasks.
Another critical element for the success of the system is load balancing. First, some tasks will take longer than others. Most obviously, moving a two hour movie will take longer than a 30 second commercial. But a well-designed storage management system should allow for partial restore.
Second, there is the issue of priority. Some tasks will be regarded as more important than others: loading material into the playout server for immediate transmission is an obvious example.
Requests for archive material into the news editing environment are often urgent.
There is also the issue of managing resources to best effect. By its nature, tape storage works best when reading or writing at its maximum data rate.
Running the tape continuously gives a better throughput, as well as less wear and tear on the tape and the drive, than starting and stopping simply because bandwidth is not available.
Content lifecycle management
The requirements of each broadcaster in terms of storage allocation, security, management, resilience and response times will be different. We have defined a central storage solution as essentially invisible to the user.
Clearly, then, it is not acceptable for the broadcaster's preferred workflow and individual requirements to be compromised by a central storage system that cannot meet them.
The approach should be to look at the lifecycle of each piece of content that the broadcaster handles, then develop rules to define how it should be handled in terms of storage and security.
The goal is to make all this possible without bottlenecks in the workflow, which compromise content delivery to critical on-air applications or risk valuable content.
A central storage management system developed with these specific requirements in mind cannot be based on a conventional IT approach.
Integrated with the highly resilient hardware configuration must be software that reflects the nature of the industry.
In particular, it has to be flexible enough to handle different types of content in different ways, to provide highly resilient storage, and access by multiple users in a timely fashion.
Bernie Walsh is director of sales for SGL.
As part of this workflow, SGL will introduce FlashBrowse, FlashBox and FlashTurbo Accelerator for Avid ISIS to the broadcast market.
FlashBrowse: SGL's new creation tool enables browse copies of archive material to be stored outside of an Avid ISIS Media Network, providing a fast, cost-effective solution for post production facilities and broadcasters.
FlashBox: SGL's new, all-in-one archive solution starts with a 4TB disk archive, a single LTO4 tape drive and 10 x 800GB tape slots, providing 12TB of entry-level storage.
FlashTurbo Accelerator for Avid ISIS: Providing a significant increase in data transfer rates, SGL's FlashTurbo Accelerator allows direct connection with Avid's ISIS Media Network, which has a distributed and open architecture that supports both Avid and non-Avid clients.