Size Matters: Gigabytes,Terabytes, Petabytes - oh my!

This latest blog is brought to you courtesy of Techolony Associate John Southern, a specialist Infrastructure Project Manager:

Size - what a wonderful thing it is! In the 1970s, we were in the realms of Megabytes - known as Big Storage. Users counted in the byte region, and while the desire was always for more, the dreams were limited to kilobytes. The 1980s came along with huge step changes. At the beginning of the decade, we were looking at single digit kilobytes for memory, and storage was peaking out in the dozens of megabytes. Just a few years in to the 80s, and we had yet another factor change - the home user. The every day non techie individuals were starting to bring this to the home; Users became familiar with memory in the hundreds of kilobytes. Storage at home was now on disk at hundreds - and soon to be megabytes.  Commercial storage was topping out in the hundreds of megabytes..  Eventually, solid state storage took over floppy disks with SSD only systems becoming commonplace for hard disk storage. Slow access storage has increased with individual tapes holding greater volumes and robot arm library arrays becoming cheaper by the day.

This is all fine - most people are happy quoting megabytes, gigabytes and terabytes. Rarely do I hear kilobytes any more and frequently architects throw around petabytes and exabytes without too many raised eyebrows.The issues begin when data needs to be moved. Within mergers and acquisitions, you are either buying or selling your data - and it needs to move from one set of storage to another. Rarely do you buy the storage units lock, stock and barrel. If you do, your team usually still want it on your better hardware, even if the “better” option is subjective. Sending storage from one business to another is the primary issue. The usual complaint centers around why it takes so long to send, when at home, I can copy X (film or photos) to W (USB pen drive) in a matter of minutes - if not seconds.

Depending on the film codecs, a HD film is between 1.6 and 9GB. Pen-drives of 128GB plus are common. Everything is about the size - or more accurate, the size range of your data.Frequently, I will throw a few hundred gigabytes out to a storage device. A USB3 gen2 connection has a maximum throughput of 10Gbit/s. Assume we live in a perfect world with no connection overhead  - we really could move a real gigabyte in under a second. Also let us ignore the real world problems such as organising permissions, needing the data static while we transfer, mix of storage such as long term tape access, similar architecture and software tools, security model for the data etc.,

A Programme Manager will usually complain that it only takes them seconds to move a film, so why are we taking longer to move data between clients?

Trying to explain is a tough road. How long does it really take then, to copy that film? Usually the first time you ask the answer is a couple of seconds. After badgering they usually say 20 seconds. So now we have just had a factor increase, but twenty seconds is still not long time. The real reason for the time difference of our data is we are not in a perfect world. Transmission overhead to check the data adds a little to the time. However, the real difference is both the read and write speeds of where your storage is. Still 20 seconds is not long when the difference is days… right?

To be continued…