Modern storage includes all manner of exotica: nanowires; HAMR; clusters and more. But research has shown that a simple storage device can save tens of thousands of lives a year in hospitals around the world. So why aren’t doctors using it?

Paper isn’t sexy, but it works
The simple storage device is a human-readable list, printed on a piece of paper. Pilots use lists to ensure that the hundreds of details required to safely pilot an aircraft are properly looked after. But doctors?

Lists? We don’t need no stinkin’ lists!
Dr. Peter Pronovost is the Medical Director of the Center for Innovation in Quality Patient Care and an Assistant Professor at Johns Hopkins University’s School of Medicine. His research has demonstrated that simple checklists can save lives - lots of lives - in Intensive Care Units (ICU).

Catheter infections in intensive care
Catheters are tubes inserted into you to monitor conditions, administer drugs or to drain. Nationwide some 11% of catheters become infected, often leading to life-threatening - or
-ending - conditions in patients who are already very ill.

Dr. Pronovost tried a simple experiment: develop a checklist for catheter insertion and see what happens. First, the list:

• wash their hands with soap
• clean the patient’s skin with chlorhexidine antiseptic
• put sterile drapes over the entire patient
• wear a sterile mask, hat, gown, and gloves
• put a sterile dressing over the catheter site once the line is in

How hard is that? Nothing any doctor doesn’t know - but all too often doesn’t do.

The results:
Quoting from a great article in the New Yorker:

Pronovost and his colleagues monitored what happened for a year afterward. The results were so dramatic that they weren’t sure whether to believe them: the ten-day line-infection rate went from eleven per cent to zero. So they followed patients for fifteen more months. Only two line infections occurred during the entire period. They calculated that, in this one hospital, the checklist had prevented forty-three infections and eight deaths, and saved two million dollars in costs.

More checklists
Dr. Pronovost and his team developed other checklists and observed similarly dramatic results:

They tested a checklist for patients on mechanical ventilation, making sure that, for instance, the head of each patient’s bed was propped up at least thirty degrees so that oral secretions couldn’t go into the windpipe, and antacid medication was given to prevent stomach ulcers. . . . . the occurrence of pneumonias fell by a quarter; and twenty-one fewer patients died than in the previous year.

Just having doctors and nurses create their own checklists cut average stays in intensive care by 50%, saving lives and money.

Wider validation
Dr. Pronovost tested his program in 103 ICUs in Michigan, including some in inner-city Detroit, the poorest large city in the US. The results were equally dramatic. According to research published in the New England Journal of Medicine the checklist program

. . . resulted in a large and sustained reduction (up to 66%) in rates of catheter-related bloodstream infection that was maintained throughout the 18-month study period.

The hospitals saved an estimated $175,000,000 in costs and over 1500 lives. With checklists.

The Storage Bits take
The US spends about a third more per capita on health care than the next most costly country, Switzerland, and our results are worse than countries that spend half of what we do. Medicare’s expected cost increases dwarf the Social Security “crisis” so beloved of fear-mongering politicians. If we can fix spiraling health care costs, Social Security is easy.

Electronic medical records are touted as a money-saving technology - and they are - but until we have a system focused on helping people get and stay well, rather than cost avoidance, American’s have little reason to trust that EMRs will help rather than hurt.

Dr. Pronovost’s checklists don’t require fancy technology or big capital expenditures to save lives and money. Which is why you don’t see them in TV ads. If I ran marketing for Hammermill . . . .

Data storage takes many forms. However it comes it extends human capabilities in ways we are still learning to appreciate.

Comments welcome, of course. Do they use checklists at your local ICU?

Massive storage has given governments worldwide the ability to record citizen activities and abuse human rights. And, as yesterday’s post on Deep Packet Inspection noted, technology marches on.

But the recording and storage revolution cuts both ways. Citizens can capture government misconduct and bring pressure to bear on those who abuse their authority.

Two cases in point
A New York 17-year old was being interrogated by police over a shooting in an elevator. As the interrogation began he pressed the record button on his music player to record the 1 hour, 15 minute session. At the trial the detective, unaware of the recording, testified that the suspect “wasn’t questioned” about the shooting.

But then the defense confronted the detective with a transcript it said proved he had spent more than an hour unsuccessfully trying to persuade Erik Crespo to confess—at times with vulgar tactics.

Once the transcript was revealed in court, prosecutors asked for a recess, defense attorney Mark DeMarco said. The detective was pulled from the witness stand and advised to get a lawyer.

The detective has now been charged with 12 felony counts of perjury.

The big game
After Hawaii’s dramatic comeback against Washington last Saturday, elated students streamed on to the field. A cop grabbed one of them, threw him to the ground and then punched him several times, evidently without provocation.

From a report:

Several fans scooted past on-duty officers in riot gear, but one young man did not make it.

“(The officer) hooks the guy around the head and brings him down,” UH freshman Miles Kreisberg said. Kreisberg zoomed in his camera to the area of the incident.

“I was like, ‘Oh, I got to catch this.’ I couldn’t believe what was going on,” Kreisberg said.

Students put the video up on YouTube and the Honolulu PD has launched a criminal investigation.

The Storage Bits take
The authors of the US Constitution knew well the range of human failings. The US government is designed to function despite human greed, dishonesty, cowardice, vanity, stupidity, arrogance and megalomania. That is why the President is sworn to defend, not the American people, but the Constitution.

In today’s world, where corporations can rival national governments in power and influence, individual liberty faces a challenge that the Founders did not foresee. Only an active and involved citizenry can continue to safeguard our liberty.

A citizen’s ability to capture misconduct in video or sound is an important check on abuse of government or corporate authority. Bravo to the technologists who are making it easier and cheaper to use.

Update: Self-surveillance The Wall Street Journal published an article on Saturday about lawbreakers taking pictures of themselves in the act - and going to jail for it:

. . . many camera-phone owners seem to think outsiders won’t have access to the photos, says Mike Schirling, deputy chief of the Burlington, Vt., police department. He says he recently helped convict a juvenile on weapons charges based on cellphone images of him brandishing a rifle at night on the roof of a school building. “Drug dealers just naturally take pictures of their drugs and their money and their significant others,” he adds. . . .

Some criminals are nabbed for taking the next technical step: distributing their camera-phone shots over the Internet. Ms. Collins, the assistant state attorney in Connecticut, says she obtained restitution payments for dozens of residents whose mailboxes had been destroyed with baseball bats. The evidence: The perpetrators — some local high school students — had posted camera-phone pictures of the deed on the MySpace Web site.

A law professor notes that people give up their constitutional right against self-incrimination by taking a picture of their lawbreaking. Truly, stupidity knows no bounds.

Comments welcome, of course. 66 years ago today my father, a newly-minted 2nd lieutenant in the Navy Medical Corps, woke up to the sounds of guns, planes and bombs in Pearl Harbor, Hawaii. He spent the next 36 hours in an operating room, patching people up. I wish I could see some video of that.

You may experience mild discomfort
The extremist “free market” crowd wants to toss hundreds of years of common and public law in the toilet so a few near-monopolies can maximize their profits and minimize your choices. Should you care? Only if you use the Internet.

“We don’t care. We don’t have to.”
That was AT&T’s unofficial motto back when they were a regulated monopoly. For decades you could buy a phone in any color you wanted, as long as it was black.

Attach a fax machine at home? Oh no! It could damage the network!

Buy a handset from a low-cost supplier? Oh no! It could be incompatible! Just rent ours in perpetuity. Much better - for our bottom line.

Telecommunications is the backbone of the information age
And it can’t be handed over to the “(un)free market” for care and feeding. In the US we did that with cellphone standards and after enormous investment we’ve ended up with a patchwork of incompatible networks and services that lag the rest of the world.

It took an act of Congress to get cell phone number portability and the telco’s dragged their feet as long as possible implementing it.

Why? So they could lock you in for a little bit longer. How can anyone trust these people to act in the public interest?

The latest abuses
Today’s Wall Street Journal - subscription required - reports on new gear that allows ISPs and telcos to track your Internet usage to provide ads based on the sites you visit. It is a fast-growing business. Per the WSJ:

The newer form of behavioral targeting involves placing gear called “deep-packet inspection boxes” inside an Internet provider’s network of pipes and wires. Instead of observing only a select number of Web sites, these boxes can track all of the sites a consumer visits, and deliver far more detailed information to potential advertisers.

According to Wikipedia:

DPI devices have the ability to look at Layer 2 through Layer 7 of the OSI model. This includes headers and data protocol structures as well as the actual payload of the message. The DPI will identify and classify the traffic based on a signature database that includes information extracted from the data part of a packet, allowing finer control than classification based only on header information.

A classified packet can be redirected, marked/tagged (see QoS), blocked, rate limited, and of course reported to a reporting agent in the network.

The “reporting agent” could be with the FBI, a divorce lawyer or maybe just someone with a grudge and access to a National Security Letter.

Getting privacy right means transparency and legal sanctions
The history of US government abuse of police powers to coerce, embarrass and control people is long and sordid. Nothing has changed. Give people power and they WILL abuse it.

We have Martin Luther King Jr. day now, but 45 years ago he was just another un-American agitator in the FBI’s view. They used phone taps to gather information to smear and discredit him.

Corporations are hardly better. General Motors sent detectives after Ralph Nader in the ’60s and HP sent detectives after their own board members just a couple of years ago. Comcast’s packet forging is just par for the course. Along with a “technical requirements” argument.

The Storage Bits take
With proper safeguards, DPI can extend advertising-supported infrastructure with broad benefits for many people. But hoping won’t make it so.

This service needs a legal framework that includes a prohibition on tracking or reporting visits to sites of a medical, religious, sexual or political nature. Information that ties surfing records to your name, address, age and the like needs to be kept separate. Government access must require warrants, not just subpoenas.

DPI will no doubt be abused no matter what the rules are. Which is why there must be criminal penalties for abuse. Stiff ones.

And net neutrality.

Comments welcome, of course. What other kinds of personal information should be protected?

Update+ Readers, thanks for flagging the missing *law* in the first sentence. AFAIK, us ZDnet bloggers are not edited. When we screw up, it is our own fault.

As a reluctant web page developer on StorageMojo and for friends, I often wonder how a page I like is done. With Safari’s cool new Inspector I don’t have to wonder any more.

Here’s is what the Inspector looks like:

It makes it easy to pick out the CSS, documents and other components of a page. Check the “Show implicit properties” box for even more data on the page.

A quick tour
This shows the Inspector as an independent window, but if you click 2nd from the left icon at the bottom of the window it will show up at the bottom of the Safari window. Click on “body” and the body code will appear, highlighted.

Enabling the Inspector functionality requires a quick trip to the Terminal (located in your Utilities folder). At the prompt enter
defaults write com.apple.Safari WebKitDeveloperExtras -bool true

Restart Safari, right click on a web page and you’ll see a new “Inspect Element” option.

Comments welcome, as always. Firefox’s Firebug add-on is an option for devoted FF users.

A mathematical analysis of the PageRank algorithm shows there are 3 simple, legal things you can do to increase your PageRank score.

These aren’t bogus SEO techniques like hidden text, link spam and page cloaking that will get you banned from Google. These techniques use the mathematical properties of PageRank to give your site more prominence.

Inside PageRank
The principle of PageRank (PR) is similar to citation indexes maintained for scientific and legal articles to measure influence. PR has to be easily computable on the fly, which has some interesting storage implications I’ll cover in another article.

Two papers, Inside PageRank by Monica Bianchini, Marco Gori, and Franco Scarselli of the University of Siena (AFAIK available only through the ACM) and Deeper Inside PageRank (pdf) by Amy N. Langville and Carl D. Meyer, cover the mathematical framework of PageRank. These suggestions are from the Inside PageRank paper.

Both are dense math papers, not CompSci, so if your linear algebra is rusty you’ll find them a hard slog. The references are great sources for further study though.

The 3 techniques

• Eliminate dangling pages
• Aggregate out links on pages with many site links
• Split long pages into several pages

Eliminate dangling pages
This is a no-brainer. A page with no out links is a dangling page. Even rarely used reference material should have links back into other pages on the site.

Aggregate out links on pages with many site links
Out links reduce what Bianchini et. al. call the “energy” - and PageRank - of a site. To reduce the lost energy aggregate the out links on pages with more internal links.

In this diagram, less energy is lost with the out link from page 1 than the one from page 2. Page 1 links to 3 internal pages, while page 2 only links to 1.

Split long pages into several pages
A web site has a “default energy” which is proportional to the number of pages in the site. All other things being equal a larger number of pages will get you a higher page rank. Here’s a graphic:

There is a usability issue with lots of pages, so be smart about how the content gets divvied up. Frustrated visitors won’t link to you, which also hurts your PR score.

The Storage Bits take
PageRank isn’t the only thing Google looks at, but it is probably the most important factor. These simple techniques will help your site do as well as it can in Google ranking.

Comments welcome

There’s a dark horse coming up on the outside
Isn’t Sun - and Solaris - almost dead? No and they’re showing quite a bit of life in the storage arena. It is amazing what a $12 billion company can do with a unique strategy and deep engineering smarts.

One big change: after winning the $1.6 billion anti-trust settlement against Microsoft, which included a 10 year cooperation agreement, the 2 companies have embraced each other in ways - like storage - unthinkable 10 years ago.

CIFS support in the Solaris kernal
After blowing billions on failed - or not very successful - storage acquisitions, I’d pretty much given up on Sun as a storage vendor. But the new CEO, Jonathan Schwartz, has a new storage strategy: move storage functionality into the operating system.

The latest piece: CIFS. There are two file serving protocols in the world, NFS, an open standard Sun invented, and CIFS, which Microsoft controls. For years I’ve listened to engineers moan about the pain of implementing CIFS on non-Windows systems.

In a blog post Sun engineer Alan Wright explains why:

There is a common misconception that Windows interoperability is just a case of implementing file transfer using the CIFS protocol. Unfortunately, that doesn’t get you very far. Windows interoperability also requires that a server support various Windows services, typically MSRPC services, and it is very sensitive to the way that those services behave: Windows interoperability requires that a CIFS server convince a Windows client or server that it “is Windows”. This is really only possible if the operating system supports those services at a fundamental level.

Solving those issues required 180,000 lines of new code in Solaris, Sun’s UNIX OS.

It gets better
OpenSolaris supports the cool new file system ZFS (see my ZFS: Threat or Menace? on StorageMojo), which is also being integrated into OS X.

ZFS eliminates most of the data integrity issues that plague older file systems like NTFS, UFS, HFS+ and the like. Sun has added important enhancements to ZFS to support CIFS, including

• Support for DOS attributes (archive, hidden, read-only and system)
• Case-insensitive file name operations.
• Support for ubiquitous cross-protocol - NFS and CIFS - file sharing.

Check out the storage community at OpenSolaris to see what else is cooking.

The Storage Bits take
OpenSolaris is becoming the finest storage platform out there. Adding CIFS support to the kernal is a Big Deal: OpenSolaris will be industry’s first OSS universal storage platform. If your shop isn’t 100% Windows this is a win for you. Even if you are 100% Windows, the low-cost Sun Thumper storage server could be a win for you.

Furthermore, the days of costly low-volume, custom hardware arrays are drawing to a close. With the massive growth in data it’s not a moment too soon. We can’t afford to keep going as we have. With OpenSolaris we won’t have to.

Update: Disclosure: I worked in Sun’s storage group for 3 years in the ’90’s. I have no financial or other affiliation with Sun.

Comments welcome, as always. Yes, I know the Samba guys aren’t happy.

Don’t keep me hangin’ on!
Given their industry leading infrastructure costs Google is the one company that could really shake up the online storage business. Only 2 questions: will they? and when?

Pimping Gdrive for the Wall Street Journal
In this morning’s Wall Street Journal is an unsourced article on Google’s supposed Gdrive ambitions. That’s right, no one from Google was quoted. But hey, slow news day.

The opportunity
Google has the cheapest infrastructure of any portal (see Killing with kindness: death by big iron), so they are well-placed to kick some serious butt in online storage.

But will Google ever pull the trigger?

Where did the WSJ get their info?
“People familiar with the matter.” Then the article positions Google against Omnidrive, Box.net, Xdrive, and the free, beta-only Microsoft Windows Live Skydrive. But these aren’t the people to beat. The $50/year, all-you-can-eat Carbonite and Mozy - the latter just bought by the fierce and deep-pocketed EMC - are the benchmark.

If Google can’t beat them they shouldn’t even get out of bed. It will be another doomed Google offering that failed to meet market demand. Like Google Video, News and Finance. Even Gmail is a weak sister against Yahoo.

When - and if - it gets announced
Just look at the pricing. If they go with tiered pricing - so many gigabytes for so many dollars - they’re doomed. With the Google brand they might even get a bit of premium over Carbonite and Mozy, but not much.

The Storage Bits take
Google has too many engineers playing at marketing. If they want to build multiple revenue streams - beyond Internet advertising - they need to get serious about actually producing high-value products. Not every competitor is as inept as Microsoft and Yahoo.

Comments welcome, of course.

Most disk drives include a feature named SMART - Self-Monitoring, Analysis and Reporting Technology - intended to tell you if your drive is dying. Can you rely on it? Sadly, no. Here’s why.

What is SMART?
SMART is a protocol for passing information from a disk drive to the CPU. The protocol is part of the ATA and SCSI standards and is based on work by IBM, Seagate and others done in the ’90s. The vendors generously placed their work in the public domain.

The protocol mandates a consistent structure for presenting drive data, but the data that gets measured is up to the drive vendor. Typically, SMART will present information on

• head flying height
• spin-up time
• bad block count
• seek time
• drive calibration retries

and more. SMART looks at the trends in these and other measures to determine if the drive is headed for failure.

Does SMART work?
According to Google’s review of 100,000 drives, the answer is a qualified no. They found that enough drives failed without a SMART warning to make SMART useless for predicting drive life. But they also found that if SMART said there was a problem the drive was much more likely to fail.

So if SMART says you have a problem, you probably do. But if SMART says you don’t have a problem, you can’t trust it.

Why doesn’t SMART work better?
Drives are complex pieces of equipment with many failure modes. The drive vendors decide what parameters are measured and what is the failure threshold. Since roughly 40% of the drives returned to vendors are NTF - No Trouble Found - vendors set the thresholds to ignore piddling errors. They might catch more failing drives, but only at the cost of even more NTF drives.

Another issue is that many drive failures can’t be predicted. SMART mostly looks at mechanical trends, but disk drives are also electronic. A cracked capacitor, power surge or interface failure can kill a drive even if the data is still safely on the disk.

Finally, there are problems in storing and interpreting the SMART data. The data is stored in a small amount of RAM so the drive a) starts from scratch each time the drive is powered on and b) trends may be missed if the RAM fills up and is purged partway through an event.

The Storage Bits take
The drive vendors are doing the best they can creating larger and higher data rate drives. The intentions behind SMART are good, but its limitations mean that a “good” drive can go “bad” without warning.

The really smart answer: only regular backups can protect your data from sudden drive failure. Accept no substitutes.

Comments welcome, of course.

Watching a stuttering YouTube video is frustrating. With all the technology at our command why can’t we do better? Starting next year, we will.

Streaming video or herky-jerky video
The Internet is a packet-based network. Each packet has a source and destination address and often a sequence number that describes where the packet belongs.

The stuttering results when video frames are dropped out of the video streams. When 2 packets can take wildly different routes or encounter sudden congestion something has to give - and your video quality is it.

This is one reason the telcos want to charge extra to deliver streaming video: it needs special treatment. But what if it didn’t?

Raptor codes
Invented in 2000 by Berkeley math professor Michael Luby, Raptor codes are a class of Fountain codes. They have fast, lightweight decoding while maintaining video quality even if more than 20% of the frames are lost.

High quality video won’t need special treatment by the network. Raptor codes have been incorporated in several video standards. Digital Fountain is offering them in products for content distribution networks.

How it works
The math is way beyond me, but the basic ideas are simple enough.

The video stream is encoded with extra data - repair data - as it is packetized and shipped over the network. The receiving decoder can can detect and often correct errors based only on the extra repair data.

This error correction is called forward error correction (FEC) because no feedback or retransmission from the sender is required. The packets contain the extra data needed to reconstruct lost packets.

The really cool part
So won’t all this extra data clog up the Internet even more? Not at all.

When packets get lost the normal recovery mechanism is to ask for the packet to be retransmitted. An extra request and a retransmission. With FEC that overhead is eliminated.

The extra redundancy can reduce latency and bandwidth use since the retransmissions aren’t required. So we get high quality video at the same network load.

The Storage Bits take
The telco’s protection racket - nice little video stream you got there. Be a shame if something happened to it. - makes even less sense in a world with widespread FEC use. Telcos should focus on adding bandwidth as demand grows. Net neutrality is a great idea and has been for 160 years.

In a few years today’s YouTube videos will look as quaint as Buster Keaton’s silent masterpiece The General does to us. Kudos to Professor Luby and the Digital Fountain team for bringing real innovation to Internet TV.

Comments welcome, of course. If you love digging into serious math, download the PDF Raptor Codes by Amin Shokrollahi, Senior Member, IEEE, a peer-reviewed article published in the IEEE TRANSACTIONS ON INFORMATION THEORY in 2006.

For a lighter-weight intro with video examples, check out DF’s presentation at the fall Demo conference here.

Update: The first couple of commenters wondered what the big deal is. To put is simply, much more efficient distribution of video across lousy networks.

Here is the abstract from the original paper on digital fountain codes:

Abstract—The proliferation of applications that must reliably distribute bulk data to a large number of autonomous clients motivates the design of new multicast and broadcast protocols. We describe an ideal, fully scalable protocol for these applications that we call a digital fountain. A digital fountain allows any number of heterogeneous clients to acquire bulk data with optimal efficiency at times of their choosing. Moreover, no feedback channels are needed to ensure reliable delivery, even in the face of high loss rates.

We develop a protocol that closely approximates a digital fountain using a new class of erasure codes that for large block sizes are orders of magnitude faster than standard erasure codes. We provide performance measurements that demonstrate the feasibility of our approach and discuss the design, implementation and performance of an experimental system.

Here’s the PDF of A Digital Fountain Approach to Reliable Distribution of Bulk Data. This is a scholarly paper.

Has Google has built its own 10 Gigabit Ethernet switch - one far cheaper and power-efficient than existing switches - as it seeks to widen its infrastructure cost advantage over Yahoo, MSN and AOL?

Andrew Schmitt, an analyst at Nyquist Capital makes the case.

Through conversations with multiple carrier, equipment, and component industry sources we have confirmed that Google has designed, built, and deployed homebrewed 10GbE switches for providing server interconnect within their data centers.

Lean and mean
Why does Google care? They’ve already made major power and cost cuts in their server infrastructure: low-end PC designs; direct-connect disks velcro’d to caseless mobo’s; and cheap, unmanaged fast Ethernet switches tying local nodes together.

After cutting the big numbers it is time to cut the smaller ones. They avoid several power and money hungry features of costly enterprise switches. Twinax cabling for short runs. Ancient 850 nm optical interfaces and cabling for longer runs up to about 100 m. No equalization. Inexpensive switch chips from Broadcom.

Why 10 gigE?
The Google File System and Bigtable are designed to use cheap, slow networks. That’s why there aren’t any NAS or RAID arrays in the main Google infrastructure: too much network overhead. Not to mention the cost.

What Andrew doesn’t explore is why Google needs 10 gigE. Because it’s there? No. They need the scale out.

Scale-out heaven
Let’s say you had a warehouse with 300,000 Core Duo 2 servers and a rapidly growing workload. Each query slogs through 70-100 MB of data to find your Phoebe Price photos. That traffic is mostly local, but more is on the way.

Machine translation. Cell phone ads. Hybrid maps. Collaboration apps. Bigger social networks. And more.

The 10 gigE switches mean that dozens of gigE uplinks from cheap unmanaged switches can be aggregated on 10 gigE links, increasing the span of 2nd level switches and eliminating the need for a 3rd switch tier. This reduces power, cost and latency, at not much more cost than standard 2nd tier switches today.

The Storage Bits take
Google’s focus on power is paying dividends for the entire industry today. Their focus on cheap, high-speed networking will also drive 10 gigE down the price curve faster than Cisco and Juniper would like.

At the same time their infrastructure cost advantage will continue to grow. Bad news if you are trying to compete with Google.

Comments welcome, as always.