If I had to pick one word to describe 2007 it would be “emergence” - lots of things that were building up to change, changed. In politics George Bush got his mojo back: from stem cells and Iraq to his policies on Russia, climate change, and oil devaluation he’s become the lame duck that stomps like an elephant. Worldwide, China edged closer to chaos, Isreal’s wall worked for peace, and old Europe awoke to the demographic threat Mark Steyn has been talking about for a decade.

Amazing things happened in IT too, and just like the changes at the political level most of the big things that happened represented the emergence of hidden longer term trends into public view.

Scale is a big issue here - a simple 2007 top ten list doesn’t show how much more important this year’s inflection points were than last year’s. Think, therefore, in terms of something like the Richter scale for earthquakes - with this year’s top ten clustered near the high end.

That said, here’s my list:

1. Tied for first place among my top ten are two technology introductions. Sun got its CMT/SMP technologies established in the market - and Apple got the iPhone out.

   Neither technology is well understood in the marketplace. Thus people compare the iPhone to things like Blackberries but in reality the Blackberry is just the current stage in the evolution of pagers while the iPhone is what the Newton set out to be: the first in a class of hand held computers capable of handling the communications and memory functions traditionally associated with desktop computing. Wait a few years and it will morph into a full replacement for the ipod/laptop/cell phone combination, probably with plugin processor cards enabling special functions from health monitoring to wall projection and full voice input.

   Similarly most people think Sun’s CMT technologies are about going multi-core and so compare it to things like Intel’s quad core Xeons. This is wrong: CMT is much more about using threading and hardware parallelism to reduce the memory bottleneck for SMP multiprocessing than it is about putting multiple cores on a chip: thus an eight core T2 runs 64 concurrent threads, not 8 - and gives each thread full concurrent access to floating point, cryptology, and packet management hardware.

2. In third place on my list is a nearly invisible change: the popularity of outsourcing IT to corporate entities indistinguishable by personnel or behavior from large internal IT groups reversed itself. Time-sharing picked up, as did both IT re-patriation and, to a much smaller extent, serious technology and IT management change.
3. In fourth place is something that didn’t happen: IBM did not introduce off the shelf super computers consisting of a rack, a customer specified number of cell processors, and a “thumper” like storage server. It did get high gigahertz Power6 cores into the market, but not the software and I/O flow control gear needed to make use of them - thus producing 4.7Ghz PPC machines that run like their 2.5Ghz predecessors and pushing its entire change strategy back far enough for internal opposition to Cell to dig in and quite possibly force another “future systems” retrenchment.
4. Number five also consists of something that didn’t happen: Windows Vista. Instead, many people forced to pay for it as part of a new computer purchase installed previously licensed Windows/XP products and the Vista product set as a whole followed the proud tradition set by Windows 3.3 and Windows ME for end of life product iterations based on copying Apple breakthroughs from years earlier - and failed.

   No “code red” class wintel upgrade enforcer emerged this year. According to Microsoft’s fifth column this was because Microsoft’s code quality and recommended operational procedures have improved - but cynics, including me, say it’s because Vista failed to catch marketing fire and the server upgrades needed for the next great sales push aren’t out yet.

5. Sixth place goes to another iceberg just starting to break the surface: identity management, customer data confidentiality, and related information retention issues are becoming primary drivers for American corporate litigation proofing.

   Meanwhile patent litigation and greenmailing continued unabated and the democrats stalled tort reform in congress - for the seventh straight year since the current administration first floated it in early 2001.

6. Number seven is the change at Intel. 2007 allowed the company to catch up with AMD - by resurrecting an abandoned multi-core line and making it smaller, and therefore faster and more power efficient, than AMD can; by using its media clout to market a plan to copy AMD on I/O flow management by late next year as if it were already embedded in its processors; and, above all, by not doing anything as stupid as buying NVidia.
7. Number eight is the hole the gaming industry dug for itself. The WII proved an enormous success - but for the hardware, not the games software. Microsoft’s X360 sold well by anyone else’s standards but lost money for Microsoft - and the company continued to demonstrate its inability to transcend the x86 by writing boring games that run slowly on a three core, six thread, 3.2Ghz PPC. Sony’s PS3, meanwhile, is catching fire as compiler improvements coupled with the sheer power of the cell engine make its versions of the same basic games everyone else sells more attractive, and Sony’s selling power starts to establish more of the related home technologies in the market.
8. Number nine is that Linux marketing didn’t change even as the annointed enemies of all things right and Linux more or less evaporated in place - Torvalds continued to support Red Hat’s anti-sun marketing by ranting against Solaris, but showed his real agenda by not taking up Jonathan Schwartz’s invitation to discuss outstanding issues over a home cooked dinner; and IBM didn’t settle with SCO but left directors at its Novell proxy praying for an SCO victory on appeal because they’re otherwise going to face a clear fiduciary duty to take up the case against IBM.

9. And, number ten is the emerging discussion of programming language support for both the grid and SMP approaches to parallel computing. Three years ago nobody, outside of some academics and a few geekheads working on OpenMP type solutions, cared - now it’s front burner stuff at industry leaders including IBM and Sun, getting “airtime” in psuedo-academic journals like Communications of the ACM, and of interest to readers of blogs like this one.

Taken all in all, 2007 was a year of big changes coming to visibility - and its hard to see how 2008 will top it - but that’s Monday’s blog.

As most people know Linus Torvalds tore a rant out of Sun last week - here’s part of what he said:

Here’s a cynical prediction (but backed up by past behaviour of Sun):

- first off: they may be talking a lot more than they are or ever will be doing. How many announcements about Sun and Linux have you seen over the years? And how much of that has actually happened?

- They may like open source, but Linux _has_ hurt them in the marketplace. A lot.

They almost used to own the chip design market, and it took quite a long time before the big EDA vendors ported to Linux (and x86-64 in particular). But when they did, their chip design market just basically disappeared: sparc performance is so horribly bad (especially on a workstation kind of setup), that to do chip design on them is just idiotic. Which is not to say that there aren’t holdouts, but let’s face it, for a lot of things, Solaris is simply the wrong choice these days.

Ergo: they sure as hell don’t want to help Linux. Which is fine. Competition is good.

- So they want to use Linux resources (_especially_ drivers), but they do *not* want to give anything back (especially ZFS, which seems to be one of their very very few bright spots).

- Ergo: they’ll not be releasing ZFS and the other things that people are drooling about in a way that lets Linux use them on an equal footing. I can pretty much guarantee that. They don’t like competition on that level. They’d *much* rather take our drivers and _not_ give anythign back, or give back the stuff that doesn’t matter (like core Solaris: who are you kidding - Linux code is _better_).

This was part of a much longer term discussion of the changes being considered for the GPL and, of course, driven by a number of “environmental” factors including speculation about Sun’s “Project Indiana” and the Microsoft patent agreements so many Linux distributors are getting forced into.

The proximate cause, however, was this bit from comment by Ingo Molnar:

Secondly, ZFS might be interesting in theory, although our prior experience of having compatibly-licensed filesystems ported over to Linux has been pretty negative: XFS ended up being an integration nightmare - and that doesnt have to do anything with the qualities of XFS (it’s one of the cleanest Linux filesystems, if not the cleanest), the problem is that components within a kernel are very tightly integrated and rarely does it make sense to port over more than just drivers or maybe libraries. And that’s i guess what OpenSolaris lacks and which i suspect it is mostly interested in: lots of nice Linux drivers ;-) XFS, the largest Linux filesystem is 100K lines of code - and ZFS (i’ve never seen it) is very likely smaller than that. Linux drivers on the other hand, as of today, are _3.7 million_ lines of code and enable Linux to run on 99% of the hardware that is produced today. Guess which one has the larger strategic significance? ;-)

You’d think Molnar’s comment would make sense but, on net, it probably does not.

There’s no doubt, of course, that a lot of the people who’re unhappy with OpenSolaris cite issues related to the absence or inadequacy of particular device drivers - I personally, for example, run MacOS X on a laptop largely because it makes working with things like cameras and other people’s overhead projectors a lot easier than Solaris does.

Molnar assumes, however, that it’s easier to port and maintain a linux device driver for Solaris than it is to write and maintain a native one - and that’s generally not true. There are very fundamental differences in how devices are conceptualised that make porting non trivial; and, more importantly, Linux driver maintenance is affected by so many external factors -including kernel change, hardware change, code fads, and licensing - that such ports require nearly continual re-invention.

At the very grossest level what’s really going on with respect to drivers is a continuation of the object versus furball debate that fired off Torvald’s first changes to the Minix kernel -changes that traded design clarity for performance by adopting x86 interrupts as the primary means of initiating a process change and that still underlie the fundamental differences between a Solaris leaf driver accessing devfs and a Linux driver that doesn’t have to - but relies on unstable kernel and hardware structures for both efficiency and process isolation.

At the more immediate level what’s going on is that Linux is under threat with these guys feeling lots of pressure and toeing the party line by acting as if Sun is the enemy: but it’s not; and with up to one third of everything in the typical Linux distribution based on code donated by Sun they have to know it.

In truth a lot of the pressure they feel is coming from Sun’s success with OpenSolaris and the CDDL - but that’s driving positive change for the entire Unix community: Solaris, BSD, and Linux. Unfortunately, however, their refusal to understand who their friends are has the unhappy side effect of making things easier for predators like IBM and Red Hat while fracturing a community whose natural external enemies are ignorance and Microsoft - not Sun and the BSDs.

I was thinking about the future of Linux when it occurred to me that one path for its future can seen as a simple consequence of what we mean by “winning.” In other words, asking whether Linux will still be a winner in ten years leads first to the question of what we mean by “winner” and then to an answer about where Linux is going.

If by “winning” we mean making your software do what you want it to for your community of interest while seeing your ideas become widely influential, then the lessons of open source history are clear: winners either build in the academic tradition of humbly taking other people’s work forward or roll history back to some branching and then set off in whole new directions.

Look at the history of widely accepted and extra-ordinarily influential IT product sets like those under the Apache, Perl, or BSD labels, and the most striking thing they have in common is that the originators set out to solve specific problems, used what other people in academia and open source had done in new ways, and judged the choices they made along the way in terms of the problem rather than in terms of the market for the solution.

If we define “winning” in terms of making money, then we’re usually talking about people who monetise other people’s work, most often by telling customers both that their products are uniquely wonderful and, at the same time, really -wink, nudge- not a copy of a proven but more expensive commercial product they’re already familiar with.

Look at Microsoft’s history: from QDOS to SQL-Server and the new Vista interface, every major product has copied someone else’s earlier work and been offered to the public simultaneously as better than an existing product, as proven by users of an existing product, and as a way of grabbing the benefit offered by that existing product with less money and less learning.

Both of these forms of winning have legs, in the sense that I’m quite confident that twenty years from now somebody will be making money packaging second hand ideas for the uninformed, while academics and others will be advancing new ideas traceable to what’s in place now.

There is a third kind of winning: one based on the fullfillment of somebody’s political agenda and not directly related to either money or technology. Thus the GNU utilities are wildly successful despite being neither innovative nor second rate, largely because they express the Free Software Foundation’s political objectives without forcing users to accept those objectives.

Linux kernel history, however, isn’t that simple - Linux started as one thing, became another, and now appears to be avoiding some hard choices. Thus the kernel started as a classic academic effort to improve on another set of ideas, specifically in response to a disagreement about the value of using, or not using, x86 interrupts in the Minix kernel, but then got pushed forward in response to an unrelated political agenda by people who have since dropped it.

What happened was simple: the mid nineties mass media developed what I call “the Myth of Torvalds” to express their own political agenda: positioning Linus as the Robin Hood of software, single handled standing up to American multi-nationals to bring free computing to the common man. Great, except that IBM’s later success in deputizing most of the merry men made it difficult for even the most dedicated fantasist to continue the charade - and the vacuum left when the hot breath of media support disappeared meant that Linux failed its tipping point sometime in 2002/3 and is now in apparent decline relative to Windows, Solaris, and the BSDs.

That doesn’t mean it will go away anytime soon, of course - IBM could keep it commercially alive for decades, the momentum built up in north American and European open source and academic usage will not disappear quickly, and the political advantages to the use of Linux across Asia will probably continue to drive acceptance there. But what’s scary for Linux is this: the applications and GNU components we think of as Linux don’t depend on the Linux kernel - meaning that the only barriers to mass migration by Linux developers to Solaris or the BSDs are psychological.

Meanwhile today’s kernel development process doesn’t fit any of the categories describing long term winners: it’s not driven by the search for excellence, by greed, or by politics. Instead the key driver right now appears to be the SCO lawsuit with lots of effort still being dedicated to proving that Linux is somehow not Unix - and not only isn’t that characteristic of long term winners, but it’s creating barriers to growth by working against standardization across both time and distributions.

The quotation is from Tim Taylor, star of TV’s long defunct sitcom: Glimpses of Heidi, but suits my purpose here, which is to discuss our tendency to remember things in terms of what’s important to us rather in terms of what really happened.

A big part of the dominant wintel mythology, for example, is based on three beliefs, all of them wrong:

1. that the people behind Intel invented the microprocessor during the early seventies;

   In reality the first microprocessors were built years earlier Ray Holt and his colleagues at AiResearch for the F-14 flight controller. Their work ended up at American Microsysystems and the staff involved, when American decided there was no future in microprocessors (!), ended up founding the MC68XX line at Motorola.

2. that Microsoft’s first commercial OS offering was MS-DOS.

   It wasn’t: they paid Toronto’s HCR to put together Xenix and sold that for over a year before paying Tim Patterson at Seattle Computer products to put together the quick and dirty CP/M rip out that became first PC-DOS and then MS-DOS.

3. that the original PC introduced in 1981 represented the state of the art at the time.

   It didn’t. In fact it under-performed IBM’s own 5220(!) from 1971/2 and used an eight bit downgrade of Intel’s 16bit 8086 -a chip that itself so under performed Zilog’s Z series that only Exxon’s timely intervention saved Intel for IBM - and all at a time when quite a few companies were building Unix desktops around Motorola’s line of 16/32bit processors and co-processors.

The use of mythic history as an IT sales tool is not, of course, limited to wintel -quick: who started Red Hat Linux and where did he work at the time? if you think it was Bob Young and he ran a software distributor, you’re wrong: it was Marc Ewing and he worked as a developer for IBM.

Revisionism gets even more interesting when we let our natural inability to see backwards mislead us in terms of where we see ourselves going. People are starting to understand, for example, that most of today’s programming reflexes don’t port well to the hottest new hardware - IBM’s cell and Sun’s CMT/SMP machines. Great, but why does the future for programming look like it’s going backwards?

More precisely, where are the new ideas? Tim Bray, the widefinder project originator I mentioned yesterday, seems to think the future might look like Erlang - put together by Armstrong, Virding et al at the Ericsson Computer Science Laboratory during the mid to late eighties. I can’t see either LISP or APL becoming any more popular today than they were in the sixties, but both fit the new hardware paradigms like custom made gloves - and while I really like what I think I understand about Sun’s Fortress language, it too seems much more about delivering on old ideas than about creating a new intellectual framework for working with either SMP or grid style parallelism.

Of course, you can argue that old ideas rule: for example, much of the intellectual capital embedded in today’s latest and greatest enterprise resource management packages comes from work on game theory done by Morgenstern and von Neuman - in 1928.

It’s just nonsense to claim that there are no new ideas - but as a practical matter we seem to recognize the value of ideas only if they correspond to our, usually incorrect, understanding of what came before them. As a result a lot of old ideas look new - not because they are, but because we’ve lost track of what came before, and after, them - so I think the lesson in all this is that if you’ve got a new idea for 2008, your success in getting it accepted will largely depend on your ability to place it in an historical context acceptable to your audience.

And on that depressing note: Merry Christmas! (best -ofs until next Friday’s attempt to “top ten” 2007.)

Here’s a comment from frequent contributor Erik Engbrecht:

Until Tim Bray started his Widefinder discussions I was a big believer in Coolthreads. The principles behind the technology just make sense. But then I noticed how badly my Intel based laptop beat the T2 when running serial code. I knew it would - but the magnitude was rather shocking.

So now I’m a skeptic. Don’t get me wrong, I would probably buy one for most general enterprise computing workloads. I still think it’s a cool technology, and in the future I’m sure those cores will get faster while Intel will just keep on growing more cores. In the end I think Sun has it right, and I think for the most part Sun has it right right now. But at this point the performance characteristics of T2 seem to be somewhat enigmatic.

Here’s how Tim Bray introduces the Widefinder project on his own website:

In my Finding Things chapter of Beautiful Code, the first complete program is a little Ruby script that reads the ongoing Apache logfile and figures out which articles have been fetched the most. It’s a classic example of the culture, born in Awk, perfected in Perl, of getting useful work done by combining regular expressions and hash tables. I want to figure out how to write an equivalent program that runs fast on modern CPUs with low clock rates but many cores; this is the Wide Finder project.

…

It’s already obvious that Web-oriented frameworks in general, and Java EE in particular, run like a bat out of hell on processors like our Niagara boxes and especially the systems that will be built around the new T2.

But you know, there’s lots of interesting computing that doesn’t happen in Web frameworks. Like, for example, logfile processing. And since more and more computers are going to start looking more and more like our Niagara-family boxes, we’re going to need to figure out how to get this kind of basic unglamorous essential job to run fast on them.

Whatever it is we use to write the many-core-friendly Intergalactic P2P Adaptive Semantic Web4.0 killer apps of the future, well, it has to be a general-purpose programming language, because they always win. Which means it has to have a good way to read a logfile and compute the most popular fetches. You might argue that this program is kind of heavy on I/O and text processing. That’s correct; but so is computing in general.

With that in mind I’d like to propose a Christmas diversion: a narrow loser programming contest in which the goal is to write the ugliest, least efficient, most obfuscated code that will do the widefinder job: list the top ten most downloaded pages from an apache log.

The rules are simple: it has to work on Linux or Solaris with only open source software.

The test is his: process an apache “combined” log file to correctly list the top ten, and only the top ten, with the correct counts.

To enter your masterpiece: put the code, the results (including run time, file size, and test hardware information), and a comment or two in the talkbacks here.

Assuming we get some entries, I’ll create a reader poll to decide who “wins” -and I’ll personally send the winner absolutely nothing.

As a serious aside, here’s Tim’s own defense of the value of his project - from his results page:

Worth Doing · There is a steady drumbeat of commentary along the lines of “WTF” This is a trivial I/O-bound hack you could do on the command line, ridiculously inappropriate for investigating issues of parallelism.? To which I say “Horseshit”. It’s an extremely mainstream, boring, unglamorous file-processing batch job. There’s not the slightest whiff of IT Architecture or Social Networking or Service-Orientation about it. Well, that’s how lots of life is. And the Ruby version runs faster on my Mac laptop than on the mighty T5120. The T5120 is what most of the world’s future computers are going to look like! Houston, we have a problem, and the Wide Finder is a useful lens to examine it.

I believe I’ve been beating the drums on the software revolution required to make use of non linear technologies like CMT or Cell for longer than just about anybody else, and I certainly agree with what he says here.

On the other hand, you rarely encounter single jobs this simple in the real world. What’s more likely to happen is that you’ll accumulate requests for custom reports from the same data - in this case an apache log file. Before today’s multi-core machines most people tried to save cycles by processing the input file only once, but with CMT/SMP that doesn’t make sense any more.

Instead you go for the maintenance and clarity advantages that go with writing separate scripts to process the file once for each report - and then run them in parallel after copying the source file to /tmp :

#!/bin/sh
#crontab 30 6 * * 1-5 2>> usr/adm/custom_reports_log
#murph 12/16/07
#
cp /var/apache2/logs/access_log /tmp
/usr/murph/custom/tims_top10.pl &
/usr/murph/custom/jacks_refs.pl &
/usr/murph/custom/harrys_crosstab.pl &

Notice first that for this to work as fast as possible your T2 has to be properly set up - with lots of swap split across the two fastest internal disks and ZFS configured for all external storage - something I don’t think Bray’s test machine is. Secondly, the important thing here is that the T2 will do many of these in parallel and first gain on, and then pass Erik’s Lintel machine as the workload increases.

For the less serious purposes of the narrow loser contest, however.. I’m thinking PostScript - i.e.

% gscript -f my_program.ps image_log |lp -dqms

prints a labelled graph.

I (ahem, cough) “may not have time” - but I’m hoping you do and looking forward to any entries I get.

I read a blog, yesterday, by one Randall Hoven, titled “Kyoto Schmyoto” which started with this:

One would think that countries that committed to the Kyoto treaty are doing a better job of curtailing carbon emissions. One would also think that the United States, the only country that does not even intend to ratify, keeps on emitting carbon dioxide at growth levels much higher than those who signed.

And one would be wrong.

The Kyoto treaty was agreed upon in late 1997 and countries started signing and ratifying it in 1998. A list of countries and their carbon dioxide emissions due to consumption of fossil fuels is available from the U.S. government. If we look at that data and compare 2004 (latest year for which data is available) to 1997 (last year before the Kyoto treaty was signed), we find the following.

• Emissions worldwide increased 18.0%.
• Emissions from countries that signed the treaty increased 21.1%.
• Emissions from non-signers increased 10.0%.
• Emissions from the U.S. increased 6.6%.

Since that’s the exact opposite of what widely quoted experts, including some prominent Americans, are saying about U.S. performance I tried to confirm the numbers - and it turns out they’re about right. During that process, however, I was struck by the similarities between what most people think they know about climate change and the Kyoto treaties on the one side and what most people think they know about the Linux vs Windows debate on the other.

What’s going on in both cases is that what people believe has little to do with facts and everything to do with individual decisions to value conformance to perceived majority opinion over conformance to objective reality. On the climate side what this means is that otherwise sane scientists and science administrators willingly pander to the media bandwagon on human responsibility for climate change while knowing perfectly well that the parallel to effects visible on other planets in our solar system prove human causation a non issue.

On the IT side this means that some of my friends willingly continue their allegiance to Microsoft while knowing perfectly well that the Microsoft client-server architecture does not work and cannot be made to work while all three of the major Unix groups: the BSDs, Linux, and Solaris, offer alternatives that are both demonstrably effective and theoretically supported.

In that same vein I’ve suffered the frustrations - and I’m sure you have too - of talking to people who seem perfectly sensible in real life but repeatedly react to IT failure by repeating what they did to create it. That’s classical insanity, but I know one guy, a person I’m happy to call a friend, who has upgraded his personal Wintel gear roughly every 12 to 15 months since the early nineties, now has Vista on a brand new quad core of some kind, still loses Word and outlook files, isn’t doing anything today he couldn’t do with a MacAV in 1991/2, and has yet to understand that spending money will never make him the instant master of the information universe he yearns to be.

( After SP1, I think I’ll suggest he try the the upgrade to dual core and XP. :) )

One of the weird things about this is that he’s consulted me prior to each upgrade - with every evidence of sincerity. In each case, furthermore, I’ve given him the same advice: just get a Mac - and fourteen times now he’s agreed that the Mac’s better but decided against buying it because Apple’s just about to go under.

The behavior looks utterly irrational - and really is inexplicable unless of course, you look at the climate change issue for a parallel because it’s pretty clear there that the movement leaders are in it for reasons having nothing to do with climate - and most of the followers are desperate to believe for political reasons having nothing to do with climate either. Apply that to the Windows vs Linux debate and you can see why the fact that Unix does more than Windows with one third the lines of code and less than one percent of the failure rate simply doesn’t matter.

So, bottom line, what’s it mean? That arguing Unix vs Windows on the facts is silly - and perhaps even counter-productive because the factual argument favoring any Unix over Windows is so overwhelming that the media, and most customers, have simply tuned out in favor of continuing uninformed opinions they’ve been led to believe lots of other people hold too.

One of the saddest, and funniest, real life stories I know involves a big dollar project, undertaken by a major international programming house under contract to a Canadian telco, to develop a line printer facility for NCR’s System VR4. It’s hard to imagine, but after this project was reviewed, funded, contracted, and started, it actually took a turn for the worse - and indeed by the time we got asked to adjudicate contractor cost over runs the usual finger pointing over the project’s failure was in full flame.

I was reminded of this a couple of months ago when some people I know got all excited about a great new freeware product they were going to launch for Solaris - a script that would let a sysadmin reliably turn off all unnecessary external network connectivity with one command.

And then yesterday I got an email pitching a data backup and recovery solution for both Solaris and Linux - a product freshly ported from Windows by “the leader in enterprise data storage solutions.” Apparently it backs up files, restores files, rejoices in a (Windows Vista required) GUI client, and “replaces the ugliness of tar” with an easy to use integrated “data vaulting solution.”

My first reaction was to think that if I were citing ugly I’d go with cpio - and that tar isn’t compatible between Linux and Solaris anyway (gtar is)- but the more interesting thing is that products like this exploit the customer’s ignorance in much the same way Linux desktop virus checkers do.

What’s most interesting about it, however, is that the backup strategies this product appeals to no longer apply to Solaris, soon won’t apply to the BSDs, and will, sooner or later, lose relevance to Linux as well.

What’s going on is that the standard process for enterprise Unix backups is really two rather unrelated processes.

One is the data processing derived disaster recovery backup expected by (data processing trained) auditors: you make daily tapes, send them off site for safe-keeping, update the colorful powerpoints defining your recovery plan every time you upgrade Powerpoint, and hope you never, ever, have to use any of it because the plans never survive the first senior IT manager to arrive on site and some critical chunk of code or data will prove unrecoverable anyway because it either didn’t get written to tape or depends on something that wasn’t written to tape or simply because the tape is unreadable.

The other one is, of course, the real one. Typically it provides for some form of processing or data continuity but is more oriented toward the kind of recovery request sysadmins face every day: some user deleted something - or ran his test program against the live data - and you could roll back the clock, please?

Until recently the best way to do this was to copy everything on the production systems to a RAID set maintained solely as a backup and then back that up to tape or another remote machine during regular working hours. This takes more effort than a commercial solution to get working because you need to customize the copy process for every application or environment you’re responsible for, but makes handling recovery requests so trivial that you can practically script them by user name, gives you a quick recovery capability if your data center does blow up, and makes it easy to find out what happened when the boss’s pet idiot gets root access to your most critical database server just long enough to emit an “# chmod 0000 /etc/init.d ”

But technology marches on, and prices continue to fall. For about the same money you’d pay for a good commercial backup and recovery package you can implement a ZFS “thumper” as your general backup server - and not taking a close look at the ZFS time machine? that’s just about as smart as expressing your disapproval of an executive decision to buy Unix by paying people who don’t know what a model script is a couple of hundred thousand bucks to add a $9,000 Sperry line printer to a machine running AT&T System VR4.

/usr/sfw/bin/gtar: /dev/rmt/0: Cannot read: I/O error
/usr/sfw/bin/gtar: /dev/rmt/0: Cannot read: I/O error
/usr/sfw/bin/gtar: Too many errors, quitting
/usr/sfw/bin/gtar: Error is not recoverable: exiting now

That’s frightening, but simple minded compared to a very different kind of recovery problem that’s becoming increasingly important in academic and other research oriented service areas. What’s going on is that researchers are coming under increasing financial and career pressure to produce either commercial success or grants at just the same time that changes in science are making simulation both riskier and more important - and some of them are finding ways to make their problems ours.

Consider these two lines:

602 192304:45./// 2512213:55/ 204096414:45/ 2512215:28/ 784608414:07/ 45122304:44/ 11215:29/ 145122305:10/ 756322812:09/ 281922304:44/ 25122614:15/ 25122304:45/ 11215:29/ 195122306:04/ 5220482812:09/ 94258752514:37/ 210242812:07/ 45122304:45/ 7988514:31/ 25122306:08/ 35122810:37/ 411024214:03/ 4510242811:00/ 6512215:29/ 466 25122305:101394/ 1302304:47..///0: 1272304:47..///0: 172304:54/0 1102304:54/0 1112305:55./// 1252304:54..///0: 1282304:51..///0: 35122304:54/ 1312304:47..///0: 1302304:47..///0: 35122304:54/ 1332304:54..///0: 1392304:54..///0: 25122304:54/ 1482304:54..//1,0/7/0,0/0,30: 1482304:540..//1,0/7/0,0/0,30: 25122304:54/ 25122307:23/ 1292304:54..///0: 1542304:540..//1,0/7/0,0/0,378:0 1292304:47..///0: 135215:29//1,0/,6413:640 182304:540/640 25122304:54/ 1312304:54..///0: 2528514:37/ 142306:0700 1482306:070..//1,0/7/0,0/0,30: 1482306:070..//1,0/7/0,0/0,30: 192306:0700 1302304:54..///0:

Now imagine first that the file this came from has slightly over 328 million lines all of which look pretty much like this - and secondly that the guy who owns the data looks at you, the departmental sysadmin, while telling your boss that someone altered the file to sabotage his research.

If this file were real it would represent the outcome of about 3,174,336,000 AMD 2.6Ghz CPU seconds and nearly seven months spent waiting for a two rack grid with 176 total cores to produce a result. Researchers, of course, don’t care about CPU seconds or hardware limitations -but they do care about waiting time and care even more if that waiting time either lets a competitor publish first or turns out to have been wasted because of an error in the setup or running code.

Combine frustration with outside financial pressure and what you get is a recipe for the expression of paranoia - hence our entirely hypothetical researcher’s assertion that the file had been sabotaged and consequent threats of both legal and direct action against the guy responsible for running the hardware.

Since I’m making this up as I go along I can imagine that this particular situation was resolved when the system logs showed that some unsung hero in the University’s central IT department had taken it upon himself to update every Solaris box on campus through an automated shutdown, patch, and reboot process running every thirty days -thereby surfacing both an IT control hidden in purchasing and a synchronization error in the panic shutdown, checkpointing, and
restart code managing the application.

In the more common case, however, the only thing between you and getting fired in disgrace will be your ability to document the stringent application of control procedures designed to safeguard both data and processes - procedures that, if actually applied, would just about cripple the research productivity you’re supposed to facilitate.

So what can you do?

There are some things you have to do - for example using cryptology and Solaris containers or similar means to ensure user privacy and denying all outsiders, particularly central IT people if you’re burdened with those, access to the machine during critical periods.

The more general solution, however, is duplication - convince management that any serious research computing is at risk and that the only known way to combine relationship based researcher access with retroactively demonstrable processing integrity is to silently copy all work to another system run by somebody else.

If you demonstrably have no access to the other guy’s system and it produces the same results, the burden of proof will shift to the accuser - meaning that, practically speaking, you’re off the hook, and so is your department chairman.

To implement this, take advantage of two things: continuously falling hardware costs and continuously improving software. If, for example, you work in biotechnology and you get along with the sysadmin over in chemistry you’ve got the basis for a deal - because he’s got the same problem.

Propose that new hardware be duplicated at both sites, use one set of containers to achieve verifiable run-time isolation, and use another pair to swap data and programs. When paranoia erupts and one of you is facing accusations of leaking, distorting, destroying, or otherwise interfering in some user’s march to the Nobel -you’ve got backup: and it’s not a tape, it’s a person who can testify that the same data with the same programs on similar hardware produced the same results.

Just make sure only you and the department heads know - or the researchers will be signing agreements promising faithfully not to whine at you if only they can, just this once, please, have access to both halves of the facility, because, you know, it’s an emergency, and you can trust them, right?

Since this week has been about green, it’s time to add some Christmass contrast: blushing red. Here are my key 2007 IT predictions from December 28th, 2006

1. Vista will make billions for Microsoft - driven by praise from the same people who hated the MacOS X interface when Microsoft didn’t sell it;
2. Itanium will continue on life support while Compaq, operating as HP, negotiates a way out with Intel;
3. By the end of the year, the super computer listings will be entirely dominated by products built using IBM’s cell processor -and the business applications performance benchmarks will be equally dominated by Sun’s second generation CMT/SMP technologies.

4. By the end of the year the OpenSolaris community will be widely recognized as larger and more active than the Linux community -and every competing OS developer community except Microsoft’s will have copied the key ideas including its organizational structure, the core provisions in the community development license, and Solaris specific technologies including ZFS and Dtrace.

On the positive side Vista is, despite its predicted relative failure making billions, Itanium is just about dead, Sun’s CMT stuff owns every business benchmark in its weight class; and just about everybody is copying Sun’s community strategies and technology.

But what happened with the supercomputer listings? Two things: first, the duration of the funding process for the most recent entries means that they implement ideas and predictions from 2003/4, not 2005/6. Second IBM appears to be well into an internal directional conflict on Cell with traditionalists hanging their hats on software acceptability - and the lab people strengthening their case by (unintentionally) lagging the hardware.

I should have known about the first of these - and not made the prediction. Ooops.

The second is more interesting because the history of companies other than IBM which fail to adapt to advances in their own premier hardware lines is dismal. DEC, confronted with Alpha, choose to continue the VMS focus over Unix and died. SGI, facing a MIPS processor redesign wimped in favor of Intel - and made itself irrelevant to the market. HP, confronted with next generation PA RISC designs, tried to partner with Microsoft and Intel instead, and died - today’s “HP” is just Compaq operating under the HP name with essentially all of the pre-merger shareholder value in HP gone.

IBM survived this once, back in 1970/1, when it put its future systems technologies on hold in order to placate its 360 customer base. Back then it took eight years for competition to ramp up to the point of becoming enough of a threat that future systems proponents within IBM were able to leverage its release as the System 38 - still outside the mainstream and still the best single product IBM has ever made. Today the company is more diversified, and only has one real competitor in high end hardware, so they can probably muddle through regardless of the decision they eventually come to - assuming, of course, that the software guys get their act together enough to let IBM shift Power8 further toward micro-scale grid computing.

Nevertheless I don’t think there’s anybody out there thinking super-computer and not looking at cell, so I’ll repeat the prediction: cell will come to completely dominate super computer listings - but this time I’ll add the CYA proviso that it will take until super-computing proposals started in 2006/7 yield working systems.

I was wrong on part of the fourth one too. The OpenSolaris community is not widely recognized as larger and more dynamic than the Linux one - and I should have known that too.

What’s going on is odd: Solaris for x86 is almost a stealth movement among developers - nearly everybody’s playing, hardly anyone’s talking. I don’t know why - it’s like they’re waiting for someone else to move first, some kind of magical tipping point perhaps, where it becomes ok to say you’re doing this?

I also made some predictions I described as “less obvious”. Some of these look okay: identity information management continues to be a disaster waiting for the legal axe to drop. My guess then: “that a bad situation will simply continue to get worse during 2007″ applies to 2008 too - as do my comments then about cheering being the main point for acronyms like “Web 2.0″ and “SOA.”

However, I also said that:

“2007 will see the beginnings of true structural convergence - meaning web based delivery of materials selected, controlled, and presented in traditional ways. Look, for example, for someone to offer a traditionally hosted “best of the web” video series comparable to TV’s original “Real People” series, but presented in TV, multicast, and custom packaging.

Ooops. I was wrong about the appearance of a “best of web” show this year, but structural convergence has clearly started - lots of newspapers, for example, are combining traditional editorial pages with on-line blogs and comments solicitation, and the big studios are all talking about responses to the new markets defined by products like the ipod video and iphone.

Overall, therefore, I’m embarrassed about not thinking through the implications the budget process has for supercomputing, for jumping the gun on public excitement about Solaris, and for not seeing that the big media players would rather drag their feet than change business processes - but, overall, I could just about live with my 2007 predictions - for 2008 :)

The big difference between a Sun Ray and a thin client is that the software on the Sun Ray does not interact with the application - meaning that someone with unrestricted physical access to the device can destroy its functionality, but not affect application integrity.

In reality this is most valuable with respect to the military concept of audit assurance and operating security: protection against unauthorized use, deliberate sabotage, and information manipulation by enemy agents. Since civilian companies rarely care about these kinds of threats the most valuable advantages offered them by Sun Ray are the same ones delivered by older smart displays like NCD’s 1980s X-terminal line: performance; reliability; cool, silent, low power operation; location and application independence; and, complete freedom from desktop viruses or other PC style “security” threats.

With that distinction between thin clients and Sun Ray in mind I’d like to challenge readers to propose a business or application demonstrating the clear superiority of the desktop architecture of your choice - Windows, Linux, MacOS X, thin clients, or Sun Rays.

The things your project has to be or achieve include:

1. comparability: the basis for comparisons is the best the other guys can do and, by default, everything will get compared to Windows client-server: the latest Microsoft software deployed to maximum advantage on the latest Intel hardware.
2. scaleablity: from a twenty-five user pre-funding demonstration to international visibility.
3. clarity: clear demonstration of the value of use - no values based ambiguities like “I can find more and cheaper people to run it…” or “it’s more secure so I don’t care if it’s slower..”

4. generality: results must have decision consequences across a broad spectrum of business uses.

I don’t have a million bucks to give away, but lets pretend that we know someone who promises to provide up to a million for research and prototyping for the right proposal and will, if that goes well, introduce you to a venture capital firm capable of providing a couple of senior advisors and the next hundred million.

Some notes:

One: scalability combines with comparability to highlight an important clarity issue here. The demonstration project has to be unambiguous about performance and value - and where ambiguity exists, that ambiguity has to be resolvable through demonstration at and above the minimum twenty-five concurrent user scale.

For example, a user on a quad core Xeon workstation running Windows can do a keyword search among 5,000 locally stored emails in significantly less time than his counterpart using a Sun Ray on a T5220 - however twenty-five Sun Ray users on that same machine will all get their results well before twenty-five Windows users sharing a single Exchange server.

Two: clarity includes visibility. For example, there’s no question Sun Ray shines in security environments - but good ones are largely invisible to others. Whatever application or business you suggest, the architecture you propose has to have unique positive value obvious to third parties -and you need to specify just how the benefits and visibility are achieved.

In addition there’s an important do-ability issue associated with clarity. Suggest a software development effort aimed at meeting the daily needs of major law enforcement agencies via smart computing and you probably have a winner on cost, technology, user satisfaction, reliability, and visibility in the IT and general business communities - but you founder on commercial do-ability because it took these guys 75 years to switch from revolvers to automatics and they won’t adapt to IT change any faster.

Note that a credible project is, I think, possible for each technology. For example, if I were to propose a demonstration business for Sun Ray my project would focus on information support within a national or international restaurant franchise catering to frequent travelers.

Good luck .. and if you want your entry to be treated as confidential, put a summary comment that doesn’t give away the store (like mine, above) here and send your entry to me directly - (winface, murph, dot.com)