Does Your System Accurately Model Its Environment?

Today’s lesson for hardware people, software people, architects, and mad scientists is brought to you by Ray Bradbury in the form of a short story called There Will Come Soft Rains.  I recommend you follow this link and read a copy of the story, as it’s only a few pages long, and well done (but I already said it was Ray Bradbury, didn’t I?).  The story, written in 1950, takes place in 2026 in a very modern, fully automated house, with all sorts of computers keeping its occupants fed, cleaned, entertained, and on schedule for work, school, and play.

Only there are no more occupants.  Because there was a nuclear war, and they’re all ashes.  The house (the last house standing), in it’s quite-less-than-infinite wisdom, has not caught on to this.  So it prepares eggs and pancakes for breakfast, and provides entertainment in the nursery, and defends itself from the few birds and animals that may approach it, and fleets of mechanical robots diligently clean up any dirt that gets in the house and dispose of the uneaten food.  In short, this system has insufficient connection to the world it interfaces with, and too few checks on expected responses, to know that something is horribly wrong.

All natural and man-made systems make assumptions.  One cannot know absolutely everything about one’s environment at all times.  Often these assumptions are good ones, but sometimes they’re more dodgy.  All of them, without exception, are compromises.  This is even true of our own bodies.  We crave salty foods, because back when the first few models came out, salt was hard to come by, so no sensor was included to to warn of too much salt.   I won’t even start on how our blood sugar levels are maintained by insulin, which can make us crave even more sugar as soon as the first wave of sugar high starts wearing off, because we have too much insulin.  I won’t even THINK of getting into how our bodies were designed, or by what.  That’s flamebait of the highest order, and not really relevant.

Examples

When we design [mechanical, electrical, hardware, software] systems, we are usually in control over those compromises.  If we need to make sure that a door is closed before a machine starts working, we can choose to put on one interlock switch or two, so if the door is warped containment is more assured.  We can iteratively analyze the weakest link in the chain looking for things we can do that pass the probability vs consequences vs cost analysis.  But we better make those decisions well-informed, and with an accurate model of the environment.  We better look for the edge cases, and either cover them, or make an informed decision not to.  And ideally, make note that we’ve done so, in case our environment changes and we need to know what past decisions were based on.  Like it’s 1997 and we might want to think about what happens to the dates in our system in three years.

I could go over a litany of cases where bad assumptions or an insufficient view of the environment have caused major problems, like Three Mile Island, the Therac-25, and the Challenger shuttle, but we all know these stories.  Yet we still develop websites that allow SQL injection, use integers when we should be using longs, use not-nearly-random-enough number generators, SysAdmins smile as server backups fail silently for months, and hardware that doesn’t account for changes in dimension and conductivity as the temperature changes.

Why does this happen?

It would be easy to blame the evil Pointy Haired Bosses and the Bean Counters.  They are often the problem.  But I have seen too many cases caused by lack of experience, lack of imagination, or plain old apathy,  on the part of the engineer. That last one is the one that concerns me the most.  Ignorance can be fixed a lot easier than apathy.

Maybe it’s a side effect of the designer being so far removed from what’s happening at the low level.  Fancy CAD programs that don’t show how a little vibration will make this bolt bang into that plate unless you happen to swing the perspective to the right angle.  Circuit routers that won’t show you that even though each power resistor should be able to lose enough heat, putting eight of them right next to each other will heat up the whole area too hot.   IDEs that don’t show the software developer that the object’s attribute is being masked by the local variable of the same name, since that’s perfectly legal.

What can we do?

I am a huge fan of automation to reduce careless errors, required memorization, and job security through obscurity.  But if we rely too much on automation, we forget that these tools don’t know everything about the operating environment and can make bad assumptions, just like us dumb old humans.  It is still the job of the human to make sure all the cases are covered, no matter how much of the process is automated.  The software won’t get fired, you will.  Dig?

  • First and foremost, we have to give a damn about doing things right, to the extent that can be commercially justified.  And when not justified, we need to put a sticky note somewhere recognizing that we’ve made the choice to not worry about a particular condition.
  • Spend a little time playing your system’s worst enemy.  Be the Devil’s advocate and think about what could go worng; what could end up borken.
  • Add more brains.  One of the things that I love about Agile software methods is that they promote shared ownership (and ideally, shared creation) of code.  Not only does that result in more people that know the code, but you’ve got more people thinking about what can happen when the beast is unleashed.  Even a much more junior developer can be a big help in this regard, because we all come from different backgrounds and experiences, and that junior developer might have been exposed to a situation the more senior developer hadn’t been.

Side note

The title of the story is a reference to a poem by Sara_Teasdale of the same name.

There will come soft rains and the smell of the ground,
And swallows circling with their shimmering sound;

And frogs in the pool singing at night,
And wild plum trees in tremulous white;

Robins will wear their feathery fire,
Whistling their whims on a low fence-wire;

And not one will know of the war, not one
Will care at last when it is done.

Not one would mind, neither bird nor tree,
If mankind perished utterly;

And Spring herself when she woke at dawn
Would scarcely know that we were gone.

She was not voted most likely to become a cheerleader in High School.  She eventually committed suicide.

Share

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.