Having spent forty years in R&D and production I’ve experienced my share of unexpected and undesired outcomes of projects, products, processes, and tasks. I’ve also been in a number of different environments where these undesired outcomes were treated very differently. I even had one boss early in my career chide me for being complacent about a successful test of a new design. He got on my case about design margins on a couple of the critical components and told me to “go break it.” Turns out he was right. Subsequent testing proved that there was not enough margin in the design. In contrast, and counter to that attitude, I had one corporate VP at another company admonish my team for a rocket test firing in which we ejected the nozzle before completion of the firing. He couldn’t accept that this was a test to examine the margins on a new design. We were attempting to determine how much material we could shave off to reduce weight, and we were operating in a regime beyond the resolution of our computer models. With customer concurrence, we had decided to conduct a test. The customer fully understood that the result was not a failure. “This company will not accept failures!” this VP proclaimed proudly at a monthly program review meeting. He was only concerned about his perception of the reputation of the company even though the customer had signed off on the test. As I indicated, our customer was perfectly satisfied with the result. The VP wasn’t. Or maybe his ego wasn’t. This was R&D and this was a test designed to push the margin. If this had been a production motor I would have been on his side about declaring it a failure, but this was R&D. Of course, this was the same VP who said he didn’t believe in his corporation investing in R&D. Instead, he believed if the company needed a technology he could just buy it. Let other companies deal with R&D failures. Well, that’s a subject for another blog.

When you get into the production arena, attitudes toward failures change. Production failures can have significant long term effects on the bottom line, customer relationships, and company reputation, depending on the industry and circumstances. Acceptance tests and inspections are conducted to ensure the quality of the unit going out the door to a customer. How does your company treat an acceptance test failure? Or an out of tolerance dimension?

I’ve worked at companies that operate on the opposite ends of the spectrum when it comes to quality and treatment of failed units. I worked for an automobile parts manufacturer delivering a $.25 spray nozzle assembled from two pressed-fit injection-molded plastic parts. The units were 100% tested. Nozzles exhibiting spray considered out of tolerance were discarded. Statistics were kept of the number of failures. No paperwork was generated for a failure. The part was just discarded. The company understood the variability in the manufacturing and assembly processes and had calculated what percentage it needed to make its required margins. Only when the rejection percentage begin to creep up (they counted the discarded units), endangering their margins would they begin an investigation. They also understood that the design was simple enough that it was highly unlikely there would be any long term latent defects hidden in the nozzle.

Contrast this to the rocket launch industry. I’ve worked for both a components supplier and for a launch integrator. The industry’s slogan concerning failures can be summed up as “one failure is a trend.” Failure during acceptance testing was seen as an indication of a potential latent failure in units, even those that passed, that could have an impact during a launch or satellite/payload operation. When you consider that the value of the payload, rocket and launch cost is typically in the hundreds of millions of dollars or more, you understand this philosophy. You only get one chance with a launch and there are no repair facilities in orbit or deep space. So the launch and satellite industry has, in general, accepted this manifesto of “one failure is a trend.” Other industries, such as healthcare, either have or are adopting similar attitudes toward failure because of the potential cost and damage impact of a systemic failure. On the other hand, I also recognize new entrepreneur-led space launch companies like Space-X are trying lower the costs of launch; how that will change the launch industry remains to be seen..

So what are the implications of the “one failure is a trend rule?” Essentially a root cause investigation must be conducted for each failure. There are many methods of performing root cause analysis (RCA) including the Five Whys, Fault Trees, “Fishbone”, and Kepner-Tregoe. It really doesn’t matter which process you use, as long as you work through the layers of design, manufacturing, human influence, etc., like peeling back an onion. The one thing you don’t want to do is take shortcuts, or jump to conclusions. I’ve been on too many of these exercises where part-way through everyone “knew the answer,” only to find, once every box was checked. that something else, often a seemingly innocuous something that no one suspected, was the cause. This is why root-cause analysis is not cheap, because it has to be comprehensive and complete.

Am I advocating using RCA in every case of a production or process failure? No, of course not. The method used on that automobile component worked for them. For the rocket launch industry, they’ve decided it’s a case of “pay something now, or pay much more later.” A company has to weigh the costs vs. the consequences and then decide how it will treat failures. If you determine an RCA is necessary, whatever method you choose, finger pointing should not be part of the process. It doesn’t mean that the consequences of a failure shouldn’t be addressed if it involves personnel. It means that everyone participating in the process understands that this is being undertaken to find and correct the cause of a failure, not to blame someone. The culture of a company will determine how this plays out.

If you have a failure and decide RCA is required, and that you need some help, Rocket Science Technologies can provide assistance. We offer a free hour of consultation with initial inquiries that may help you decide whether you need to proceed with RCA and what method suits you the best. There are no one size fits all answers but there are no shortcuts either. Rocket science involves the science of getting the details right and that is our goal in helping you.