Having spent forty years in R&D and production I’ve experienced my share of unexpected and undesired outcomes of projects, products, processes, and tasks. I’ve also been in a number of different environments where these undesired outcomes were treated very differently. I even had one boss early in my career chide me for being complacent about a successful test of a new design. He got on my case about design margins on a couple of the critical components and told me to “go break it.” Turns out he was right. Subsequent testing proved that there was not enough margin in the design. In contrast, and counter to that attitude, I had one corporate VP at another company admonish my team for a rocket test firing in which we ejected the nozzle before completion of the firing. He couldn’t accept that this was a test to examine the margins on a new design. We were attempting to determine how much material we could shave off to reduce weight, and we were operating in a regime beyond the resolution of our computer models. With customer concurrence, we had decided to conduct a test. The customer fully understood that the result was not a failure. “This company will not accept failures!” this VP proclaimed proudly at a monthly program review meeting. He was only concerned about his perception of the reputation of the company even though the customer had signed off on the test. As I indicated, our customer was perfectly satisfied with the result. The VP wasn’t. Or maybe his ego wasn’t. This was R&D and this was a test designed to push the margin. If this had been a production motor I would have been on his side about declaring it a failure, but this was R&D. Of course, this was the same VP who said he didn’t believe in his corporation investing in R&D. Instead, he believed if the company needed a technology he could just buy it. Let other companies deal with R&D failures. Well, that’s a subject for another blog.
When you get into the production arena, attitudes toward failures change. Production failures can have significant long term effects on the bottom line, customer relationships, and company reputation, depending on the industry and circumstances. Acceptance tests and inspections are conducted to ensure the quality of the unit going out the door to a customer. How does your company treat an acceptance test failure? Or an out of tolerance dimension?
I’ve worked at companies that operate on the opposite ends of the spectrum when it comes to quality and treatment of failed units. I worked for an automobile parts manufacturer delivering a $.25 spray nozzle assembled from two pressed-fit injection-molded plastic parts. The units were 100% tested. Nozzles exhibiting spray considered out of tolerance were discarded. Statistics were kept of the number of failures. No paperwork was generated for a failure. The part was just discarded. The company understood the variability in the manufacturing and assembly processes and had calculated what percentage it needed to make its required margins. Only when the rejection percentage begin to creep up (they counted the discarded units), endangering their margins would they begin an investigation. They also understood that the design was simple enough that it was highly unlikely there would be any long term latent defects hidden in the nozzle.
Contrast this to the rocket launch industry. I’ve worked for both a components supplier and for a launch integrator. The industry’s slogan concerning failures can be summed up as “one failure is a trend.” Failure during acceptance testing was seen as an indication of a potential latent failure in units, even those that passed, that could have an impact during a launch or satellite/payload operation. When you consider that the value of the payload, rocket and launch cost is typically in the hundreds of millions of dollars or more, you understand this philosophy. You only get one chance with a launch and there are no repair facilities in orbit or deep space. So the launch and satellite industry has, in general, accepted this manifesto of “one failure is a trend.” Other industries, such as healthcare, either have or are adopting similar attitudes toward failure because of the potential cost and damage impact of a systemic failure. On the other hand, I also recognize new entrepreneur-led space launch companies like Space-X are trying lower the costs of launch; how that will change the launch industry remains to be seen..
So what are the implications of the “one failure is a trend rule?” Essentially a root cause investigation must be conducted for each failure. There are many methods of performing root cause analysis (RCA) including the Five Whys, Fault Trees, “Fishbone”, and Kepner-Tregoe. It really doesn’t matter which process you use, as long as you work through the layers of design, manufacturing, human influence, etc., like peeling back an onion. The one thing you don’t want to do is take shortcuts, or jump to conclusions. I’ve been on too many of these exercises where part-way through everyone “knew the answer,” only to find, once every box was checked. that something else, often a seemingly innocuous something that no one suspected, was the cause. This is why root-cause analysis is not cheap, because it has to be comprehensive and complete.
Am I advocating using RCA in every case of a production or process failure? No, of course not. The method used on that automobile component worked for them. For the rocket launch industry, they’ve decided it’s a case of “pay something now, or pay much more later.” A company has to weigh the costs vs. the consequences and then decide how it will treat failures. If you determine an RCA is necessary, whatever method you choose, finger pointing should not be part of the process. It doesn’t mean that the consequences of a failure shouldn’t be addressed if it involves personnel. It means that everyone participating in the process understands that this is being undertaken to find and correct the cause of a failure, not to blame someone. The culture of a company will determine how this plays out.
If you have a failure and decide RCA is required, and that you need some help, Rocket Science Technologies can provide assistance. We offer a free hour of consultation with initial inquiries that may help you decide whether you need to proceed with RCA and what method suits you the best. There are no one size fits all answers but there are no shortcuts either. Rocket science involves the science of getting the details right and that is our goal in helping you.
“This is not rocket science…” How many times have you heard that expression? In general, that statement is used to indicate that whatever you’re doing is not overly complex. It’s a tribute to the perceived complexity of rocket science. But just what is rocket science? Is it some arcane form of engineering that doesn’t relate to the things done in the commercial world? Or is there more to it? And, more importantly, can rocket science be relevant in today’s fast-paced market?
Dictionary.com’s first definition of rocket science is “rocketry” (English teachers used to scream at me for using different forms of the word in the definition but dictionaries seem to get away with it). Rocketry, in turn, is defined as “the science of rocket design, development, and flight.” The website’s second definition of rocket science is “something requiring great intelligence, especially mathematical ability.” So, on the surface, it appears rocket science is just that, the science of building and launching rockets with a nod toward things being complicated. Neither of those definitions satisfies me. Based on my experience in the industry, I believe they are incomplete. Only when you get into the nitty-gritty of a space launch does the essence of rocket science become clear. Rocket science is all about getting the details right.
With a space launch there are no second chances. There are no do-overs. If the launch fails, that’s it. A billion dollars may end up in the ocean, or scattered in pieces around the launch pad, or in a useless orbit around the Earth. No second chances. Once in space you can’t bring your malfunctioning satellite or probe into a local garage for repairs. You build in redundancies when you can and work to reduce the risk as much as you can.
The launch vehicle and its payload combined have hundreds of thousands of parts, subassemblies, and assemblies that must all work and function together for success to occur. A system with a million parts and 99.99% reliability can still exhibit a hundred malfunctions, of which any one may lead to a catastrophic failure. So the emphasis in the commercial space launch world is on the details. You have to get them all right. So when it comes down to it, rocket science is really the science of managing millions of details, while also working to bring operational risk down.
How does that relate to you? Your project of replacing a machine on your assembly line, or developing a new drug, or testing your electronics package certainly doesn’t involve millions of details. True, but it still may entail hundreds, or even thousands, of interrelated tasks, components, and tests or inspections when you add up everything that has to be done. These are the details you must account for in what you do. Furthermore, there may be one detail that is ignored because your team members believe someone else must be paying attention to it. You find this out after it rises up and bites you in the butt. Or there may be a detail you just didn’t think of. There’s a reason why project management is one of the core competencies in rocket science. But it’s far from the only one.
Every component, subassembly, and assembly used in a rocket launch is either analyzed or tested to determine its suitability for use during the launch or on the payload. For many projects and products this idea of analyzing and testing everything may seem like overkill and too expensive and time consuming. In the commercial world it probably is. Until you have a problem. Let me give you a real life past example. An automobile OEM supplier couldn’t seem to get its electronics to pass its shock or vibration testing. Since the electronics were packaged the way it was done previously they were confident in the design. However, to be on the safe side, because this new version was slightly different in size and shape, they decided to run some tests. They used the same fixture they always used that never had a problem. They made what seemed like a very slight modification to that fixture to accommodate new mounting holes. Yet the parts failed.
In discussions with them, the question arose whether it was the actual electronic hardware that failed or whether it was something in the test set-up causing the failure. They didn’t have the capability to run the analysis to know whether the fixture, the environment, or the part design was the cause. My company had the capability. Our engineer on the project had done this sort of analysis countless times. We proved the fixture was resonating, adding higher loads than the electronic components would see in real life. We helped them redesign the fixture and their parts passed.
Rocket Science Technologies, Inc. has the knowledge and experience to help you in situations like this. We have the experience to guide you through this kind of failure recovery in an efficient manner to find a solution. We can also help you plan your next new product design and development to help avoid these kinds of issues. We can help you corral those details so the risk of failure or issues is significantly decreased. We can’t guarantee success, but we can improve the chances dramatically. And, in the case of something going wrong, we can help you get back on track and recover.
Rocket Science Technologies, Inc., has gathered engineers, physicists, mathematicians, and project managers as associates, available on an as-needed basis, to add their know-how to help you get past even the most challenging technical obstacles. All of RST’s personnel have shown through their careers a propensity for taking on tough problems and solving them. We relish solving technical challenges. We also understand the needs of the commercial market for reduced costs and higher velocity to production and market. Check us out at http://rocketscitech.com.