Probable Cause

After USAir Flight 427, a Boeing 737-300, crashed outside Pittsburgh on September 8, 1994, St. Petersburg Times reporter Bill Adair was granted unprecedented access to the National Transportation Safety Board’s investigation. Boeing claimed the crash—which killed all 132 people on board—was caused by pilot error; the pilots’ union claimed the Boeing 737 was defective. A series of clues unearthed through meticulous detective work pointed to a problem in Flight 427’s power control unit, a hydraulic device that controls the movement of the rudder.

When the investigation concluded in March 1999, Adair published a vivid four-part narrative in the St. Petersburg Times. An expanded account of the investigation, including the players and politics involved, was published in April 2002 by Smithsonian Institution Press in the book The Mystery of Flight 427. In this excerpt, investigators zero in on a suspicious servo valve in the power control unit.

A HYDRAULIC VALVE HAD TO pass a battery of tests to get accepted by Boeing. One test shook it violently, like a can of house paint in a mixer. Another test moved the valve back and forth five million times. The most brutal test froze the valve to –40 degrees Fahrenheit and injected it with hot hydraulic fluid. That represented the worst imaginable condition—a hydraulic pump overheating when the plane was in frigid air at 35,000 feet. Hot fluid would shoot into the frozen valve, causing it suddenly to expand. The test was called thermal shock.

Boeing did not manufacture its own valves, just as it didn’t build most of the parts for its planes. Instead, it relied on hundreds of suppliers such as Bendix Electrodynamics. The company was bidding to make a similar one for Boeing’s giant new plane, the 747. Bendix engineers built a prototype of the valve to undergo the standard battery of tests—the paint shaker, the marathon, and thermal shock.

The tests for the 747 valve were conducted in a gray stucco building in an industrial section of North Hollywood, not far from the Burbank airport. The lab, which took up most of the first floor, was filled with a thick, oily smell from all the hydraulic fluid. The room was a veritable torture chamber for a hydraulic valve. The lab even had special steel containers called crash boxes that were used the first time a valve was pressurized, in case it exploded.

Upstairs was a man named Ralph Vick, an engineer who worked on some of the company’s most important projects. Vick was not directly involved in the bid for the 747 valve, but he kept close tabs on the tests because he—like everyone else in the company—desperately wanted to win the big Boeing contract.

The torture tests on the 747 valve were no different from hundreds of others performed in the Bendix lab that year. The technicians placed the valve in a tiny freezer and hooked up the hydraulic lines. Once the valve had cooled to sub-zero, they flipped a switch and heard the steady whine of the hydraulic pumps. They moved the valve back and forth, as if a pilot were stepping on the pedals. Then someone flipped another switch, and piping hot fluid shot inside. Usually the valve kept moving. But this one strained and then stuck for a few seconds.

It had failed the test.

When Vick heard about the results, he knew it was a setback but not a catastrophe. The valve was an amazingly tight device, with only a few millionths of an inch between each slide, so a very tiny design error could cause a jam. The Bendix engineers went back to their drawing boards and redesigned the tolerances. The new valve passed without problems.

Thirty years later, Vick unpacked his suitcase in his hotel room and sat down at the desk with a legal pad. He had come to Washington for the first meeting of the “Greatest Minds in Hydraulics” to review the work of the National Transportation Safety Board on the Flight 427 case. The safety board had hit so many dead ends in the case that the panel had been assembled to look for new tests that the investigators should try. At 67, Vick was a quiet, serious man, a good choice for the group because he had designed dozens of valves and had been awarded 25 patents. He was quite familiar with the unique valve-within-a-valve used for the 737 rudder.

Sitting in his hotel room, he recalled the Bendix test 30 years earlier, when hot fluid hit cold metal and the prototype valve stuck for a few seconds. That jam turned out to be no big deal—a redesign took care of the problem. But he wondered if the rudder valve on the USAir plane had stuck the same way. He sketched a brief outline of the test on a piece of paper and gave it to NTSB investigator Greg Phillips the next day.

“I think we should look at this,” Vick said. “It may be something.”

The NTSB had not done a thermal shock test on Flight 427’s valve because there had been no comments on the cockpit tape about a hydraulic problem. If one of the pumps had broken, it would have triggered a warning light in the cockpit and the pilots would likely have mentioned it. But Phillips agreed to try the test. He was open to any suggestion.

The power control unit from the USAir crash, manufactured by Parker Hannifin, would be frozen to –40 degrees, similar to the outside temperatures at 30,000 feet, and then would be pumped with hot hydraulic fluid.

No one expected a breakthrough. The 737 valve had passed its own thermal shock test when it was certified in the 1960s. Besides, the temperature range was far more extreme than anything the PCU encountered in real life. Boeing officials viewed the test as a waste of time. Boeing’s Jean McGrew, chief engineer for the 737, said the airplane would encounter thermal shock conditions only if it flew to the moon.

On August 26, the Greatest Minds in Hydraulics and Phillips’ systems group gathered at Canyon Engineering, a tiny hydraulics company in an industrial park in Valencia, California. They had chosen Canyon because the chairman of the hydraulics panel worked there, but the company did not have the sophisticated test equipment that Boeing and Parker Hannifin, the unit’s manufacturer, did. Phillips brought the PCU in a sturdy navy blue chest, like a violinist carrying his prized Stradivarius. He took the 60-pound case to his hotel room each night to make sure that no one could tamper with the device.

At Canyon, the PCU was placed in a big white Coleman cooler, the same kind you would take on a picnic. Holes were cut in the cooler for pipes and tubes and then sealed with gray duct tape. John Cox, the pilots’ union representative in the investigation, and several others in the room said they were concerned that the temperatures were not controlled closely enough to produce legitimate results. But they forged ahead with the tests to see what would happen.

The group tested two PCUs—a new one straight from the factory and the one from the crash. To make sure that the hydraulic fluid was similar to Flight 427’s, they used fluid drained from other 737s. They used a pneumatic cylinder to act like the pilot’s feet, pushing the valve back and forth. The room filled with a steady rhythm of clicks and hisses as the cylinder moved the valve left and right.

Click, hiss, click, hiss. They put the factory PCU through its calisthenics at room temperature, testing it 50 times. It responded normally. They let gaseous nitrogen into the cooler and watched the temperature gauges plummet to –30 or –40 degrees to simulate the air at 30,000 feet. Click, hiss, click, hiss. Finally, they tried two tests to simulate an overheated hydraulic pump, heating the fluid to 170 degrees. Click, hiss, click, hiss. The hot fluid hit the cold valve, but there were no problems. The factory PCU worked great.

They removed it from the Coleman cooler and installed the PCU from Flight 427. Click, hiss, click, hiss. No problems at room temperature. Click, hiss, click, hiss. The frigid unit was blasted with hot fluid, but it still worked fine.

It was the investigators’ last day in Valencia, and the tests were going so smoothly that several people started to pack up and say goodbye.

They had reached the most extreme condition. The PCU was depressurized, frozen with the nitrogen gas, and then injected with piping-hot fluid.

The hot fluid hit the cold valve. Click, hiss, click, hiss, click, hiss, click, hisssssssssssssssssss.

The hissing changed pitch. The valve had jammed.

“It didn’t come back,” said someone in the room.

“That’s interesting,” said someone else. “Reeeeaaalllll interesting.”

A second later, the arm went back to neutral and began cycling again. Click, hiss, click, hiss.

They stopped the test and talked about what had happened. Did they have a breakthrough? The test conditions were so poorly controlled that any result was questionable. A computer operator who had been collecting test data had mistakenly deleted everything, so the team had little evidence of what they had seen. Everyone agreed to try it again.

Click, hiss, click, hisssssssssss. Click, hisssssssssss. The valve was moving slower than it was supposed to. Click, hisssssssssssssssssssssssss. It stuck again.

The group agreed that the test should be done again in a more controlled setting. The Boeing team criticized the tests, saying they were too extreme and that the valve could have been damaged. So the next morning, Phillips woke up at 4 a.m. and drove to Parker Hannifin in order to perform a test to make sure the valve was okay.

The test was crucial. When the group had first examined the valve after the crash, they had not found any scratches inside it. If they found scratches now, it would prove that a jam had occurred, which would indicate there had not been a jam on Flight 427. Also, a scratch would mean that the valve had been altered since the crash, which would rule out any further tests. The whole theory about a valve malfunction would go down the drain.

The Parker technicians took the valve apart, measuring and documenting each piece. They put them under a microscope, examining each surface for scratches or scrapes. They found none and no evidence of a jam. Phillips breathed a sigh of relief.

They had proved that the valve could jam—and leave no evidence behind.

Six weeks later, Phillips’ group reconvened in a Boeing laboratory in Seattle. This time, instead of testing the PCU in the Coleman cooler, they used a specially designed foam box with a window on top. The box’s cooling system was more powerful and precise, with temperatures closely monitored by a computer.

They ran through the same tests they had done in Valencia, starting with the factory PCU at room temperature and then trying a variety of thermal shocks. Once again, the factory PCU passed every test.

The technicians removed it and replaced it with the PCU from Flight 427. It passed the first tests with no trouble. Then came a repeat of the most extreme test in the Coleman cooler. They removed hydraulic pressure from the PCU and let it soak in the cold air until it reached –40 degrees. The hydraulic fluid was heated to 170 and shot directly into the PCU. The technician moving the valve back and forth felt it slow down. He didn’t notice it bind, but a computer showed it had jammed momentarily. He repeated the action, and felt the lever kick back when he tried to move it to the right. When he tried again, he felt it stick to the left and then jam.

Once again they had shown that the 427 valve was unique. It jammed when the factory unit did not.

Yet Boeing was right. The extreme temperature range necessary for a thermal shock just wasn’t present in real life. And there was no proof that it had happened on the USAir plane. Despite their skepticism, Boeing engineers said they would examine the charts from the tests for anything unusual.

A few days later, in a building overlooking Paine Field in Everett, a young Boeing engineer named Ed Kikta sat at his desk, reviewing the charts. He could see the test data on his computer screen, but he liked to print the results so he could study them more closely. The charts showed the flow of hydraulic fluid during each test: higher when it was pushing the rudder and down to zero when it was not. Kikta expected that when the outer valve jammed during the thermal shock, the inner valve would compensate and send an equal amount of fluid in the opposite direction, which would keep the rudder at neutral. That was the great safety feature of the 737 valve. It could compensate for a jam.

But as Kikta studied the squiggly lines for the return flow, he saw dips that were not supposed to be there. When he matched them to another graph showing the force on the levers inside the PCU, he made an alarming discovery. When the outer valve had jammed, the inner valve had moved too far to compensate. That meant the rudder would not have returned to neutral, the way it was supposed to.

The rudder would have reversed.

That could be catastrophic. A pilot would push on the left pedal, expecting the rudder to go left, but it would go right.

To make sure he hadn’t made a mistake, Kikta showed the results to the other engineers in the room. They agreed with his interpretation. It appeared that the valve had reversed. Kikta looked up and saw that his boss, Jim Draxler, was putting his coat on, getting ready to leave. Kikta stopped him.

“I think I’ve found something in the data,” Kikta said. “We might have a problem here.” Draxler took his coat off, set down his briefcase, and listened to what Kikta had to say. The consequences of his discovery were enormous. If he was right, the PCU was not performing the way Boeing had promised. The valve-within-a-valve was supposed to provide redundancy if one slide jammed. But this result meant a single jam could cripple a plane.

The next morning Draxler convened a group that he called his “grizzled veterans,” engineers who had lots of experience with flight controls. Kikta explained his findings and showed them the charts. Draxler went around the room, asking each engineer about the significance of Kikta’s discovery. They were unanimous: It was a serious problem that needed to be fixed quickly.

Boeing sprang into action. The company ordered Parker Hannifin to run its own tests to check Kikta’s conclusions. Parker engineers confirmed the results and discovered that when they jammed the outer valve, the levers in the PCU appeared to flex slightly, which allowed the inner valve to line up with the wrong holes.

Boeing was notorious for being the slow-moving “Lazy B,” but not this time. Fear was a powerful motivator. Engineers usually needed weeks to get an airplane for a test, but now they got one off the assembly line in just 24 hours. The plane landed at Boeing Field and was pulled into a company hangar. As a cold rain fell outside on the night of October 29, 1996, the 737 was rigged with the special device that Parker had built to simulate the jam. Michael Hewett, a Boeing test pilot, climbed into the cockpit while Kikta stood on a platform on the tail of the plane, watching the rudder and the PCU. Hewett pushed on the pedals, moving the rudder from side to side. The first two tests went smoothly, and the rudder operated as intended.

Then came a more rigorous test. Hewett slowly stepped on the left pedal and the rudder moved properly. He then jammed his foot on the right pedal as hard as he could. It kicked back with tremendous force.

The rudder swung in the wrong direction.

Further tests showed that the likelihood of the rudder reversing depended on where the outer slide jammed. If it jammed closer to its neutral position, the rudder was less likely to reverse. But if it jammed when it was farther from neutral, a reversal was more certain.

It was about midnight now and everyone was exhausted. They all drove home worrying about what they should do to fix the plane.

The next day, Boeing notified the FAA that the company had found a problem with the rudder PCU but wanted 24 hours to figure out how to deal with it. The FAA agreed.

Intense meetings went on all day in Renton and Everett, Washington, as the Boeing engineers discussed how to respond. They broke into two teams, one to come up with a plan to enable pilots to detect and respond to a jam, and another to look at long-term design changes to the PCU. They worked into the night. By 11 p.m., they got approval from senior management for a pilot test and some short- and long-term changes to the PCU.

The next day, Halloween, about 10 Boeing officials drove to the FAA office in Renton, a big mirrored cube of a building beside Interstate 405. They weren’t sure what the FAA would do. Would the agency want to ground the airplane? The PCU no longer protected against jams the way it was supposed to, so the plane might no longer meet certification standards.

About 25 Boeing and FAA officials gathered in a conference room. Draxler began by explaining what they had found in the tests, with Hewett frequently interrupting to give his perspective. It took a unique kind of windup to trigger the phenomenon, they said. You had to press on one pedal and then stomp hard on the other to make the primary slide line up with the wrong holes and cause the reversal.

An FAA official asked: Did this match what had happened to the USAir plane?

The Boeing engineers said all they knew from the test was that if you jam the outer slide, you could get a reversal. Jams were extremely unlikely because of the many filters in the hydraulic system, which removed particles before they caused problems. In 30 years and more than 50 million flights, there had been only seven confirmed jams. None had resulted in an accident or injury. And there was no evidence that one had occurred on the USAir plane.

Another FAA official pointed out that the new evidence seemed to counter Boeing’s claims that the pilots had caused the crash.

Jean McGrew spoke up. “We’ve received a lot of public criticism about hiding things and not wanting to spend a lot of money,” he said. “But I frankly don’t care [what it costs]. If there is something wrong with the airplane, I want to fix it.”

The meeting ended. Boeing said it would issue a bulletin to warn airlines about the condition. The bulletin would require mechanics to perform a test every 250 hours, stomping on the pedals to check for jams. The FAA planned to issue an emergency airworthiness directive that mandated the tests. Boeing also said it would develop a long-term plan to redesign the valve to prevent a reversal. That fix was likely to take several years.

These emergency directives were more symbolism than real action, designed to reassure the public that the FAA was taking action. The engineers knew the tests would not be very effective. They would catch a jam if it occurred at the precise moment of the test, but a jam could still occur at any time.

Despite Boeing’s discovery, FAA officials say they did not give serious thought to grounding the 737 fleet. The plane had a good safety record, they said, and a jam was still considered highly unlikely.

While the Boeing-FAA meeting was going on in Renton, Phillips and Tom Haueter, the lead NTSB investigator, were 2,000 miles away in Pittsburgh, unaware of the developments. They had returned to the Holiday Inn near the Pittsburgh airport to meet with all the parties. Haueter and each of his group leaders gave updates on the investigation. Phillips reviewed the results of the thermal shock tests (without knowing of Boeing’s finding) and discussed what work still needed to be done. Rick Howes, the Boeing coordinator for the investigation, sat through the all-day meeting without saying a word about the company’s big discovery.

When the meeting broke up, Haueter, Phillips, and Tom Jacky, the NTSB performance chairman, took a flight back to Washington. As they got off the plane at National Airport and walked toward the subway station, Haueter’s beeper went off. The NTSB had a new pager system that could transmit words as well as phone numbers.

Haueter glanced down at it. “major finding rel to pit / defect found on servo valve,” the pager said.

“This is a joke,” he said. “This isn’t real. Some jerk has figured out our paging system.” They went their separate ways and headed home.

The message had come from Ron Schleede, Haueter’s boss, who had been working late in the NTSB office when McGrew and John Purvis, the head of Boeing’s accident investigations, called to tell him about the finding. Schleede transmitted the message to Haueter and then walked downstairs to the bar at the L ’Enfant Plaza Hotel, where NTSB chairman Jim Hall was having a drink.

“Jim,” Schleede said, “I think we’ve got it.”

The next day, the FAA briefed Haueter, Phillips, and other NTSB officials about the finding. Haueter realized that it was a major piece of his puzzle.

“This isn’t the way I thought it would end,” he told Phillips as they walked back after the meeting. “I expected it was going to be a fight all the way to the end, putting all these little pieces together, with people saying we wouldn’t have enough evidence. And all of a sudden here is something no one expected.”

That day, Boeing sent a telex to every airline in the world that flew 737s:

Alert Alert Alert Alert Alert Alert Alert Alert
Boeing Alert Service Bulletin 737-27A1202
November 1, 1996.
The dual servo valve is designed to overcome the effects of a jammed primary or secondary slide. Although there has never been a report of a secondary slide jam, tests just completed at Boeing have shown that, under certain conditions, some jams of the secondary slide can result in anomalous rudder motion.

Anomalous rudder motion: It was a Boeing euphemism for a catastrophic situation—a rudder jam and a reversal.

John Cox of the Air Line Pilots’ Association’s John Cox heard rumblings about the discovery on Halloween night but didn’t hear the news until the morning of November 1, when the alert was issued. He had spent an extra day in Pittsburgh and was summoned to the office of William Barr, USAir vice president of flight operations. A group of pilots and safety officials were meeting to discuss the service bulletin and how USAir should respond. The airline had the third-largest fleet of 737s in the world.

Barr asked Cox point-blank, “Is the airplane safe?”

“Yes,” Cox said. He was convinced that a jam was still highly unlikely and that, even if one occurred, pilots could recover. USAir had been the first airline to raise its minimum speed above 190 knots (220 mph), so that a rudder hardover could be countered by turning the wheel; therefore, USAir pilots had an extra cushion of safety. And the airline’s pilots were already doing a rigorous rudder check, so they were effectively conducting the test before every flight.

Just before Thanksgiving, Phillips went back to the Parker plant in Irvine to compare the valve from the USAir plane with the ones from an Eastwind Airlines 737 that had had a rudder anomaly and the factory PCUs. He wanted to find out if there was something that made the Flight 427 valve jam when the others would not.

Every rudder valve was slightly different. All valves had to meet certain Boeing and FAA standards, but none of the holes for hydraulic fluid on each one were exactly the same. The tests so far suggested that some valves could be more prone to reverse than others.

At Parker, the three valves were each disassembled and examined and then hooked into a test rig to see how far off neutral they had to be moved before the rudder would reverse. The factory valve performed the best. It would not reverse until the outer slide was 38 percent extended. But the USAir and Eastwind valves would reverse more easily, when the slides were extended 12 and 17 percent, respectively. Also, a measurement of the distance between the valve slides found that the USAir unit was considerably tighter than the other two.

That was the final piece of evidence that Haueter had waited for. At last he had proved that the USAir valve was unique. After three years and hundreds of tests, he now had a scenario for what had happened to Flight 427.

It went like this: It was a smooth flight from Chicago to Pittsburgh, so there was not much movement of the yaw damper, which automatically moves the rudder to compensate for the onset of yaw. That lack of movement might have allowed particles to build up in the hydraulic fluid. There could have been a modest thermal shock to the PCU because of overheated fluid from a hydraulic pump—not enough to set off a warning to the pilots but enough to make the cold valve suddenly expand.

The PCUs on other 737s might have tolerated that without trouble. But the valve on this particular USAir plane was especially tight. The thermal shock and the contaminants caused a jam. And the jam happened when one slide inside the valve was slightly off center and more likely to reverse. The pilot or the yaw damper was commanding the rudder to go right, but it went hard over to the left.

All of this occurred at the most vulnerable speed for a 737: 190 knots. The pilots compounded the problem when they pulled back on the control column, which made the plane lose speed and stall. The plane spiralled down and crashed into a hill.

It always takes a chain of events to cause a crash. In this case, it took wake turbulence from a Delta 727, the startling of the pilots, the fact that the plane was flying at the crossover point, the uniqueness of the valve, the jam and reversal, and the mistake of pulling back on the stick. If any one of those things had been different, the plane would not have crashed.

“What the hell is this?” Captain Peter Germano had said on the cockpit voice recorder as USAir 427 plunged toward the ground. Haueter thought he finally knew the answer.�