When the Error Message Comes From 50 Million Miles Away

The Curiosity Rover, NASA’s Mars Science Laboratory, has been grabbing attention since August 2012, when it landed in Gale Crater in a spectacular fashion. The rover began its trek toward Mount Sharp, about five miles away, and almost immediately began sending back discoveries from the Martian surface. Then one night last February, project mission manager Jim Erickson was awakened by a call. “Ground control said they were getting some interesting telemetry from the Mars rover relay,” Erickson says.

Team members at NASA’s Jet Propulsion Laboratory in California had noticed the problem when Curiosity failed to send back the data it had recorded for the day and go into regular sleep mode. That meant something was wrong with the rover’s main computer. The engineers woke up Erickson and switched the rover over to its secondary computer, which they call the B-side, a move that automatically puts the rover into safe mode so the team can root around for the problem.

Erickson was concerned, but he was a veteran at space hardware troubleshooting, having worked on the Mars rovers Spirit and Opportunity—and Galileo. Sure enough, his team located damaged memory hardware in the A-side primary computer. The team could reprogram the A-side to not recognize the damaged parts, but Curiosity was working just fine using the B-side, so they decided to keep it as the rover’s primary system. In the space business, it’s always good to fly with a spare.

From the very beginning, Hayabusa was plagued with problems. The spacecraft, launched by the Japan Aerospace Agency (JAXA) in May 2003, had a bold mission: Rendezvous with the asteroid Itokawa, 200 million miles away, collect a piece of it, and bring it to Earth.

Just a few months into its two-year voyage to Itokawa, a solar flare hit Hayabusa, severely damaging its solar batteries. The reduction in electricity hindered the efficiency of the spacecraft’s ion engines, delaying arrival by three months. Hayabusa’s relatively small target was racing through space at 57,000 mph, and since the spacecraft had to leave Itokawa by November 2005 to be on the proper orbital path to reach Earth, the delay forced engineers to recalculate the probe’s trajectory and shorten its stay.

Shortly before the probe arrived at Itokawa, two of its four reaction wheels failed. Loss of these fuel-conserving attitude adjusters forced the team to make creative use of the chemical thrusters to maintain its orientation. “There were some long days and a few sleepless nights wondering if the spacecraft would survive,” says Don Yeomans, a scientist at the Jet Propulsion Laboratory and member of the Hayabusa team.

By September 2005, Hayabusa was on final approach to Itokawa. The plan was for the spacecraft to release its grasshopper-size mini-lander, MINERVA, to collect data and images from the surface. The main spacecraft would then approach and bump the asteroid to kick up and capture particle samples. But because of the communications delay due to Hayabusa’s distance from ground control, the command to release MINERVA arrived just after the main spacecraft’s automatic altimeter fired its thrusters to keep the craft away from the asteroid. Result: It was too high for MINERVA’s launch. The tiny lander drifted off into space. (contd.)

(contd.) That was not the end of the mission’s woes. Instead of “bumping” the asteroid, Hayabusa strangely hovered about 30 feet above it, until scientists sent a command for it to abort so they could try again. Later they realized the spacecraft hadn’t been hovering—it had landed, but failed to fire the two “bullets” designed to chip flakes into a sample collector. Project scientists were hopeful that enough dust had been kicked up to take a sample, so they sealed the container, but attempted one more try at the maneuver. The bullets failed to fire again, and then Hayabusa’s thrusters sprung a leak. The spacecraft went into safe mode, and as it drifted away from Itokawa, all communication was lost.

The team found Hayabusa two months later, as a small blip on a radio-wave screen at the Usuda Deep Space Research Center in Saku City, Japan. It took another four months to fully recover control and reestablish communications. As Hayabusa limped toward Earth powered only by the weakened ion engines, its arrival date was pushed from 2007 to 2010. Some doubted that the spacecraft could make it at all.

“But the Hayabusa spacecraft and the JAXA-led team answered every challenge,” says Yeomans. The spacecraft burned up over southern Australia on June 13, 2010, and the sample container capsule floated intact to the ground as planned and was recovered by scientists (above). Inside the container was about 1,500 grains of dust. In spite of all the near catastrophes, the mission became the first to return samples from an asteroid, providing scientists valuable first-hand study of these solar system objects.

Sending a machine out into space requires taking many, often mysterious, risks, so preventable human error makes a failure that much more painful. In June 1998, Bernhard Fleck and his wife had just moved near NASA’s Goddard Space Flight Center in Maryland to work on the Solar and Heliospheric Observatory, or SOHO, which had already completed its primary two-year mission: studying the structure and changes of the sun. Fleck, a project scientist with the European Space Agency, which had joined NASA on the solar probe, was unpacking boxes when he got a call from his team. During a routine maneuver conducted every three months, SOHO lost its attitude lock on the sun. The loss triggered a safe mode called Emergency Sun Reacquisition, or ESR, which fires the spacecraft’s thrusters in an attempt to get the craft back facing the sun.

“At first I wasn’t worried,” Fleck says. “I knew it would be a long night, but SOHO had gone into ESR before. We were bringing it back, and had a second ESR. Then a third. After five minutes someone in the control room said, ‘Oh shit! We made a mistake!’ It was our fault.”

Recovering SOHO from safe mode would have gone smoothly the first time if it weren’t for two crucial errors. A pre-programmed command code made it so one of the gyroscopes, which was supposed to activate during ESR to get the spacecraft realigned with the sun, failed to turn on. A second pre-programmed code was sending the team incorrect readings from another gyroscope. To make matters worse, the team had rushed to get SOHO out of safe mode instead of analyzing the problem, during which time they might have found the command errors. With the thrusters firing but the gyroscopes uncontrollable, SOHO was left spinning wildly, unable to collect solar energy. Eventually, the spacecraft switched off, terminating the communication link. One can imagine a shrug and maybe a robotic wish that the humans would get it together.

“The feeling was very grim,” Fleck says, “But the team did not give up.” The first “long night” turned into long months searching the skies to find the spacecraft, and Fleck updated a website every day on the recovery process. Then came the update on August 4: “The search was finally successful!” Using a combination of the Arecibo Radio Telescope in Puerto Rico and NASA’s Deep Space Network, the team found SOHO floating out in space and reestablished communications.

It was a few more months before the team could return SOHO to full operations, slowly charging its solar batteries and defrosting its hydrazine fuel. At any time, the tank or pipes could have ruptured. As the spacecraft emerged from its icy state, the team was thrilled to find that all 12 science instruments were still operational. The gyroscopes, however, could not be salvaged, so Fleck’s team improvised a unique stabilizing system using the spacecraft’s reaction wheels, a type of flywheel.

Seventeen years after launch, SOHO continues to make significant contributions to our understanding of the sun. “It was a blunder with a very lucky outcome,” Fleck says. “Though it was a nightmare, I learned you do everything you can, and you can do what you think is impossible.”

In 1995, NASA kicked off the New Millennium Program, an initiative to test technology in space to benefit future space science research. The first platform was Deep Space 1, which launched in October 1998 with innovative systems like autonomous navigation, miniaturized cameras, and an ion propulsion engine that needed to prove its usefulness for long-duration missions.

The primary 11-month technology demonstration went well, according to NASA project manager Marc Rayman. Then the mission team sent Deep Space 1 off on a two-year mission to fly by the comet Borrelly (pictured) in September 2001. But just two months into the trip, the spacecraft’s star tracker, a critical space navigation tool, failed. The initial reaction from NASA officials was that the incident was fatal to Deep Space 1. “But when you have a team of passionate and ambitious explorers,” Rayman says, “you don’t give up easily.”

Indeed, the team saw the situation as an entirely new test for their suite of equipment, and dreamed up ways to jury-rig the systems to replace the star tracker. After seven months, engineers had developed new command software that would make use of the miniature camera and the autonomous navigation system—transmitting the software update to the spacecraft, now a couple of hundred million miles away, at a painfully slow rate. At one point, the data link to the spacecraft was interrupted, causing everything that had been uploaded up to that time to be lost. That was the lowest point for Rayman and his team, he says. Still, they started again.

On September 21, 2001, Deep Space 1 flew into the icy dust cloud of Borrelly and took the first images of the nucleus of a comet.

Freelance writer Zoe Krasney lives in New Mexico, surrounded by the history and future of aviation and space exploration.