Real-Life Stress Test

Did you ever break that shiny new system in the middle of a major crisis?

I have.

This is how I remember it.

As every sailor’s story starts, so there I was….
It was December 2004, the M/V Selendang Ayu was on its way along the Great Circle Route when it lost propulsion North of Unalaska Island in the Aleutian Chain. The ship began drifting as the weather, as weather does in Alaska, began to worsen.

The call went out, the ship was in trouble. Tugs from nearby Dutch Harbor and the USCGC Alex Haley raced to the scene to try and bring the stricken vessel under control.

Meanwhile, our media relations team scrambled to coordinate messaging for the quickly evolving case. At the forefront of the messaging effort was our brand spanking new webtool, PIERS (Public Information and Emergency Response System), an all-in-one tool that replaced our old type it in word and fax it out system for how we did press releases. It was a system that, for the first time, allowed our three offices spread across the state to work together in real time no matter where we were. I looked at my subordinate and said, “Grab your stuff. I don’t care how you do it but get out to Dutch Harbor. We are going to need boots on the ground.” She raced out the door and press releases and inquiries began to fly. The plan came together fast. I’d hold the fort in Kodiak. She’d stand up a field office in Dutch Harbor for the duration of the response, backed by public affairs personnel mobilizing from as far away as Elizabeth City, NC. A group of our district staff would run the Joint Information Center in Anchorage, where the media gathered. A skeleton crew in Juneau would handle everything else.

Things on-scene continued to deteriorate. The sea was angry that day and the tugs and cutter were unable to check the progress of the drifting ship. Thick tow lines snapped. Anchors couldn’t hold. It was time to abandon ship. Two USCG helicopters from Kodiak headed out to start the hoist operations. The first, CG-6020 lifted 9 of the 27 crew out and deposited them on the deck of the Alex Haley, a feat that was determined to be too dangerous to attempt a second time. A second helicopter brought nine more of the crew to shore.

As the ship began grounding on the Northern shore of Unalaska Island the final hoists began. The last eight crewmen aboard the Selendang Ayu were hesitant to get in the little basket. The hovering helicopter lowered the rescue swimmer to hasten their departure. Slowly, each man was lifted from the deck as the ship ground on the rocky shore. During the next to last hoist a rogue wave hit the ship and washed over the helicopter. The windshield went white as the engines flamed out from the water. As the flight mechanic called on the intercom, “We’ve got water in the cabin sir! Up! Up! Up”, the helicopter spun, breaking off its tail and shattering the main rotor, knocking it out of the sky and into the frigid waters below.

Miraculously, the smaller H-65 Dolphin, CG-6513, launched from the Alex Haley in a hail Mary maneuver that used the ever increasing ship movement from the waves to catapult them from the deck. This put the 6513 in position to see the crash and begin rescue efforts right away. The three USCG helicopter crew were safely rescued along with one of the seven crewmen. Unfortunately, the ship’s crew were not in survival suits and they were quickly lost to the frigid, churning waters.

The 6513 had to head to Dutch Harbor leaving the ship’s captain and the rescue swimmer aboard the now grounded Selendang Ayu. They would remain aboard the ship for nearly four hours until the 6513 could come back to rescue them.

My phone rang in Kodiak. The voice on the other end said, “This is the CDO (Command Duty Officer) at Air Station Kodiak. You need to get over here. One of the helicopters went down”. This would be the first real-world test of PIERS for our district. Since it let us work from anywhere, I quickly shifted my operations to the air station command center and knocked out a bare-bones press release saying that one of the helicopters went down during the rescue. If only I had known how badly this real world test would go.

As happens, information slowly made its way to us. Being in the command center sped that up somewhat, allowing me to get the news as it came in to the air station. The news coming in was bad and not getting any better. A helicopter had definitely gone down. We knew which one but didn’t want to release it until we knew how the crew was and families had been notified. Imagery from the case began to come in. We had dramatic photos of the 6020 dropping off the crew on Alex Haley along with some video of the evolution. Phones were ringing off the hook asking about the crash. Another call, the ship had broken in half with the Captain and the rescue swimmer still on board. More calls, the USCG crew had been rescued but only one of the ship’s crew had survived and he was injured. Despite 6513’s search, no other crew could be rescued from the water.

More footage came in, the 6513 launch this time, and we put out another update with imagery and video. The phones were still ringing, and the next question was already coming: what happened, and what were we going to say about it. That’s when our fancy new system came to a crashing halt….only not just for us…but for all users, everywhere. Turns out, embedding the video in the press release and then having people around the world downloading it as they followed the news of this ship caused a lot of traffic and bandwidth.

While the rescue swimmer and captain held on to a broken ship in freezing waves and driving blizzards, back in Kodiak and Anchorage we were fighting for as much information as possible while also trying to fix the tool we needed to push that information out.

Frantic calls with the company began trying to nail down what was going on. No one could log in. If you did get logged in, the system was crawling or lethargic. Things weren’t working and we needed to find out what was going on quickly. Turns out it was the video. We quickly came up with an alternative way to send it out. The fix was made and we were up and running in an hour.

Except, being down hard like that in the middle of an emergency is not ideal and it happened not once, but twice. The second time, the next day after first light revealed a ship broken in half, oil and soybeans polluting the waters and shore, and the broken, oil stained hull of the CG-6020 washed ashore. Once again, as soon as we cleared the video for release we put it out, following the fix we had identified. Soon, the system was down again, moving like frozen molasses as the world watched and downloaded the video of the 6020 on the rocky beach getting hit by waves.

This time the bandwidth was so great that we were getting hit by overage charges (yes that was a thing back then). Again, I got on the phone to figure out what was going on and get it fixed.

You see, the shiny new toy had been used for situations like this. It was specifically designed for situations like this but no one had ever really put it to the test this hard before. Our real world use broke it when we needed it the most. This is why it’s important to test things as close to a real world use as possible. This is why we teach people to “train as you fight”. You need to know how your people and systems will operate in a real world crisis. You need to exercise working without power, or phones, or internet. You need to have a backup plan to the backup plans so you can pivot quickly and keep moving when there may be lives on the line.

That day, when things went down, we pivoted back to word, emails, and phone calls until the system could be brought back online. We relied on our training to respond to the situation and it got us through.