Photo Info

What is it like to fly a helicopter with a tablet?

By Eric Adams

Published on: April 30, 2019
Estimated reading time 12 minutes, 3 seconds.

Having spent more than five years honing its Matrix autonomous technology in its SARA S-76 demonstrator, Sikorsky recently invited three non-pilot journalists to see just how far the system has come.

In a recent flight aboard Sikorsky’s autonomy testbed — an S-76 outfitted with a supercomputer, electrically actuated servos, and myriad sensors sprinkled around the fuselage exterior — the most eye-opening moments didn’t necessarily come during the crisp liftoff, precision maneuvering, or brilliantly smooth landing. No doubt, each was hugely impressive, and that capability was the core reason I and just two other aviation journalists were invited to Sikorsky’s Stratford, Connecticut, headquarters to become the first media to “fly” the self-flying helicopter.

Nor did my “aha” moment come while I effortlessly tapped my instructions into a tablet computer strapped to my knee, or even when I got my hands on a pair of inceptors aboard SARA (Sikorsky Autonomy Research Aircraft) that interpreted my intentions and automatically balanced the controls typically modulated by the collective, cyclic, pedals, and throttle, allowing me to bank easily through the sky or fine-tune my position above the airfield.

While SARA has been used to mature the Matrix technology, it will see its first commercial application in a Sikorsky S-92. Ted Carlson Photo

No, the most vivid part of the flight actually came during the in-between moments. That is, the periods when I jumped in to make a change, or when Sikorsky test pilot Mark Ward — my “copilot” on this mission and the only other human on board, given that the cabin is packed with test gear — executed a rare input of his own using the conventional controls. Usually he did this merely to demonstrate the handoff process between pilot-control and computer control, between hands-on and hands-off, and indeed between brain-on and brain-off. The transition requires no thought, just action. It’s immediate and, most importantly, easy, allowing you to seamlessly refocus your attention between tasks. Though I’m not a certificated pilot, I have flight experience and readily grasped the workload-reduction potential this innovation offers.

The system, called Matrix Technology, is designed to be instantly interruptible simply by taking over the control inputs, and it re-engages once control is relinquished, either continuing with its programmed mission or maintaining your most recent heading and attitude. It can lurk in the background as an always-on virtual co-pilot or ever-vigilant emergency backup during manual flying, or be used to manage most or all of the flight while the crew communicates, strategizes, or simply monitors. Sikorsky debuted the system in 2013 and has been honing it aboard SARA ever since, to the point now where it’s about to be deployed in military and commercial helicopters.

The Sikorsky Autonomous Research Aircraft (SARA) is a modified S-76B. Sikorsky has used it to extensively test its autonomy technology. Ted Carlson Photo

My 30-minute flight in the vicinity of Sikorsky’s headquarters was revelatory on several levels, from the low level of training required to fly SARA to the indisputable sophistication lurking in plain sight. It began with about 45 minutes of instruction on how to use Matrix, via a simulator in a motor coach parked adjacent to the S-76. There I learned how to use the tablet, with its moving map — alas, no pinch/zoom capability, since that might be inadvertently activated in flight — central helicopter icon representing SARA, and control widget that operators summon to input instructions. You tap on that, set your speed, altitude, and direction, or just tell it to liftoff and establish a hover at a prescribed altitude. You can load preset flight plans, or simply tell it to fly to a point on the map, prompting the computer to calculate the best route and approach strategy. I also familiarized myself with the two inceptors. A preflight briefing followed with Ward, lead autonomy engineer Igor Cherepinsky, and the key engineers and flight-test personnel working on the program.

When we finally boarded SARA, the tablet and inceptors were of course immediately familiar. Ward started up the helicopter’s dual turbines and taxied us out to our liftoff point — ground operations are still fully manual — and handed control over to me. Communicating with Cherepinsky in the trailer to confirm everything was set properly, I tapped “execute” on the tablet. A few seconds later, we lifted off slowly from the ground and the system put us into a steady 60-foot hover. I then used the inceptors to maneuver us around the field for a few minutes, marveling at the ease with which I was able to do so, particularly given my previous helicopter flight experience that amounted to precisely zero hours. Each time I let go, the helicopter steadied itself precisely. I kept looking to make sure Ward wasn’t secretly helping the system out, but he clearly wasn’t. The only reason his hand was even on the stick was to activate the mic trigger when talking to Cherepinsky or myself. When we eventually headed out to our waypoint above the countryside, the helicopter managed every aspect of the flight, motoring us along with spooky precision.

This is the tablet I used to input my flight plan and execute it. It’s hard-wired into the system’s computer, which itself takes data from the helicopter’s systems and then executes the flight via electrical servos attached to the controls. The stick to my right is the inceptor that controls left/right and forward/reverse movement – essentially, of course, heading in whatever direction you point. Eric Adams Photo

While the system operates relatively simply from the pilot’s perspective, it is packed with algorithms and powered by steroidal hardware able to manage huge volumes of data coming from multiple cameras, laser scanners, radar, and onboard sensors. As you watch the moving map, you see the lidar hard at work scanning the terrain and the air for obstacles and identifying suitable landing zones, calculating the approach to each on the fly in the event one is needed.

Eventually, the kind of computerized automation demonstrated with Matrix will permit such things as optionally-piloted flight and even fully autonomous air-taxi missions for civilian passengers, with no trained pilots aboard at all. That specific vision still sits well over the horizon, dependent as it is on a staggering gamut of successes in electric vertical-lift, regulatory approval, and financial management, among other things. But in the near term, crews flying with the features developed in this program will find themselves able to more readily attend to mission-specific tasks — communication, observation, strategizing — rather than managing the minutiae associated with helicopter piloting. Not only that, but as a baseline safety tool, Matrix could become your hyper-vigilant, never-distracted co-pilot that will help keep you out of trouble and bail you out of whatever jams you do find yourself in.

The SARA simulator, which is located in a motor coach. It’s an exact duplicate of the system on board SARA, and is in fact linked with it during test flights, so engineers on the ground can make adjustments, monitor and even fly the helicopter remotely. Eric Adams Photo

Ultimately, Ward said, the goal is not just for this to enhance safety, but to help redefine mission capabilities. “This allows operators to reduce crew requirements, yet still handle complex missions,” he explained. “It creates flexibility and the potential for true single-pilot operations.”

Cherepinsky added that this potential exists even in the face of the most challenging circumstances, such as poorly lit landing zones or otherwise degraded visual environments. Matrix technology can already handle landings on barges at sea, and it’s been proven in simulation to be able to manage autorotations as well as approaches to ships and oil rigs at night and in severe weather. “In these cases, pilots will essentially monitor the situations and step in as needed,” Cherepinsky said. “So for now it’s absolutely designed for professional pilots. The next step is to make it usable by minimally trained crews, where it will have much more control over the missions and will handle much more of the communication and other responsibilities.” That will involve further honing the user interface, which Cherepinsky said is designed for professional pilots, and it still needs to make progress in terms of interpreting and executing casual verbal or keyed-in commands. “Nailing the interface is really hard,” he added. “We can have all these great algorithms, but how do we enable the human operators to convey their intent for each flight? That’s taking a lot of time to get right.”

Then there’s the fundamental challenge of convincing professional pilots that the system won’t itself hinder them in some fashion. But Ward said those who’ve tried it, while occasionally resistant to the idea of computers taking control — given the instincts and training they’ve honed to execute complex, often highly nuanced missions — have been consistently won over, just as previous generations were quickly won over by simple autopilot. “People may automatically assume we’re trying to kick them out of the cockpit,” Ward said. “We’re not. We’re trying to make it so you can do more while in the cockpit, without worrying about the thousands of things you usually need to worry about at any given moment.”

Sikorsky chief test pilot Mark Ward, monitoring the Matrix system in the air. His hand is on the stick mostly to activate the mic button, but the system takes care of everything else. If he wanted to jump in, he could simply take over control and his movements would override any system inputs, deactivating the autonomous flight mechanisms until he relinquished control back to the helicopter. Eric Adams Photo

Features found in Matrix Technology will begin to appear first in the UH-60 Black Hawk helicopters the U.S. Army flies — in fact, a group from the Army tried it out at Fort Eustis, Virginia, late last year, with highly favorable reactions — and then its features will be available in Sikorsky’s commercial helicopters, including the S-92 heavy-lift model. After that, well, the sky’s the limit, including eventually being a potential key enabler of civilian aerial mobility. Even if that long-promised vision doesn’t materialize, there will likely still be enormous benefits to pilots wishing to manage their missions as efficiently and safely as possible.

Leave a comment

Your email address will not be published. Required fields are marked *

Notice a spelling mistake or typo?

Click on the button below to send an email to our team and we will get to it as soon as possible.

Report an error or typo

Have a story idea you would like to suggest?

Click on the button below to send an email to our team and we will get to it as soon as possible.

Suggest a story