Sponsored Content
Sergeant Wade Bourassa of the Saskatoon Police Service in Saskatchewan flies his Wescam MX-10 electro-optical/infrared (EO/IR) camera on board the aviation unit’s Cessna 182. The unit recently adopted Sentient Vision Systems’ artificial intelligence (AI) software, named Kestrel, which provides a moving target indicator (MTI) that automatically detects object movement and places a red box around it.
Bourassa explained that before he would use the EO/IR camera to search street to street and now he can search an entire block. In one example, he coordinated the takedown of an armed and violent escaped convict at night hiding in the forest. Using the Kestrel overlay, he tracked the entire tactical team and their K9 dogs using the MTI software during the apprehension of the suspect, which drastically increased his situational awareness.
Flying a camera with the Kestrel, whether on a fixed-wing aircraft or helicopter, enables a search area 20 times larger than would be possible without such sophisticated software. The EO/IR sensor can be operated in a wide field of view to effectively search a large swath of land or ocean, which enables the software to automatically detect object movement independent of a human observer. The user can then zoom in digitally using a mission computer while maintaining a wide search area with the EO/IR payload.
Furthermore, the user can then initiate “breadcrumbs” on individual or multiple detections which displays the object’s location where it started, drawing a path of the detection on the user interface. The user can then download the information on each breadcrumb with positional data within the EO/IR’s field of view, enabling forensic review or providing data for prosecutions if needed.
Kestrel can track over 400 targets at once, be they individual vehicles or people. The software works equally well at night with infrared cameras as it does during the day. Kestrel can even spot objects on the ocean’s surface, either floating or moving.
“We can have the best operators and the best equipment but it comes down to reducing stress on the operator and alleviating the high probability of gaps in our search area to ensure effective coverage.”
Kestrel has the added benefit of reducing operator fatigue and increasing situational awareness by automating the search functionality. The software can be used on nearly any camera system or video feed, and Kestrel will notify the user of any movement within the area being scanned.
“For electronic search you’ve got a dedicated payload operator. Your job is just to operate the EO/IR sensor payload and not transition your eyes away,” said Sentient’s lead trainer, Mick Sheehy, who has more than 20 years of experience flying for Australian and international maritime agencies conducting intelligence, surveillance and reconnaissance (ISR) and search-and-rescue (SAR) missions.
“But inherently, you become fatigued. Inherently you will miss the object because you can’t examine the whole area. We can have the best operators and the best equipment but it comes down to reducing stress on the operator and alleviating the high probability of gaps in our search area to ensure effective coverage, and have the highest levels of confidence that this computer vision, this AI, this algorithm is examining this whole scene for me.”
Sentient’s beginnings
Sentient, from Melbourne, Australia, has been developing airborne surveillance solutions for over 20 years and was founded in 1999 by Dr. Paul Boxer. Boxer left his engineering job to pursue a doctorate in AI because “I wanted to be educated in the next big thing and that was 20 years ago,” he said.
Boxer and his co-founder developed a computer vision product and showcased it to the Defence Science and Technology Group. That initial product became Kestrel, which was 100 times faster than the existing software, providing the moving target indictor solution for airborne assets. And Sentient has been perfecting computer vision for aviation and surface-based applications since then. The company offers a range of solutions to integrate computer vision (CV) and AI to enhance the capabilities of the airborne and land- and maritime-based camera systems.
As Sentient’s original software application for object detection, Kestrel integrates into various EO/IR systems to support airborne assets including fixed-wing and rotary-wing aircraft and drones. Multiple European and U.S. law enforcement agencies have adopted Kestrel to enhance the overall effectiveness of their ISR operations. Kestrel enables the operator to open the field of view to 20 degrees, and the MTI software locates everything moving, day or night.
“Kestrel is designed to detect any object moving on land as small as two-by-two pixels in size, and sometimes down to sub pixel,” said Sentient general manager Stewart Day. This roughly translates to a six-foot object from 8,000 feet, depending on the resolution of the camera utilized.
“Kestrel is also available as a maritime solution detecting objects on the surface of the ocean such as a small boat, life raft or person in the water,” Day added. “This greatly enhances situational awareness for mission operators enabling faster and better decision-making to protect against threats or save lives.”
The building blocks to ViDAR
In 2016, Sentient took its Kestrel software one giant step forward, introducing deep learning algorithms to develop ViDAR, or Visual Detection and Ranging, basically making a camera into a visual radar with a 180-degree field of view. ViDAR has a nominal maritime search swath of 20 nautical miles, where it detects everything in the search area and reports it back to the operator.
The Australian Maritime Safety Authority (AMSA) adopted ViDAR in 2016 as the agency upgraded its SAR fleet with four Challenger 604s. Their sensor package included a three-camera ViDAR system with an EO/IR turret, and a maritime search radar.
This turned out to be a good thing for Nigel Fox, who was sailing almost 60 miles (95 kilometers) off the north coast of Australia battling a storm in January 2021 when he fell overboard, leaving his cat, Stinky, on board. Fortunately, Fox, a skilled sailor, had donned a life jacket equipped with a personal locator beacon (PLB) which he activated, initiating the call to AMSA which is responsible for the largest search-and-rescue region in the world.
The Challenger launched on the case on Jan. 4, when Fox activated his PLB. Despite the maritime search radar and the PLB going off, the Challenger crew could not locate Fox because of heavy seas. They turned to ViDAR, and flew at 500 feet and 190 knots.
ViDAR processed the imagery feed frame by frame in real time with every pixel being analyzed for a range of characteristics with a 180-degree field of view. If ViDAR detects an object, it alerts the operator by sending a thumbnail image to the mission system display and a geolocation marker on a moving map display that provides the object’s updated location, bearing and range from platform. Within that swath, ViDAR can spot small, semi-submerged objects the size of a human head. It has demonstrated detection rates of up to 96 percent.
One of those thumbnail images showed Fox floating with his life jacket. The operator then cross-cued the platform’s inspection turret to direct the pilot to his location. The AMSA Challenger dropped Nigel a life raft; he was able to scramble on board and wait for a rescue boat. After his rescue, AMSA recovered his boat 150 miles (240 km) from his location with Stinky still on board.
Because of the success of ViDAR, Sentient developed a modular pod, the VMS-5 POD, or ViDAR Maritime Search POD. The VMS-5 POD is able to move from asset to asset, rotary-wing to fixed-wing or unmanned aerial system (UAS). The versatility of ViDAR enables multiple configurations from a group 1 UAS — less than 25 pounds (11 kilograms) — up to any size of manned or unmanned aircraft.
The maritime sector
At the same time the Challenger was being outfitted with ViDAR in 2016, Sentient signed an agreement with Insitu, a subsidiary of Boeing, to provide detection capability onboard the ScanEagle, a small UAS (sUAS) used regularly for maritime surveillance. Sentient capitalized on its technological advances and developed a solution to use ViDAR on the ScanEagle by placing a rotating ViDAR turret on the sUAS.
The ViDAR turret rotates, then stops for a few seconds, moves and stops again, providing similar swath coverage as the five-camera system while the targeting turret provides the visual picture, taking cues from the ViDAR detections. ViDAR detects any anomalous movement, static or moving on the water within 45 miles (around 75 km) of the platform, depending on its altitude.
The head of Insitu’s business development at the time was a retired U.S. Coast Guard rescue swimmer, Ron Tremain. Over 23 years of flying SAR, he knew the value of ViDAR, and how difficult it is to find anything in the wide-open ocean, especially a person in the water. His efforts were instrumental in the U.S. Coast Guard adding ViDAR on the Scan Eagle in 2016. Tremain saw ViDAR adding enormous value to the ScanEagle, which helped the sUAS win the U.S. Coast Guard’s $117 million baseline contract to support its new national security cutters.
“The skippers of the national security cutters, they didn’t want to sail without ViDAR, it had that big of an impact,” Tremain said.
Tremain went on to explain that the U.S. Coast Guard’s Research and Development Center ran ViDAR through a series of tests in 2016, and ViDAR found objects with a greater than 90 percent level of accuracy. It even found several objects in the test area which weren’t known, including a submerged testing torpedo and vessels greater than 35 miles (55 km) away.
ViDAR provides a 180-degree swath of coverage enabling a 300- to 600-times improved searched capability compared to using EO/IR alone, depending on on-scene factors. Helicopters employing ViDAR will be able to expand their search areas significantly, adding in a layer of AI to change the game completely.