Satellite
Imagery
Social Media
Metadata
Photogrammetry
3D Modeling
Civil Data
Performance
Modeling
The Tools
Open Source Tool Box
In an era of ubiquitous data, open source analytical tools have the power to take researchers anywhere in the world without leaving the office. To get a sense of just how powerful these "tools" – like satellite imagery, social media, 3D modeling, etc., can be – analysts at the James Martin Center for Nonproliferation Studies (CNS) analyzed a HIMARS (High Mobility Artillery Rocket System) Rapid Infiltration Exercise conducted by the United States and the United Arab Emirates outside of Dubai in 2018.
There are thousands of satellites in orbit above the earth collecting images of the planet using various kinds of sensors. Once only available to governments, today a number of commercial companies make satellite imagery available to all – a boon for open source researchers.
Diamond Tempest
"This is a rehersal of how we use U.S. Army and Emirati HIMARS to competitively posture across the Mideast"
-Col. Steven Carpenter, U.S. 75th Brigade commander
On April 18, 2018 the 79th Heavy Rocket Regiment of the Emirati Land Forces and the 1st Battalion of the 14th Field Artillery Regiment of the U.S. Army participated in a HIMARS Rapid Infiltration Exercise (HIRAIN). The exercise took place at Al Minhad airbase outside of Dubai, in the United Arab Emirates. The exercise was called Diamond Tempest, and was one of a series of similar exercises conducted in the Middle East as part of Operation Spartan Shield. After the exercise, over 50 images and three videos were released, depicting the events of April 18th. Upon reviewing the images and video, it became clear that the sequence of events put forth in the videos was not the true order of events.
"It's All About Speed: U.S. and Emirati Forces Practice Rapid Infiltration of HIMARS"
Creating Order Out of Chaos
After reviewing all of the images and video clips, analysts’ first step was to geolocate the events depicted in these materials in order to familiarize themselves with the exercise maneuvers and key locations at Al Minhad. Next, the team set out to put the day's events in proper order, using a variety of open source techniques.
The Diamond Tempest HIRAIN exercise took place at Al-Minhad Air Base in Dubai. Using a combination of publicly released images, videos, and satellite imagery, CNS analysts were able to pinpoint the exact locations of various segments of the exercise. The Diamond Tempest exercise took place primarily in the highlighted area of interest seen below.
Below are two examples of how analysts geolocated the images and video clips. Hover over each photo on the right-hand side to view the image annotations.
Wall
Wall
Tent
Structure
Shipping Containers
Shipping Containers
Structure
After geolocating all of the images, the team set about reordering them to reflect actual events using the methods outlined below.
The metadata, filenames, and watch faces seen in the images contributed to a full timeline of the exercise. For many images and all of the video clips, CNS analysts had to use photogrammetry to determine the precise time of day. Here is an example using a still from the Department of Defense's official video of Diamond Tempest.
Data Verification
Over the course of its timeline reconstruction, CNS analysts discovered several video clips that were inconsistent with known facts about the exercise. Below is a breakdown of the two instances of anomalous video footage.
This still from the video captures two HIMARS (one American and one Emirati) preparing for loading on to a C-17 airplane that is just out of frame on the right-hand side of the image. The American HIMARS backs up towards the C-17, while the Emirati HIMARS idles behind it, in the foreground. This series of events took place at precisely 22.022°N, 55.346°E on the apron at Al-Minhad airbase.
Aircraft Type: Boeing C-17A Globemaster III
Registration: 09-9209
ICAO Address (hex code): AE49C5
Base: McChord AFB, Tacoma, WA
Aircraft Type: Boeing C-17A Globemaster III
Registration: 04-4130
ICAO Address (hex code): AE123B
Base: McGuire AFB, Burlington County, NJ
Aircraft Type: Boeing C-17A Globemaster III
Registration: 06-6166
ICAO Address (hex code): AE1464
Base: Dover AFB, Dover, DE
Exact clip of 06-6166 seen in "MIGHTY C-17 Globemaster III Landing/Takeoff On A Dirt Airfield"
Tonopah Test Range, NV
Posted to YouTube by AirSource Military
May 1, 2013
Flight path of 09-9209 based on Automatic Dependent Surveillance Broadcast (ADS-B) data
Over the Atlantic Ocean
Data acquired through ADS-B Exchange
April 18, 2018
Location of 04-4130 as it pulls off the runway towards the apron on April 18, 2018
Al Minhad Airbase, Dubai, UAE
Satellite Image: Google Earth
The press release for Diamond Tempest clearly stated that the exercise would include a simulated fire drill -- not a live fire drill. The launch pod seen in the video was clearly marked "trainer launch pod."
Despite the “trainer launch pod” image, two videos of Diamond Tempest released by the 14th Field Artillery Regiment included clips of rockets being fired. Because CNS analysts knew both the direction the launchers were facing during the fire drill and the technical specifications of the GMRLS-Unitary rocket, they could measure the acceleration of the rocket as it was launched from the pod and estimate where it would land. Had these clips been a real part of the exercise, the rocket would have traveled about 70 km south and landed near a populated are called Sweihan. This helped the team conclude that the clips of rocket launches were taken from some other exercise video and added for dramatic effect.
What is Left?
After organizing, analyzing, and discarding what did not fit, CNS analysts were left with a pretty clear sense of the day’s activities. Below is a timeline of events, based on CNS’s use of the wide variety of open source tools outlined in this interactive.
9:30-9:45 am
HIMARS staging
for exercise
10:00-10:15 am
American and Emirati officials gather
10:45 am
Official welcome
and briefing
11:30 am
First pair of HIMARS load on C-17
2:27 pm
First pair of HIMARS unload from C-17
2:30 pm
Second pair of HIMARS load
2:49 pm
First pair of HIMARS simulate fire drill
Area of Interest
10:45am - Welcome and briefing
2:45pm - Simulated fire drill
HIMARS 1
HIMARS 2
HIMARS 2
HIMARS 1
Most digital images contain "metadata" - information such as the time and location of the photograph. It is possible to strip out metadata, but efforts to eliminate metadata are not always successful. Here, DVIDS, replaced some metadata, but did not realize that the time stamp remained deep in the file and could be read by software such as ExoMakina’s Tungstene. This image was taken at 11:32am local time.
When images are downloaded from a camera they usually have a filename that reflects the order in which they were taken (e.g., IMG_0844, IMG_0845, etc.). Such was the case when the public affairs officer for the 14th Field Artillery Regiment uploaded images from Diamond Tempest to his Flickr account. While there were gaps in coverage, the filenames helped CNS analysts to determine an approximate timeline of events.
There were several batches of photos that came from social media like Twitter and Facebook that did not contain helpful metadata or filenames. However, in multiple instances, participants' watch faces were clearly visible in the high-resolution images. Here, you can see the watch of Brigadier General Sultan Mohammed Rasid Al Habsi, Deputy Commanding General of the Emirati Land Forces. The time is 10:13am.
Metadata
Image Filenames
Watches
CNS analysts used Adobe Dimension to determine the perspective of the photographer, inserted an accurate model of a HIMARS vehicle, and moved the sun to match the shadow. This enabled the team to calculate both the altitude of the sun and its azimuth. The altitude of the sun can be determined by measuring the ratio of the vehicle to its shadow, while the azimuth can be determined from the angle of the shadow on the ground. In both cases, it is important to correct for the effect of perspective in the image.
Finding the Altitude and Azimuth of the Sun
3D Modeling:
For the second method, we used Adobe Dimension. Typically a product for graphic designers, Dimension allows you to accurately fit a 3D object into a 2D scene. Here, we imported an accurate model of the HIMARS and aligned it to scale and at the camera's perspective. In Dimension, we added a virtual sun to cast a shadow across the HIMARS, matching it to the still image. After matching the shadows, Dimension records the virtual sun's azimuth and altitude.
The last step for both methods was to take the height to shadow ratio, and then the calculated azimuth and altitude, and input them into a free publicly available program called SunCalc. SunCalc shows the sun’s movement and sunlight-phase for a certain day at a certain place. Given the location (22.022°N, 55.346°E), date (April 18, 2018), and information gathered from the photo, SunCalc generated an approximate time of between 2:26 and 2:31 pm.
SunCalc
Live fire footage taken from "Steel Warriors" video posted to YouTube by a public affairs officer for the 14th Field Artillery Regiment.
The footage in Tracker, a program by Physlets.
Sweihan, approximately 70 km south of Al Minhad airbase.
Al Minhad
Sweihan
3:10 pm
Third pair of HIMARS load
4:00 pm
Fourth pair of HIMARS load
There is a lengthy gap in photos and videos from 11:30 am to 2:30 pm. Exercise participants likely took a break during the hottest period of the day. Diamond Tempest included a brief flight, which also may have occurred during this time period.
*
*
We live in an era of ubiquitous images and videos – 350 million photographs are uploaded to Facebook every day! Governments, too, rely on social media to announce military activities and signal to friends and foes. The images and videos they post can be used to better understand military capabilities around the world.
Today, digital images often contain “metadata,” which is information about the photograph that is stored in the file but not readily visible to the viewer. There are three categories of metadata: (1) administrative, including the time and location of when and where the image was taken; (2) descriptive, such as headlines of a caption, which can be added after the fact; and (3) rights, which may identify the creator, any copyright information, or property rights.
Photogrammetry is the technique of obtaining reliable data about three-dimensional objects seen in two-dimensional images and videos. For example, it is possible to accurately measure the length and diameter of North Korean missiles seen in parade footage without prior knowledge of the missiles’ dimensions.
Creating three-dimensional models helps analysts better understand the relationship of objects in a photograph to one another, and to assess the capabilities of those objects (e.g., missiles’ performance data or the plutonium production capacity of a nuclear reactor).
Performance modeling software is most useful to assessing the range and other capabilities of ballistic missiles and rockets. After using photogrammetry to estimate the size and mass of a missile, it is possible to use performance modeling programs to measure its acceleration and calculate the engine’s thrust.
While not its intended use, civil data – such as weather, traffic, and air traffic data – is immensely useful for open source researchers looking at nonproliferation problems. For example, most ships and aircraft are required to broadcast their position as a safety measure, enabling researchers to track illegal coal shipments from North Korea or track cargo plane movements.
Video released on DVIDS - Defense Visual Information Distribution Service
"HIRAIN: HIMARS RAPID INFILTRATION"
Video posted on YouTube and the public Facebook account of the 1st Battalion, 14th Field Artillery Regiment
STRUCTURE
STRUCTURE
SCROLL
DOWN
Mouseover or touch the icons on the right
Analysts were able to geolocate one of the C-17 aircraft (04-4130) to Al Minhad airbase and determine that it was in the region at the time of the exercise.
The two other aircraft shown in the video were not in the region during the time of the exercise. Using flight-tracking data, CNS analysts determined that one of the C-17s (09-9209) was flying over the Atlantic Ocean at the time of the exercise. A reverse-image search revealed that clips of two of the C-17s (9-9209 and 06-6166) were ripped from footage of a 2012 exercise at the Tonopah Test Range in Nevada and reused in this video.
Diamond Tempest demonstrated the ability of the U.S. and Emirati forces to deploy HIMARS vehicles to a battlefield. Videos and still images heavily featured C-17 Globemaster cargo planes. Not all the footage, however, was from the exercise. The markings on the aircraft showed three different C-17s (04-4130, 9-9209 and 06-6166).
After geolocating all of the images, the team set about reordering them to reflect actual events using the methods outlined below.
Metadata
Image Filenames
Watches
Watches
Watches
There were several batches of photos that came from social media like Twitter and Facebook that did not contain helpful metadata or filenames. However, in multiple instances, participants' watch faces were clearly visible in the high-resolution images. Here, you can see the watch of Brigadier General Sultan Mohammed Rasid Al Habsi, Deputy Commanding General of the Emirati Land Forces. The time is 10:13am.
There were several batches of photos that came from social media like Twitter and Facebook that did not contain helpful metadata or filenames. However, in multiple instances, participants' watch faces were clearly visible in the high-resolution images. Here, you can see the watch of Brigadier General Sultan Mohammed Rasid Al Habsi, Deputy Commanding General of the Emirati Land Forces. The time is 10:13am.
Metadata
Metadata
Image Filenames
Image Filenames
Most digital images contain "metadata" - information such as the time and location of the photograph. It is possible to strip out metadata, but efforts to eliminate metadata are not always successful. Here, DVIDS, replaced some metadata, but did not realize that the time stamp remained deep in the file and could be read by software such as ExoMakina’s Tungstene. This image was taken at 11:32am local time.
Most digital images contain "metadata" - information such as the time and location of the photograph. It is possible to strip out metadata, but efforts to eliminate metadata are not always successful. Here, DVIDS, replaced some metadata, but did not realize that the time stamp remained deep in the file and could be read by software such as ExoMakina’s Tungstene. This image was taken at 11:32am local time.
When images are downloaded from a camera, they usually have a filename that reflects the order in which they were taken (e.g., IMG_0844, IMG_0845, etc.). Such was the case when the public affairs officer for the 14th Field Artillery Regiment uploaded images from Diamond Tempest to his Flickr account. While there were gaps in coverage, the filenames helped CNS analysts to determine an approximate timeline of events.
When images are downloaded from a camera they usually have a filename that reflects the order in which they were taken (e.g., IMG_0844, IMG_0845, etc.). Such was the case when the public affairs officer for the 14th Field Artillery Regiment uploaded images from Diamond Tempest to his Flickr account. While there were gaps in coverage, the filenames helped CNS analysts to determine an approximate timeline of events.
Interactive by Jeffrey Lewis, Anne Pellegrino, David La Boon, Adam Martyn,
David Steiger, Fabian Hinz, Jake Hulina, and Jessica Varnum