Enhancing HMD-Based F-35 Training through Integration of Eye Tracking and Electroencephalography Technology05 Apr 2015 Meredith Carroll, Glenn Surpris, Shayna Strally, Matthew Archer, Frank Hannigan, Kelly Hale, and Wink Bennett | Design Interactive, Oviedo, Florida"
Abstract. The ever increasing complexity of knowledge, skills and abilities (KSAs) demanded of Department of Defense (DoD) personnel has created the need to develop tools to increase the efficiency and effectiveness of training. This is especially true for the F-35, the first 5th-generation aircraft to use an HMD as the primary instrument display. Additionally, the F-35 can perform operations previously performed by multiple operators, which potentially places incredible strain on the pilot’s cognitive resources by exposing him to large amounts of data from disparate sources. It is critical to ensure training results in pilots learning optimal strategies for operating in this information rich environment. This paper discusses current efforts to develop and evaluate a performance monitoring and assessment system which integrates eye tracking and Electroencephalography (EEG) technology into an HMD enabled F-35 training environment to extend traditional behavioral metrics and better understand how a pilot interacts with data presented in the HMD....
...The current training program for F-35 transition pilots is 8 weeks long. The transition pilots are comprised of legacy aircraft experts such as experienced F-16 or F-22 pilots. These pilots will become F-35 instructors upon the completion of the program. Training begins with a week of military lectures, followed by 3 weeks of lectures and academic courses specific to the F-35. A pilot training aid (PTA) laptop simulator is flown by transition pilots during these early phases of the course. The last phase of the training program is a mixture of 8-10 F-35 Full Mission Simulator (FMS) sessions and 4-5 actual flights in the F-35. The PTA and the FMS are the two main simulators used in the transition curriculum. The PTA has a large touchscreen monitor that displays both the out-the-window view of the aircraft as well as the touchscreen instrumentation (i.e. Main Forward Display). In addition to the touchscreen monitor, the PTA also has a full replication of the F-35 Hands-On Throttle and Stick (HOTAS). The PTA is mainly used during academic lectures to familiarize the pilot with the controls and procedures for the F-35. An HMD is not used in conjunction with the PTA.
The FMS is a high fidelity flight simulator which contains a full 1-to-1 replication of the F-35 cockpit surrounded by a dome with almost 360 degrees of visual coverage. The pilot trainee is outfitted with an HMD visor that reveals a HUD fixed on the center windscreen. Additionally, a de-cluttered, un-fixed version of the main HUD with a reduced selection of essential symbols (e.g., airspeed, altitude) appears on the HMD when the pilot turns his/her head off bore-sight (i.e., left, right, up, or down). The simulator sessions in the FMS are 1.5 hours in duration and are preceded by a 1 hour pre-brief and followed by a 1 hour debrief. Each trainee in the FMS has the individualized, one-on-one attention of an instructor. The instructor has an operator station where he can launch scenarios and insert abnormal aircraft conditions. During the training session, the instructor can also view the pilot’s performance unfolding from a series of view, including the field of view (FOV) in the cockpit due to a head-tracker associated with the HMD.
The debrief then provides the opportunity for the instructor to playback any flight segment during the simulator session and review notes, exceptional performance, and trainee performance errors. Control inputs, the pilot’s FOV, and other simulator information can be accessed by the instructor to facilitate this debrief. Instructors depend on overt behavioral actions and communications to identify performance errors. One limitation of this approach is the inability of the instructor to determine the specific instruments the pilot is monitoring, both within the HMD and on the MFD. Heads up/heads down status can typically be inferred based on the FOV presented by the HMD, however, the specific information that the pilot is visually integrating is not accessible. Given that a large portion of the task is monitoring information presented by a range of instruments; this limits the instructors understanding of how pilot performance is unfolding....
...
4 HMD ASSESS Use Case (pp.28-29) HMD-ASSESS is designed to be utilized during the actual training session and debrief. A use-case was developed to demonstrate the HMD ASSESS concept of operations for F-35 FMS training sessions and is presented in summary in this section.
A typical training session in the FMS may include several abnormal malfunctions from which a pilot must attempt to recover. During this particular training session, the instructor has inserted an Integrated Power Package (IPP) failure into the scenario. As the pilot trainee attempts to recover from the IPP failure, he performs three key errors: 1) the pilot misses a critical checklist item (i.e., arming the backup oxygen system); 2) the pilot spends too much heads down time looking at his checklist and fails to scan his primary flight instruments (altitude, attitude, airspeed) at the necessary intervals; 3) the pilot develops tunnel vision on an area of the cockpit irrelevant to the appropriate task, e.g., determining the best place to land, resulting in a delay in conducting a critical checklist item (i.e., open RAM door).
After the training session in the simulator has ended, the instructor uses the HMD ASSESS after action review displays to facilitate his debrief to the pilot trainee as follows. The instructor is interested in assessing the students handling of the IPP failure, so the instructor clicks on this segment of the timeline and the timeline automatically zooms into the IPP failure event. The instructor points out overall timing summary for that segment to the pilot, including total time heads up vs. heads down and total time in high priority areas. The instructor can illustrate to the pilot trainee that he spent a large amount of time heads down while handling the IPP Failure.
The instructor then clicks on the first system identified error, which automatically zooms the timeline down to a system default of 30 seconds on either side of the error. The instructor plays back the error and points out that, based on the eye tracking data, the pilot was distracted from reading the checklist by focusing on blinking lights on the IPP Panel.
The instructor then moves on to the next error (i.e., breakdown in a periodic eye scan of flight instruments), by selecting the error from the error summary list. The instructor wants to show the pilot how he failed to scan his primary flight instruments frequently enough. By using the Overview mode containing a summary of all eye tracking data for 30 seconds on either side of the error, the instructor illustrates to the trainee that a scan of these three primary flight instruments did not occur during this time period. The instructor confirms this by pointing out the timing summary which shows that the pilot spent very few seconds looking at the altitude, attitude, and airspeed instruments for the specified window of time.
The instructor then points to the section of the timing summary that shows the total time spent on each MFD page for the segment of flight in focus. He uses this data to illustrate that the pilot spent only 30 seconds looking at the navigation page and flight instruments because he started to look for the nearest airport to land too early, instead of following the checklist steps. This caused the pilot to delay in opening the RAM (i.e., air intake) door, which resulted in systems overheating more quickly.
As illustrated in the se case, HMD ASSESS will allow an instructor to more accurately and efficiently diagnose a performance issue. Instructors will be better able to direct a pilot’s attention during overwhelming flight scenarios and prevent pilots from making common mistakes with regard to visual attention allocation...."
Source: download/file.php?id=20448 (PDF 200Kb)
05 Apr 2015, 05:52
"...most helmet visors are cut to properly fit the pilot's nose ridge..." aaahh the wonduhs of mod tech eh. 
Yes I do think the Display Visor would be relatively INexpensive - anyone know? Buehler? Anyone? 
