VR Smoke School Student Accuracy Results from an EPA Study

Can someone learn to accurately read smoke opacity using a virtual reality headset and then apply that training to real smoke in the field?

That was one of the most important questions we had to answer before VR Smoke School could be approved as a provider under EPA ALT-152.

The results were clear.

In a comparative study reviewed by the U.S. Environmental Protection Agency, students trained using the VR Smoke School program were compared with observers trained at an in-person smoke school.

Students trained with VR Smoke School were:

• 55.1% more accurate when reading white smoke
• 25.5% more accurate when reading black smoke

And importantly, the VR-trained students did not demonstrate that accuracy by viewing additional plumes on a VR headset.

They were trained in virtual reality and then evaluated in person, outdoors, using actual smoke produced by a smoke generator.

Why ALT-152 Providers Must Demonstrate Accuracy

EPA Method 9 is the procedure used to visually determine the opacity of emissions from stationary sources.

EPA ALT-152 provides an alternative to the traditional in-person certification process by allowing qualified systems to use virtual reality for Method 9 observer training and certification.

But simply developing a VR smoke school program does not qualify an organization to be an ALT-152 provider.

An ALT-152 provider must demonstrate to the EPA that its system can train observers to determine opacity with accuracy that meets or exceeds the benchmark established for observers trained at traditional in-person smoke school.

That benchmark comes from EPA-650/4-75-009, an EPA publication from 1975.

Establishing the In-Person Smoke School Benchmark

EPA-650/4-75-009 evaluated the accuracy of observers trained at traditional smoke school.

Nine observers were trained to make opacity observations using smoke produced by a smoke generator.

Following their training, the observers completed approximately:

• 4,500 white smoke observations
• 3,600 black smoke observations

Their readings were compared with the actual opacity measured by an in-stack transmissometer.

The study calculated the mean absolute deviation between the observers' readings and the measured opacity.

For white smoke, the mean absolute deviation was 3.74.

For black smoke, the mean absolute deviation was 3.33.

With absolute deviation, lower is better. A smaller number means the observer's opacity determination was closer to the opacity measured by the transmissometer.

These results established the accuracy benchmark against which a VR-based ALT-152 training system could be compared.

How VR Smoke School Was Evaluated

To qualify as an ALT-152 provider, we conducted a similar evaluation using students trained through VR Smoke School.

Nine students were trained and certified using the VR Smoke School program while wearing a virtual reality headset.

During training, the students learned to estimate the opacity of black and white smoke using recorded smoke plumes displayed in VR. They were shown standard opacity values, allowed to practice, and then completed the certification test.

Afterward, they were taken completely out of the virtual environment.

At an in-person field event, the nine VR-trained observers viewed actual smoke produced by a smoke generator.

They completed approximately:

• 4,500 white smoke observations
• 3,600 black smoke observations

Their observations were compared with the true opacity measured by the smoke generator's in-stack transmissometer - the same method used to establish the original in-person benchmark.

VR Smoke School vs. In-Person Smoke School: The Results

The results showed lower mean absolute deviation for observers trained with VR Smoke School for both smoke colors.

The VR Smoke School-trained observers achieved a mean absolute deviation of 1.68 for white smoke, compared with 3.74 for the in-person benchmark.

For black smoke, VR-trained observers achieved a mean absolute deviation of 2.48, compared with 3.33 for the in-person benchmark.

That means students trained through VR Smoke School were:

55.1% more accurate when observing white smoke and 25.5% more accurate when observing black smoke.

You can review the complete EPA-published study, including the individual observer deviation data, test conditions and EPA's approval letter.

Why Were the VR Smoke School-Trained Observers More Accurate?

The comparative study establishes the difference in accuracy. It does not attempt to prove exactly why that difference occurred.

Based on our experience developing ALT-152 and operating both traditional and virtual smoke school programs, however, we believe several characteristics of the VR training environment contributed to the results.

Controlled Viewing Conditions

Outdoor smoke school is inherently variable.

Wind changes. Clouds move. Lighting changes. Backgrounds vary. Smoke plumes can shear, roll, dissipate, or move toward or away from the observer.

It is sometimes suggested that these changing conditions are an advantage of in-person smoke school because they expose students to more realistic field conditions. In practice, that is not how an in-person smoke school is designed or operated.

A smoke school operator generally wants the same things we want when creating VR training plumes:

• A stable smoke plume traveling as close to straight up as possible
• A high-contrast background
• Clear skies and favorable sunlight
• A consistent viewing angle
• Minimal wind

If an operator could choose perfect conditions for every training event, they would.

The problem is that outdoor smoke schools cannot control those conditions.

Smoke generators use a relatively weak fan to move the plume vertically. Even a light breeze can overpower that airflow and push the plume sideways. That behavior is largely a characteristic of the smoke generator itself and is not representative of the industrial sources that Method 9 observers evaluate in the field.

Many industrial emission sources exhaust through large stacks or use substantial fans to move exhaust gases. Their plumes do not respond to a light breeze in the same way as smoke leaving a relatively small smoke generator.

Backgrounds create another limitation.

Operators try to position students so they can view the plume against a high-contrast background. But the physical layout of an in-person event does not always make that possible. Sometimes an appropriate background simply is not available. At larger events, there may not be enough room to position every student where the contrast is optimal.

Those conditions are not intentionally created to improve the student's training. They are compromises caused by the location, weather, equipment and number of students attending the event.

The same is true when students have to reposition themselves because the wind changes or when they are forced to observe smoke against a poor background. Those situations may happen during an outdoor smoke school, but they are not features an operator would deliberately select if better conditions were available.

More importantly, inconsistent conditions can make it harder for a new student to understand what they are supposed to be learning.

If the plume is changing at the same time the lighting, background, viewing angle, and wind conditions are changing, the student has multiple variables affecting what they see. Instead of learning how a change in opacity affects plume appearance, the student may simply be forced to guess.

With VR Smoke School, we can separate those variables.

Videos with excessive wind shear or turbulence can be excluded. Camera position, sun angle, viewing angle, background, and lighting can be controlled. Students can therefore concentrate on learning the effect of opacity rather than trying to distinguish opacity changes from changes caused by the training environment.

This also creates a more consistent certification experience.

An in-person smoke school conducted in Phoenix may have completely different lighting, weather, backgrounds, and plume behavior than one conducted in Chicago. Even two events held at the same location on different days can look very different.

With virtual training, students can be presented with smoke under consistent conditions regardless of where or when they complete their training.

That consistency is intentional.

The purpose is not to suggest that real-world opacity observations always occur under ideal conditions. They do not. The purpose is to first give students a controlled environment in which they can clearly learn what opacity does to the appearance of a plume.

Once they understand the effects of opacity, they can apply that knowledge when backgrounds, weather, plume behavior, and other field conditions change.

More Consistent Opacity Values

This may be one of the most important differences between virtual and in-person smoke school.

Under ALT-152, each video used by our system must represent a specific opacity value within ±2.5%, based on transmissometer data.

That is a very tight tolerance.

Anyone who has operated a smoke generator knows how difficult it can be to maintain a consistent plume outdoors. Even under good conditions, smoke output can fluctuate during an observation.

Traditional in-person training plumes are not subject to the same ±2.5% stability requirement imposed on ALT-152 video recordings. In fact, there is no defined opacity tolerance for in-person training plumes.

I have personally overseen the operation of more than 5,000 in-person smoke schools, and maintaining a perfectly stable smoke value during every observation is simply not realistic.

With VR training, we can select only portions of recorded smoke that meet the required tolerance.

If we tell a student they are looking at 40% opacity, we can provide a plume that consistently represents 40% opacity.

That gives the student a much cleaner reference point from which to learn.

Students Can Learn From Their Mistakes

Feedback is an important part of learning opacity.

With our virtual smoke school process, students have opportunities to practice, compare their estimates with actual opacity values and learn where they tend to overestimate or underestimate a plume.

Students can also instantly review readings they missed and see how their estimate compared with the actual value.

Instead of simply seeing more smoke, they receive information that helps them understand why their estimate was wrong.

That feedback turns each mistake into a training opportunity.

Consistency May Be the Most Important Difference

The most interesting part of the study is not that the VR-trained observers produced lower deviation numbers.

It is that they were trained on recorded smoke viewed through a VR headset and then successfully transferred what they learned to real smoke generated outdoors.

That's the important part.

The purpose of smoke school should not be to teach students to memorize what a particular smoke generator looks like at 20%, 40% or 60% opacity.

The objective is to teach them to recognize the effects of opacity itself.

By providing stable smoke, controlled lighting, consistent viewing conditions and precisely measured opacity values, we believe students can be trained to recognize those effects.

Instead of relying primarily on visual memory, which fades over time, students can apply what they have learned when the background, weather, plume behavior and other field conditions change.

The results of the comparative study support our approach.

What Did the EPA Conclude?

After reviewing our comparative study, certification program and ALT-152 compliance documentation, the EPA approved VR Smoke School as an ALT-152 provider.

The EPA concluded that our VR headset-based alternative method produces observers capable of conducting opacity observations with equivalent or better accuracy than observers certified using existing Method 9 procedures.

That conclusion is important because VR Smoke School is not simply intended to make Method 9 certification more convenient.

The goal is to train accurate opacity observers.

The comparative study demonstrated that students could learn opacity in a controlled virtual environment and then take that knowledge into the field and accurately evaluate real smoke under real-world conditions.

A Different Approach to Smoke School

Traditional smoke school has been used for decades, but virtual reality gives us the ability to control important parts of the learning environment that are difficult or impossible to control outdoors.

Stable smoke.

Consistent lighting and wind.

Consistent viewing conditions.

Immediate feedback.

The results suggest that when students are given a consistent environment in which to learn opacity, they can develop skills that transfer successfully to actual field observations.

And improved accuracy is only one of the benefits of our smoke school program.

To learn more about the people who developed VR Smoke School and helped author the original ALT-152 method, visit our About Us page.

If you have questions about EPA Method 9, ALT-152 or VR Smoke School, feel free to Contact Us.