Simulated Phishing Tests: Should You Run Them on Your Team

Somebody has probably already offered to sell you simulated phishing tests. The pitch is easy to repeat: your provider sends your staff a fake scam email, whoever clicks gets a short lesson, and you end up with a number you can put in front of your insurer. Whether that exercise protects your company depends almost entirely on how it is run.

Simulated phishing tests are worth running, and they are also very easy to run in a way that produces a comforting number and changes nothing. Below is what the independent research found, what the vendors report, and the conditions that separate a program worth paying for from a line item.

What a simulated phishing test actually involves

Your IT provider writes an email built to look like a real scam and sends it to your team without warning them. Nothing is installed and nobody is attacked. Whoever clicks the link lands on a page telling them it was a test, usually with a short lesson attached. A few days later you receive a report showing how many people opened the message, how many clicked, and how many went as far as typing a password.

The cost is low, usually a few dollars per person per month, and setup takes an hour or two. That low price is part of the problem, because it makes the exercise easy to sell and easy to leave running on autopilot for three years.

What the independent research on simulated phishing tests found

The most serious study on this was run by researchers at the University of California San Diego. Over eight months, they sent ten waves of fake scam emails to more than 19,500 hospital employees, then cross-checked the results against who had recently completed the organization’s annual security course. People who had just finished that course failed at roughly the same rate as everyone else.

The short lesson shown right after a click did not help much either. The difference in failure rates between people who received it and people who did not was very small, and the researchers noted that most employees spent almost no time reading it. Their conclusion was blunt. As these programs are commonly deployed today, they are unlikely to produce broad protection against phishing.

The vendor numbers, and why both can be true

The companies that sell simulated phishing tests publish more encouraging figures. The largest of them analyzed 42 million simulated emails sent to 14.8 million people at 64,000 organizations. Before any training at all, about one in three employees engaged with a scam email. After a full year of continuous training, that fell to roughly one in twenty-five.

Both sets of findings hold at once, because they measure different things. The university study measured what the little lesson after a click adds on its own. The vendor measured what twelve months of repeated practice produces, using its own simulations as the yardstick. What neither one supports is the annual ritual, one course and one test a year, which is what most small companies are actually buying.

The lure decides the score more than your team does

The same university study reported failure rates ranging from 1.8% to 30.8% across its ten campaigns. A message asking people to update their email password caught almost nobody. A message announcing a change to the vacation policy caught close to a third of the workforce.

That range matters when you read your own report. A flattering result can mean your staff are careful, and it can just as easily mean your provider sent an easy message that month. Before you take a 3% click rate as good news, ask what the email was about and who chose it.

The version of this that costs you money

The moment results are read out by name, or attached to a performance review, your team learns that admitting a mistake is expensive. In a real incident, the hour between somebody clicking and somebody telling you is the most valuable hour you will have. A program that teaches people to stay quiet has taken that hour away from you in exchange for a slide.

There is a second failure mode, quieter and more common. Some companies run the same easy simulation every quarter, watch the numbers improve, and conclude their people are getting sharper, when what improved was familiarity with one template.

The conditions that make simulated phishing tests worth running

1. Results are reported by department and never by name, and nobody is disciplined for a click. Say that out loud to the team before the first campaign, because they will assume the opposite.
2. Reporting is what you measure. Ask for the share of people who forwarded the message to whoever handles your IT, and how many minutes passed before the first report arrived. That number predicts how a real incident will go far better than the click rate does.
3. The messages get harder over time. A generic package delivery notice tells you nothing after the second round. The useful ones reference your own suppliers, your own invoice cycle, your own busy season.
4. They run every month or two rather than once a year. The researchers found no measurable effect from the annual course, and a once-a-year test inherits the same problem.
5. A click alone cannot lose you an account. That means a code at sign-in on every account in the company, and somebody watching for a sign-in from an unusual place rather than waiting to be told.

What this looks like for a business around Augusta

Two local realities push simulated phishing tests past the level of a nice idea. A large share of companies in Columbia County and the wider Augusta area sell into the defense supply chain, and those customers increasingly ask smaller suppliers to describe how their staff are trained before renewing a contract. Insurance renewal questionnaires ask a version of the same thing, usually in a box that expects a yes or a no.

Answering yes about staff training is a statement you may have to defend later. If your program is one course every January, it is better to say so and price the gap now than to discover during a claim that your yes meant something narrower than the insurer assumed.

The questions to ask your IT provider

1. What do we run today, how often, and who writes the messages?
2. Do we measure how many people report the email, and not only how many click it?
3. Who sees the individual names in the results, and what happens to those names afterwards?
4. If somebody hands over a password to a real one, what stops the attacker from getting into the account?
5. When somebody reports a suspicious message on a Saturday morning, who reads it, and how quickly?

A provider who answers the second and the fifth question clearly is running a program. A provider who only shows you a click rate is running a report.

The value of simulated phishing tests shows up as a faster phone call during a real incident, which is the outcome worth paying for.