The Security Bug Your Tests Cannot See

Picture an office building with a guard at the front desk. You show your badge, the guard waves you through, and in you go. Deeper inside there is a server room that only IT staff are meant to enter.

Now picture how that building's security gets tested. One person walks up to the front desk, shows a badge, gets waved through, and goes home. They file a report saying security works.

Nobody ever tried the server room door.

That is roughly how most modern software gets tested for permissions, and it is what our paper at IEEE SOSE 2026 in Fukuoka, Japan set out to fix.

The PDF above is the accepted version of the manuscript. © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The final published version is available from IEEE at doi.org/10.1109/SOSE71128.2026.00014.

One Click, Many Doors

Modern apps are rarely one big program. They are built as microservices, which means the app is chopped into dozens of small programs that call each other over a network.

Booking a train ticket in one click might quietly involve a login service, an order service, a payment service, and a seat service, each one handing the request along to the next.

Every one of those services decides for itself who is allowed to do what. That decision is called authorization, and it is made independently in dozens of places, by different teams, on different schedules.

So the rules drift apart. One team tightens a rule. Another team rewrites a service and accidentally stops passing along proof of who you are. Nothing appears broken, because the front desk still says yes.

Why Today's Testing Tools Miss It

There are good automated tools that generate security tests. The best known are EvoSuite and EvoMaster. They share one habit: they knock on the front door and stop there. Everything behind it is a black box to them.

Both halves of that picture show the same app. On the left, the tool can only see the first step, so it raises an alarm it cannot back up. On the right, the tool follows the request through every step and gets the answer right.

A tool that only sees the entrance fails in two directions at once. It cries wolf about problems it cannot confirm, and it stays silent about real ones happening out of sight.

The Idea: Work Out the Answer First, Then Let AI Write the Test

The obvious move is to point an AI model at the code and ask for security tests. That does not work well. The code is far too big to fit, most of it is irrelevant, and the model has no reliable way to know which service calls which.

So we did it the other way around.

Step one is arithmetic, not AI. We read the source code automatically and build a map of every service, every door, and the permission rule on each one. Then, for any journey through the app, we ask a simple question:

Does this person's badge open every single door along the route?

If yes, they should get in. If any one door refuses, they should be turned away. There is no judgement involved. It is the same logic as a padlocked chain: one locked link stops the whole thing.

Step two is where AI comes in. By this point we already know the right answer for every person and every route. The AI is never asked what should happen. It is only asked to write the actual test that checks it, which means filling in a realistic request with sensible data.

Drawing the Map

To make any of this possible, we first have to figure out which service calls which, automatically, from the source code alone.

The tricky part is spotting the moments when one service reaches across the network to call another. Those calls are the doors nobody tests.

On our test system, this found 399 of 418 such jumps, about 95%. The 19 it missed were cases where the destination is decided while the program is running, so it cannot be worked out just by reading the code. We deliberately leave those marked as unknown rather than guessing, because a confident wrong answer is worse than an admitted gap.

Some of these journeys are surprisingly deep. The longest ran through 17 services before finishing.

What Happened

We tested this on Train-Ticket, a realistic 20-service demo train booking system that researchers use as a standard yardstick. Then we deliberately broke it, planting faulty permission rules the way a careless code change would, and checked whether the tests caught them.

Against other testing tools

EvoMaster is a genuinely good tool. It always aims at the right place and writes sensible checks. It simply has no way of knowing what happens after the first step, so fewer than half its tests line up with a real journey through the system. EvoSuite produced four tests in total, none of them meaningful for this kind of app.

To be fair in the other direction: EvoMaster's tests ran without technical errors 100% of the time, against 97.4% for ours. On raw stability, it beat us.

Against the mathematical approach

There is a rival school of thought that proves security properties mathematically instead of testing them. We compared against one such tool, using 16 planted flaws.

The proof-based tool declared all 16 broken cases secure. Ours caught every one, at the cost of a single false alarm.

That sounds like a rout, and in this test it was. But the fair reading is narrower: a mathematical proof describes what the design should do. It cannot tell you what the software running on real servers, with real configuration and half-finished changes, is actually doing. The two approaches answer different questions, and this result is an argument for using both.

Being honest about the rough edges

Not everything scored 100%. When assigning which roles to test, the system got it right 82% of the time. Coverage of routes and roles was complete, but that role-matching gap is real and we report it as such.

The tests themselves ran cleanly 97.4% of the time, and the whole process takes about 16 seconds per scenario, which is fast enough to run automatically every time code changes.

The Bug We Found by Accident

Beyond the flaws we planted on purpose, the system turned up a real one we did not know about: a service that forgot to pass along proof of identity when calling the next service down the line. Legitimate, properly logged-in users were being turned away deep inside the app.

No tool that stops at the front door could have found that.

It also taught us something honest about the limits of this work. Once you start testing deep inside an app, you run into ordinary bugs as much as security ones. Requests often fail not because permission was refused, but because the data they needed was not there.

What This Means If You Build Software

1. Test the inside doors, not just the entrance. If your security tests all stop at the API gateway, they are testing the lobby.
2. Treat "allowed in, blocked later" as a red flag. It is either a genuine security hole or a feature nobody can actually reach. Both are worth a look.
3. Check what your public endpoints expose. Our crude keyword scan for words like payment and profile still turned up things that should not have been public.
4. A crash is not a rejection. If your tests count any error as "correctly blocked", some of them are passing for the wrong reason.

And the broader lesson about AI: models are far more reliable when you hand them a fact to verify than when you ask them to supply the fact. Pin down the answer with ordinary analysis first, and use the model for the part it is genuinely good at.

What This Does Not Prove

I would rather state the limits plainly than oversell it.

It is one system, and a small slice of it. The results come from 13 entry points on a single test application. Train-Ticket is a respected benchmark, but there is no standard collection of real permission bugs to test against, so we checked every finding by hand.

It only works on one kind of codebase so far. The code reader is built for Java Spring, a common but far from universal way of building services. The underlying idea carries over; the tool would need rewriting for each new language.

It cannot see rules enforced outside the code. Plenty of permission rules live in network infrastructure rather than in the application itself. Those are invisible to this approach.

It does not scale for free. Checking every role against every route across every path multiplies quickly. On a large system you would need to prioritise rather than test everything.

And there is a trust question. Companies are understandably wary of letting an AI model gate their releases, and calling a large model on every code change costs real money. The sensible path is probably to shrink this down into a small, specialised model.

For anyone who wants the full numbers, the exact statistical breakdowns, and the formal write-up, all of it is in the published paper, available from IEEE at doi.org/10.1109/SOSE71128.2026.00014. The prompts, generated tests, and raw results are separately available on Zenodo under an open licence.

If you would like the more technical version of the ideas behind this, my earlier post on graph-based prompting for microservice API testing covers how we make this scale.

Common Questions

What is a microservice?

One small program that does one job inside a bigger app. Rather than building an app as a single large program, teams split it into dozens of small services that talk to each other over a network.

What is an authorization blindspot?

A part of an app where the permission rules disagree between the entrance and the services deeper inside. Because most tools only check the entrance, that mismatch is never tested and stays hidden.

What is authorization drift?

When the permission rules in running software slowly stop matching what the team intended. Each service gets updated on its own schedule, so one team can change or remove a check without anyone else noticing.

Can AI be trusted to write security tests?

Here it is never asked to decide what secure behaviour is. That is worked out in advance by analysing the code. The AI only writes the test that checks an answer we already have, which is a translation job rather than a judgement call.

Citation

If you find this work useful, please cite:

@INPROCEEDINGS{11652948,
  author={Uddin, Md Arfan and Weerasinghe, Shakthi and Wojtak, Connor and Cerny, Tomas
          and Silva-Junior, Deuslirio and Ribeiro, Mateus Eduardo S.
          and Dos Santos Neto, Manoel Ver{\'i}ssimo and Graciano-Neto, Valdemar V.
          and Galv{\~a}o, Arlindo and Abdelfattah, Amr S.},
  booktitle={2026 International Conference on Service-Oriented System Engineering (SOSE)},
  title={Automated Generation of Microservice Authorization Tests Using Large Language Models},
  year={2026},
  pages={41-50},
  doi={10.1109/SOSE71128.2026.00014}}

About This Research

This work was done at the University of Arizona with collaborators at the Federal University of Goiás in Brazil, supported by the National Science Foundation under Grant No. 2409933. All experiments and data processing were carried out by the university collaborators on university-managed resources.

Thanks to my advisor Dr. Tomáš Černý and the research group. The prompts, generated tests, and raw results are published openly on Zenodo.

If you work on this kind of problem, or just want to talk about it, get in touch.