On this page
Controlled, automated penetration testing should be a normal part of a security program. Use it for repeatable validation as systems change, and spend specialist time on the questions that need deeper investigation. The buying decision is about how those two kinds of work fit together.
Start with the release calendar
Put the date of the last penetration test next to the releases and infrastructure changes made since it finished. Which of those changes have been tested? Which are waiting for the next engagement? The answers are a good starting point for deciding whether the current program keeps pace with the business.
An annual test may have been well scoped and well executed. Trouble starts when its report is still cited as grounds for confidence in an environment that has changed materially. The NCSC says plainly that the confidence a point-in-time assessment gives you decays over time, because code keeps evolving and new attacks and vulnerabilities are discovered.
The UK Government Service Manual’s vulnerability and penetration testing guidance recommends testing frequently as services are built, automating as much testing as possible to find basic vulnerabilities, and using exploratory testing to find weaknesses that more advanced attackers could exploit. It is written for government services, and the combination works well elsewhere too.
We would make automated penetration testing the baseline for repeatable, authorized validation within an agreed scope. That is an operating recommendation. Regulators have not adopted it as a new standard, and automated products differ widely in what they cover. The program still needs specialist testing and any independently required assessments.
Ask what the service tests
“Automated penetration testing” is a broad label, so buyers need to establish what a proposed service does. Discovering an asset, detecting a possible vulnerability and demonstrating a permitted action are different capabilities. A product may offer some of them or all of them.
A scanner is still useful. It gives a team a repeatable way to identify potential weaknesses across many systems. Controlled exploit validation asks a different question: can this particular weakness be used under the conditions tested? Neither result shows that every relevant attack has been tried. Our post on exposure management and vulnerability management covers the same distinction from the prioritization side.
Ask for a demonstration in an agreed environment, then inspect the evidence it produces. Which identities and credentials does it need? Which platforms and vulnerability classes can it assess, and what is excluded? Can an engineer reproduce the finding? Can you tell a completed test from an inference or a failed run?
Keep those answers in the buying decision, next to the price. The NCSC’s guidance on vulnerability scanning tools and services says automated scanning cannot match manual penetration testing for coverage, and positions scanning as a cost-effective way to catch common issues so that testers can focus on harder problems. A new product name doesn’t remove the need to check what is covered.
Build safety into the commissioning decision
A useful automation program starts with permission and boundaries. Agree the systems, identities, environments and techniques in scope. Identify shared infrastructure and supplier services before anyone tests them. The GOV.UK guidance says teams must get explicit consent from any third-party supplier whose software they use, and should check the contract to confirm it.
Production testing deserves an operational conversation. Which actions could interrupt a service, lock an account, create a transaction or expose sensitive records? Where must the tool stop? Who can pause it, and how quickly? These questions belong in the agreed rules of engagement.
We prefer a staged rollout. Establish how the tool behaves in a representative environment, start with a limited production scope where appropriate, then expand against agreed acceptance criteria. Keep an accountable service owner involved throughout. When a supplier calls its testing safe, treat that as the opening of the discussion.
Evidence needs protection as well. Findings can reveal privileged routes and sensitive configuration, so agree access, retention, redaction and deletion alongside the test itself. Check what happens to temporary accounts and any artifacts when the engagement ends. CREST’s guide to penetration testing treats identifying testing constraints and implementing management control processes as steps in a penetration testing program.
Use specialist time where context matters
Automation should change how experienced testers spend their time. Give them the map, the repeatable findings and the known gaps before they arrive. Then commission work that needs deeper investigation, an understanding of the business process or a fresh look at the architecture.
Business-logic testing is a good example. The OWASP Web Security Testing Guide’s introduction to business logic describes this kind of testing as a manual art that relies on the tester’s skill and knowledge of the complete business process and its rules. Consider an illustrative refund workflow. Each function might look sound on its own, while one particular sequence lets a user get a refund and keep a benefit that should have been withdrawn. Finding that inconsistency takes someone who understands what the transaction is meant to achieve.
Once a team understands an established abuse case, it can often turn it into a repeatable regression check. Discovering an unfamiliar abuse case is a different job. Ask how much informed exploration an engagement includes and who will carry it out.
Keep specialist coverage for critical applications, significant architectural changes and open questions that automation doesn’t answer. Preserve any independently required testing arrangements. The practical aim is a program where routine validation and skilled investigation make each other more useful.
Let change trigger the next test
A testing schedule needs a planned cadence and specific change triggers. A new public service, an identity configuration change, an API integration or a major release may justify a fresh look at scope and new testing. The right depth depends on the service and on what changed. The GOV.UK guidance asks teams to assess security regularly, especially during major changes to the codebase, such as introducing a new dependency or integration.
Agree those triggers with engineering and operations. Without them, a platform can run continuously against yesterday’s asset list while the organization assumes everything important is covered. Ownership of scope changes should be as clear as ownership of findings.
Start with one consequential service. Document the baseline and run the program through a meaningful change. Watch how quickly the new exposure becomes visible, whether the result reaches the right owner and whether the team can act on it. That tells procurement more than a feature comparison.
Measure how the program handles change: how long an agreed asset waits to be assessed, which planned tests completed and which could not run, and whether the evidence reached its owner. Track unresolved material findings and retest outcomes too. A rising test count can hide gaps when scope or completion is unclear.
Agree closure before the first finding
A test becomes useful when someone owns the response. Agree urgency, escalation and closure evidence before commissioning it. A finding may need immediate containment, a permanent fix or an explicit risk decision. Don’t hold back urgent mitigation when the available evidence already justifies action.
Price the follow-through along with the initial run. Ask what triggers another charge, who investigates an unclear result and how failed or partial retests are handled. A proposal that looks inexpensive can turn out hard to use if those responsibilities are left open.
Use the commissioning sheet below to compare proposals against the service you need to protect. Ask each provider to take one finding all the way through to retest. Record the gaps that remain and decide who will cover them.
A commissioning sheet for ongoing testing
Complete this for one service before choosing a provider. Require an answer or a stated exclusion for every row, and don’t accept a broad capability label as evidence.
| Decision | What to agree | What to inspect |
|---|---|---|
| Coverage | Assets, identities, business functions, environments and exclusions | A scope map and the tests it supports |
| Authority and safety | Permissions, production limits, stop conditions and escalation contact | Rules of engagement and the pause procedure |
| Cadence and change | Scheduled checks, change triggers and time allowed for reassessment | A sample record of new scope entering the testing cycle |
| Depth and gaps | Automated work, specialist investigation and required independent assessments | Example evidence, handling of incomplete tests and responsibility for gaps |
| Closure and cost | Remediation owner, retest terms, evidence retention and additional charges | A finding linked to its fix and retest result |
Agree the acceptance criteria before a pilot. Afterward, review what was covered, what was learned and what is still unresolved.
Where Panop fits
Panop’s autonomous penetration testing brings ongoing discovery, controlled validation and retesting into one workflow. Its testing matrix sets out, for each vulnerability class, how Panop confirms it and what evidence it records. Use that as the starting point for a scoped demonstration. Ask what it can validate in your environment, what access it needs, and which questions remain for specialist investigation.
Sources
- NCSC: Continually test your security, February 20, 2019
- NCSC: Vulnerability scanning tools and services, January 19, 2021
- GOV.UK Service Manual: Vulnerability and penetration testing, updated October 23, 2024
- CREST: A Guide to Penetration Testing, December 2022
- OWASP: Web Security Testing Guide v4.2, Introduction to Business Logic