Vulnerability Scanning, Penetration Testing and Red Teaming: How They Differ, How Often, and Which to Buy

Three kinds of testing, three different questions

Vulnerability scanning: automated matching against known flaws

A vulnerability scanner identifies hosts, operating systems, applications and open ports, then matches them against the known vulnerabilities in its database to find outdated software versions, missing patches and misconfigurations^[1]. It is fast and repeatable, which makes it the right tool for regularly and comprehensively measuring your exposure to known problems^[1]. When buying, be clear about two dimensions:

• External or internal: an external scan looks in from outside the organization's perimeter (usually the internet), is blocked by firewalls and can only reach the ports allowed through; an internal scan usually finds more vulnerabilities than an external one, and NIST considers both viewpoints important^[1].
• Unauthenticated or authenticated: an unauthenticated scan uses no credentials and sees only what an external attacker without credentials would see; an authenticated (credentialed) scan logs in with a privileged account, can see configurations, installed software and permissions, and reveals many more vulnerabilities^[2].

Scanning also has clear blind spots:

• It only sees systems that are active and relies on a signature database that must be updated before every scan; it has a high false positive rate (reporting vulnerabilities that do not exist) as well as false negatives, so the results need interpreting by someone with networking and OS security expertise^[1].
• It cannot find risks that only emerge when several vulnerabilities are chained: each one may look low risk on its own while the combination is high risk^[1].
• The scanner's rating is not your risk. NIST's example: a scanner may rate an FTP server as moderate risk because it sends passwords in cleartext, but if that server is only an anonymous public server that uses no passwords, the actual risk may be considerably lower^[1].
• Business logic and custom code. OWASP points out that automated tools do not think creatively; an administrative backdoor hidden behind a 30-character parameter value, for example, is practically impossible for a scanner to guess^[3].

Penetration testing: people attacking within an agreed scope

NIST describes penetration testing as security testing in which assessors mimic real-world attacks to find ways around the security features of an application, system or network, often launching real attacks on real systems and data with the tools and techniques attackers commonly use^[1]. The key difference from scanning is that most penetration tests look for combinations of vulnerabilities that can be chained to gain more access than any single flaw would allow; a test can also show how well defenders detect attacks and respond^[1]. A test can simulate an outside attacker (given only target IP ranges and starting from public information) or an insider already on the internal network with ordinary employee access who tries to escalate privileges^[1].

Penetration testing has its own limits:

• It is labor-intensive, needs deep expertise and can damage systems or make them inoperable, a risk that can never be fully eliminated^[1].
• It is a sample. OWASP cites Gary McGraw's view that a penetration test can only identify a small representative sample of all possible security risks, and because web applications are almost always custom-built, web application penetration testing is closer to research^[3].
• It does not replace scanning. The PCI Security Standards Council (PCI SSC) states plainly that a penetration test is not a vulnerability scan: the two complement each other, and the former does not replace the latter^[2].

Red teaming: objective-driven, testing detection and response

In the definition cited by the NIST glossary, a red team exercise reflects real-world conditions and is conducted as a simulated adversarial attempt to compromise organizational missions or business processes, giving a comprehensive assessment of the security capabilities of the organization and its systems^[4]; the red team's purpose is to demonstrate the impact of successful attacks and what actually works for defenders in an operational environment^[5]. In other words, a red team's scope is usually an objective (for example, obtaining payment authority in the finance system), not a list of IP addresses. What NIST SP 800-115 calls covert testing is close to this: only senior management knows about it, IT staff do not, and it tests technical controls, how IT staff respond to perceived incidents, and how well staff follow security policy; its focus is the damage an adversary could cause, so it does not test every control or identify every vulnerability^[1]. Because testers have to slow down to stay under the radar, covert testing is often time-consuming and costly^[1].

CISA has published the story of one red team assessment: in 2023 it ran a no-notice, long-term simulation against a US federal civilian agency. The red team gained initial access by exploiting a known vulnerability in an unpatched web server and, separately, through phishing; it went on to compromise the entire domain and pivoted through trust relationships to an external partner organization. Throughout the first phase, network defenders did not detect the red team^[6]. No scan report can give you that kind of finding: it measures how long it takes you to notice once someone is already inside.

Security health check: Taiwan's regulation-defined review

The security health check (資安健診) is a required item in the annexes to Taiwan's Regulations Governing the Classification of Cyber Security Responsibility Levels. It covers network architecture review, network malicious activity review, client endpoint malicious activity review, server malicious activity review, and directory service and firewall connection configuration review; levels A and B also include a database security review of core information systems^[7]. It looks at architecture, configuration and whether malicious activity is already present in the environment, while vulnerability scanning and penetration testing are listed separately in the annexes under "security testing"^[7], so one cannot stand in for the other. For a periodic review of overall architecture, configuration and processes, see our security health check service.

Which to choose, and how often

Start with the question you need answered

• "Which known vulnerabilities do we have, and have we fixed them?" Choose vulnerability scanning, run regularly and comprehensively.
• "Can an attacker get from this entry point to our core systems?" Choose a penetration test with a clear scope, carried out by people.
• "If we are breached, will we notice, and can we stop it?" Choose a red team exercise, but only once you have monitoring and response capabilities worth testing.

NIST recommends regularly scheduled network and vulnerability scanning interspersed with periodic penetration testing; because penetration testing is costly and carries real risk to systems, NIST considers an annual test may be sufficient^[1]. If you do not yet have an asset inventory and routine scanning, build those first and consider a red team later: SP 800-115 notes that covert testing normally identifies and exploits the most rudimentary vulnerabilities to gain access^[1], so you would be paying a lot to learn something routine scanning could have told you.

Minimum frequencies under Taiwan's responsibility level regulation

Government agencies and specific non-government agencies covered by Taiwan's Cyber Security Management Act must carry out the items in the regulation's annexes according to their responsibility level; the current version was amended and promulgated on January 7, 2026^[7]. Government and specific non-government agencies at the same level have the same frequencies^[7]:

• Level A: vulnerability scanning of all core information systems twice a year and penetration testing once a year; a security health check once a year.
• Level B: vulnerability scanning of all core information systems once a year and penetration testing once every two years; a security health check once every two years.
• Level C: vulnerability scanning and penetration testing of all core information systems once every two years; a security health check once every two years.
• Level D: none of the three; only antivirus software, a network firewall and security training are required.

The annexes also state that these items start from the year after initial approval, a change of level or a change to core information systems; specific non-government agencies may instead use other measures of equal or greater effect for the health check if the central competent authority for their sector approves them^[7]. For who is covered and other obligations, see Taiwan's Cyber Security Management Act.

Private companies that are not on the list may still take on testing obligations through outsourcing contracts. Article 7 of the Act's Enforcement Rules requires a contractor that develops a custom information system to provide proof of security testing; where the system is a core information system of the commissioning agency, or the contract is worth NT$10 million or more, the agency must test it itself or commission a third party to do so^[8].

PCI DSS: organizations handling payment card data

Requirements 11.3 and 11.4 of PCI DSS version 4 set out the following^[9]:

• Internal vulnerability scans at least once every three months and after any significant change; high-risk and critical vulnerabilities must be resolved and rescans must confirm it^[9].
• Internal scans must be authenticated, and systems that cannot accept credentials must be documented^[9]; PCI SSC notes this was new in v4.x and became mandatory after March 2025^[2].
• External vulnerability scans at least once every three months by a PCI SSC Approved Scanning Vendor (ASV)^[9]; a passing scan means all vulnerabilities with CVSS scores of 4.0 through 10.0 are resolved^[2]. PCI SSC also strongly encourages scanning more often than every three months^[2].
• Internal and external penetration tests at least once every 12 months and after any significant infrastructure or application upgrade or change, performed by a qualified internal resource or qualified external third party with organizational independence^[9].

Triggers beyond regulation

Regulation sets a floor. CISA's Binding Operational Directive 23-01 requires US federal civilian agencies to perform automated asset discovery every 7 days and vulnerability enumeration across all discovered assets every 14 days, using privileged credentials wherever possible^[10]. By comparison, a scan every two years may satisfy an audit but falls far short of what day-to-day patching needs. Beyond the fixed schedule, plan testing before new systems go live, after major releases or architecture changes (PCI DSS also treats significant changes as a trigger for scans and penetration tests^[9]), when you add internet-facing services, after mergers or acquisitions, and after recovering from a security incident.

Before you hire: scope and rules of engagement

NIST defines rules of engagement (ROE) as detailed guidelines and constraints for executing a security test, established before testing starts, which give the test team authority to carry out the defined activities without needing further permission^[1]. The ROE template in an appendix of SP 800-115 works well as a checklist for a procurement specification^[1]:

• Targets and exclude list: list the IP addresses, domains and applications authorized for testing, and the systems explicitly not to be tested^[1]. If you are given a list of public IP addresses, confirm before testing that they really belong to the organization^[1]; check each one with extra care in cloud environments where public IPs change.
• Test type and timing: external or internal, authenticated or not, whether test accounts are provided, and whether to simulate an outside attacker or an insider; production systems may need to be tested outside peak business hours^[1].
• Allowed and disallowed activities: for example, whether social engineering or physical intrusion is allowed, whether testers may create, modify or execute files on systems, and what happens to those files afterwards^[1]. Scanners' denial-of-service tests can be suppressed as a group to reduce the risk of affecting hosts^[1].
• Stop criteria and emergency contacts: when to halt and whom to notify if testing affects a system or a real attack happens during the test; include the incident response call tree^[1].
• Data handling: how test data is collected, stored, transmitted and destroyed; test results are exactly what an attacker would want and should be treated as sensitive^[1].
• Reporting and sign-off: the minimum content of each report and how often reports are delivered (for example, daily status reports); at a minimum the test team leader and the organization's senior management sign to confirm they understand the scope and boundaries^[1].

Also require the vendor to log every test with a timestamp, test type, tool, commands and the IP address of the testing equipment^[1], so that if something breaks you can tell whether the test caused it. For red team exercises, a trusted third party can act as intermediary so that when defenders spot suspicious activity they can first verify whether it is a test before launching a response^[1].

Accepting the report: telling good from bad

OWASP says a good report should work for both executive management and technical staff^[11]. Check it item by item:

1. Scope and limitations are stated: the report should give the scope, the test period and limitations such as lack of access or credentials, lack of time, or broken functionality^[11]. Results reflect a single point in time and do not guarantee that every issue was found^[11]. "No high-risk findings" without a stated scope and limitations tells you nothing.
2. False positives have been removed: findings from automated tools usually need validation to weed out false positives, and manual examination is more accurate than cross-checking with a second tool^[1]. OWASP also says tool output in appendices should be cleaned up rather than dumped^[11]; if the whole report is scanner output, this step was skipped.
3. Evidence is reproducible: each finding should include everything needed to understand, reproduce and fix it, including a reference ID, exploitability, impact, how to exploit it and remediation steps, with sensitive data masked^[11]. Your engineers should be able to follow the steps and reproduce it.
4. Risk ratings are tied to business impact: scanner ratings should not be accepted as-is^[1]. The report should weigh how easy exploitation is, whether working exploit code exists, the access required and the impact on the system, and explain in an appendix how each rating is determined; the executive summary should describe impacts such as compliance issues or reputational damage in business terms^[11].
5. Root causes and remediation advice are included: every finding should come with root cause analysis and mitigation recommendations, both technical (for example, applying a particular patch) and process-related (for example, updating the patch management process)^[1].
6. A retest is included: PCI DSS requires exploitable vulnerabilities found in penetration testing to be corrected according to their risk and the test to be repeated to verify the corrections^[9]; a retest report should also summarize the current status of previous findings^[11]. Write at least one retest into the contract.

Feeding results into remediation

Test results serve as a reference point for corrective action and a benchmark for tracking progress^[1]. Delivery of the report is not the finish line. Next:

• Load findings into your asset inventory and ticketing system: give every finding an owner and a remediation deadline, and merge and de-duplicate it with existing scan results.
• Prioritize by exposure and exploitation intelligence: PCI SSC also stresses that severity is not risk without context and recommends combining KEV and EPSS^[2]. For a full prioritization method and remediation deadlines, see Patch Management: How to Prioritize When You Can't Fix Everything; for what the scores mean, see Reading Vulnerability IDs and Scores.
• Verify fixes: follow the PCI DSS approach of rescanning to confirm high-risk and critical vulnerabilities are resolved and handling the rest according to a targeted risk analysis^[9]; a closed ticket is not proof of a fix.
• Turn red team results into detection: CISA's red team assessment found that a "known-bad" detection approach caused defenders to miss alternative techniques; the assessed organization only understood the full extent of the compromise after analyzing host, internal network, egress network and authentication logs, and CISA recommends primarily behavior-based detection rather than relying on indicators of compromise (IOCs)^[6]. For how to use IOCs and their limits, see How to Check Your Environment Against Advisory IOCs.

If you lack the in-house staff to interpret scan results, weed out false positives and set remediation priorities, see our vulnerability assessment service.

Action checklist

• Pull out the test reports from the last three years and confirm whether each was really a scan, a penetration test or a health check, which systems it covered and whether it included a retest.
• Determine whether the responsibility level regulation, outsourcing contracts or PCI DSS apply to you, and list the required frequency and next due date for each type of test.
• Set up an internal and external scanning schedule more frequent than regulation requires, switch internal scans to authenticated scanning, and keep a register of systems that cannot accept credentials.
• Before commissioning a penetration test, write rules of engagement from the SP 800-115 template: targets, exclude list, allowed activities, timing, stop criteria, contacts and sign-off.
• Require in the contract that reports include scope and limitations, reproduction steps and evidence, risk ratings tied to business impact, root causes and remediation advice, and at least one retest.
• Schedule testing before new systems go live and after significant changes, not only on the annual calendar.
• Run a red team exercise only once you have monitoring and response in place, and afterwards turn every undetected technique into a detection rule.
• Follow vulnerability advisories and make actively exploited vulnerabilities a focus of your next scan and test.