Manual vs Automated Analysis: Strengths and Limits

The previous lesson showed what automated scanners do well and where they fall short. This lesson zooms out and gives you a practical framework for deciding, at each stage of a test, how much to automate and how much to do by hand, so that your limited time produces the most value for the client.

Two complementary skills, not two camps

It is tempting to see "manual testing" and "automated testing" as a rivalry, with one side being "the real hacking" and the other "just running a tool". In practice, professional testers use both constantly, often in the same hour, because they answer different questions:

• Automation answers "where might something be wrong, across a large surface, quickly?"
• Manual analysis answers "is this specific thing actually wrong, and what does it really mean for the business?"

A simple decision framework

Ask three questions for any given task during an engagement:

1. Is the surface large and the check repetitive? (for example, checking thousands of URL parameters for a known pattern) Favour automation.
2. Does the check require understanding what the application is supposed to do? (for example, "should this user be allowed to see this data?") Favour manual analysis, since business meaning is exactly what tools cannot infer.
3. Is time limited and the target important? Use automation first to triage quickly, then spend your remaining, more valuable time on manual verification of the most promising and most critical candidates.

Time allocation on a real engagement

A common and reasonable pattern for a time-boxed test (say, five working days) looks roughly like this, and your fil rouge project will follow a similar shape:

Phase	Share of time	Mostly
Reconnaissance	~20%	A mix, with manual judgement on prioritisation
Vulnerability analysis (this chapter)	~20%	Automated scanning, then manual triage
Exploitation and post-exploitation	~40%	Mostly manual, guided by the prioritised list
Reporting	~20%	Entirely manual

These numbers are only a planning guide, not a rule. A simple external black box test with a small scope may spend relatively more time on reconnaissance; a well-scoped grey box test of a single complex application may spend almost all its time on manual exploitation and logic testing.

Signals that tell you to switch from automated to manual mode

• A scanner flags something with a high severity score but a vague description: verify manually before trusting the number.
• The target has custom, bespoke software rather than well-known off-the-shelf products: automated databases of known weaknesses are far less useful, since there is no large body of public research about this specific, unique code. Manual review becomes proportionally more important.
• You are testing a business process (checkout, loan approval, access request) rather than a single technical component: this almost always requires a human to think like both a legitimate user and an attacker at once.
• A finding, if real, would be very high impact (for example anything touching payments or health data): confirm it manually, carefully and conservatively, before it goes anywhere near a report.

Signals that tell you to lean on automation

• A large number of hosts or pages need the same basic check (is a known, patched weakness present; is a default credential still in place).
• The engagement is recurring (for example a quarterly external check), where consistent, repeatable automated scanning lets you compare results over time and quickly spot what changed.
• Time is genuinely too short to review everything manually, and a triaged, imperfect list is better than no list at all, as long as the client understands the trade-off you made and why.

Documenting your method, not just your findings

A report that only lists findings leaves the client unable to judge how thorough the test really was. Professional reports (which you will study in detail in chapter 7) usually include a short methodology section stating which tools were used, which parts were tested manually, and which were out of reach given the time available. This transparency is part of professional honesty: a client should never believe a test was exhaustive when it was, necessarily, time-boxed.

Your fil rouge: planning the rest of Project Lighthouse

Using the candidate list and triage notes from the previous two lessons, write a short plan in your journal: for the remaining chapters of this course (exploitation, passwords, post-exploitation), which of your fictional candidate findings deserve manual, in-depth attention, and which would realistically be "flagged but not pursued further" in a real, time-boxed engagement, with one sentence of justification for each decision.

Your turn: allocate a short engagement

You have exactly two working days to test a mid-sized web application for a client, in a grey box approach with one user account. List, in order, the first four activities you would do, and say for each one whether it is mostly automated or mostly manual, and why.

See a solution

A reasonable plan: (1) quick automated reconnaissance and port scan of the application's infrastructure, mostly automated, to map the surface fast; (2) an automated web vulnerability scan, authenticated with the provided account, mostly automated, to triage obvious candidates quickly; (3) manual testing of access control and business logic around the account's privileges, mostly manual, since this is exactly what scanners cannot assess; (4) manual verification of the two or three highest-severity scanner findings, mostly manual, to confirm they are real before writing the report. With only two days, broad, deep manual review of every page is unrealistic, so automation does the first pass and the limited manual time is spent where it has the most impact.

With a clear method for analysing vulnerabilities, the last lesson of this chapter helps you decide, among everything you found, what to attack first.