Automated ECU Security Testing vs Manual Penetration Test

What is the difference?

Automated ECU security testing is software on the bench asking an ECU thousands of questions whose answer pattern is already known: which sessions open, which of the 255 service identifiers answer in each one, which DIDs are readable or writable, whether the SecurityAccess seed repeats, what happens to the ECU when an ISO-TP first frame claims an impossible length. A manual penetration test is an engineer asking questions nobody has written down: why does this routine exist, what does the bootloader accept after a reset, can the calibration interface be combined with a writable DID into a persistent change.

The two are not competitors. Automation produces coverage and evidence on every firmware version. The pentester produces understanding and finds the attack path that crosses components. A good ECU penetration test already contains both: the tester runs enumeration and fuzzing in the background and spends the paid days where tools are blind.

Side by side

	Automated ECU testing	Manual penetration test
What it finds	Services reachable in the wrong session, readable and writable DIDs, weak or constant seeds, missing attempt counters, parser crashes and hangs under fuzzing, exposed XCP/CCP, DoIP routing to blocked ECUs, open SOME/IP services, ADB and debuggable apps on IVI units	Logic flaws in the diagnostic state machine, seed/key algorithms recovered from firmware, bootloader and update weaknesses, chains across interfaces, design errors in the security concept, anything novel
What it misses	Context, intent, chained attacks, firmware internals, anything without a pattern	Breadth at scale, regressions between tests, long fuzzing runs; limited by the booked days and the agreed scope
Who does it	Your test or QA engineers, with the ECU on a bench	External or internal security engineers with a scope and rules of engagement
Bench time	Minutes for enumeration, hours for fuzzing, unattended	Two to six weeks per ECU including reporting
Cost	Licence per tested component, price on request; the bench hardware you already have	20,000 to 45,000 euros per ECU, typical range for Germany and the EU, not an offer
Fits a release cycle	Yes, on every build via API or CI plugin	No, a point in time, usually once per platform generation and after major changes
Required by	ISO/SAE 21434 Clause 10.4.2 [RC-10-12] recommends component testing with fuzzing and vulnerability scanning; UN R155 asks for appropriate and sufficient testing	ISO/SAE 21434 Clause 11 [RQ-11-01] names penetration testing for validation; OEM requirement sets and customer contracts
Output	PDF report per run, SARIF export, scan history, a ranked Fix Plan	A report with reproduction steps, severity in context, attack chains and fixes; a debrief and a retest

Choose automated testing when

• You ship the same ECU in many variants or release firmware several times a year. Running the same checks by hand each time is slower, less consistent and nobody does it after the second release.
• The weakness class is one that tools are good at: sessions and services, DIDs, SecurityAccess counters and delays, robustness under malformed UDS and CAN traffic, DoIP routing, SOME/IP exposure, Android hardening.
• You need evidence more than a story. An auditor asking “what did you test, when and what happened to each finding” is better served by dated scan history and a SARIF file than by a PDF from two years ago.
• You have no budget for a pentest yet. Fix what automation finds first; a tester who spends paid days on a writable serial number DID is a waste of a pentest.

For regression testing and protocol robustness, automation is simply the better choice.

Choose a manual penetration test when

• The platform is new: a new microcontroller family, a new bootloader, a new diagnostic stack, a first Ethernet ECU. Somebody has to understand it before anything about it can be automated.
• The security goals involve logic: who may unlock which level, what the update path trusts, how the gateway decides what to route. No tool knows what is supposed to be allowed.
• A customer, an OEM or an assessor asks for a penetration test with a named tester and a method description. That is what Clause 11 of ISO/SAE 21434 means by validation.
• Findings have to be chained and judged. Automation reports a writable DID and a weak seed as two findings; a tester shows that together they give persistent code execution.

Both together

The sensible setup is automated testing on every build plus a manual penetration test per platform generation and before start of production, with the automated results handed to the tester. The tester skips the obvious, the days go into firmware, bootloader and chains, and the automated suite keeps watch between tests so the pentest report does not quietly expire.

Zyberum builds AutoST and still puts an engineer on every new platform. We are specific about what the product does: AutoST finds the known, the regressions and the crashes and writes the evidence. It is not a penetration test and we do not sell it as one. If a vendor promises you a full ECU pentest from a tool alone, ask who read the results.