CVE-2017-8540

What Is Microsoft Malware Protection Engine?

The Microsoft Malware Protection Engine (MsMpEng) is the core scanning component of Windows Defender, Microsoft Security Essentials, Microsoft Forefront Endpoint Protection, and other Microsoft security products. MsMpEng runs as a SYSTEM-level service and automatically scans files as they are downloaded, copied, or accessed — without user interaction. Because MsMpEng processes untrusted content with SYSTEM privileges and is always running, memory corruption vulnerabilities in the engine achieve SYSTEM-level code execution just by sending a malicious file to a Windows system. Project Zero's Tavis Ormandy extensively researched MsMpEng vulnerabilities in 2017, reporting multiple critical issues.

Overview

CVE-2017-8540 is a memory corruption vulnerability (out-of-bounds write, CWE-787) in the Microsoft Malware Protection Engine. When MsMpEng scans a specially crafted file, it performs an out-of-bounds write that can lead to remote code execution as SYSTEM. Because Windows Defender automatically scans incoming email attachments, downloaded files, and USB-connected media without user interaction, an attacker can trigger exploitation simply by sending a target a malicious file via email — the act of delivery causes the antivirus to scan and execute the exploit. Fixed via an automatic engine update (version 1.1.13804.0+) deployed through Windows Update without user action. CISA added CVE-2017-8540 to the KEV catalog in March 2022.

Affected Versions

Microsoft Malware Protection Engine versions before 1.1.13804.0 on:

• Windows Defender (Windows 10, Windows 8.1, Windows 7)
• Microsoft Security Essentials (Windows 7)
• Microsoft Forefront Endpoint Protection, Forefront Security for SharePoint
• Microsoft Exchange Server 2013, 2016 (with Exchange malware scanning)

Technical Details

Root Cause: Out-of-Bounds Write During File Scanning

CVE-2017-8540 is an out-of-bounds write vulnerability (CWE-787) in MsMpEng's file parsing code. When the engine scans a specifically crafted file format — likely a malformed executable, document, or script — it writes beyond the bounds of a heap or stack buffer, corrupting adjacent memory. This corruption enables control of instruction execution, leading to code running as SYSTEM (the MsMpEng service user).

No user interaction beyond file receipt: Windows Defender scans email attachments before they reach the inbox (via Exchange integration) and scans downloaded files immediately; just receiving a malicious email or downloading a malicious file triggers the vulnerable scan — the victim does not need to open or interact with the file.

Attack Characteristics

Exploitation Context

• Antivirus as attack surface: Project Zero's research in 2017 demonstrated that complex file parsers in security software running with elevated privileges represent an especially dangerous attack surface — every file processed by a vulnerable AV engine is a potential exploit vector
• SYSTEM privileges without elevation: Unlike most local vulnerabilities requiring privilege escalation, MsMpEng runs as SYSTEM, so exploitation immediately yields maximum privileges on the target machine
• Email-based delivery: Nation-state actors can exploit CVE-2017-8540 by sending a targeted phishing email with a malicious attachment — when the Exchange server or endpoint AV scans the attachment, code executes; this achieves remote code execution without the target opening anything
• CISA KEV (2022): Added March 3, 2022 reflecting confirmed exploitation of MsMpEng vulnerabilities in targeted attacks

Remediation

1. Verify Windows Defender engine is updated — ensure the Microsoft Malware Protection Engine is at version 1.1.13804.0 or later; check Windows Defender in Settings → Windows Security → Virus & Threat Protection → Protection Updates; the engine should auto-update via Windows Update.

2. Do not disable Windows Update or signature updates — Windows Defender engine updates are delivered automatically and do not require full OS updates; ensure Windows Update is enabled and Windows Defender updates are not blocked by enterprise policy.

3. Verify Exchange malware scanning engine — organizations running Exchange Server with built-in malware scanning should verify that the MsMpEng version on Exchange servers is also updated; Exchange engine updates may require separate verification.

4. Keep antivirus engines updated — as a general principle, security product engines that parse untrusted content must be kept on the latest version; MsMpEng and other AV engines have had multiple memory corruption vulnerabilities and auto-update is critical.