New Intel Vulnerability Foreshadow L1 Terminal Fault VMware Affected

0

Yet another security vulnerability has been discovered that is based on the Spectre and Meltdown attacks found in Intel processors. The new exploit is called Foreshadow and exploits Intel’s SGX technology which is ironically designed to protect sensitive data such as credit card numbers, social security numbers, etc from alteration or unauthorized outside access. This is supposed to protect the data from malware or attackers in general. What is concerning as well is that virtual machines running on enterprise hypervisors can be vulnerable. Attackers can compromise one virtual machine and then read memory from another virtual machine hosted on the same server. Specific to VMware, let’s take a look at New Intel Vulnerability Foreshadow L1 Terminal Fault VMware Affected and see what steps VMware has already take to mitigate the flaw.

New Intel Vulnerability Foreshadow L1 Terminal Fault VMware Affected

Intel has disclosed a new type of CPU speculative-execution vulnerability that is collectively known as L1 Terminal Fault or “Foreshadow” that has vulnerable any past and current Intel processor from at least 2009-2018.  The vulnerability happens when affected Intel microprocessors speculate beyond the authorized data access.  This allows Intel microprocessors to expose a new side-channel for attack.  This most importantly affects virtual machines running in hypervisor environments such as VMware.  This allows a malicious virtual machine running on an affected CPU to speculate contents of the hypervisor or another VM’s privileged information that resides at the same time in teh same core’s L1 Data cache.

Below are the supported VMware Intel processors affected:

Intel Code Name FMS Intel Brand Names
Nehalem-EP 0x106a5 Intel Xeon 35xx Series;
Intel Xeon 55xx Series
Lynnfield 0x106e5 Intel Xeon 34xx Lynnfield Series
Clarkdale 0x20652 Intel i3/i5 Clarkdale Series;
Intel Xeon 34xx Clarkdale Series
Arrandale 0x20655 Intel Core i7-620LE Processor
Sandy Bridge DT 0x206a7 Intel Xeon E3-1100 Series;
Intel Xeon E3-1200 Series;
Intel i7-2655-LE Series;  Intel i3-2100 Series
Westmere EP 0x206c2 Intel Xeon 56xx Series;
Intel Xeon 36xx Series
Sandy Bridge EP 0x206d7 Intel Pentium 1400 Series;
Intel Xeon E5-1400 Series;
Intel Xeon E5-1600 Series;
Intel Xeon E5-2400 Series;
Intel Xeon E5-2600 Series;
Intel Xeon E5-4600 Series
Nehalem EX 0x206e6 Intel Xeon 65xx Series;
Intel Xeon 75xx Series
Westmere EX 0x206f2 Intel Xeon E7-8800 Series;
Intel Xeon E7-4800 Series;
Intel Xeon E7-2800 Series
Ivy Bridge DT 0x306a9 Intel i3-3200 Series; Intel i7-3500-LE/UE, Intel i7-3600-QE,
Intel Xeon E3-1200-v2 Series;
Intel Xeon E3-1100-C-v2 Series;
Intel Pentium B925C
Haswell DT 0x306c3 Intel Xeon E3-1200-v3 Series
Ivy Bridge EP 0x306e4 Intel Xeon E5-4600-v2 Series;
Intel Xeon E5-2400-v2 Series;
Intel Xeon E5-2600-v2 Series;
Intel Xeon E5-1400-v2 Series;
Intel Xeon E5-2600-v2 Series
Ivy Bridge EX 0x306e7 Intel Xeon E7-8800/4800/2800-v2 Series
Haswell EP 0x306f2 Intel Xeon E5-2400-v3 Series;
Intel Xeon E5-1400-v3 Series;
Intel Xeon E5-1600-v3 Series;
Intel Xeon E5-2600-v3 Series;
Intel Xeon E5-4600-v3 Series
Haswell EX 0x306f4 Intel Xeon E7-8800/4800-v3 Series
Broadwell H 0x40671 Intel Core i7-5700EQ;
Intel Xeon E3-1200-v4 Series
Avoton 0x406d8 Intel Atom C2300 Series;
Intel Atom C2500 Series;
Intel Atom C2700 Series
Broadwell EP/EX 0x406f1 Intel Xeon E7-8800/4800-v4 Series;
Intel Xeon E5-4600-v4 Series;
Intel Xeon E5-2600-v4 Series;
Intel Xeon E5-1600-v4 Series
Skylake SP 0x50654 Intel Xeon Platinum 8100 (Skylake-SP) Series;
Intel Xeon Gold 6100/5100 (Skylake-SP) Series
Intel Xeon Silver 4100, Bronze 3100 (Skylake-SP) Series
Broadwell DE 0x50662 Intel Xeon D-1500 Series
Broadwell DE 0x50663 Intel Xeon D-1500 Series
Broadwell DE 0x50664 Intel Xeon D-1500 Series
Broadwell NS 0x50665 Intel Xeon D-1500 Series
Skylake H/S 0x506e3 Intel Xeon E3-1500-v5 Series;
Intel Xeon E3-1200-v5 Series
Kaby Lake H/S/X 0x906e9 Intel Xeon E3-1200-v6

 

Current Intel processors share the physically-address L1 Data Cache across both logical processors of a Hyperthreaded enabled core, simultaneous scheduling of software threads on both logical processors can create the potential for security compromise and the stealing of information.

There are two attack vectors that are described here:

  • Sequential-context – A malicious virtual machine can infer contents of L1 data of previous thread
  • Concurrent-context  – A malicious virtual machine can infer contents of L1 data of concurrently executing thread

Below is the current information regarding the Foreshadow Intel flaw from a VMware perspective.

VMware has already outlined details of the flaw in the official KB article covering the security advisories here:

The L1 Terminal Fault is covered in the following summaries:

Mitigation of CVE-2018-3646 requires Hypervisor-Specific Mitigations for hosts running on Intel hardware.
Mitigation of CVE-2018-3620 requires Operating System-Specific Mitigations.
CVE-2018-3615 does not affect VMware products and/or services. See KB54913 for more information.

VMware Mitigate of Foreshadow L1 Terminal Fault Vulnerability

VMware outlines a two-fold approach to mitigating the new security vulnerability:

  • vSphere updates and patches
  • Enabling a new feature known as “ESXi Side-Channel-Aware Scheduler”

What does the ESXi Side-Channel-Aware Sheduler do?  Well in basic terms, it will only schedule the hypervisor and Vms on one logical processor of an Intel Hyperthreading-enabled core.  This means the ESXi scheduler will not make use of all the hyperthreading cores presented.  The feature has made mention that it will impose a non-trivial performance impact.  You can imagine that only scheduling of ESXi and VMs on a number of cores will certainly lower the performance of virtual machines as they won’t be able to take advantage of all the hyperthreaded cores available.

VMware has outline a (3) phase deployment of the mitigation process:

  • Update Phase –
  • Planning Phase
  • Scheduler-Enablement Phase
Phased-Mitigation-approach-for-VMware-Intel-Foreshadow-L1-Terminal-Fault New Intel Vulnerability Foreshadow L1 Terminal Fault VMware Affected

Phased Mitigation approach for VMware Intel Foreshadow L1 Terminal Fault (image courtesy of VMware)

The two attack vectors mentioned above are mitigated in the following ways:

  • Sequential-context – mitigated by vSphere patches
  • Concurrent-context  –  mitigated by ESXi Side-Channel-Aware Scheduler

Update Phase:

With the update phase, the sequential-context attack vector is mitigated by simply applying the relevant vSphere patches.  A note here is the mitigation releases for Foreshadow are dependent upon Intel microcode updates that have already been released as vSphere update patches which are documented here:  VMSA-2018-0020

The workflow is basic update flow, however, it needs to be mentioned:

Updating ESXi before vCenter may result in very cryptic notification messages.

Planning Phase

With the planing phase, you start to plan on implementing the Concurrent-context attack vector mitigation.  With this phase, VMware has introduced an HTA Mitigation Tool that has as its purpose the complete mitigation of the L1 Terminal Fault (L1TF) vulenerability found in CVE-2018-3646.  The HTA Mitigation Tool requires enabling the ESXi Side-Channel-Aware Scheduler.  The scheduler is not enabled by default.

The HTA Mitgation Tool is available for download at the bottom of the page found here:

With the assistance of the HTA Mitigation Tool, the planning phase helps to establish an understanding of the current environment and if sufficient CPU capacity is available to enable the scheduler without operational impact.

As already mentioned, disabling hyperthreaded cores from the scheduler can certainly lead to a performance impact, but also other areas of impact to be aware of before enabling including:

  • Any virtual machine that has more vCPUs assigned than the number of physical cores on the ESXi host
  • Any virtual machine that may have custom affinities assigned or NUMA settings
  • VMs that are latency sensitive
  • ESXi hosts that have 70% or higher CPU utilization
  • ESXi hosts with custom CPU resource options
  • HA Clusters where rolling upgrades would result in higher than 100% CPU usage
Running-the-VMware-HTA-Mitigation-Tool-scanning-for-Foreshadow-Mitigation New Intel Vulnerability Foreshadow L1 Terminal Fault VMware Affected

Running the VMware HTA Mitigation Tool scanning for Foreshadow Mitigation

Scheduler-Enablement Phase:

In this phase, the actual mitigation of the concurrent-context vulnerability is implemented.  There are multiple ways to implement the ESXi Side-Channel-Aware Scheduler:

  • Set the hyperthreadingMitigation advanced setting on a ESXi host
  • vSphere Web Client or vSphere Client
    • Connect to the vCenter Server using either the vSphere Web or vSphere Client.
    • Select an ESXi host in the inventory.
    • Click the Manage (5.5/6.0) or Configure (6.5/6.7) tab.
    • Click the Settings sub-tab.
    • Under the System heading, click Advanced System Settings.
    • Click in the Filter box and search VMkernel.Boot.hyperthreadingMitigation
    • Select the setting by name and click the Edit pencil icon.
    • Change the configuration option to true (default: false).
    • Click OK.
    • Reboot the ESXi host for the configuration change to go into effect.
  • ESXi Embedded Host Client
    • Connect to the ESXi host by opening a web browser to https://HOSTNAME.
    • Click the Manage tab
    • Click the Advanced settings sub-tab
    • Click in the Filter box and search VMkernel.Boot.hyperthreadingMitigation
    • Select the setting by name and click the Edit pencil icon
    • Change the configuration option to true (default: false)
    • Click Save.
    • Reboot the ESXi host for the configuration change to go into effect.
  • ESXCLI command line utility
    • esxcli system settings kernel list -o hyperthreadingMitigation
    • esxcli system settings kernel set -s hyperthreadingMitigation -v TRUE

Takeaways

The New Intel Vulnerability Foreshadow L1 Terminal Fault VMware Affected is certainly a developing story.  VMware has reacted quickly and provided an interesting tool in the HTA Mitigation Tool.  There is one thing for sure, the Spectre and Meltdown fallout is continuing to add to the saga of worse performance.  With the mitigation for the concurrent-context, the utility enables scheduling of only one logical processor (Hyper-Thread) of a Hyper-Thread-enabled core.  This means as VMware stated, the performance impact will be non-trivial meaning it will certainly impact performance.  Keep an eye on this post as I will keep it updated if any new information comes to light.