چگونه سرورهای ESXi سازمان خود را در مقابل حملات باج افزاری تحت Shell یا SSH ایمن نمائیم ؟1401-04-22

برگرفته از کانال کامیران: با توجه به گسترش حملات باج افزاری تحت ESXi در پست بعدی به نکات امنیتی این بستر مجازی سازی خواهیم پرداخت.

اولین اصل امنیت ESXi همان Vlan بندی است ! یعنی IP مربوط به ESXi در Vlan مجزایی از سرورها و کلاینت ها قرار گرفته باشد و تنها سیستم های مشخصی به IP اصلی ESXi و سامانه های مدیریتی آن دسترسی داشته باشند. نباید کلیه سیستم ها و سرورها به IP اصلی ESXi دسترسی داشته باشند. همچنین هیچ یک از پورتهای مدیریتی ESXi روی اینترنت پابلیش نشود.

▪️ سرویس های ESXi Shell و SSH باید در حالت عادی غیر فعال باشد و فقط در مواقع نیاز فعال شده و بلافاصله غیر فعال گردد. خط فرمان محلی ESXi میباشد و SSH این خط فرمان را به صورت ریموت در اختیار شما قرار میدهد. پورت پیش فرض SSH روی 22 است لذا اگر پورت 22 سرور ESXi شما باز است یعنی SSH سرور فعال میباشد ( راهنمای این کار)

▪️ گزینه Lockdown Mode را در ESXi خود فعال نمائید . در این حالت تمام دسترسی های راه دور (Remote) به هاست ESXI مسدود می شود و تنها از طریق vSphere vCenter , DCUI می توانید به ESXi دسترسی داشته باشید.

▪️ از اکانت های مدیر شبکه دامین برای ورود و دسترسی به ESXi یا vSphere استفاده نکنید. در حملات اتفاق افتاده ، هکرها دسترسی Administrator را در اختیار دارند و به راحتی وارد ESXi خواهند شد.

▪️ از پسورد های ساده برای اکانت root سرورهای ESXi استفاده ننمائید و اصول امنیتی پسورد های قوی را رعایت نمائید.

▪️ سرورهای vCenter، ESXi و VMware Tools را به محض انتشار پچ های امنیتی ، آپدیت و وصله نمائید.

▪️ در حملات باج افزاری تحت ESXi فقط و فقط آپدیت آفلاین میتواند شما را از جهنم ایجاد شده رهایی دهد. لذا سامانه های بکاپ آفلاین خود را تقویت نموده و صحت بکاپ های گرفته شده را بررسی نمائید.

▪️ در این سبک حملات، هکر خود را به هر روشی به داخل شبکه میرساند و سپس با اسکن شبکه و پورتهای باز سرورها ، ESXi شما را یافته و برای ورود به آن تلاش میکند . هکرها برای به دست آوردن پسورد ESXi از نرم افزارهایی مانند Mimikatz در سیستم های تحت کنترل خود استفاده میکنند.

▪️ روش های نفوذ اولیه هکر در شبکه میتواند یکی از موارد زیر باشد :

باز بودن پورت های آسیب پذیر و قابل نفوذ روی اینترنت همانند RDP,SQL,Oracle,Exchange Web و …

استفاده از نرم افزار های آسیب پذیر یا کرک شده در لبه شبکه و انتشار آنها به صورت آنلاین که به هکر دسترسی RCE بدهد.

ارسال فایل های آلوده Word یا Excel از طریق ایمیل یا USB برای کاربران.

استفاده از سیستم های آلوده و فاقد آنتی ویروس در شبکه به عنوان سیستم زامبی و تسخیر شده.

نفوذ به سیستم کاربران شبکه از طریق وب سایت های آلوده و جعلی.

آلوده کردن و تسخیر سیستم های شبکه با کرک های جعلی از طریق سایت های ساختگی و ….

در زیر فایلی که توسط شرکت VMware منتشر شده را برای شما قرار می دهم جهت مطالعه بیشتر در این خصوص

VMware Security Best Practices

Securing a virtual environment running VMware vSphere is not as simple as accepting the default setting and policies during installation. While the defaults provide partially hardened settings, there are many ways to improve upon them. By following VMware security best practices, you can help protect your VMware infrastructure against cybersecurity threats, including both malicious attacks and careless mistakes.

This article explains how to reduce the risk to your enterprise security by properly securing your VMware vCenter server and your ESXi hypervisor, and details the key best practices for secure deployment, operations and networking.

Handpicked related content:

• [Free Download] VMware Auditing Quick Reference Guide

VMware vCenter Server Security

VMware vCenter server is the main control center of your vSphere environment. Whether you install it on a Windows or Linux operating system, the following best practices can help you maintain it in a secure state:

• To help keep VMware secure, make sure your vCenter Server systems use static IP addresses and host names. Each IP address must have a valid internal DNS registration, including reverse name resolution.
• By default, the password for the vpxuser account expires after 30 days. Change this setting if necessary to comply with your security policy.
• If you are running vCenter Server on Windows, make sure the remote desktop host configuration settings ensure the highest level of encryption.
• Make sure that the operating system is up to date on security patches.
• Install an antivirus solution and keep it up to date.
• Make sure that the time source is configured to sync with a time server or a time server pool, in order to ensure proper certificate validation.
• Do not allow users to log directly into the vCenter server host machine.

VMware ESXi Security

To secure your ESXi hypervisor, implement the following best practices:

• Add each ESXi host to the Microsoft Active Directory domain, so you can use AD accounts to log in and manage each host’s settings.
• Configure all ESXi hosts to synchronize time with the central NTP servers.
• Enable lockdown mode on all ESXi hosts. That way, you can choose whether to enable the direct console user interface (DCUI) and whether users can log in directly to the host or only via the vCenter Server.
• Configure remote logging for your ESXi hosts so you have a centralized store of ESXi logs for a long-term audit record.
• Keep ESXi hosts patched to mitigate vulnerabilities. Attacks often try to exploit known vulnerabilities to gain access to an ESXi host.
• Keep secure shell (SSH) disabled (this is the default setting).
• Specify how many failed login attempts can be made before the account is locked out.
• ESXi version 6.5 and later supports UEFI secure boot at each level of the boot stack. Use this feature to protect against malicious configuration changes within the OS bootloader.

Secure Deployment and Management

Here are the principal best practices for secure deployment and management of a VMware vSphere environment:

• Keep your virtual machine templates up to date with guest OS security patches.
• Deploy new VMs only from your VM templates. This practice helps ensure that the base OS is properly hardened before applications are installed and that all VMs are created with the same baseline level of security.
• Minimize use of the Virtual Machine console, since it allows users to use power management and removable device connectivity.
• Disable unnecessary functions inside each VM, including system components that are not necessary for the application you’re running. You can also disable CD/DVD drives, floppy drives and USB adapters.
• Restrict datastore browser access to limit risk to the virtualization files stored on your datastores.
• Prevent users from running commands inside a VM by disabling the command line window.
• Install an antivirus solution, set it to the highest protection possible and keep it up to date.

Secure Operations

Here are the key best practices for secure operations:

• VMware vSphere uses role-based management control (RBAC) to manage permissions. You can easily create, clone and modify roles in the vCenter Server system to implement the least-privilege principle.
• Integrate vCenter with Microsoft Active Directory. This can be done not only for vCenter running on Windows but also for a vCenter Server Appliance (VCSA) running on Linux PhotonOS. This integration enables you to use AD authentication for existing Microsoft AD users within vSphere, and enables vSphere administrators will be able to use a common identity source to grant access to vSphere objects.
• Create a named account for each user with the vCenter server administrator role. Grant this role only to administrators who need it; other users should have access to only the VMs and other resources they need to do their jobs.
• Resource privileges control the creation and management of resource pools. You can set privilege at different levels (for example, at the folder level) and let them propagate inside the object.
• Audit network traffic, firewall activity and other critical events. vSphere enables you to view the basic change log; for example, you can use the Tasks and Events tab in the vSphere Client to review all changes to any object in your vSphere hierarchy during a certain time period. However, third-party security solutions offer additional functionality, such as custom alerting on critical changes like the deletion of VMs or changes to resource pools. They also provide easy-to-read reports on changes and logon activities that make it easier to monitor your virtual environment and prepare for compliance

Secure Networking

The VMware vSphere virtual networking layer includes multiple elements, such as a virtual network adapter, virtual switches (vSwitches), distributed virtual switches (DVSs), ports and port groups. The ESXi hypervisor uses those elements to communicate to the outside world. The following the best practices will help you improve network security:

• Isolate network traffic based on type. You can use virtual LANs (VLANS) for this purpose.
• Secure virtual storage network traffic:
  • Isolate storage traffic on separate physical and logical networks.
  • Use CHAP authentication in iSCSI environments.
  • Use Internet Protocol security policy (IP Sec) when possible. This network security mechanism allows authentication and encryption of packets of data sent over a network.
• Use firewalls to help secure virtual network elements and filter VM network traffic. Don’t modify the default ESXi firewall configuration.

Conclusion

Securing your virtualized environment is a complicated task. Following the best practices detailed here to design, configure and monitor your environment will take you a long way toward reducing risk and ensuring regulatory compliance.

Secure Your VMware ESXi Hosts Against Ransomware

We have recently seen an increase in ransomware targeting VMware vSphere ESXi hosts and encrypting all virtual machines at once. You can secure your ESXi hosts from ransomware execution by following these three simple steps, using ‘execInstalledOnly’ and (optionally) TPM 2.0 and Secure Boot.

[UPDATE 2021-11-15]: VMware support has now confirmed that execInstalledOnly can be used without having TPM 2.0 and/or Secure Boot, so if you want to get protected quickly you can jump directly to setting it to TRUE. Don’t forget to test it properly before rolling it out broadly. They also confirmed that it’s supported on vSphere 6.5 and 6.7.

Why we should use execInstalledOnly to protect ESXi against ransomware

• Ransomware executing inside a VMware vSphere ESXi host can encrypt all the virtual machines at once, without having to compromise each guest operating system. An example of this is RansomEXX a.k.a. Defray777. More info on it can be found in this Crowdstrike writeup.
• This attack vector is possible because once attackers get control of an ESXi host, they are by default allowed to upload and execute any custom binaries they want.
• We can fairly easily prevent this by using the relatively unknown ESXi setting VMkernel.Boot.execInstalledOnly (optionally in combination with TPM 2.0 and UEFI Secure Boot) which is described in the ‘Three steps to protect ESXi against ransomware’ section below. (Jump straight there)

Background

We have recently seen an increase in ransomware attacks where the encryption is executed from the virtualization platform (ESXi or Hyper-V hosts) rather than from inside each guest operating systems (Windows, Linux etc). The benefit of this method from the attackers’ side is that they can encrypt numerous systems without having to reach them all over the network and obtain administrative privileges. This can greatly increase the scope and speed of the attack, which is bad news for us.

This blog post won’t go into the technical details on how the attacker gets into the ESXi hosts to execute the actual ransomware. This could for example be done through an RCE vulnerability such as the one for SLP in ESXi or through Active Directory->vCenter Server->ESXi, but also in other ways. A future blog post will analyze this in more detail and provide more suggested protections.

The ransomware will encrypt all virtual machines’ vmdk files on all attached datastores. It will also encrypt the ESXi host itself including all log files, so unless you have central tamper-proof logging in place it will be very difficult to secure forensic evidence regarding how the attack was carried out.

Despite the encryption, the ESXi hosts will usually remain running since they have already loaded the system files into memory. However, they will usually not survive a reboot, and will need a complete reinstallation. As long as the host is still running, the ransomware monitors the virtual machines and will encrypt any new vmdk or other virtual machine files that are put on shared datastores that it can reach.

VMware has a good technical post about this ransomware at Deconstructing Defray777 Ransomware, which goes through the technical details, but doesn’t mention specifically how to protect the ESXi hosts.

They also have some good videos covering the basics of ransomware protection on vSphere (but doesn’t mention execInstalledOnly):https://www.dideo.ir/extension/embed?url=Ly93d3cueW91dHViZS5jb20vZW1iZWQvN053SjlCLVNMYjg=

When we at Truesec perform Security Health Checks of customers’ vSphere environments, we always give everyone the following fundamental recommendations, so do make sure you also work towards getting these under control:

• Keep your systems (vCenter Server, ESXi hosts, VMware Tools etc.) up to date when there are security patches released.
• Use unique, strong passwords for administrative accounts and handle them securely. Consider not using Active Directory for administrator level access to vSphere.
• Segment your networks so that vCenter Server and ESXi administrative interfaces are not reachable for non-administrative computers and users. Use dedicated workstations and MFA for administrators.
• Configure central logging so that you have tamper-proof logs of all administrative actions and changes in your environment.
• Make frequent backups, and make sure they can not be deleted even if the attacker gets complete control over the rest of your environment.

Three steps to protect ESXi against ransomware

To prevent the actual ransomware execution, we recommend our customers to take the following steps for all ESXi hosts:

• (Optional) Use TPM 2.0 chips
• (Optional) Enable UEFI Secure Boot on the physical servers
• Prohibit execution of custom code inside ESXi (VMkernel.Boot.execInstalledOnly)

As always, make sure you test and evaluate the consequence of any upgrades and changes on a non-critical part of your environment before rolling it out in production. Also make sure your third-party vendors support the new configuration.

(Optional) Step 1: TPM 2.0

TPM 2.0 is a hardware chip that most modern physical servers have. It allows the operating system (ESXi in our case) to store secrets, keys, measurements etc. in a secure manner. This is used by vCenter Server to make sure the ESXi hosts’ boot files haven’t been tampered with, and works with vSphere/ESXi 6.7 and newer. More info here: https://blogs.vmware.com/vsphere/2018/04/vsphere-6-7-esxi-tpm-2-0.html

For the vSphere Attestation, there isn’t any specific configuration that needs to be set. If your ESXi hosts have active TPM 2.0 chips, vCenter Server will automatically display their attestation status in the ‘Monitor->Security’ tab of the clusters:

In vSphere 7.0 U2 and newer, the TPM 2.0 chip is also used to encrypt the configuration of the ESXi host as well as protect some settings from tampering (called ‘enforcement’). This is described in detail in the vSphere documentation.

(Optional) Step 2: Secure Boot

Secure Boot is a UEFI BIOS feature that strengthens the security of the operating system (ESXi in this case) by making sure that all code that is loaded at boot is digitally signed and has not been tampered with. Unlike some other operating systems, ESXi can have Secure Boot enabled retroactively without having to perform a complete reinstallation.

However, there might be some third-party packages (‘VIBs’, in vSphere language) that have the wrong ‘Acceptance level’ and can prevent Secure Boot from working correctly. To check if your ESXi host already has Secure Boot enabled, and whether there are any obstacles to enabling it, run the following two commands from an ESXi command line (SSH or ESXi Shell):

/usr/lib/vmware/secureboot/bin/secureBoot.py -s

/usr/lib/vmware/secureboot/bin/secureBoot.py -c

As we can see in the output above, Secure Boot is currently Disabled, but there are no obstacles preventing us from enabling it. We can now put this ESXi host in Maintenance Mode, reboot it, enter the server’s BIOS setup, enable Secure Boot, and boot up the ESXi host again. If you’re unsure on how to enable Secure Boot, check with your hardware server vendor on how to do this.

In vSphere 7.0 U2, the Secure Boot setting can be protected from tampering using the ‘enforcement’ capability. This is set using the following command line:

esxcli system settings encryption set --require-secure-boot=TRUE

More information is available in this VMware Documentation page.

Step 3: execInstalledOnly

—
[UPDATE 2022-04-28]: IMPORTANT: If you are updating/patching ESXi hosts using vSphere Lifecycle Manager (formerly known as Update Manager) using the old fashioned Baseline method rather than the newer Image method (link to article describing the difference) you will bump into problems when having execInstalledOnly set to TRUE.

The recommended workaround is to switch to the Image method, since it will also bring other benefits. If you can’t switch, you will unfortunately need to wait until vSphere 8.0 before being able to enable execInstalledOnly.

When using the baseline method and enabling execInstalledOnly, the error message you will get when scanning an ESXi host for patch compliance is:

Cannot deploy host upgrade agent. Ensure that vSphere Lifecycle Manager is officially signed. Check the network connectivity and logs of host agent and vpxa for details.

If you are getting this error message, either switch to the vLCM Image method or follow the instructions in https://kb.vmware.com/s/articl… to revert the execInstalledOnly setting and the enforcement of the setting.

If you set execInstalledOnly back to FALSE but keep the enforcement at TRUE, you will get a purple screen when rebooting the ESXi host. The purple screen is by design, and is described at the end of this blog post.

# Revert the enforcement of the setting
esxcli system settings encryption set --require-exec-installed-only=FALSE 

# Revert the setting itself
esxcli system settings kernel set -s execInstalledOnly -v FALSE

The reason for the Lifecycle Manager problem is that when using Baselines, VMware is apparently using an unsigned VIB, which is the update agent that Lifecycle Manager pushes out to the ESXi hosts when scanning or updating them. This is a very unfortunate mistake, which I hope they will fix before vSphere 8.0. I recommend you open a support case and tell VMware they need to fix this in 7.0 as well.
—
(Now back to the blog text..)

The execInstalledOnly setting prohibits execution of custom code inside ESXi and will make the ESXi host simply refuse to execute anything that was not installed through a signed VIB package from a certified partner.

This is very easy to achieve in ESXi compared to a general purpose operating system like Windows or Linux. ESXi is by design an “appliance” which doesn’t require any custom code to be run on it apart from VMware’s own code and the drivers and utilities of certified partners. Combining execInstalledOnly with TPM and Secure Boot which tamper-proofs the existing VIBs gives us an excellent combination of protective measures against ransomware and other malware executing inside the ESXi hosts.

The setting is found in ESXi under Manage->Advanced Settings at VMkernel.Boot.execInstalledOnly and it can be set without having to open a CLI to each ESXi host. We can set it for individual ESXi hosts using the vSphere Web Client or for multiple ESXi hosts at a time using PowerCLI (VMware’s PowerShell modules). The examples below use the CLI over SSH, since this gives us some additional information that is good to have when explaining the inner workings of the setting.

The default setting of an ESXi host is that execInstalledOnly is set to FALSE. We can verify this by simply listing the setting using

esxcli system settings kernel list -o execinstalledonly

and then checking that we are allowed to execute a custom binary. Rather than executing a real ransomware, we used a test binary that displays a ‘Hello world’ message to indicate that it was allowed to run:

Now, we set the execInstalledOnly setting to TRUE using

esxcli system settings kernel set -s execinstalledonly -v TRUE

Note that the ‘Configured’ value has changed to TRUE, but that the ‘Runtime’ value is still at FALSE. This is because the system requires a reboot for the setting to start working. We can verify that the protection is still not in effect by successfully executing our test binary again.

After rebooting the system, we can now see that the setting’s ‘Runtime’ value is TRUE, and we can also see that we are no longer allowed to execute our custom test binary.

Make sure you test your ESXi hosts properly after changing this setting on a test host. Some hardware vendors might have agents and/or utilities running inside ESXi that perform monitoring, central configuration, firmware upgrades etc.

In vSphere 7.0 U2, the execInstalledOnly setting can be protected from tampering using the ‘enforcement’ capability. This is set using the following command line:

esxcli system settings encryption set –require-exec-installed-only=TRUE

More information is available in this VMware Documentation page.

Putting it all together

Putting TPM, Secure Boot and execInstalledOnly together, we can now establish the following order of checks and changes to each ESXi host:
(Note that steps 1-3 and 6-7 are optional)

1. /usr/lib/vmware/secureboot/bin/secureBoot.py -s
2. /usr/lib/vmware/secureboot/bin/secureBoot.py -c
3. Make sure there are no warnings or errors above
4. esxcli system settings kernel set -s execinstalledonly -v TRUE (or set on multiple hosts using PowerCLI)
5. Put the ESXi host in Maintenance Mode and reboot it
6. At boot-up, enter the BIOS setup and enable UEFI Secure Boot (if not already enabled according to step 1)
7. (If running vSphere 7.0 U2 or newer and having a TPM 2.0 chip):
   1. esxcli system settings encryption set --require-secure-boot=TRUE
   2. esxcli system settings encryption set --require-exec-installed-only=TRUE
   3. /sbin/auto-backup.sh
8. Check that everything works, and take the ESXi host out of Maintenance Mode

Again, make sure you properly test these changes on non-critical ESXi hosts before rolling them out to the entire environment. Especially, test an ESXi update/upgrade using Lifecycle Manager to make sure it works well.

Once set, monitor any changes to these settings using central logging and set alerts so that you get notified if anyone tries to change them. Also monitor events around VMs getting unexpectedly shut down, since a ransomware needs to perform this step before being able to encrypt the files, since they’re locked by the hypervisor while running.

Tamper-proofing ESXi

If you’re using the vSphere 7.0 U2 additional protections to ‘enforce’ the execInstalledOnly setting and Secure Boot, your ESXi hosts will fail to boot and display a purple screen (which is good) if TPM, Secure Boot or execInstalledOnly is tampered with.

This is described in the first example in following VMware KB article: Boot time failures due to ESXi configuration encryption (81446)

Fix the problem by going back and re-checking all the above steps:

• If TPM 2.0 has been disabled, re-enable it.
• If UEFI secure boot has been disabled, enable it back.
• If execInstalledOnly boot option is set to FALSE, change it back to its initial value (i.e. TRUE).

If you have any questions or bump into any problems, don’t hesitate to contact us and we will try to help you out. Watch this space for more blog posts and videos about ransomware and vSphere/ESXi security.

مشاوره