Backing up VMware ESXi VMs with ghettoVCB

Pre-requisites

  • SSH access to your ESXi host.
  • A datastore, preferably a remote NFS or iSCSI one, where backups can be stored.

Installation

  1. Download the ghettoVCB VIB and then using your tool of choice (e.g. the vSphere Client or FileZilla over SFTP), transfer the VIB to your ESXi host.
  2. Connect to your ESXi host over SSH and enable the installation of “community” packages using the command esxcli software acceptance set level=CommunitySupported . By default, ESXi will only allow the installation of official packages, so unless the acceptance level is set to Community Supported, installation of ghettoVCB will fail.
  3. Install the ghettoVCB VIB using the command esxcli software vib install v /path/to/vghettoghettoVCB.vib f . If you used the vSphere Client to upload it to a datastore, you can find it at /vmfs/volumes/DATASTORENAME/vghettoghettoVCB.vib.

You can verify that the installation was successful by navigating to /opt/ghettovcb/bin . If successful, there should be two files in here – ghettoVCBrestore.sh and ghettoVCB.sh scripts.

Configuration

  1. Using the text editor of your choice (e.g. vi ), edit /opt/ghettovcb/ghettoVCB.conf .
  2. Copy the example configuration from https://github.com/lamw/ghettoVCB/blob/master/ghettoVCB.conf and paste it in to the new file.
  3. Adjust the configuration values as desired. Important values are;
    • VM_BACKUP_VOLUME – The path to your datastore where backups will be stored. For example, if your datastore was called “ISCSI-DATASTORE” and you wanted to store backups inside a folder called “Backups”, you would set this to /vmfs/volumes/ISCSIDATASTORE/Backups .
    • VM_BACKUP_ROTATION_COUNT – The number of backups you wish to retain for each VM. Once more than this number of backups exist, the oldest one will be deleted automatically.
    • EMAIL_SERVER and EMAIL_SERVER_PORT  – The address and port of the email server you want to send backup reports to. ghettoVCB uses netcat to directly speak the SMTP protocol to your email server and as a result authentication and SSL/TLS are not supported! If you don’t want email reports on backups, delete these configuration lines.
    • EMAIL_TO – The email address to send backup reports to. If you don’t want email reports on backups, delete this configuration lines.
  1. Save and quit.

Run

Now that ghettoVCB is installed and configured, we can run a backup. Firstly, try backing up a single VM using the command /opt/ghettovcb/bin/ghettoVCB.sh g /opt/ghettovcb/ghettoVCB.conf mVMNAME , replacing VMNAME with the name of your VM. ghettoVCB will then start backing up this VM to your configured datastore directory by creating a snapshot and then cloning the VM’s disks. Once complete, ghettoVCB will remove the snapshot and return you to the console.

Review the output of ghettoVCB for any errors. If the backup was successful, you should see the line ###### Final status: All VMs backed up OK! ###### in the output. Now, try backing up all of your VMs using the command /opt/ghettovcb/bin/ghettoVCB.sh g /opt/ghettovcb/ghettoVCB.conf a . Again, monitor the output of ghettoVCB and make sure that the backup completes successfully.

Schedule

Running backups on a manual schedule is no fun though, so instead it’s possible to run ghettoVCB automatically using a cron job.

  1. Using the text editor of your choice (e.g. vi ), edit /var/spool/cron/crontabs/root .
  2. Add a new line containing 0 1 * * * /opt/ghettovcb/bin/ghettoVCB.sh g/opt/ghettovcb/ghettoVCB.conf l /tmp/ghettoVCB.log a > /dev/null .
  3. Save and quit – as this file is typically read-only you may need to force your editor to save (in vi , use the command :wq! ).

What does this new line do?

    • At 1am every day, executes the backup command. You can change the time that the backup command is executed by changing the cron expression from 0 1 * * * . If you’re unfamiliar with cron-syntax, you can use a tool such as Cron Maker to generate a suitable cron-syntax.
    • Uses the configuration file /opt/ghettovcb/ghettoVCB.conf to adjust how the backup runs.
    • Outputs a log of the backup process to /tmp/ghettoVCB.log , which can be viewed later.
    • Includes all VMs on the host as a backup target. If you only want to backup a specific VM, you can change a to m VMNAME . To backup a list of VMs, change a to f/opt/ghettoVCB/vmlist and edit /opt/ghettoVCB/vmlist so that it contains a list of the VM names you wish to backup, one per line.
    • Redirects the output of the script to /dev/null , as some issues have been reported with running ghettoVCB with a cron job when the output isn’t redirected.

Slipstream Windows 2000/XP/2003

The other day I had to install Windows XP on a very old laptop and it took quite a bit to get a bootable USB stick created and slipstream the necessary drivers to recognize the IDE drivers.

These instructions are for Windows XP:

Extract your copy of Windows XP to a local drive, use DriverPacks BASE to slipstream (inject) the DriverPacks you require. Use nLite to create an ISO from the directory. Finally use WinToFlash to burn the image to USB.

Do this in the sequence above, accept the defaults in the GUI and it should just work. A good source for help and discussion is here and here.

I tried Rufus and it did not work, WinToFlash does some additional magic to make the USB drive bootable on very picky motherboards.

Update VMware ESXi host management IP in vSphere 6.7

Recently I had to update all the hosts in my cluster to a new IP to move them over to a different VLAN. There is a lot of information out there on how to do this, here is what worked for me.

  1. Turn off vSphere DRS/HA to prevent any kind of failover
  2. Migrate all VMs from host
  3. Put host into maintenance mode
  4. Disconnect host from vCenter
  5. Change management network TCP/IP settings using DCUI (KVM)
  6. Update DNS records for the host to reflect new IP
  7. SSH to vCenter and clear cache by restarting these services:
    systemctl restart systemd-resolved.servic
    systemctl restart dnsmasq
  8. Verify that you can ping host DNS from vCenter and that it resolves to correct IP
  9. Re-Connect host in vCenter
  10. Take host out of maintenance mode
  11. Load balance your VMs

Before you do this also review the relevant VMware KB here: https://kb.vmware.com/s/article/2084629

IBM Power9 9009-22A & StorWize V7000 Gen3 Benchmarks

IBM Power9 9009-22A

  • 512GB DDR4 Memory
  • 2x 8-Core Power9 Processor
  • 2x400GB NVMe M.2 SSD for rootvg
  • 16Gb FC 2 Port Adapter Pair
  • FW910.20 (VL910_122) firmware
  • AIX 7.2 ML3
    • AIX_AAPCM mpio driver
    • algorithmset set to shortest_queue
    • reserve_policy set to no_reserve
    • queue_depth set to 32

IBM StorWize V7000 GEN3

  • 17x 1.92TB 2.5In NVMe Flash Drive
  • 16Gb FC 4 Port Adapter Pair
  • 120 GB M.2 Boot Drive Pair
  • Spectrum 8.2.1.1

Benchmarks

  • Random I/O and Sequential I/O with 8K 1 Thread
  • Random I/O and Sequential I/O with 8K 32 Threads
  • Random I/O and Sequential I/O with 1M 1 Thread
  • Random I/O and Sequential I/O with 1M 32 Threads
  • 100% Read and 100% Write
  • 50% Read and 50% Write (backup I/O profile)
  • 80% Read and 20% Write (OLTP database I/O profile)

The tests will be done using XDISK V8.6 for AIX 7.2 with -ORW flags specific for AIX to benchmark the storage array and bypass VMM cache.

Summary

  • As we increase the block size and number of threads so does the performance up to a certain point. The hardware is capable of high performance without having to go and tweak a lot of things.
  • Technical information related to setting up FC-NVMe instead of FC (SCSI) seems to be non existing for the GEN3 and/or AIX at time of writing.
  • DRAID6 seems like a good alternative to the traditional RAID10 even for critical workloads, if you are concerned about disk failures then configure it for two rebuild areas which allows you to loose two disks.
  • If you are upgrading from a RAID10 spinning disk (15K) array to NVMe SSD array, strongly consider DRAID6 as an alternative.
  • It was not possible to assign a stand-by hot-spare in a DRAID6 configuration, you can still mark an SSD as a spare but the software will not automatically consume it if a disk fails.
  • No issues whatsoever failing one of the active/active storage nodes and monitoring the events inside AIX, everything worked as expected in a redundant path host and storage configuration.

Traditional Raid 10 Benchmarks
(two mdisk devices, total of 16 drives + 1 spare)

Random I/O
8K Block Size
1 Thread

100% Read

BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

 8K    1   0   100 R    -RW  236587   1848   0.001   0.157   0.003   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0     0 R    -RW    4015   31.4     0.0     0.0     0.0     0.0   0.001    3.24   0.247   0.242

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0    50 R    -RW    7832   61.2   0.001   0.026   0.004   0.005   0.001    4.48   0.248   0.242

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0    80 R    -RW   18509  144.6   0.001   0.053   0.003   0.005   0.001    9.25   0.248   0.242

Sequential I/O
8K Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0   100 S    -RW  345942   2703   0.001   0.949   0.002   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0     0 S    -RW    3123   24.4     0.0     0.0     0.0     0.0   0.001    3.97   0.319   0.315

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0    50 S    -RW    6710   52.4   0.002   0.030   0.003   0.005   0.001    4.14   0.293   0.277

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0    80 S    -RW   17871  139.6   0.001    1.60   0.003   0.005   0.001    3.73   0.265   0.241

Random I/O
8K Block Size
16 Threads

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0   100 R    -RW 1505683  11763   0.001    1.90   0.003   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0     0 R    -RW   33141  258.9     0.0     0.0     0.0     0.0   0.001    8.78   0.480   0.399

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0    50 R    -RW   62571  488.8   0.001    15.7   0.022   0.005   0.001    6.34   0.485   0.401

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0    80 R    -RW  151339   1182   0.001    6.95   0.010   0.005   0.001    6.68   0.479   0.398

Sequential I/O
8K Block Size
16 Threads

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0   100 S    -RW 3581072  27977   0.001    90.0   0.004   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0     0 S    -RW   28645  223.8     0.0     0.0     0.0     0.0   0.001    7.44   0.557   0.527

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0    50 S    -RW   58762  459.1   0.001    9.32   0.027   0.005   0.001    7.25   0.515   0.460

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0    80 S    -RW  148521   1160   0.001    8.02   0.010   0.005   0.001    7.43   0.495   0.390

Random I/O
16K Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0   100 R    -RW  178765   2793   0.001   0.258   0.004   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0     0 R    -RW    3966   31.0     0.0     0.0     0.0     0.0   0.001    9.54   0.250   0.241

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0    50 R    -RW    7289  113.9   0.002   0.032   0.004   0.005   0.001    3.64   0.265   0.256

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0    80 R    -RW   17327  270.7   0.002   0.143   0.004   0.005   0.001    3.76   0.263   0.254

Sequential I/O
16K Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0   100 S    -RW  238870   3732   0.001   0.259   0.003   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0     0 S    -RW    3366   52.6     0.0     0.0     0.0     0.0   0.001    3.94   0.295   0.281

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0    50 S    -RW    7028  109.8   0.002   0.047   0.005   0.005   0.001    3.67   0.277   0.263

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0    80 S    -RW   17454  272.7   0.002   0.176   0.004   0.005   0.001    3.69   0.265   0.253

Random I/O
1MB Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0   100 R    -RW    5847   5847   0.001    1.52   0.169   0.169     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0     0 R    -RW    4161   32.5     0.0     0.0     0.0     0.0   0.001    3.55   0.238   0.232

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0    50 R    -RW     756  756.0   0.058   0.249   0.173   0.173    1.32    5.61    2.40    2.38

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0    80 R    -RW    1589   1589   0.039    1.65   0.172   0.172    1.29    5.39    2.38    2.37

Sequential I/O
1MB Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0   100 S    -RW    5215   5215   0.001    1.50   0.191   0.188     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0     0 S    -RW     410  410.0     0.0     0.0     0.0     0.0    1.21    5.20    2.36    2.35

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0    50 S    -RW     767  767.0   0.103   0.323   0.173   0.174    1.26    5.01    2.36    2.35

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0    80 S    -RW    1409   1409   0.001    1.51   0.200   0.192    1.70    5.11    2.67    2.66

Random I/O
1MB Block Size
16 Threads

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0   100 R    -RW   78682  78682   0.001     130   0.201   0.195     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0     0 R    -RW   33581  262.4     0.0     0.0     0.0     0.0   0.001    5.88   0.474   0.391

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0    50 R    -RW    2325   2325   0.001    80.8   0.616   0.201    2.94    81.2    13.0    11.5

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0    80 R    -RW    6610   6610   0.001    49.7   0.346   0.197    2.84    34.4    10.6    10.8

Sequential I/O
1MB Block Size
16 Threads

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0   100 S    -RW   76001  76001   0.001    57.8   0.210   0.205     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0     0 S    -RW    1351   1351     0.0     0.0     0.0     0.0    2.98    43.2    11.7    11.0

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0    50 S    -RW    2659   2659   0.001    70.1   0.612   0.231    2.96    43.1    11.3    11.0

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0    80 S    -RW    6602   6602   0.001    51.5   0.349   0.200    3.03    34.6    10.6    10.8

Distributed Raid 6 Benchmarks (DRAID6) 
(1 mdisk device, 16 drives, 2 rebuild areas, 1 spare)

Name: Pool0
State: Online
Capacity: 15.49 TiB / 20.23 TiB (77%)
Volumes: 1
MDisks: 1
Encryption: Not Encrypted
Data reduction: Enabled
Easy tier: Balanced
Extent size: 1.00 GiB
Free physical capacity: 4.75 TiB
Reclaimable capacity: 0 bytes

Name: mdisk0
State: Online
Capacity: 20.26 TiB
Pool: Pool0
Mode: Array
Write protected: No
Tier: Tier 0 Flash
Encryption: Not Encrypted
Deduplication: Not Active
Fast-Write state: Empty
Thin-Provisioned: No
Supports unmap: Yes
RAID level: Distributed RAID 6
Redundancy: 2
Rebuild Areas goal: 2
Rebuild Areas Total: 2
Strip size: 256 KiB

Random I/O
8K Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0   100 R    -RW  228787   1787   0.001    1.38   0.003   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0     0 R    -RW    4061   31.7     0.0     0.0     0.0     0.0   0.001    4.84   0.244   0.238

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0    50 R    -RW    7766   60.7   0.001   0.036   0.004   0.005   0.001    4.84   0.250   0.242

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0    80 R    -RW   18293  142.9   0.001   0.024   0.003   0.005   0.001    5.03   0.252   0.242

Sequential I/O
8K Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0   100 S    -RW  354277   2768   0.001    9.62   0.002   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0     0 S    -RW    3065   23.9     0.0     0.0     0.0     0.0   0.001    6.87   0.325   0.317

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0    50 S    -RW    6629   51.8   0.001   0.020   0.003   0.005   0.001    5.65   0.297   0.275

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0    80 S    -RW   16971  132.6   0.001   0.021   0.003   0.005   0.001    9.35   0.278   0.253

Random I/O
8K Block Size
16 Threads

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0   100 R    -RW 1501107  11727   0.001    40.0   0.003   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0     0 R    -RW   27052  211.3     0.0     0.0     0.0     0.0   0.001    8.38   0.589   0.490

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0    50 R    -RW   46707  364.9   0.001    22.9   0.027   0.005   0.001    7.00   0.653   0.581

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0    80 R    -RW  115231  900.2   0.001    10.4   0.012   0.005   0.001    9.63   0.635   0.565

Sequential I/O
8K Block Size
16 Threads

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0   100 S    -RW 3475273  27151   0.001    60.1   0.004   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0     0 S    -RW   25121  196.3     0.0     0.0     0.0     0.0   0.001    10.9   0.635   0.616

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0    50 S    -RW   52680  411.6   0.001    17.6   0.027   0.005   0.001    11.0   0.578   0.499

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0    80 S    -RW  123603  965.6   0.001    12.4   0.011   0.005   0.001    13.2   0.597   0.452

Random I/O
16K Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0   100 R    -RW  181688   2839   0.001    1.52   0.004   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0     0 R    -RW    3934   30.7     0.0     0.0     0.0     0.0   0.001    5.78   0.252   0.239

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0    50 R    -RW    7076  110.6   0.002   0.020   0.004   0.005   0.001    3.48   0.274   0.259

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0    80 R    -RW   16432  256.8   0.002    3.87   0.005   0.005   0.001    4.14   0.275   0.264

Sequential I/O
16K Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0   100 S    -RW  224488   3508   0.001   0.152   0.004   0.005     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0     0 S    -RW    3184   49.8     0.0     0.0     0.0     0.0   0.001    12.1   0.312   0.287

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0    50 S    -RW    6931  108.3   0.002   0.024   0.005   0.005   0.001    6.58   0.281   0.258

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

   16K    1   0    80 S    -RW   16945  264.8   0.002   0.294   0.004   0.005   0.001    4.58   0.274   0.256

Random I/O
1MB Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0   100 R    -RW    5741   5741   0.001    1.45   0.172   0.173     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K    1   0     0 R    -RW    3989   31.2     0.0     0.0     0.0     0.0   0.001    5.12   0.248   0.242

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0    50 R    -RW     722  722.0   0.044   0.213   0.175   0.175    1.38    8.22    2.52    2.50

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0    80 R    -RW    1532   1532   0.038    1.42   0.172   0.173    1.37    7.15    2.49    2.48

Sequential I/O
1MB Block Size
1 Thread

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0   100 S    -RW    5768   5768   0.001    1.39   0.173   0.173     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0     0 S    -RW     379  379.0     0.0     0.0     0.0     0.0    1.55    9.51    2.57    2.54

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0    50 S    -RW     741  741.0   0.153   0.317   0.190   0.179    1.57    8.72    2.44    2.42

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M    1   0    80 S    -RW    1555   1555   0.001    1.48   0.172   0.173    2.16    8.61    2.45    2.44

Random I/O
1MB Block Size
16 Threads

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0   100 R    -RW   80170  80170   0.001     105   0.197   0.195     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    8K   16   0     0 R    -RW   27300  213.3     0.0     0.0     0.0     0.0   0.001    10.8   0.583   0.490

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0    50 R    -RW    1622   1622   0.039     263   0.779   0.209    3.37     207    18.8    16.0

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0    80 R    -RW    5137   5137   0.001    75.4   0.372   0.187    3.10     226    14.0    14.7

Sequential I/O
1MB Block Size
16 Threads

100% Read

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0   100 S    -RW   76222  76222   0.001     120   0.209   0.206     0.0     0.0     0.0     0.0

100% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0     0 S    -RW    1173   1173     0.0     0.0     0.0     0.0    3.50    3024    13.5    12.3

50% Read 50% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0    50 S    -RW    2168   2168   0.001    68.2   0.671   0.241    3.90    3019    14.0    13.3

80% Read 20% Write

    BS Proc AIO read% IO  Flag    IO/s   MB/s rMin-ms rMax-ms rAvg-ms   WrAvg wMin-ms wMax-ms wAvg-ms   WwAvg

    1M   16   0    80 S    -RW    5192   5192   0.001    45.9   0.396   0.201    3.44    2694    13.7    13.7

Installing Exchange Server 2013 CU21 requisites on Windows Server 2012 R2

We are going to install Exchange 2013 CU21 ( combination of Mailbox and Client Access  role) on a single box.

Install the Unified Communications Managed API 4.0 Runtime, and the Visual C++ 2013 redistributable package.

Install the required features using:

Install-WindowsFeature AS-HTTP-Activation, Desktop-Experience, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation, RSAT-Clustering-CmdInterface, RSAT-ADDS

Make sure you check for updates after you install these as typically the UCM API has some post-updates.

If you are installing Exchange Server in the AD forest for the first time run the following Exchange 2013 setup command to prepare Active Directory:

Enable Copy & Paste in VMware vSphere 6.7

If you want to enable copy & paste on VMware vSphere 6.7 do the following:

  1. Disable Lockdown Mode if it is enabled and start the SSH service
  2. Log in to the ESX/ESXi host as a root user.
  3. Take a backup of the /etc/vmware/config file.

Open the /etc/vmware/config file using a text editor and add these entries to the file:

vmx.fullpath = "/bin/vmx"
isolation.tools.copy.disable="FALSE"
isolation.tools.paste.disable="FALSE"

Or you can just copy and paste this:

grep -i 'vmx.fullpath = "/bin/vmx"' /etc/vmware/config || echo 'vmx.fullpath = "/bin/vmx"' >> /etc/vmware/config
grep -i 'isolation.tools.copy.disable="FALSE"' /etc/vmware/config || echo 'isolation.tools.copy.disable="FALSE"' >> /etc/vmware/config
grep -i 'isolation.tools.paste.disable="FALSE"' /etc/vmware/config || echo 'isolation.tools.paste.disable="FALSE"' >> /etc/vmware/config

You must reboot each VM on the host (or use vMotion to move the VM back and forth).

Headless Windows Server, how to manage firewall remotely?

I noticed that once you deploy headless windows servers (domain controllers as an example) you can manage everything remotely (mmc, event viewer, rsat, etc.) except Windows Firewall right out of the box.

To be able to manage the firewall remotely, you need to open additional ports on the headless instance using:

netsh advfirewall firewall set rule name="Windows Firewall Remote Management (RPC)" new enable=yes
netsh advfirewall firewall set rule name="Windows Firewall Remote Management (RPC-EPMAP)" new enable=yes