Search the Community

I have to maintain the VDA software on roughly 100 persistent MCS desktops. Every time Citrix came out with a new version of Virtual Apps and Desktops (XenApp/XenDesktop), I would cringe at the thought of having to "touch" every one of them. It usually took me two or more weeks to get through them all. I finally got tired of that process and decided to script it. It has saved me a ton of time (and from pulling out all of my hair), and I hope it can help others as well. The really short version of how it works: Gets a list of computers from the DDC (or you can manually input the computer names into the script). On each computer it will create two scripts, and two scheduled tasks. The first script loads auto logon information into the registry (local administrator account) Executed one time to restart the machine The second script handles the VDA removal and install of the new version. Executes when the local administrator logs on At the end, it removes the scheduled tasks, the PowerShell scripts copied to the VDA, and auto logon information. Here is the script - read through all the comments, so you know how to edit it for your environment. I also created a from an older version of the script if you want to get a better idea of how it works. It was designed to work with Citrix Hypervisor (XenServer), but the new one uses a mapped drive instead.

Introduction Charles Dickens most certainly got it right when he wrote in his quintessential novel: “It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only.” Dickens obviously was not confronted with configuring servers and many of the nuances they seem to require to get them running well. He most certainly had enough troubles of his own, just different ones from the sort that system architects and administrators now encounter some 160 years later. This tale is about two specific servers, the Dell R720 and R730, of which we still have quite a few in production (along with some newer R740 units). It could also be said that it’s about two R730 servers, with different BIOS settings. It also involves GPUs, to thicken the plot. This is above all a cautionary tale, for if nothing else, the lesson for us all is to not assume that the out of the box BIOS/firmware settings of a server are going to yield the best results. And the corollary is that one should not assume that the same settings will hold true over different controller model releases, in particular if the controller release (iDRAC in this case) changes in major ways from one version to another, not to mention that BIOS updates can obviate earlier settings and functionality. But, I get ahead of myself. Do note that while I’m going to be talking about experiences with running XenServer 7.X on these servers, the same caveats can pertain to whatever operating system or hypervisor might be installed. The Plot Thickens It all started with experimenting with turning on and running turbo features with various C-states enabled on a number of R720 and R730 units running XenServer. Citrix has gone back and forth on the pros and cons of turbo boosts, and some say to just run all virtual CPUs with the maximum fixed highest available clock rate is the way to go, while others counter that turbo boost – if properly implemented – can make a big performance difference. The specific applications run on the unit also weigh in, of course. I’m not going to debate this here, but rather investigate what happens if you do leverage turbo mode, whether you’re a fan of it or not. In any case, there are a number of options to view and adjust settings with a routine called xenpm and these were working perfectly well on the R720 servers, but not on the R730 units. It was not as if I needed this on the R720, for it was behaving as expected; I wanted to see how the R720 settings compared to those of the R730 making use of the xenpm utillity. I went back to look at the BIOS settings on the R730 machines, which are quite different given that the R720 uses an iDRAC7 whereas the R730 makes use of the iDRAC8. In the case of the R730, I made sure turbo mode was enabled on my test server, as well as C-states, yet I was getting either errors or no output at all from various forms of the xenpm command. The lack of output, according to what I had read, meant something was amiss in the configuration. Furthermore, the R720 had some settings that are apparently not replicated on the R730, such as “Performance per Watt(OS)." This, however, provided in and of itself a bit of a clue. To exacerbate my frustration, I had seen a number of articles about Linux where CPU frequency scaling was mentioned and where people discussed entries that controlled turbo functions in files such as: /sys/devices/system/cpu*/cpufreq/scaling-governor and no such entries were found on XS 7.1 or 7.6 – not even close. Plus, with the latest R730 BIOS at the time of this writing updated to 2.9.1, the iDRAC interface itself seemed a bit different from earlier versions. And in fact, to give away the main issue about settings early on in this blog, thanks to some good conversations with some of the smart folks I know, the main issue was attributed to the BIOS being set to “System DBPM” when it needed to be “OS DBPM” (Demand Based Power Management)! Recall the “Performance per Watt(OS)” setting mentioned above; the clue is in the “OS” portion. In fact, Red Hat states specifically in the article https://access.redhat.com/articles/2207751 that a server should be set for OS (operating system) performance as otherwise, the operating system (in this case, XenServer) cannot gain access to control the CPU power management, which ties in with the ability to manage also the CPU frequency settings. Voila! Note that on the R730, to make this setting, in the BIOS setup you apparently have to pick the “Custom” System Profile option so that access to that option becomes visible, as seen below: There is a great article at: https://xenserver.org/partners/developing-products-for-xenserver/19-dev-help/138-xs-dev-perf-turbo.html which discusses turbo boost in the XenServer environment, specifically. Among other important factoids, it mentions that while C-states do not necessarily need to be enabled to allow for turbo boost to kick in, without them enabled, turbo mode being leveraged is “significantly less likely to occur.” P-states and C-states should be fully enabled. One article referred to within that document (CTX132714) about removing manufacturer’s settings that disable these states seems to have unfortunately disappeared. Standard vs. New Settings Before getting into details, let’s first take a look at how a R730 server without turbo boost enabled and with a fixed maximum CPU speed on all processors compares with running the same benchmarks on an R730 server with turbo mode enabled, but incorrectly configured. First off, here are the specs of the R730 servers, which are identical except for some BIOS settings: Dell R730, dual Intel Xeon E5-2680 v4 CPUs @ 2.40 GHz, 14 cores (28 total), 28 hyperthreads (56 total), 256 GB 2400 MHz RAM, XenServer 7.6 fully patched as of this writing, NVIDIA M60 GPU running GRID 7.1 drivers. Do I wish I had higher intrinsic clock speeds at the expense of fewer cores? Yes, but this is what I had to work with. For the main benchmark, I was running the free Windows version of the Unigine Superposition benchmark (https://benchmark.unigine.com/superposition) on Windows Server 2016 Standard virtual machines (VMs) that are configured identically. No other VMs were running on the XenServer instances. Here are the results of two runs, one with “1080p medium,” the other “1080p extreme” benchmarks. Above all, the desire was to see how the GPU performance was impacted by factors dictated in part by the CPU and server configuration, overall. Note that one set of benchmarks was run ten times to check for consistency and other than some variation in the minimum frames per second (FPS) metric and GPU load for the “1080p medium” test, all others had standard deviations from the mean of less than 1%. In other words, the consistency was very good. Here are the results of running on a server with all virtual CPUs running fixed at 2.40 GHz and turbo mode disabled: These are the results from the incorrectly configured server that was set up to supposedly make use of turbo mode: There is actually a reduction in performance of around 48% and 3%, respectively, in the turbo version – not what I initially expected. The first benchmark metric (1080p medium) is actually more important, in my opinion, because it should be more representative of an actual user environment where a single process is less likely to consume the entire GPU and may need to share it even with other users. The CPU load is also important as the overall performance of the server will also weigh in. With big data sets, there is a fair amount of I/O needed to keep the flow of data in and out of the GPU memory moving and hence faster CPUs are generally more desirable. The scope of this article would get way too large if these sorts of details were discussed at length, hence I will leave those topics for another blog or discussion, as interesting as they are. Fixing Turbo Mode The next step is to fix the turbo boost so that it’s functional. On many servers, this can be done with a BIOS setting; it certainly works that way on an R720. To do this on the R730, we first of all set the CPU Power Management parameter in the BIOS to “OS DBPM” and reboot. We also have in the case of the R730 no obvious way I could at least find to adjust the frequency governor in the BIOS, hence I did this with the CLI command (to change it from its initial value of “ondemand”): xenpm set-scaling-governor performance This is put in place and is effective immediately. To make the setting permanent over reboots, you need to run: /opt/xensource/libexec/xen-cmdline --set-xen cpufreq=xen:performance This needs be run on each XenServer host. To test if turbo mode is enabled, use the command: xenpm get-cpufreq-para | grep turbo and it should state “turbo mode: enabled” in the output for each virtual CPU. Note that there is an interesting convention that if a CPU has turbo mode available, it will show in addition to its highest frequency that frequency plus 1 MHz. For example, for a 2.40 GHz system, you see for the scaling frequency output from the “xenpm get-cpufreq-para” command the following values which are in kHz: scaling_avail_freq : *2401000 2400000 2300000 2200000 2100000 2000000 1900000 1800000 1700000 1600000 1500000 1400000 1300000 1200000 The starred entry shows what’s currently in use and in this case it’s indeed 1 MHz (1000 kHz) higher than the nominal maximum CPU clock frequency. Other useful commands to check the overall state include: xenpm get-cpuidle-states xenpm get-cpufreq-states xenpm get-cpufreq-average To monitor the full state and frequency usage, the command: xenpm start [seconds] can be used, where “seconds” is an integer in seconds to collect Cx/Px statistics. To monitor this over the time the benchmarks were run, I used: watch -n 5 'xenpm start 1|grep "Avg freq" |sort | tail -16' to check on the largest average frequency states seen every five seconds with a one-second integration time. I also verified that all P-states and C-states were enabled and that the governor had been properly set to “performance”. These are the results of these changes: The metrics compared to those of non-turbo mode are now under 2% different for both the benchmark itself and the GPU utilization. What a difference! The maximum CPU frequency in both these cases rose from 2.40 GHz to 2.881 GHz, an improvement of around 20%. With further tweaks -- that will remain the topic of a different discussion -- I was able to better these benchmarks to: This translates to an increase over the non-turbo version benchmark by around 36% and nearly 2% for the “1080p extreme” case and the GPU utilization rises from 82% to 97% for the “1080p medium” run. Furthermore, in this more optimized configuration the maximum CPU frequency rises from 2.40 GHz to 3.145 GHz, or over 31%. Summary First and foremost, take the time to review your hardware and make sure it’s configured such that it will be as tailored as best as possible to your specific environment. Things change over time, so review periodically the settings and see if they still make sense with your current needs. That requires monitoring your setup and tracking the performance over time. There are a plethora of good tools and utilities to make that possible. The GPU performance does depend on the CPU and various other server and system settings; don’t take for granted that it’s going to be just fine out of the box. Experiment a sufficient amount of time with various settings until you have it working as best as you think possible. Remember, though, that a test environment will not behave the same way as your production environment, hence take what you come up with with a grain of salt and expect to monitor and refine settings over time. GPUs can make a huge difference in the performance of applications, but they cannot do so if the environment in which they are used isn’t optimized. Your setup is only as good as the weakest link in the chain. Of course, some compromises will often need to be made based on cost, space, time, power, and other factors. Bear in mind what apps are going to push the limits and when and where bottlenecks might take place. Finally, keep your end users in the loop. You want to keep them happy and they will be the first to let you know if something isn’t looking normal, and take them seriously no matter what the metrics might report. Listen to their input. Acknowledgements I would like to thank a number of folks of the NVIDIA vGPU Community Advocate (NGCA) group for very useful tips and enlightening discussions, in particular Ben Jones, Ronald Grass, Rasmus Raun-Nielsen, and Jan Hendrik Meier.

Servers can only do their job well if they are optimized and tuned to perform their duties that best suit the applications they run and fulfill the expectations of those who use them. This short series of articles strives to address in particular how the increasingly more common combination of servers and embedded GPUs deserves special attention to bring out the best in both. This is a follow-up to the original article, which dealt primarily with BIOS settings and how they affect CPU and GPU performance. In that article, it was mentioned that further tweaks resulted in an increase over the non-turbo version GPU benchmark by around 36% for the “1080p medium” and nearly 2% for the “1080p extreme” case and the GPU utilization rising from 82% to 97% for the “1080p medium” run. Furthermore, in this more optimized configuration, the maximum CPU frequency rose from 2.40 GHz to 3.145 GHz, or over 31%. In this installment, I will go into a few more details and clarifications regarding the BIOS settings and discuss a bit more the roll of the GPU. In the next installment, there will be a deep dive into the topics of non-uniform memory access (NUMA) and server architecture, sockets/cores-per-socket settings, virtual machines (VMs) and their vCPU and memory assignments, and a summary of how this all comes together to play a role in the end result. The details will also be revealed in the next installment that led to the above-mentioned improvements in the overall VM/GPU performance. In this and the follow-up article, I’d like to go into a bit of more detail how those metrics were achieved as well as dive more deeply into various configuration settings that all contribute to the overall performance of the system. Above all, with server performance suffering as a consequence of necessary BIOS and OS patches due to security vulnerabilities such as Spectre and Meltdown, getting as much out of one’s servers as possible is one way to provide at least partial compensation. A number of items that influence server and GPU performance will be discussed. Revisiting the BIOS Going back to the BIOS settings discussed in the original article, there are a number of settings that ought to be clarified as they are in some cases quite esoteric, yet can make a big difference in how the server performs. Here is a screen shot of those settings from the other article as a reference: Energy Efficient Turbo allows a processor’s core frequency to be adjusted within the turbo range based on the workload. I have left it at this value as it seems it would probably work well in conjunction with enabling C-states. Here is a little bit more on what the “Uncore Frequency” is and why the choice was made to go with “Dynamic” vs. “Maximum”. The “Uncore Frequency,” known sometime later in history also as the “System Agent,” came about when the Intel Nehalem microarchitecture introduced a flexible architecture allowing optimization for different segments. Core processing functions (ALU, FPU, L1 and L2 cache) are split from the “uncore” functionality; “uncore” is invoked whenever the CPU needs to access data.) Note the “uncore” (System Agent) setting difference: “Dynamic“: the uncore frequency appears to match the frequency of the fastest core “Maximum“: the uncore frequency typically remains fixed It would appear that “Maximum” would seem like it would give you the best overall performance, but in fact, it can limit the ability of the “uncore” to make use of higher clock rates when those are taking place and hence block its ability to take advantage of the CPU’s highest current clock speed, resulting in a mismatch and a slowdown. Because the CPU is responsible for feeding data to the GPU, it can become a bottleneck, probably most noticeable in the FPS video rates, but also important when a GPU has a lot of data flowing in and out of it because of its own memory limitations. Gamers have been aware of this for a long time and hence have resorted to overclocking of their CPUs to push the envelope. Hence, “Dynamic” appears to be overall a better choice. For more about the “uncore” see for example this article. Power management profiles that are characteristic of a typical Dell server look similar to the following diagram and these are the recommendations for customizing the server to get maximum performance: The “uncore” parameter was already discussed above. Worth mentioning in a little more detail is the C1E parameter, which also reduces the CPU voltage in addition to stopping the CPU internal clock. It is known as the “enhanced C state”. If this mode is enabled in the BIOS, the CPU will enter this mode rather than make use of the default Halt (C1) mode when a HLT instruction is issued (see this article). See also this additional article which discusses but does not show in tabular form the “Custom Profile” shown above. One deviation in the recommended custom setting from the “Performance” profile is to enable C-states, which allows the turbo mode to be maximized as it allows cores that are not as busy to idle better and save power all while providing the opportunity to the CPU to run as fast as possible with maximum power. In that article in Figure 14 on page 24, note that other than the dense configuration, which makes use of the System DPM with turbo disabled, the overall power usage averages out to be within a percent of the various other settings. On the other hand, the dense configuration will decrease performance by up to 10% and the average power usage up to around 20%. In our case, however, we are interested in maximizing the performance. Note, however, that there is a certain risk here of encountering a certain degree of performance inconsistency since pushing the turbo to the maximum cannot typically be sustained for long periods of time due to either competition for GPU resources from other users and/or the shift of CPU resources to other tasks. If end user experience (EUX) consistency is desired, turbo mode might actually not be the best setting. The reader is reminded that what may turn out the best for the particular servers discussed here may not be the case on others with different architectures. Always test on equipment as close to the final production environment as possible. The same EUX issue also becomes a matter to take into consideration when looking at NVIDIA GPU scheduling, as will be touched on in the next section. For more on Dell R730 profile settings, see for example this reference. GPU Settings Another factor, that was not a concern in the very controlled test environment I used, is that of the GPU scheduling settings. Depending on the GPU model, there are up to three options available in NVIDIA GRID (now rebranded as vGPU)/Tesla GPUs: Best Effort is the oldest, introduced with Kepler and still the only option on Maxwell GPUs and is based on time-sliced round-robin. Scheduling is on a first come, first served basis. An advantage includes the use of all idle cycles, but it has an “unfair” mechanism for competing requests which have to wait, so performance may suffer. No cycles go idle in this model – some GPU gets them if any activity is going on. Also, there is no way to pre-allocate resources. This is the only scheduler supported on Kepler and Maxwell GPUs. Equal Share strives to allocate similar resources to all tasks. Each task gets a turn. If, however, there is no activity taking place for a task, it still gets a time slot and the GPU idles during this time, which is not ideal as the time slice is wasted. It is also a round robin style of scheduler and does offer protection from “noisy neighbor” effects. It works also with CUDA computational tasks and is supported on Pascal and newer architectures. Fixed Share is the newest scheduler. It’s also based on a round robin scheduling mechanism, but here assures a fixed rather than equal share of the GPU cycles. This can result in more idle cycles compared to the equal share mechanism, however. For guaranteeing minimum performance, for QoS, it may be the best choice, despite its limitations. For customers using these in the cloud, it may be a good choice. It is also available only on Pascal and later GPU architectures. Depending on your environment, GPU scheduling should be taken into consideration as GPU behavior will be different depending on which model is chosen and what applications your end users are running and how end users perform their work. Note that the default is Best Effort across the board. Here are illustrations of how the Equal Share and Fixed Share schedulers operate: The table below shows the various NVIDIA GPU models along with their scheduling options and associated parameters: Scheduling Mode: Best Effort Equal Share Fixed Share Supported Hardware Maxwell, Pascal, Volta, Turing Pascal, Volta, Turing Pascal, Volta, Turing Primary Use Cases Enterprise Enterprise Cloud vGPU Aware No Yes Yes Mixed computes/graphics Supported Recommended Recommended Idle Cycle Redistribution Yes No No Guaranteed QoS No Yes Yes Noisy Neighbor Protection No Yes Yes Frame Rate Limiter Required Yes No No An informal poll of a few experienced folks who design and deploy this technology resulted in the majority sticking with the tried and proven “Best Effort” scheduler. The most compelling explanation is that it is the only one where idle cycles are not wasted and in industries where every end user has a lot of work to do and GPU loads are heavy, this makes as full use of the GPU cycles as possible. Note, however, that similar to how turbo mode on a CPU can lead to too much undesirable variation in EUX, the Fixed Share scheduler may work out better in some cases, in particular for cloud-based installations since there are more variables involved in the communication links between the server and end user and issues such as latency and packet losses can factor in. The topic on which GPUs are available from different vendors and how to make such selections could easily be a whole separate set of topics to discuss. My personal opinion is that currently, NVIDIA offers the most variety and versatility and is the one vendor I have the most direct experience with. Which model GPU to choose therefore becomes another major topic, along with in the case of NVIDIA GPUs what so-called “profile” to use and whether to dedicate a GPU or portion of it to each VM or to share it among multiple VMs. The ability to carve a GPU into segments of various sizes and licensing types according to ”profiles” and assign them separately to one or more VMs is an important part of the design and implementation process for NVIDIA GPUs. It may well turn out that a single GPU model is not the best option, especially if you have users whose needs cover a broad range of applications and datasets. In some cases, it may be adequate as well as more economical to only provide GPU back ends to the most demanding applications and leave all the work to CPUs for less demanding applications. The combination of XenDesktop and XenApp (Citrix Virtual Desktops and Applications) can work well in such a case as you can map specific XenApp applications to the XenDesktop and users will rarely even notice that those applications are being run on some remote server, yet benefit from the GPU back end that is working behind-the-scenes. It should be mentioned that there are also GPU offerings from AMD, including the S7100X, S7150, S7150x2 and V340 (the latter of which is currently not compatible with XenServer). Support for the AMD FirePro series started with XenServer 7.2 They range from 8 to 32 GB of memory. Intel offers its IRIS Pro graphics series of GPUs that are embedded in its newer E3-xxxx series CPUs. Everything is already integrated and GPU acceleration takes place for any application running on these processors. XenServer 7.1+ supports Intel GVT-d (virtual dedicated graphics acceleration, one VM to one physical GPU) and GVT-g (virtual graphics processing unit, multiple VMs to one physical GPU). In 2020 Intel will supposedly release its Xe line of GPUs, which could become serious contenders in the GPU field. For more about the Xe GPUs, see here. Note that currently, none of the AMD or Intel GPU offerings require any software licensing. As competition heats up, it can be expected that AMD will follow up with new offerings at some point, as well. The choice of server and which type and how many GPUs can be accommodated should also be carefully considered. The choice of server has to ensure that whatever GPUs it makes use of won’t hold back GPU performance and also vice-versa. There will always be a limit on how far you can push a component somewhere in the chain of dependencies and the idea is to strive towards not letting a single component dominate that limitation. There are cases where more, less powerful servers can be a better choice than fewer, more powerful servers, as well as the exact opposite. In some cases, it may take some experimentation to find what works best. Whether purchasing for an on-premises installation or figuring out what to rent in the cloud, it may take some experimentation to find a good configuration that best fulfills one’s needs. Cloud-hosted options make changes a lot easier, of course, since they are virtualized instances. No matter what operating system or hypervisor one uses, there are a number of commonalities that make for similar configuration guidelines. The following book contains a lot of very good information on this topic, even for those making use of servers that are not running VMware components, and I can highly recommend it: Johan van Amersfoort's VDI Design Guide. To maximize end-user density and light to moderate loads, one might decide that the NVIDIA M10 is still the best option for the price, or if greater power, memory or additional processing options are more important, perhaps the newer P4 or T4 units. Two T4 units will provide the same or possibly even greater end-user density while taking up the same amount of server space (two slots) as an M10, however with only about half the power consumption and quite a bit more GPU processing power. Cost, power, available slots, and GPU performance and what is supported on various servers will all play a role in the decision-making process. It is also important in the case of NVIDIA GPUs to have a thorough understand of the licensing models, which impact both the server and client sides and also require running a separate license server. The details are far beyond the scope of this article, but can be seen in detail in the recent document revision. If making a purchase of hardware, be aware that many vendors will sell only specific configurations that are known to work reliably and they are willing to support. If you are retrofitting a server with GPUs, be aware of its limitations and the possibility of voiding the warranty if certain limits are exceeded. Before Moving On To the Next Installment This article covered some additional details on the effects of various BIOS settings, a bit more about system internals, and GPU scheduling. The variety of GPUs to choose from makes the decision more difficult these days, but competition is generally a good thing for many industries. Good research and detailed discussions with sales engineers and possibly consultants should be planned for. All the tweaks possible may still not be enough to provide a solid enough platform if inadequate components are chosen at the onset. When testing, be careful to change if possible only one parameter at the time and run application benchmarks preferably that are sometimes available for the particular applications, themselves. Consider running a variety of benchmarks and repeat often enough to be able to get similar results. Load testing software can also be beneficial if used and interpreted correctly. Testing can be a very time-consuming process, but can make all the difference in terms of EUX as well as being able to right-size the system holistically. As mentioned in the beginning of this blog post, the next installment will go into the details that led to significant improvements in overall VM performance and will cover a variety of configuration topics that are important once your servers and GPUs are set up and ready to host VMs. Tobias Kreidl can be reached on Twitter at: @tkreidl

If you've been following my blog posts over the last couple weeks, you have probably seen two major products have released to web, one shortly after the other. Here are two of my more recent posts: NVIDIA Virtual GPU Software v8.0 Just Released! Now Supporting Windows 10 1809 and Windows Server 2019! - Dated April 16th, 2019 Citrix Hypervisor (formerly XenServer) 8.0 Just Released! Now Supporting Windows Server 2019! - Dated April 25th, 2019 As you can see from the dates on these two blog posts, XenServer 8.0 was released within two weeks of NVIDIA vGPU Software v8.0. Many of us in the Citrix/NVIDIA community were left wondering if this meant that we would have to wait another couple months for NVIDIA vGPU Software vNext, to see support for Citrix Hypervisor (XenServer) 8.0 added. As you can see from this screenshot taken at the time of launch, NVIDIA vGPU for XenServer 8.0 is not listed: http://blog.youngtech.com/wp-content/uploads/2019/04/Download_List-1.png Much to my surprise, when I checked the NVIDIA license portal over this last weekend, NVIDIA vGPU for XenServer 8.0 was already added and is now available for download! (Insert applause sound effects here...) http://blog.youngtech.com/wp-content/uploads/2019/04/Download_List_2019.4.28.png Seeing that both NVIDIA v8.0 and Citrix Hypervisor v8.0 are now available, I felt it would be appropriate to write a quick blog post to show how simple and easy it is to get started with these latest and greatest versions for a basic Proof of Concept environment. In my lab, I have a number of older K1/K2 cards, but my latest system is a server with an NVIDIA Tesla P4, which you can read about here: https://images.nvidia.com/content/pdf/grid/data-sheet/nvidia-p4-datasheet.pdf In this blog post, I will show you getting started steps for installation and configuration of Citrix Hypervisor 8.0 from a bootable USB drive, NVIDIA vGPU 8.0 Host Manager (hypervisor software), Windows Server 2019, and NVIDIA Driver installation. At that point, the basic POC gold image will be ready to install the Citrix Virtual Delivery Agent (VDA), connect to Citrix Cloud, and build the first Catalog/Pool of Server OS machines for App or Desktop publishing. Those steps are not included in this guide, as I have covered these concepts in previous blog posts, which you can find here and here. To get started, you will need to download the latest Citrix Hypervisor software, which you can obtain for free here: https://www.citrix.com/downloads/citrix-hypervisor/ Be sure to grab the Hypervisor installer (ISO) as well as the XenCenter installer for Windows: CitrixHypervisor-8.0.0-install-cd.iso, and CitrixHypervisor-8.0.0-XenCenter.msi respectively. Next, obtain the latest NVIDIA vGPU Software v8.0 installation ZIP from NVIDIA here: https://docs.nvidia.com/grid/ If you do not already have a license, you can obtain a Free Evaluation License for 90-Days, using the link provided at that site. Be sure to use the correct link for "NVIDIA vGPU for XenServer 8.0", which will have a filename of: NVIDIA-GRID-XenServer-8.0_418.66-418.70-425.31.zip Let's get started! To build a bootable USB installer for Citrix Hypervisor, I like to use this free utility, Universal USB Installer: https://www.pendrivelinux.com/universal-usb-installer-easy-as-1-2-3/ Configuring for Citrix Hypervisor is as simple as selecting the ISO and the following options: http://blog.youngtech.com/wp-content/uploads/2019/04/CTXHYP-0001.png Once Create is clicked, the formatting and installation process to the USB device will take 3-5 minutes, at which point it's ready to plug into the server and boot. You will notice a little Citrix rebranding has occurred with the splash screen now showing the following. Press ENTER to install http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0002-1024x873.png Select the appropriate keymap: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0003-1024x873.png OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0004-1024x873.png Accept EULA to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0007-1024x873.png Select the appropriate disk and check the box next to Thin Provisioning. OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0009-1024x873.png OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0010-1024x873.png Select Verify and OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0011-1024x873.png After a minute or two, media verification is complete: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0012-1024x873.png OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0013-1024x873.png Enter and confirm a password for the Citrix Hypervisor 'root' account: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0014-1024x873.png Select Static configuration and enter IP, Subnet Mask, and Gateway information. If appropriate, enter tagged VLAN information. OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0015-1024x873.png Enter a Fully Qualified Hostname (as registered in forward/reverse lookup in DNS), and a primary/secondary DNS Server. OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0016-1024x873.png Select the appropriate geographical area: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0017-1024x873.png Select the appropriate time zone: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0018-1024x873.png Select Using NTP: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0019-1024x873.png Enter NTP information. I typically prefer to use tick.usno.navy.mil and tock.usno.navy.mil; 192.5.41.40 and 192.5.41.41 respectively. OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0020-1024x873.png Select Install Citrix Hypervisor to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0021-1024x873.png The precheck and installation process will take a couple minutes to completed: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0022-1024x873.png After several minutes, you will be asked if you would like to install any Supplemental Packs. This would be a perfectly acceptable time to mount the NVIDIA Virtual GPU Software .ISO file, if you want to install the software during installation to save a couple steps and a reboot. For now, I'd like to demonstrate how easy it is to install through XenCenter, so I'll select NO for Supplemental Pack installation: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0023-1024x873.png Another couple minutes, depending on the speed of your server and local storage, and the installation will complete: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0024-1024x873.png Unmount the Citrix Hypervisor installation media (CD-ROM or USB) to proceed. OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0025-1024x873.png During boot, you'll notice the new Citrix Hypervisor splash screen: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0027-1024x873.png Once booted, a Citrix Hypervisor configuration console is displayed. If you properly set everything during installation, nothing should be required from this console. All additional configurations can be performed through XenCenter. http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0029-1024x873.png From a Windows admin system, I have downloaded and unzipped the NVIDIA installer, and downloaded the Citrix XenCenter installation MSI. First, we'll install XenCenter by double clicking on the installer. http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0030-1024x873.png Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0031-1024x873.png Select "All Users" and Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0032-1024x873.png Install to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0033-1024x873.png Finish: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0034-1024x873.png Launch Citrix XenCenter from the Start Menu: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0035-1024x873.png Click Yes to check for updates: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0036-1024x873.png Select Add Server: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0037-1024x873.png Enter the IP Address, root username (default root) and Password as specified during installation. Add to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0038-1024x873.png Click Save and restore to auto-connect XenCenter when launched. OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0039-1024x873.png You can use this screen to learn more or Enroll for Health Check. Otherwise, click Close. http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0040-1024x873.png First, I'll license the Citrix Hypervisor by connecting to an existing Citrix Licensing server in the environment. If you don't already have a Citrix Licensing server, you can download one and request a trial license as part of the Citrix Virtual Apps and Desktops evaluation process. I won't cover details on that process, but simply show how to connect Citrix Hypervisor to an existing Citrix Licensing Server. To do this, go to Tools -> License Manager: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0041-1024x873.png Click Assign License: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0042-1024x873.png Enter the IP Address (or DNS Name), and appropriate License version that aligns to the License Server. OK to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0043-1024x873.png If configured correctly, server will now show Licensed. Close to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0044-1-1024x873.png Next, we'll install the NVIDIA Software, using the folder previously extracted from NVIDIA-GRID-XenServer-8.0-418.66-418.70-425.31.zip. Navigate to Tools -> Install Update: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0045-1024x873.png Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0046-1024x873.png Select the bottom option (Select update or supplemental pack from disk). Browse to the Supplemental Pack .ISO file (should be the only ISO image in the extracted ZIP folder). Click Next: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0047-1024x873.png Select the appropriate server(s) to apply the update. Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0048-1024x873.png The file will take just a moment to upload from disk to the Citrix Hypervisor's local storage. Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0049-1024x873.png Installation prechecks will now take place. At this point in the installation, you may receive the following errors as show below: http://blog.youngtech.com/wp-content/uploads/2019/05/image-1.png The error messages may say "The update is not compatible with this server version." or "You need to resolve the issues found before you can continue." IF THIS HAPPENS...YOU MUST REDOWNLOAD THE NVIDIA VIRTUAL GPU SOFTWARE FROM THE FLEXNET LICENSING PORTAL There was an issue with the original NVIDIA .RPM file that was uploaded over the weekend. I discovered this issue, along with others in the community, reported the issue to NVIDIA Enterprise Support, and the .ZIP file has now been re-uploaded. Sidenote: The original/bad zip file has an dash between the 8.0 and 418 in the file name (NVIDIA-GRID-XenServer-8.0-418.66-418.70-425.31.zip). The new/good zip file has an underscore between the 8.0 and 418 in the file name (NVIDIA-GRID-XenServer-8.0_418.66-418.70-425.31.zip). If you downloaded the right zip file and goes well in the precheck, you should see the following all clear messages! Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0050-1024x873.png Since this is a non-production system, it's safe to let XenCenter carry out all tasks automatically, which involve restarting the server. Install update to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0051-1024x873.png As the update is installed, you can monitor the progress, including rebooting of the server: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0052-1024x873.png After a couple minutes, the installation is complete. Finish to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0053-1024x873.png Now that the NVIDIA Host Manager is installed, I'll mount an SMB Share where my Windows Server 2019 media is located. To do this, you can right click the server and select New SR: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0054-1024x873.png Select ISO library - Windows File Sharing (SMB/CIFS). Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0055-1024x873.png Name the Storage Repository. Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0056-1024x873.png Provide the path, username and password to connect to the SMB share. Finish to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0058-1024x873.png The new MDT SR shows available on the left panel. Next, we'll build our first Windows Server 2019 virtual machine. Right click the host and select New VM: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0059-1024x873.png Select Windows Server 2019 (64-bit) from the list. Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0060-1024x873.png Provide a Name for the new VM. Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0061-1024x873.png Select the ISO from the newly created SR in the drop-down. Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0062-1024x873.png Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0063-1024x873.png Change the VM specifications for CPU and Memory if desired. Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0064-1024x873.png Review the list of available GPU Profiles from the drop-down. This is also a good sanity check to confirm that the NVIDIA Host Manager is functioning correctly on the Citrix Hypervisor! Since I want to deploy two VMs on this host, I'm selecting the P4-4Q profile. Next to Continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0066-1024x873.png Click Edit to change the size of the system disk (60GB for example is a good starting point for Windows Server 2019). Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0067-1024x873.png Change the networking properties if necessary. Otherwise, next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0068-1024x873.png Review the configuration and click Create Now: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0069-1024x873.png Once created, the VM will boot automatically, and in my case, using the LiteTouch ISO, will connect to my MDT server to pull down the server image. Alternatively, you could mount the Windows Server 2019 installation media, it just requires more manual configuration than MDT. http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0074-1024x873.png Once booted to the image selection menu, I'll choose Deploy Windows Server 2019 Datacenter: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0075-1024x873.png After a couple minutes, the installation and Windows Update process will complete. Windows has detected the new network, Citrix Hypervisor tools installation, and MDT task sequence is finished. I'll click Yes to all prompts, which will reboot the VM. After a reboot, I'll login again, and download the NVIDIA package to the Desktop. http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0076-1024x873.png From the NVIDIA folder, I'll install the Windows drivers using 425.31_grid_win10_server2016_64bit_international.exe: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0077-1024x873.png Yes to the UAC prompt: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0078-1024x873.png OK to extract: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0079-1024x873.png Agree and Continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0080-1024x873.png Next to continue: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0081-1024x873.png Installation will take a couple minutes, and the screen will flicker/blank during driver installation: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0082-1024x873.png Once installed, the screen resolution will be nearly unintelligible. You'll need to resize if you intend to use the console. http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0083-1024x873.png Right click and select Display Settings: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0084-1024x873.png Select a resolution close to or equal to 1024 x 768 to make the Console easy to see. Close Display Settings: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0086-1024x873.png Close the Driver installation window: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0087-1024x873.png Open Device Manager and confirm the vGPU Profile and respective driver show up properly: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0088-1024x873.png Right click on the desktop and select NVIDIA Control Panel: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0089-1024x873.png Navigate to the Manage License section under Licensing. Specify the Primary License Server (IP or DNS) and Port Number. Much like the Citrix licensing server, this was previously configured in my environment. Click Apply. http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0090-1024x873.png For detailed information on the installation and use of the NVIDIA License Server, see the following: https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html After Apply has been clicked, a popup will appear in the bottom left "Acquiring NVIDIA license": http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0091-1024x873.png When a license is properly acquired, the popup will change to "NVIDIA license acquired.": http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0092-1024x873.png Under the licensed clients section of the NVIDIA License Server, the new client will be registered automatically: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0093-1024x873.png To verify the build of the new Windows Server 2019 Datacenter VM, right click the start menu and click System: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0095-1024x873.png Scroll down to the bottom and you'll see the Windows specifications, specifically the Edition, Version (1809), and OS build (17763.437) for example: http://blog.youngtech.com/wp-content/uploads/2019/05/CTXHYP-0096-1024x873.png At this stage in the process, we're ready to continue forward with the setup and configuration for Citrix Virtual Apps and Desktops using Windows Server 2019 with NVIDIA vGPU as our master gold image! Summary That concludes my overview guide for installing and configuring Citrix Hypervisor 8.0 with NVIDIA vGPU 8.0 and Windows Server 2019! I hope this guide has been helpful on your journey toward virtualizing the most graphics intensive Windows workloads. I am very excited to see these technologies are now available, as it's been a long time coming for Citrix Hypervisor/XenServer and NVIDIA GRID support of Windows Server 2019. Follow me on Twitter (@youngtech) or other social media platforms, or check back in for more news and announcements. Thanks for reading! Dane Young, MBA My Twitter | My LinkedIn This content was originally posted by Citrix Technology Professional, Dane Young, on youngtech.com. Click here to read the original post.

The previous installment discussed some more details of the influence of BIOS settings on performance and also discussed the importance of the choice and configuration of GPUs, including some discussion of the GPU scheduler, which is specific to the NVIDIA Tesla line of GPUs. In this follow-up blog to Part 2, the long-awaited detailed discussion of how the overall performance gains were realized over the stock configuration are presented. A number of interesting side topics are also brought into the discussion, and a summary of the overall tale is given at the end as a set of guidelines that may prove useful to those dealing with CPU/GPU configurations. That goal has certainly been my intention all along. NUMA, CPUs and Sockets/Cores-per-Socket Settings This covers a lot of territory, but will be kept relatively brief in the interest of not getting too long and too detailed. There are certainly a number of excellent articles on these topics. Many of these settings can make a difference in performance and it gets tricky because they are all inter-related to one degree or another, hence changing one parameter will likely have some sort of influence on the others. This, in turn, makes testing harder as ideally, you want to only alter one variable at the time so that you don’t have to try to untangle cross-correlations. In reality, it’s not always possible to do so, but fortunately, it is possible to see in some cases where another parameter starts to play a significant role after a change is made. Here is a table showing comparisons of the benchmark metrics with different vCPU and sockets/core-per-socket counts: These were all run with no other activity present on the server and under as identical conditions as possible. With the exception of the FPS minimum, a test of ten runs with one of the settings resulted in standard deviations from the mean of less than 1%. The various maximum values achieved are highlighted. Contrary to expectations, some of the results come close, despite quite different parameters. There are a number of very interesting observations that can be gleaned from this table. First of all, note that this particular server has 14 cores and 28 hyperthreads per physical CPU. There was also less memory assigned than half the total system memory (256 GB) to the VM, which would otherwise force virtual NUMA (vNUMA) to kick in regardless. In this case 119 GB was chosen and hence vNUMA will still kick in as soon as a VM is assigned more cores than are available on a socket. Note also that even if all the vCPUs can fit within a single NUMA node, depending on how much memory the VM uses and also when the VM was started up, there might not have been adequate memory available for it to all fit on the same NUMA node and hence the decreased performance due to the memory interconnect to the other NUMA node kicks in. I was made aware by Frank Denneman that for example with VMware ESX that the allocation of VMs is driven by trying to place them as best as possible to be contained within a single NUMA node without factoring in that the memory requirements may still force vNUMA to be invoked because of there not being enough available on a NUMA node. As will be emphasized later, the order in which VMs are started up therefore can be important! This topic goes deeper, though, because if hyperthreading is enabled, can’t you double the number of vCPUs and still avoid NUMA since the vCPUs can all be assigned to the same physical CPU socket (using, for example CPU pinning)? Yes, while this is true, the disadvantage of doing so can come about because you are trying to run processes on twice as many physical cores as exist and hence you are over-loading the number of execution resources. This is discussed and illustrated nicely in this article. Note that this depends further on whether or not the L1TF (L1 Terminal Fault speculative execution side-channel exploit – see for example this article) patch is in place or not as if so, no two workloads will share the same core. However, most systems will nevertheless recognize this as being non-optimal and therefore by default, the cores-per-socket count will create extra vNUMA nodes if the number of total vCPUs exceeds the number of cores (not hyperthreads!). Note that while this appears to be the case for XenServer/Xen, it differs among ESXi versions. Frank Denneman suggests in this case to look at PPD and VPD mappings and investigate with the vmdumper utility. The job of assigning vCPUs to cores is up to the CPU scheduler. Bearing that in mind, let’s see how badly this affects performance as well as what happens when NUMA kicks in. With 8 vCPUs, everything should fit nicely on one socket and this configuration, where they fit nicely, wins overall. With 14 vCPUs, this should just fill the cores of one CPU’s socket, and it does just about as well as an 8 vCPU configuration. With 16 vCPUs, vNUMA should kick in (16 exceeds the physical core count), but the hyperthreading effect is apparently still mostly avoided. The result is performance also very close to that of the 8 vCPU configuration. Note that configuring a VM with any odd number of vCPUs that exceeds the number of cores (or hyperthreads, in this case) on a socket can make for a suboptimal configuration and should generally be avoided (though there are some strange exceptions!). If the amount of memory exceeds that assigned to a socket, any odd number of vCPUs for a VM should typically be avoided since it can cause vNUMA to be invoked in such cases and in addition, create an imbalance in how the vCPUs are divided between CPU sockets and create potentially an even less ideal competition for memory (see, for example, this article). Even an even number of vCPUs can lead to poorer performance, as seen in the case of 26 vCPUs as it results in an odd number of cores being assigned to each virtual socket, which is bad for this particular CPU architecture and in addition, NUMA is apparently triggered. Note also that with this configuration, the GPU utilization falls from 97% to between 81% and 91%, and hence the GPU is being underutilized. I ran this test configuration again a couple of times to convince myself this was really the case, and the CPU utilization of the top two cores was really averaging to more like 60% and 50%, as opposed to the other cases where at least two CPUs were carrying a significant load and where the top two were both very close to the 60% mark. Exactly how this transpired isn’t clear. With 32 vCPUs, the number of cores (and even hyperthreads, in this case) per socket is exceeded, resulting in a double hit (invoking vNUMA and possibly making heavy use of hyperthreads on the same socket, depending on how the scheduler allocates vCPUs) and overall performance falls below that of the VM running with 8 or 14 vCPUs. It’s still evidently not as heavy a penalty when running with 26 vCPUs and the ensuing odd core count split between physical sockets. In addition to the vCPU count, note also the differences among the different settings for sockets and cores-per-socket. These settings will also influence the distribution of the vCPUs. It is evident in cases where vNUMA is avoided or barely present that the optimal configuration confines the vCPUs best as possible onto a single socket, hence the socket count is set to 1 and the cores-per-socket value to the total number of vCPUs. However, this is a virtual socket count that is not to be confused with the actual assignment of cores by the CPU scheduler. Note that in the case of 32 vCPUs, a single virtual NUMA (vNUMA) client is not even possible since 32 exceeds even the hyperthread count of a single socket (32 > 28), and therefore on XenServer, the minimum number of sockets assigned is two. It should also be noted that when more sockets are specified than there are physical sockets, this leads to the creation of so-called virtual sockets. The impact of this depends on various factors, and one good source to read more about this in detail is here. One comment regarding vCPU assignments should be mentioned, namely how to check what the actual corresponding CPUs are. The Xen/XenServer utility “xl” is very helpful here, since the Linux utility “numactl” is not part of the official XenServer distribution. The very useful utility “coreinfo” can be installed on Windows VMs. With “xl info –n” the general topology of the XenServer’s hardware can be displayed along with the mapping of the individual CPUs (including hyperthreads, if hyperthreading is available and enabled). One of the most useful commands for listing VM CPU associations is “xl vcpu-list <VM1> <VM2> <…>” which displays the domain ID (the same ID delivered from the command “list_domains”), the vCPU, assigned CPU, processor state and some additional information. If one or more XenServer VM “name-labels” are given, this restricts the display to just the VM(s) specified, otherwise all are displayed, including that of dom0. This allows one to immediately equate which CPU a XenServer vCPU is associated with and given the knowledge of the topology, the distribution can readily be seen and monitored and verified if NUMA has kicked in or not. Perhaps in a different article, various more advanced topics involving vCPU pinning, CPU pools, the consequence of VM migration, NUMA memory allocation, and more can be discussed, but this has already grown beyond the original intentions! Putting It All Together Here are a few additional points that cover some miscellaneous, but important configuration aspects. First of all, unlike the benchmarks run here, real-world situations will involve typically multiple VMs competing for resources, hence even if avoiding hyperthreads on a core, this is going to be offset by other VMs making use of the same cores and hyperthreads if there is any over-commitment of vCPUs for all VMs compared to the number of cores and hyperthreads. Performance gets then very complex, because a lot will depend on the nature of what the VMs are doing and how they compete for resources. I have often suggested that it’s probably best to try to group VMs together on a server that are doing similar sorts of tasks, as the distribution of resources might be easier to understand. If of course they are all very heavy-weight processes, this isn’t going to help the situation any and the only solution is probably running tasks on separate servers or possible pinned to specific CPUs. Still, storage and networking loads also factor in, thus even that may not result in much improvement. As to vCPU over-commitment, Nick Rintalan pointed out back in his Citrix blog how NUMA is unfortunately often ignored. He also came up with a “1.5:1 rule” in terms of how much over-allocation of total vCPUs vs. server physical cores. In his case, he illustrated this with a two-socket server with 8 cores per socket, hence 16 physical cores altogether. With hyperthreading this makes the total number of virtual CPUs (vCPUs) 32. As to the ideal total number of vCPUs to run on that server, Nick stated an oversubscription of vCPUs ought to lie somewhere between these two values of 16 and 32. His assertion was, “The short answer is somewhere in between and most of the time the best ‘sweet spot’ will be an over-subscription ratio of 1.5:1, meaning 24 vCPUs in this case.” Since then, he’s released an updated blog stating now that the “sweet spot” probably lies somewhere between 1.5 and 2. Given that, let’s now do a comparison. In my particular case, we have Intel Xeon E5-2680 v4 CPUs with 14 cores per socket x 2 sockets x 1.5 oversubscription factor = 42 vCPUs or with an oversubscription factor of 2, that would be 56 vCPUs. I am running two Citrix Virtual Apps (XenApp) instances per server (to leverage the two available GPU engines) and a Citrix Virtual Apps (XenApp) controller consuming just two vCPUs. That leaves in theory around 20 vCPUs (or 27 vCPUs in the case of an oversubscription of 2) per XenApp VM. A choice of 16 vCPUs per VM therefore seems like a good compromise. The benchmarks come within 3% of the 8 vCPU case and well under a difference of 1% of the 14 vCPU configuration (and recall, these tests were run with no other VMs at all active on the server!). The test with 26 vCPUs resulted in clearly inferior results. In addition, Nick released a rather interesting article that talks about the peculiarity of the Intel Haswell-EP v3 chips with their odd cluster on die configuration, and that’s something else to take into account. The lesson here is that one needs to know one’s CPU architecture and all its possible quirks! Nick Rintalan and Mark Plettenberg presented a Webinar in April 2019 entitled “Citrix and Login VSI: Scalability Update” which can be reviewed and which goes into detail about some of the impacts of newer server architectures and how scaling is impacted. It is well worthwhile spending an hour or so to watch. I would also like to cite the recent investigation that evidently ran parallel to this one, coming from Sven Huisman. His investigation involved loading multiple VMs onto a server with different vCPU counts to see what the impact would be; there were no GPUs involved in his study. His conclusion was the following: “With other hardware specifications, the advice is to start with half the number of cores of the physical CPU per VM with a maximum of 8 vCPUs per VM. The total number of vCPUs should be equal to the total of number of hyperthreaded cores.” Furthermore, it should be pointed out that the newer Intel Skylake architecture introduced a mesh-based architecture, which incorporates a number of advanced features not available in earlier chip sets. Without access to such a system, I was not able to yet run tests to see how different the results would be, but this points out the need to not assume that newer generation CPUs are going to behave the same and that tests should be run independently to find the optimum settings. For some details on the Skylake architecture, see for example this article. There will also be differences in the Cascade Lake, Cooper Lake and Ice Lake chipsets as the evolution continues. It should be noted in particular that Cooper Lake and Ice Lake are based on an all-new hardware platform. As to the ideal number of cores vs. CPU frequency, this is a subject of much debate and I’ll leave it pretty much at that. Clearly, for single-threaded applications, a faster clock rate is better and the load on the server should be such that the process stays in execution mode as long as possible. When considering ordering servers with GPUs or adding GPUs to existing servers, do note that many manufacturers will not support systems containing GPUs where the CPU power draw exceeds a certain wattage, primarily because of limitations on cooling. Some servers can be ordered with higher capacity cooling fans. Fortunately, some of the more contemporary GPUs also have much lower power requirements and in many cases, will operate with just passive cooling. Be sure to check the specifications on your servers carefully to avoid potential issues that void service agreements or may cause operational issues. To summarize: VMs are by default automatically distributed by the server’s CPU scheduler to be optimized as best as possible and avoid NUMA, if it can possibly be avoided If not enough cores and/or memory associated with a node are available, the VM will be split between NUMA nodes The order in which you start VMs can matter! To avoid vNUMA in some cases, launch your most important VMs first On XenServer, you can over-allocate vCPUs, but not memory Choosing the right number of vCPUs and sockets and cores/socket matters With more cores/socket, the CPU load gets split and apparently does not saturate a CPU as with a single core Extra vNUMA nodes are created if the number of total vCPUs exceeds the number of cores (not hyperthreads!), hence strive for one vNUMA node for a VM If not enough resources are available, a VM’s vCPUs and memory are evenly split between two physical CPUs on a two-node server, which can reduce efficiency Sometimes, you may be better off with more and smaller VMs Don’t overload a server with VMs that take up a total of too many vCPUs Caveat: some products license by socket count! Test your system with various configurations to find best optimization/compromise There is a limit on total CPU power when installing one or more GPUs: you balance the maximum CPU clock speed vs. the number of cores Why Are All These Details So Important? The question should not have to be asked, but it needs to be, especially since in my experience, when this topic is brought up, it seems to be something that relatively few have really thought through in detail or done much to address. As mentioned above, with changes to the BIOS and operating system as well as with more GPUs getting integrated into servers these days, being able to get the most performance possible out of your servers is important to not only deliver as good as possible end-user experience, but to also save money by optimizing equipment instead having to buy or lease more of it. Cache for the Merchandise The NUMA issue can actually be critical with large amount of data being processed. This can be made more evident by examining how CPUs deal with cache. There are several different cache layers, the types and sizes of which vary quite a bit over various architectures, but as early as the Sandy Bridge (v1) Intel CPUs, there were LLC (last level cache), L2 and L1, where the LLC is shared among cores and factors into the ring-based interconnect that takes place among the various components consisting of cores, graphics, the LLC, and the system agent (“uncore”). Details on that process can be found for example here. In one of his articles, Frank Denneman discusses the issue of “cache coherency” and illustrates the difference in access to various cache levels using the Sandy Bridge architecture. Latency is measured as the time needed to load data from the cache. Here is a table, showing typical cache latency values for this particular architecture: Cache Type Typical cache latency (CPU clock cycles) L1 4 L2 12 L3 26-31 Local memory 190 (from the local CPU) Remote memory 310 (up to this, from the other CPU) To quote Frank Denneman from the reference above: “When data is fetched from memory it fills all cache levels on the way to the core (LLC->L2->L1). The reason why it’s also in the LLC is because the LLC is designed as an inclusive cache. It must have all the data contained in the L2 or L1 cache.” This is why NUMA can take a big hit on application execution efficiency. While some of this can be improved upon through so-called cache prefetching and cache snooping, it’s still going to slow things down when any less efficient data transfer needs to take place. Summary Various tips and clarifications on topics such as best practices, evaluating settings, test results, and a variety of other bits and pieces of information have been pulled together here. The interdependencies complicate things, plus the real-world behavior of VMs will be always somewhat different than any test environment can try to replicate. Nevertheless, planning and testing can go a long way towards optimizing one’s configuration. These discussions have not even gone into taking into consideration other important factors, including storage and networking, which also influence the overall system performance. Storage can be a huge restricting element as can overloading servers with too many VMs. The XenServer dom0 instance not having enough memory resources, hence causing swapping, can be an easily overlooked bottleneck. Spending some time evaluating the way your servers are configured and loaded can make a large difference and the time invested is well worth the effort. Acknowledgements Thanks are due to a number of individuals who freely contribute their time and knowledge and are the source of many invigorating discussions. I would like to specifically thank Frank Denneman and Nick Rintalan for their publications that act as both as remarkable references as well as inspirations for others to dig deeper into the topics they so adeptly handle, as well as Frank for his willingness to scrutinize these two manuscripts and provide valuable feedback. Constructive suggestions were also provide by fellow NVIDIA vGPU Community Advisors (NGCA) Johan van Amersfoort, Thomas Poppelgaard, and Rasmus Raun-Nielsen. Tobias Kreidl can be reached on Twitter at @tkreidl

Citrix announced Workspace, Analytics, Security Practices and NetScaler Secure Web Gateway during the first day of Synergy 2017. See below a comprehensive list : Citrix Workspace Service (a.k.a. StoreFront++ or Codename Project Frontdoor) A new cloud-based service to deliver ALL (not just Citrix) apps and data on a unified and secure Workspace (portal) - software defined perimeter Citrix Analytics Service is cloud-based security and performance analytics service to get visibility across the XenApp, ShareFile, XenMobile, and NetScaler ecosystem Security Practices is a new Citrix consulting offer focused on Security assessments NetScaler Secure Web Gateway helps you deal with encrypted traffic and allows you to use NetScaler as a forward proxy XenApp/XenDesktop 7.14 (which was later pulled out from the Citrix website) StoreFront 3.11 with only bug fixes XenServer 7.2 including NVIDIA GRID vGPU XenMotion and Windows Continuum for Virtual Desktops Bitdefender Hypervisor Introspection (HVI) Security Virtual Appliance - Only available for XenServer 7.x App Layering 4.2 adding Windows 10 64-bit support into Labs Citrix System Optimizer (not available at this time) an equivalent of the VMware OS Optimization tool Citrix HDX Ready Pi 3 (SC-T25) with support for Dual Monitors and integrated Skype for Business optimization pack Citrix Workspace IOT including Smart Spaces for Collaboration Workspace Environment Management 4.3 Support for the new Windows 10 S operating system with Citrix Receiver UWP Secure Browser Essentials on Microsoft Azure is a simple cloud-based offering for secure remote access to the Internet that isolates end user internet browsing from enterprise endpoints and networks Support for Microsoft Skype for Business on Google Chromebooks Already released features/products that were also part of the announcements included: XenApp and XenDesktop Essentials on Microsoft Azure XenMobile integration with Microsoft InTune G-Suite integration with Sharefile Cisco and Nutanix joined HCI Ready Workspace appliance program NetScaler 12.0 & NetScaler MAS 12.0

There are various ways to update the BIOS and firmware on Dell servers, such as with the Dell Repository Manager (https://www.vmadmin.co.uk/other/332-dellbiosfirmwareupdaterepomanager) and even without needing Windows as a means of booting up and performing the updates by creating a bootable ISO, e.g. http://www.dell.com/Support/Article/us/en/04/SLN296511 or other means, such as listed in https://wiki.ubuntu.com/DellBIOS. For XenServer, these procedures require extra work and in some cases, multiple reboots. You really don’t want to incur any more downtime than necessary nor do you want to create problems by making changes on dom0 that could cause issues later with support or functionality. One procedure to streamline this was mentioned a number of years ago in the very nice article by Neil Spellings, CTP, at http://neil.spellings.net/2012/03/03/updating-dell-firmware-from-within-xenserver-dom0/ which entails creating a supplementary yum repository that can be used to leverage the Dell Linux Repository (DLR) utilities. Unfortunately, this has issues under XenServer 7.0, which jumped from being based on CentOS 5 to CentOS 7. The way yum repositories are now set up makes it harder (deliberately, of course, as Citrix doesn’t want people creating problems with foreign installs) to download or update “foreign” RPMs via yum. One such workaround was posted at http://discussions.citrix.com/topic/378880-xenserver-70-repositories/ but this doesn’t work as it stands any more, as for one, the release version is now different (and even making it a generic “7” which is then pointed to by the latest release isn’t all that’s necessary). It also calls for an additional xfsprogs utility to be installed. The DSU Utility One option under newer installations of Red Hat or SUSE is in lieu of DLR to instead use the DSU (Dell System Update) utility. This is a more flexible, newer option which is unfortunately currently limited to only the following operating systems: Red Hat Enterprise Linux 6.8 (x86_64) Red Hat Enterprise Linux 7.2(x86_64) SUSE Linux Enterprise Server 11SP4 (x86_64) SUSE Linux Enterprise Server 12SP1 (x86_64) That means of course that at present, only XenServer 7.0 falls into that category and also from my tests, is only compatible (at least for BIOS updates) with PERC 7 and higher controllers (7, 8. 9 and including the H700 and H800 series, as well as potentially some of the more recent H300 series), so likely just Dell 11th generation and higher servers. Even though the package can be made to install correctly on older servers, you get an error message indicting the DSU software is not compatible with the controller. I’m not sure about the H310, but it and the H330 are new enough to possibly also be compatible. Details on Dell server controllers can be found here: http://www.dell.com/support/article/us/en/19/SLN292279/. The DSU utility (aptly named dsu) has certain other advantages over the Dell Linux Repository (DLR) utility. First off, you can select individual updates to apply. It will also show you the current and latest versions of what’s installed and what is available. Under certain circumstances, packages can even be reverted (though I didn’t encounter any in my investigation that were). Instead of being built on the Open Build System, the packages are based on Dell Update Packages (DUP). It is even possible to create selected packages bundled into a bootable ISO or a directory. Setting up the repository on standard Linux boxes with the operating system listed above should be as simple as running the commands: #wget -q -O - https://linux.dell.com/repo/hardware/dsu/bootstrap.cgi | bash #yum install dell-system-update Unfortunately, this is not so simple given the XenServer 7.0 environment and how even enabling base and mirror collections isn’t always successful. Doing so can be quite messy and the thought here is to be as nonintrusive as possible. I have worked out a way to accomplish this quickly and easily with minimal changes to dom0. The goal is to keep the server from getting “polluted” with unnecessary changes and prevent any interference with the intended functionality of XenServer. Here are the steps: 1) Create the file /etc/yum.repos.d/dell-system-update.repo and insert the contents: [dell-system-update_independent] name=dell-system-update_independent baseurl=http://linux.dell.com/repo/hardware/dsu/os_independent/ gpgcheck=1 gpgkey=http://linux.dell.com/repo/hardware/dsu/public.key enabled=1 exclude=dell-system-update*.i386 [dell-system-update_dependent] name=dell-system-update_dependent mirrorlist=http://linux.dell.com/repo/hardware/dsu/mirrors.cgi?osname=el$releasever&basearch=$basearch&native=1 gpgcheck=1 gpgkey=http://linux.dell.com/repo/hardware/dsu/public.key enabled=1 (Note that if need be, the wget and yum commands can be run on a different server and the contents of the resulting repository copied and pasted into the file on the target XenServer, which would also be a good way to check for possible updates in the future. Also, should there be an issue resolving linux.dell.com, try substituting the IP address 143.166.82.43 for it in the four occurrences above.) 2) Run the command “yum install dell-system-update” and respond “y” to the various questions. 3) If necessary, in particular if you have private network, firewalls or border router limitations, you may need to make use of a yum proxy server. Assuming there is such machine available and is accessible, on your XenServer, edit the file /etc/yum.conf and enter at the end of the file the one line: proxy=http://IP-address-of-proxy-host:3128 for example, proxy=http://10.11.12.13:3128 ,where “3128” is the standard squid Web proxy port used by yum (change the port number to match your site’s requirements, if needed). That’s it. You should now have access to the “dsu” utility. The various options can be seen by running “dsu –help”. More about dsu and its options can be found here: https://linux.dell.com/repo/hardware/dsu/ . While the configuration of a yum proxy server is beyond the scope of this article, there are a number of good guides describing how to create reverse proxies, for example, this one for a squid proxy server: https://www.cyberciti.biz/tips/howto-rhel-centos-fedora-squid-installation-configuration.html. Depending on the environment, this last step may not be required at all. An Example Here is an example of the output when running the "inventory" option. Each of the individual components is identified and numbered and shown along with the current firmware level: # dsu --inventory Verifying catalog installation ... Installing catalog from repository ... Fetching dsucatalog ... Reading the catalog ... Installing inventory collector ... Fetching invcol_WF06C_LN64_16.12.200.896_A00 ... Verifying inventory collector installation ... Getting System Inventory ... 1. BIOS ( Version : 2.5.2 ) 2. Lifecycle Controller, 1.4.2.12, A00 ( Version : 1.4.2.12 ) 3. Dell 64 Bit uEFI Diagnostics, version 4247A0 Installer Revision 14.05.00, 4247.1 ( Version : 4247A0 ) 4. Power Supply ( Version : 09.13.59 ) 5. Power Supply ( Version : 09.13.59 ) 6. CPLD ( Version : 1.0.3 ) 7. PERC H710 Mini Controller 0 Firmware ( Version : 21.3.0-0009 ) 8. Firmware for - Disk 0 in Backplane 1 of PERC H710 Mini Controller 0 ( Version : DB08 ) 9. 12G SEP Firmware ( Version : 1.00 ) 10. iDRAC ( Version : 1.66.65.00 ) 11. NetXtreme BCM5720 Gigabit Ethernet PCIe (eth0) ( Version : 7.10.18 ) 12. NetXtreme BCM5720 Gigabit Ethernet PCIe (eth1) ( Version : 7.10.18 ) 13. NetXtreme BCM5720 Gigabit Ethernet PCIe (eth2) ( Version : 7.10.18 ) 14. NetXtreme BCM5720 Gigabit Ethernet PCIe (eth3) ( Version : 7.10.18 ) 15. Intel® Ethernet Converged Network Adapter X540-T2 ( Version : 0.0.0 ) 16. Intel® Ethernet Converged Network Adapter X540-T2 ( Version : 0.0.0 ) Here is a list of dsu options available: # dsu --help Usage: dsu [OPTION]... Options: -h, --help Help -v, --version Version -i, --inventory Perform Inventory -n, --non-interactive Non-interactive mode -g, --get-categories Get category values -c, --category=<CATEGORY1,CATEGORY2,...> List only updates of <CATEGORY1,...> categories -u, --apply-upgrades-only List only upgradable updates -d, --apply-downgrades-only List only downgradable updates -e, --apply-equivalent-updates Enable equivalent updates -l, --update-list=<FILENAME1,FILENAME2,...> Apply <FILENAME1,...> updates --config= Configuration file --source-type= TYPE=PDK --source= Source location --destination-type= TYPE=ISO|CBD --destination-location= Output location when destination-type is provided --bootable-log-location= File to write logs during server update --ic-location= Inventory Collector binary --input-inventory-file= Inventory XML file --output-inventory-xml= Path to save the Inventory XML file --preview Preview updates Typically, you would run dsu with option "a" (select all) and then "c" (commit). Below is the appearance of a different machine after an update was performed. Afterwards, the output below from the plain command “dsu” shows most are now at the highest rev level (the exception in this example is the iDRAC): # dsu Verifying catalog installation ... Installing catalog from repository ... Fetching dsucatalog ... Reading the catalog ... Verifying inventory collector installation ... Getting System Inventory ... Determining Applicable Updates ... |-----------Dell System Updates-----------| [ ] represents 'not selected' [*] represents 'selected' [-] represents 'Component already at repository version (can be selected only if -e option is used)' Choose: q - Quit without update, c to Commit, - To Select/Deselect, a - Select All, n - Select None [-]1 BIOS Current Version : 6.4.0 same as : 6.4.0 [-]2 Dell 32 Bit Diagnostics, version 5162 Installer Revision 14.05.00, 5162A0, 5162.1 Current Version : 5162A0 same as : 5162A0 [*]3 iDRAC6 Current Version : 1.97 Upgrade to : 2.85 [-]4 Firmware for - Disk 0 in Backplane 0 of PERC 6/i Integrated Controller 0 Current Version : HS11 same as : HS11 [-]5 PERC 6/i Integrated Controller 0 Firmware Current Version : 6.3.3.0002 same as : 6.3.3-0002 [-]6 Dell LifeCycle Controller v1.7.5, 1.7.5.4, A00 Current Version : 1.7.5.4 same as : 1.7.5.4 [-]7 PowerEdge R710 BCM5709 Gigabit Ethernet rev 20 (eth3) Current Version : 08.07.26 same as : 08.07.26 [-]8 PowerEdge R710 BCM5709 Gigabit Ethernet rev 20 (eth2) Current Version : 08.07.26 same as : 08.07.26 [-]9 NetXtreme II BCM5709 Gigabit Ethernet rev 20 (eth9) Current Version : 08.07.26 same as : 08.07.26 [-]10 PowerEdge R710 BCM5709 Gigabit Ethernet rev 20 (eth1) Current Version : 08.07.26 same as : 08.07.26 [-]11 NetXtreme II BCM5709 Gigabit Ethernet rev 20 (eth8) Current Version : 08.07.26 same as : 08.07.26 [-]12 PowerEdge R710 BCM5709 Gigabit Ethernet rev 20 (eth0) Current Version : 08.07.26 same as : 08.07.26 [-]13 NetXtreme II BCM5709 Gigabit Ethernet rev 20 (eth7) Current Version : 08.07.26 same as : 08.07.26 [-]14 NetXtreme II BCM5709 Gigabit Ethernet rev 20 (eth6) Current Version : 08.07.26 same as : 08.07.26 [-]15 Firmware for - Disk 2 in Backplane 0 of PERC 6/i Integrated Controller 0 Current Version : D598 same as : D598 Enter your choice : 3 [*]3 iDRAC6 Current Version : 1.97 Upgrade to : 2.85 Enter your choice: c (It turns out it’s not compatible, since this is a PERC 6 controller, and hence it stays as the level it already is. On servers with newer controllers, it would have allowed you to proceed.) Finally, here is an example of performing a successful update of an individual component. The inventory report showed the following for item 15: [ ]15 Dell Lifecycle Controller, 1.6.5.12, A00 Current Version : 1.6.5.12 Upgrade to : 1.7.5.4 To perform the update you would enter 15 to select that item,which then appears with an asterisk, indicating it has been selected: [*]15 Dell Lifecycle Controller, 1.6.5.12, A00 Current Version : 1.6.5.12 Upgrade to : 1.7.5.4 To then apply the update to this item, enter option "c". The results are as follows: Fetching Lifecycle-Controller_Legacy_Application_0WFGM_LN_1.7.5.4_A00 ... Installing Lifecycle-Controller_Legacy_Application_0WFGM_LN_1.7.5.4_A00 ... Collecting inventory... ......................................... Running validation... Dell Lifecycle Controller, 1.6.5.12, A00 The version of this Update Package is newer than the currently installed version. Software application name: Dell Lifecycle Controller, 1.6.5.12, A00 Package version: 1.7.5.4 Installed version: 1.6.5.12 Executing update... WARNING: DO NOT STOP THIS PROCESS OR INSTALL OTHER DELL PRODUCTS WHILE UPDATE IS IN PROGRESS. THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME UNSTABLE! ......................................................................................................................................... ......................................................................................................................................... ......................................................................................................................................... ......................................................................................................................................... ....................................... The update completed successfully. Done! Please run 'dsu --inventory' to check the inventory (Note that a reboot should be performed first or in one case I encountered, the inventory command just hung. I found several instances in which a cold boot was necessary.) Conclusions As with BIOS/firmware updates, a reboot at the end is necessary. A nice aspect of DSU is that the updates can be applied while the host is running and the reboot can be scheduled to take place at any convenient time. The DSU process is simple and flexible. It is, unfortunately, not functional currently with any XenServer version prior to 7.0, partly because of a library incompatibility with libcurl.so, and as mentioned above, Dell only officially supports DSU on RHEL 6.8 and 7.2, and given that XenServer 6.5 SP1 is based on CentOS 5, the incompatibility is not surprising. Looking forward to XenServer 7.1 and beyond, DSU can provide the desired functionality and versatility to perform firmware updates and above all, with a more pronounced degree of simplicity compared to DLR. The installation is also simplified by not having to manipulate any of the existing repositories, which can definitely be seen as a bonus as it is much less intrusive than what otherwise would be necessary. Disclaimer: Citrix frowns upon modifications to, as well as the installation of, additional packages onto XenServer. Do so at your own risk, knowing that such actions might void any service contracts and support.