XenApp Server Showing Load of 10000

Received an Incident, where users are not able to launch applications or while doing weekly health check ups, found load of the Xenapp servers are showing full load, i.e. 10000…… what to do…..

Please go through the below steps one by one to troubleshoot the issue and yes, it will get resolved for sure, though I am aware that it’s an old one, but sometimes everything consolidated under one page, makes things easier for everyone.

Steps

1. Please check the Load Evaluator assigned to the server. It shouldn’t be on Maintenance Mode.

2. If it’s not on maintenance, then find the rules of the load evaluator, which has been assigned to the particular server and check if the server is reaching its Threshold limit and if it is, then find in case anyway is there to reduce the load below Threshold.

e.g.- The server is having load evaluator set with respect to “CPU utilization” & “Memory utilization”, then check the Threshold value and check whether the CPU & memoty are being utilised until that value.

3. If the usage is normal and users are connected, then check the load again through “qfarm /load” command. It might be a spike in usage of the server as per the load evaluator assigned to it, which reflected in the report.

4. If the usage is normal and no users are connected, but still the Load is showing 10000, then stop the IMA service and start it again.

5. If still the Load is showing 10000, then go ahead and recreate the LHC through below command after stopping “IMA” and “MFCOM” service.

Dsmaint recreatelhc

Start the “IMA” and “MFCOM” services and check the load again.

6. If still the load is on 10000, then stop the “IMA” & “MFCOM” service and recreate the LHC and radecache through below command:

Dsmaint recreatelhc

Dsmaint recreaterade

And reboot the server and the automatic services like IMA and MFCOM should start themselves. Please check the load now.

7. If still not resolved, then assign another Load Evaluator to the server and reboot and check. If the load changes, then revert back to the original load evaluator and check again.

8. If still not resolved, then please check the below Registry Keys:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfDisk\Performance
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfProc\Performance
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfOS\Performance

And confirm that the following value does not exist:
disable performance counters = 1

If it is there, then change the value to 0 or delete the value altogether. No reboot is required for the changes to take effect.

9. Stop the Windows Management Instrumentation (WMI) Service and run the following command:

winmgmt /salvagerepository

winmgmt /resetrepository

And restart the XenApp Server. Please start the WMI service after reboot and check the load now.

10. If still not resolved, then Stop “Citrix WMI Service” & “Performance Logs & Alerts” service

a) Go to “C:\Windows\System32\”

b) Rename the files “perfc009.dat” & “perfh009.dat” to _OLD

c) Copy “perfc009.dat” & “perfh009.dat” from a working Citrix w2K8 R2 server to the same location.

d) Reboot the server

e) Once the server is rebooted, Start “Citrix WMI Service” & “Performance Logs & Alerts” service.

This should resolve the issue.

Links which contributed to this post

http://support.citrix.com/article/CTX132837

http://support.citrix.com/article/CTX129350

http://support.citrix.com/article/CTX131584

http://support.citrix.com/article/CTX115686

XenDesktop 7

Ok…. There is again a new version of Citrix Product in the market and as a technical person and being part of Citrix family, we ought to know what it is and why they have come up again with a new product like this, what are the benefits etc etc.

Well! In brief, this changes the whole concept & fundamentals of XenApp & XenDesktop, as they have removed IMA, removed FARM concept and merged both XenApp & XenDesktop into a single environment called XenDesktop7 and lots more.

Below are the key points and brief overview of the product. Haven’t included the in-depth analysis of each & everything, as lot many things are still a mystery, though not completely, until the product is launched in the market for download and tested thoroughly in & out.

What Is XenDesktop7

XenDesktop7 is a unified environment, where Applications & Desktops are delivered from the same platform.

This is the latest product in the XenApp & XenDesktop family, which combines both the famous previous products of Citrix, XenApp (Application Virtualization) & XenDesktop (Desktop Virtualization).

Before going through the below things, would advise to clearly understand XenDesktop architecture first, so that it will be easier to understand how the merger happened.

Key Differences Between XenApp And XenDesktop7

1. Unified Environment

    –   Previously, the application virtualization environment & the desktop virtualization environment, both were different products having their own different architecture and different nomenclature for their environment. But in XenDesktop7, both the virtualization environments are now delivered from the same unified single environment, providing both Applications & Desktops to the users.

2. Citrix Studio

    –   Again a name change to the AppCenter. The new name is Citrix Studio. You will use Studio for all the activities like configuring environments, provide accesses to users on applications/desktops etc. In XenDesktop 7, we have one more Citrix Director to monitor Health Checks and other HelpDesk related Activities like session monitring etc. So, in XenDesktop 7, we have 2 separate consoles to use, Citrix Studio & Citrix Director.

3. No IMA. It’s FMA

    –  The IMA(Independent Management Architecture) concept have been removed with  XenDesktop7, along with IMA service. FMA (FlexCast Management Architecture) replaces all the functionality and service of IMA.

4. Mobilization

   –   Citrix XenDesktop7 enables Mobile Workstyle, by delivering Windows Apps, Desktops from almost any device such as Tablets, Smartphones, Macs, PCs, Thin Clients etc.

5. No FARM concept, It’s Sites

   –   From this version onwards, the XenApp Farms are called as Sites.

6. No IMA Data Store For Applications like it used to be in XenApp

   –   From XenDesktop7 onwards, there is no data store concept like it used to be in all the XenApp previous versions. Everything will be stored in the Database in the similar way the architecture of XenDesktop 5 suggests.

7. No Support For Oracle as a Database

   –   It’s only SQL Server editions which is being supported by XenDesktop7. No more Oracle databases are supported in Citrix.

 8. No Separate Console For EdgeSight

   –   There are lot many features of EdgeSight related to Performance Management have been integrated with Citrix Director. No more EdgeSight installation option is provided while installing XenDesktop 7.

And the Network Analysis part has been integrated with Netscaler HDX Insight application. So, no more edgesight as a separate console or entity.

9. No Dedicated Zone Master

   –   No more ZDC in XenDesktop7. The function of user connection requests & communication with hypervisors have been distributed among all the Controllers in the site.

10. Support for W2K12

   –   XenDesktop7 can only be installed on W2K8 R2, W2K12 Server OS.

11. No Shadow Taskbar

   –   To view and interact other user’s sessions remotely, you have to use the Shadow Feature in the Citrix Director or Citrix Studio.

12. Support for 3D Graphics Applications

   –   Citrix has struggled a lot till XenApp 6.5 to enable IT persons to deliver High-end 3D Graphics softwares over it, as they are very much CPU & Memory intensive like AutoCAD etc. This time they have found the solution through GPU utlization, instead of CPU, while working on 3D Pro Graphics.

And there are lot more……… but the above ones, definitely are the major key differences, as it seems like point to be noted down.

Architecture

Hope the below diagram clarifies the whole architecture of XenDesktop7. If you have experience in XenApp or XenDesktop, it should be easier for you understand.

(Image Courtsey “citrix.com”)

Soon will be publishing a detailed Enumeration Process and Architecture details of XenDesktop7, once the E-Docs and product is available for download on Citrix official website.

 Benefits Of Using XenDesktop7

1. Cloud Ready Platform

   –   XenDesktop7 is built on a new powerful delivery platform, which means it is designated for IT-as-a-Service, where self-service is there for both users & IT persons for accessing their applications & desktops along with the flexibility of deploying infrastructure on all the types of clouds (private, public & hybrid).

What do we mean by the above is, XenDesktop7 delivers your Applications/Desktops on a secured environment, which is filtered through policy, and can be accesses through any device from anywhere. It reduces Cost and improves accessibility to your work, no matter wherever you are, whichever device you are using (BYOD).

2. Any Device Anywhere Anytime

–   BYOD is revolutionary, where you can “Bring Your Own Device” to office or can work from anywhere with a decent network connection. With XenDesktop7, accessing your work applications and desktop from any device you like, just makes work life balance a lot easier.

3. PC Refresh

–   Many organizations are facing aging PC fleets and to deal with that requires both labor & huge cost investment.

While developing XenDesktop7, Citrix has partnered closely with Thin client manufacturers to reduce costs and BYOD of course reduces the cost of labor involved from office point of view.

4. Better HDX

–   Professional graphics applications can now be delivered with high performance GPU sharing and H.264-based Deep Compression at frame rates more than 50% higher than previously possible, providing a smooth experience with excellent bandwidth efficiency, even with very large 3D models. For less demanding 3D use cases, new software rasterization technologies for DirectX and OpenGL provide a cost-effective alternative to hardware GPUs. And Desktop Composition Redirection leverages the DirectX graphics processing capabilities of Windows devices to deliver a highly interactive user experience.

–   Mobile workers will notice that the video frame rate on 3G connections at 800 Kbps is more than double what it was before. And HDX Windows Media Redirection, previously available for Windows and Linux, is now offered on Mac, iOS and Android, further improving server scalability by shifting the rendering to the user device. Furthermore, this technology can leverage hardware outboard from the CPU to conserve battery power on mobile devices.

–   Multicast support has been extended to Windows media, dramatically reducing the bandwidth required to deliver live video events to branch offices. And video caching with CloudBridge has been extended to support content fetched directly by the user device, for even more bandwidth savings.

–   Major enhancements to the delivery of Unified Communications applications from XenDesktop have been announced by Microsoft (Lync), Cisco (Jabber) and Avaya (one-X Communicator). And Local App Access provides a new solution for Windows apps that need or benefit from local resources, including Citrix GoToMeeting with HD Faces. (More on Local App Access in a future blog post.)

–   XenDesktop 7 introduces full support for Windows 8 with a high level of interactivity and graphics quality.

And lot more…. Seems like we have encountered a huge change in the Virtualization Environment with the launch of XenDesktop7, as it clubbed two virtualization technologies into one unified environment along with that reduction of cost for IT, enabling Cloud ready platform, delivering your office environment to the device you like are all what we can dream for……….. and with Flexcast management is just icing on the cake.

Few more links which you can refer to:

XenDesktop Architecture : http://www.packtpub.com/article/xendesktop-architecture

BYOD : http://en.wikipedia.org/wiki/Bring_your_own_device

FlexCast : http://www.citrix.com/products/xendesktop/features/flexible-delivery/flexcast-calculator.html

Reference : Introduction To XenDesktop7 Training

Update :

XenDesktop 7 is ready to be downloaded from http://download.citrix.com/downloads/xendesktop/product-software.html

And they have also added all the details to the Citrix E-Docs, which can be downloaded as a .PDF like, we used to do for XenApp 6.5 and XenDesktop 5.0 onwards:

http://support.citrix.com/proddocs/topic/xendesktop/cds-xd-7landing-page.html

It’s a mind boggling 1861 pages in total, but when we combine 2 major virtualisation products into one, you tend to expect that….. 🙂

Troubleshooting Clipboard Issue

Troubleshooting Clipboard issues has been a nightmare for all the Citrix admins since the beginning. There might not be anyone, anywhere in the whole world that hasn’t faced this issue in the Citrix environment.

Clipboard’s dependency on so many things makes it more difficult to troubleshoot and resolve the issues within minutes. Here are the steps which you can follow at your environment to get the issues fixed permanently. Please be careful with the Registry tweaks, as using Registry Editor incorrectly can cause serious problems that may require you to re-install your operating system.

To understand how the Clipboard thing works, the best explanation can be found out in the below link:

http://www.dumpanalysis.org/blog/index.php/2006/12/09/clipboard-issues-explained/

Steps To Be Followed To Resolve The Issue

1. Reset the user’s session in Citrix management Console and ask user to re-launch applications and check. The issue should get resolved if the issue is with the Citrix application, if not, then follow the next steps.

2. Check the permissions on the Citrix client on the user’s machine to check the Clipboard mapping permissions are enabled.

3. Check on the user’s machine whether RDPCLIP.EXE is active or not, if yes, then kill that particular process. (Though RDPCLIP.EXE is associated with RDP sessions, but sometimes it’s found that user’s machine does have RDPCLIP.exe running at the background, which causes this issue)

4. If no RDPClip process is found on user’s desktop, then search on the Citrix server on which the user is having active sessions. If found, kill the process.

5. Check on the Citrix server to which user is connected to and you should find Wfshell.exe running with the user’s name.

If not, then verify that the registry includes a subkey under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Citrix\Wfshell named Virtual Clipboard and the subkey has two values: DllName and NotifyEvent, both of type REG_SZ and respective entries of Vclipbd.dll and WfshellVclipbdNotify.

Verify Vclipbd.dll is on the system drive.

6. Tell the user to reboot the machine. Delete the temporary files & cookies from user’s TS Profile after the user is logged off from all Citrix sessions.

7. Log onto the Citrix server on which the user has got active sessions. Go to “Administrative Tools -> Terminal Service Configuration -> Right click on ICA-TCP -> Properties -> Client Settings” and uncheck, if “Clipboard mapping” under “Disable The Following” has been checked.

8. Ask the user to log off and replace the NTUser.DAT file with a new one.

9. Provide a new TS profile to the user to fix the issue. (You never know where might be the exact issue in the profile. Just to be on the safer side.)

This should fix the issue for the user with the above steps, but if lot many users are facing Clipboard issues frequently, then finding out the root cause for this issue is the only permanent solution, for which you need to follow the below steps and it may require approval from your Leads & the Change Management, as you are going to make some major changes on the Citrix servers and with the registry.

10. Publish “Repair Clipboard Chain” application for the user in Citrix. Can be downloaded from http://support.citrix.com/article/CTX106226

(The Repair Clipboard Chain must be installed on the same Citrix server, on which the user is facing issues with. If user is facing issues with a particular application, then Repair Clipboard Chain must be installed on all the servers hosting the particular application, so that it can be triggered on the same Citrix server, which user is connected to, at the time of error. Please follow the above URL to find out the process to install & uninstall the Repair CBD Chain)

11. There are lot many 3rd party applications available over internet which can get the issue resolved. One   of those is RDP-Clipboard-Fix which can be downloaded from here: http://www.remkoweijnen.nl/blog/2007/10/25/rdp-clipboard-fix/

This tool needs to be installed on the user’s machine, whenever the user gets the issue, he/she needs to run this application.

12. If still not resolved, then take few test applications & few test Citrix servers.

On the user’s local machine & Citrix server, make the below entries onto their registry through command or manually:

reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management” /v SessionPoolSize /t reg_dword /d 64 /f

reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management” /v SessionViewSize /t reg_dword /d 64 /f

And then observe for few days, whether the issue gets resolved for all the users or not.

13. If everything fails, then the issue can well be with the user’s local machine or 3rd party applications. Add the below registry value to the user’s machine and Citrix server hosting the 3rd party application is a must to permanently resolve the issue.

Under [HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ICA Client\Engine\Configuration\Advanced\Modules\WFClient], add the following REG_SZ value:

CbChainInterval=<value>

Where <value> is the interval, in milliseconds, at which checks are performed. Supported values range from 0 to 2000, inclusive.

And reboot the machine and server. This registry fix has resolved the issue in more than 95% of the cases, as the 3rd party applications are responsible for the Repair Clipboard Chain to be broken.

Links which contributed towards the content of this blog are:

http://support.citrix.com/article/CTX106226

http://support.citrix.com/article/CTX086028

http://blogs.msdn.com/b/rds/archive/2006/11/16/why-does-my-shared-clipboard-not-work-part-1.aspx

http://www.remkoweijnen.nl/blog/2007/10/25/rdp-clipboard-fix/

http://blogs.citrix.com/2009/05/28/how-to-repair-clipboard-functionality-in-citrix-xenapp-and-is-it-time-to-enable-the-fix-by-default/

http://www.dumpanalysis.org/blog/index.php/2006/12/09/clipboard-issues-explained/

http://support.citrix.com/article/ctx112434

http://support.citrix.com/article/CTX102977

And A Big Thanks to my colleague Gordon Wilton as well, for guiding & helping me with this issue.

Enumeration & Application Launch Process In XenApp 6.0/6.5

As we all are aware that when XenApp 6.0 came to the market, the coders rewrite the whole code for the new version and made significant changes to a lot of things including few steps in enumeration process as well. This blog takes you deep down into the back-end process including all the steps, which makes things possible for a user to launch citrix application through web interface.

Application Enumeration Process

The application enumeration process is as follows:

1. A user launches a Web browser then types the web URL.
2. It connects to Web Interface & the Web Interface returns the logon page.
3. The user types in credentials.
4. The user’s credentials are then forwarded from Web Interface server to XML service of the XML Broker (if there is no XML Broker configured separately, then Data Collector acts as the XML Broker) in plain text form with the exception for passwords only, which gets sent using obfuscation.
5. XML service then sends that information to the local IMA service in the form of HTTP/HTTPS.
6. The IMA service then forwards them to the local server’s (XML Broker/Data Collector) Lsass.exe (Local Security Authority Subsystem Service), which resides in C:\Windows\System32 folder.
7. The Lsass.exe encrypts the credentials and then passes them to the Domain Controller of the local domain and then to the relevant domain if required (in case the user is logging onto a multiple domain environment).
8. The domain controller returns the SIDs (user’s SID and the list of group SIDs) back to Lsass.exe of the XML Broker/Data Collector and then to the IMA service.
9. IMA service uses the SIDs to search the Data Store for the list of applications and the Worker Group Preference Policy for the authenticated user.
10. If Data Store is not available, then the communication fail over to Local Host Cache (LHC) in the XML Broker/Data Collector to get the details for the user.
11. The list of the applications together, with the user’s worker group preference policy, is returned to the XML service and then on to the Web Interface.
12. The Web Interface uses its Java objects to create the web page which contains the application set for the user and then user’s Worker Group Preference Policy get cached in the Web Interface server.
13. Then web interface server returns that web page to the user’s web browser.

(This completes the application enumeration process).

Application Resolution Process  

The application resolution process is as follows:

1. User selects the application by clicking the application icon (such as Microsoft Word).
2. The selected application data (Microsoft Word) is passed back to Web Interface.
3. Web Interface passes the application (Microsoft Word) information, together with the user’s Worker Group Preference policy, back to XML service on XML broker server/Data Collector.
4. XML service then forwards the information to the IMA service.
5. Then the information is forwarded to the IMA service of the Zone Data Collector (If the XML Broker & Data Collector is the same server, then this part will not exist)
6. The Zone Data Collector tries to find out a least loaded server as per the Worker Group preference list.
7. When it finds the least loaded server, it sends a query to the Citrix Service Manager of that server to verify whether the server has the required application installed, if the answer is no, then it searches next least loaded server as per the worker group.
8. If the answer is yes, then IMA service provides the host ID details to the IMA service of the ZDC.
9. ZDC then transfers this information to the XML broker through IMA service.
10. The XML broker translates this host ID into its relevant IP address by searching in the Local Host Cache of itself.
11. The IP address is then provided to Web Interface by the XML service of the XML Broker server.

(This completes the application resolution)

Application Launch Process  

The application launch process is as follows:

1. Web Interface uses this IP address to create the Launch.ICA file. (Please note that if the user is coming from an external environment through internet, then the Launch.ICA file contains the Fully Qualified Domain Name (FQDN) or Domain Name Server (DNS) name of the server running the Secure Gateway or Citrix Access Gateway. IP Address of the XenApp server is never revealed to the Citrix client.)
2. The ICA file is then returned to Web browser on the client machine.
3. Citrix Web plug-in/Citrix Online Plug-In uses the ICA file to launch an ICA connection to the least loaded XenApp server.
4. The XenApp server requests the License Server to provide a license.
5. Once the license is allocated by the Citrix.exe (Vendor Daemon) after checking the availability, then the server launches the application and then sends it to the user’s screen to interact.

(This completes the application launch process)

The below diagram shows the process for the users coming from outside of the environment, but for the domain users authenticating from inside, also go through the same procedure where as DMZ zone is not involved and the web interface servers reside inside the Internal firewall.

Few links which contributed towards the content of this blog are:

http://support.citrix.com/article/CTX129585

http://support.citrix.com/article/CTX133307

http://support.citrix.com/proddocs/topic/web-interface-impington/wi-how-web-interface-works-gransden.html