Blog Archives
The Return of the Search Application Topology Component Health State Error
Today I was prepping a SharePoint 2013 VM and noticed the Search Service Administration page in Central Admin didn’t display the Search Application Topology health indicators. Instead it showed an error I hadn’t seen in a while:
“Unable to retrieve topology component health states. This may be because the admin component is not up and running”
I had encountered this error several times during the SharePoint 2013 preview phase. Then it turned out to be a genuine bug and fixing it required the installation of several hotfixes as detailed in the SharePoint Server 2013 Known Issues article. However, on this particular VM I had installed the RTM versions of Windows Server 2012, SQL Server 2012 and SharePoint 2013. I also had installed the latest updates and the SharePoint 2013 Prerequisites Installer should have taken care of other missing stuff. So this bug shouldn’t have happened again.
Just to be sure I tried installing the four hotfix packages, but (of course) this failed as they were either already installed or not relevant anymore. Now, I was pretty sure I had seen fully working Search Application Topology health indicators the night before (yeah, we SharePointies are nightly creatures). So what could have changed?
Then I remembered I had performed a little tweak to minimize SharePoint 2013’s memory usage. As you may know SharePoint 2013 has some very high memory requirements, especially around Search. Specifically the Search Engine’s NodeRunner.exe processes are memory hungry beasts. A SharePoint 2013 server configured for Search typically has five such NodeRunner.exe instances running, each consuming hundreds of megabytes of memory. For production environments this may be just fine, but on my development and demo VM’s I like them a little less greedy.
From earlier blog posts I learned it is possible to limit the amount of memory a NodeRunner.exe instance uses, by adjusting the memoryLimitMegabytes parameter in the C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\NodeRunner.exe.config XML file. In fact, in the SharePoint 2013 Preview version the NodeRunners suffered from a memory leak and adjusting this config was a quick hack to fix this leak. One of the last things I did last night was change this value from 0 (unlimited) to 100 MB. However, I hadn’t actually come around to testing the outcome of this change (yes, I do need some sleep).
...
<nodeRunnerSettings memoryLimitMegabytes="100" />
...
So the first thing I did was starting SysInternals Process Explorer (which I always use instead of the built-in Task Manager) and searched for the NodeRunner.exe processes.
To my surprise there weren’t any…
The NodeRunner.exe instances are spawned by SharePoint’s Search Host Controller Service, so the first thing I did was go to the “Manage Services on Server” page in Central Admin and restart that service. Thanks to Process Explorer’s handy ‘Difference Highlighting’ feature it was clearly visible that several NodeRunner instances got started, but almost immediately stopped running.
To get a little more information I started SysInternals Process Monitor (another indispensable tool) and logged the system while restarting the Search Host Controller Service. From Process Monitor’s ‘Process Tree’ dialog it was clear (too) many NodeRunner instances were started, but somehow weren’t able to survive for long.
Process Explorer also showed me the NodeRunner processes were started using the following command line parameter:
"C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\NodeRunner.exe" --noderoot "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\9E4952\IndexComponent1" --addfrom "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\9E4952\IndexComponent1\Configuration\Local\Node.ini" --tracelog "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\9E4952\IndexComponent1\Logs\NodeRunner.log"
From the bold piece above I learned NodeRunners keep a trace log. When I checked that file, the cause for the failing of the NodeRunners became pretty clear:
Caught exception in node activator: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Microsoft.Ceres.CoreServices.Node.NodeActivator.ActivateNode(IDictionary`2 configuration)
--- End of inner exception stack trace ---
Since I figured this must be due to me changing NodeRunner’s memory limits, I reset the NodeRunner’s config file (hey, always keep a backup!) and again restarted the Search Host Controller Service. This time the NodeRunner instances continued to run just fine. So it became clear to me there apparently is some minimum amount of memory a NodeRunner instance needs to be able to run and in my case it needed more than 100 MB.
With the NodeRunner instances running just fine, the Topology Health indicators also worked again, showing my farm’s Search stuff was in perfect health. I experimented with the NodeRunner memory limits some more and finally settled for a 250 MB limit. Not as small as I’d liked them to be, but still a significant decrease when compared to their original memory footprint.
I do want to make clear that even with this 250 MB limit I experienced some NodeRunner crashes. The general advice is to NOT change the NodeRunner memory limit configuration. And NEVER EVER do this in a production environment!
Hopefully this blog post helps out other people who might stumble upon this problem.
Building a SPQuery ViewFields string
If you’re querying SharePoint content using a CAML query from code it’s a good habit to always populate the SPQuery instance’s ViewFields property. Otherwise the returned SPListItem instances might not contain any data for certain fields and throw an exception when you try to access those fields.
Specifying ViewFields involves creating a string of CAML FieldRef elements. For instance if we want our query to return items that contain data for the Title, Created and ID field we use this:
1: <FieldRef Name='Title'/><FieldRef Name='Created'/><FieldRef Name='ID'/>
As you can see there’s some overhead of boilerplate markup involved. I’ve written a small piece of code that I always use to make my life a little easier. Today I happened to post this code in a reply I wrote on the MSDN forums and also decided to submit it as Community Content to the official SPQuery docs on MSDN. Then I thought I might as well share it with you here. So here it is:
1: public static string BuildViewFieldsXml(params string[] fieldNames)
2: {
3: const string TEMPLATE = @"<FieldRef Name='{0:S}'/>";
4: StringBuilder sb = new StringBuilder();
5: foreach (string fieldName in fieldNames)
6: {
7: sb.AppendFormat(TEMPLATE, fieldName);
8: }
9: return sb.ToString();
10: }
11:
12: // Use it like this:
13: SPQuery query = new SPQuery();
14: query.ViewFields = BuildViewFieldsXml("Title", "Created", "ID");
15:
16: // Note that you can specify a variable amount of string parameters, i.e.
17: query.ViewFields = BuildViewFieldsXml("Title", "Created", "ID", "Author", "Gender");
Yeah, you’re right. This piece of code isn’t exactly rocket science. But you might appreciate it anyway 🙂
SharePoint Frustrations #1: The undocumented “IncludeTimeValue” CAML attribute
I’m planning on doing some posts about frustrating things I have encountered (and still do!) during my SharePoint development efforts. Here’s the first one:
Last year while working on a MOSS 2007 project for one of our customers I stumbled on what I thought was a bug in SharePoint 2007. I had created a custom list that was filled with Electronic Program Guide (EPG) information for the streaming video media that that site contained. I then created a Webpart that used ASP.Net AJAX to continually show the actual EPG information below the video stream.
In order to obtain the EPG items I used a CAML query to query the EPG list. The list was simply a custom list containing amongst others a column of type DateTime that was called "ProgramEnd". As the name suggests it contained the time the program ended. I then used the following code to create a query that was supposed to obtain the currently broadcasted item and all future items.
1: const string EPG_QUERY_TEMPLATE = @"
2: <Where>
3: <Geq>
4: <FieldRef Name='ProgramEnd' />
5: <Value Type='DateTime'>{0}</Value>
6: </Geq>
7: </Where>";
8:
9: SPQuery query = new SPQuery();
10: query.Query = String.Format(EPG_QUERY_TEMPLATE,
11: SPUtility.CreateISO8601DateTimeFromSystemDateTime(DateTime.Now));
12: // ...
To my surprise this query always returned too many items. Further investigation showed that time part of the query seemed to be ignored completely, so it returned the items as if no time was specified! So instead of getting the current item and all future items it returned all items broadcasted for that day.
After wasting a lot of time debugging, using the U2U Caml Query Builder and looking for an answer / solution on the internet I gave up and wrote a quick hack around it, which fortunately wasn’t very difficult. It would just run the query and run the results through some additional code that checked the ProgramEnd DateTime field and filter out the wrong items, like this:
1: SPListItemCollection results = ...; // The results from the query mentioned above
2: // Trim the results to only include the current and future items
3: List<SPListItem> trimmedItems = new List<SPListItem>();
4: foreach (SPListItem result in results)
5: {
6: DateTime programEnd = (DateTime)result["ProgramEnd"];
7: if (programEnd >= DateTime.Now)
8: trimmedItems.Add(result);
9: }
Fortunately this worked just fine and the customer was happy. I was not… 😦
Today while surfing the Net I stumbled on this entry in the MSDN forums, from which I learned it wasn’t a bug, but that you need to included the "IncludeTimeValue" attribute to the CAML query, like this:
1: <Where>
2: <Eq>
3: <FieldRef Name='programEnd' />
4: <Value Type='DateTime' IncludeTimeValue='TRUE'>
5: 2008-07-30T12:00:00Z
6: </Value>
7: </Eq>
8: </Where>
If only I had known it was this simple… What bothers me is that this little, but very important attribute seems to be totally undocumented. I couldn’t find any information about it in the WSS / SharePoint SDKs.
Ofcourse once I knew what to look for I found some other blogs and forums mentioning this issue. It turns out the UCSharp blog had already blogged about this way back in October 2007, only a few months after I searched for it. Even Karine Bosch, U2U’s "CAML Girl" and author of the famous U2U CAML Query Builder, says she only recently found out about this. Fortunately she has included support for the "IncludeTimeValue" attribute in her latest version of the U2U Caml Query Builder, which I know a lot of SharePoint developers use to construct and test their CAML queries.
I also noticed that someone called puneetspeed has added some Community Content to the online SharePoint SDK’s SPQuery docs explaining this issue. So hopefully Microsoft will add information about the "IncludeTimeValue" attribute to the official SDK text in the near future.
Do not reuse SPQuery!
Last week I was refactoring some of my SharePoint code. I stumbled on a loop that created a new SPQuery instance for each iteration. The code was something like this:
1: SPList list = GetCommentsList();
2: const string VIEWFIELDS = "";
3:
4: foreach (string param in params)
5: {
6: // Create fresh new SPQuery instance...
7: SPQuery query = new SPQuery();
8: query.ViewFields = VIEWFIELDS;
9: query.Query = BuildQuery(param);
10:
11: // And use it...
12: SPListItemCollection items = list.GetItems(query);
13: ProcessItems(items);
14: }
As the value of the ViewFields property remained the same for each iteration I decided to create just one SPQuery instance and reuse it, like this:
1: SPList list = GetCommentsList();
2: const string VIEWFIELDS = "";
3:
4: // Create just one SPQuery instance...
5: SPQuery query = new SPQuery();
6: query.ViewFields = VIEWFIELDS;
7:
8: foreach (string param in params)
9: {
10: // And reuse it...
11: query.Query = BuildQuery(param);
12: SPListItemCollection items = list.GetItems(query);
13: ProcessItems(items);
14: }
To my surprise this code didn’t work! The first time the SPQuery instance was used it worked just fine. However, during the next iterations of the foreach loop it didn’t seem to get updated.
So, what have we learned today? Never reuse SPQuery instances!
Well… That’s not entirely true. You can reuse SPQuery instances for a very valid reason. If you use the RowLimit property you can limit the number of items returned in the query, which is useful for paging as seen in this sample code (taken from MSDN):
1: using (SPWeb oWebsiteRoot = SPContext.Current.Site.RootWeb)
2: {
3: SPList oList = oWebsiteRoot.Lists["Announcements"];
4: SPQuery oQuery = new SPQuery();
5: oQuery.RowLimit = 10;
6: int intIndex = 1;
7:
8: do
9: {
10: Console.WriteLine("Page: " + intIndex);
11: SPListItemCollection collListItems = oList.GetItems(oQuery);
12:
13: foreach(SPListItem oListItem in collListItems)
14: {
15: Console.WriteLine(oListItem["Title"]);
16: }
17: oQuery.ListItemCollectionPosition =
18: collListItems.ListItemCollectionPosition;
19: intIndex++;
20: } while(oQuery.ListItemCollectionPosition != null);
21: }
So, only reuse SPQuery instances if you use paging. If you change the actual CAML query you should create a new SPQuery instance for it.
Generating GUIDs in the Visual Studio IDE
If you’re programming on Windows you’ll often come across a situation where you have to generate a new GUID. In fact I find myself generating new GUIDs more than ever! I used to use them only for COM/ActiveX and Windows shell integration stuff. Nowadays however I spit out several GUIDs a day, especially for projects involving WiX and SharePoint. In fact, I tend to use them everytime I need a unique identifier. Here’s one I’ve generated especially for this blog post:
{1B87C5E2-0818-4c21-ADFC-741E8D8C1B3D} 😉
The classic tool of choice for generating GUIDs is ofcourse GuidGen.exe, which ships with all the major Microsoft SDK’s and can also be started from the Tools menu of Visual Studio. GuidGen is a small tool which gets the job done, but somehow I have never gotten used to it. I don’t like the fact that I have to leave the Visual Studio IDE to use it and that I cannot control the output format of the GUIDs GuidGen generates. For instance when using WiX you need to use GUIDs without the accolades, which means that a GUID generated by GuidGen needs some editing before it can be used. Yawn…
In the past I was a pretty avid Borland Delphi programmer. One of the nice things of the Delphi IDE, and one that I used very often, was that you could press the keyboard shortcut <ctrl> + <alt> + G and it would place a fresh new GUID at the cursor’s position in the source code editor. I really missed this functionality in Visual Studio.
Fortunately Visual Studio can be easily customized using macro’s and it was easy hacking together a little macro that does the job. I’ve used it for several years now and shared it with many colleagues and friends. Here it is:
Sub Create_GUID()
Dim sel As TextSelection
sel = DTE.ActiveDocument.Selection
sel.Text = System.Guid.NewGuid.ToString("D").ToUpper()
End Sub
Just add this macro to Visual Studio and assign it to the keyboard shortcut of your choice. I use <alt> + G, since the original Delphi shortcut I mentioned above is already in use by Visual Studio by default (it point to the Debug.Registers function).
Ofcourse you can easily modify this macro to suit your personal needs. If you like the way GuidGen bakes ’em, here it is:
Sub Create_GUID
Dim sel As TextSelection
sel = DTE.ActiveDocument.Selection
sel.Text = "{" + System.Guid.NewGuid.ToString("D").ToUpper() + "}"
End Sub
Or maybe you prefer underscores instead of hyphens:
Sub Create_GUID
Dim sel As TextSelection
sel = DTE.ActiveDocument.Selection
sel.Text = Replace(System.Guid.NewGuid.ToString("D").ToUpper(), "-", "_")
End Sub
Or you might be a lower-case fanatic:
Sub Create_GUID
Dim sel As TextSelection
sel = DTE.ActiveDocument.Selection
sel.Text = System.Guid.NewGuid.ToString("D").ToLower()
End Sub
Anyway, I think you get the point…
Splitting a Virtual PC VHD
Two weeks ago me and some colleagues attended the U2U course Developing Microsoft SharePoint 2007 Solutions Part 2: Advanced, led by the legendary Jan “SmartPart” Tielens. I can recommend this course to everyone interested in SharePoint development. It’s filled with interesting topics and it was put together by the also legendary Patrick Tisseghem. During the course we had to practice the stuff we had learned by going through several hands-on labs. For this U2U provided us with a Virtual PC virtual hard disk image (VHD), that contained a complete SharePoint 2007 developer environment (Windows Server 2003, WSS 3.0, MOSS 2007, Visual Studio 2005, SQL Server 2005, Office 2007 etc.).
A couple of days ago, back at our office, me and my colleague Harmjan had some spare time and wanted to practice some of the hands-on labs from that course. Unfortunately the hard drive of my Harmjan’s laptop didn’t have enough free space to contain the U2U course VHD, which is about 11GB in size. He did however bring his brandnew iPod, which still had plenty of free disk space. So he asked me to copy the VHD to his iPod, so he could run it from there.
To my surprise the file copy operation failed! As it turned out Harmjan’s iPod was formatted with the FAT32 file system. One of the main limitations of this file system is, that the maximum size of a file you can put on it is 4GB (well, it’s 232-1 actually…). That meant the U2U VHD would never fit. Ofcourse I could have reformatted the iPod to NTFS, but since an iPod’s firmware doesn’t support this file system it would mean the iPod couldn’t be used as an iPod anymore.
So there had to be an alternative approach. I remembered I had once encountered a situation where a VHD was splitted into several smaller chunks. I searched for this behaviour in the Virtual PC help file and found this:
In some situations, Virtual PC might automatically split a virtual hard disk file into smaller files because of file-size limitations imposed by the host operating system. For example, if the virtual hard disk is stored on a FAT32 volume of the host operating system, it is split into multiple 4GB files.
So this was it! We just had to find a way to split the U2U VHD into several smaller chunks (each less than 4GB). But how to do it? As far as I know Virtual PC has no option to split an existing VHD into smaller chunks. The Virtual Disk wizard only supports this process in the opposite direction, i.e. merging several VHD chunks into one big VHD. I searched the internet, but didn’t find a solution. I did encounter several posts of people who had asked this question before. One of these posts was located on Ben Armstrong’s Virtual PC Guy blog. Unfortunately, from the comments on that post I concluded even he didn’t seem to have an answer. I still sent him an e-mail, asking how to do it, though.
While waiting for a reply from Ben I got bold and downloaded a copy of the official VHD specs from Microsoft. Maybe I could hack together a little tool that would do the job. The only information I could find about splitted VHD’s was this:
Versions prior to Microsoft Virtual Server 2005 supported splitting of disk images, if the disk image grew larger than the maximum supported file size on the host file system.
Some file systems, such as the FAT32 file system, have a 4-GB limit on file size. If the hard disk image expands more than 4 GB, Microsoft Virtual PC 2004 and previous versions will split the hard disk image into another file. The split files do not have any headers or footers, just raw data. The last split file has the footer stored at the end of the file. The first file in the split disk image has an extension of .vhd. The following split files use the .v01, .v02, … filename extension. The split files will be in the same directory as the main hard disk image. The maximum number of split files that can be present is 64.The size of the split file cannot be altered.
There it was! As it turns out a split VHD is nothing more than a normal VHD, chopped up into smaller chunks. No extra headers or footers, just a raw file splitting operation. I couldn’t believe it was this simple.
So I quickly downloaded a copy of the popular HJSplit file splitting utility and split the U2U VHD into several chunks, using a maximum chunk size of 3.5GB (I could have used a larger chunk size, but I wanted to be on the safe side). I then renamed the chunks, so they matched the pattern required by Virtual PC (i.e. U2U.vhd, U2U.v01, U2U.v02, U2U.v03, …). I copied the chunks to the iPod and created a new Virtual PC instance (.vmc), that used the newly, splitted, VHD. And guess what… It worked! I experimented with several other chunk sizes and it doesn’t really seem to matter what chunk size you choose.
I immediately notified Ben Armstrong of my findings and in his reply he said he too was surprised this approach had worked. By now he has already spoken to his colleagues at the Microsoft virtualization team and they confirmed that splitting VHD’s this way is safe and he said he planned to blog about this sometime in the near future.
Well, this concludes my first real blog post. It got a little longer than I had expected, but I hope you found it a good read.
Greetz,
Leon
LeonZandman.com 