Category Archives: Uncategorized

Hacking Cortana: meet *all* of Cortana’s personas!

Last week after Microsoft had released the Windows Phone 8.1 update there was a lot of buzz around Cortana, Windows Phone’s brandnew personal digital assistant. People found out Cortana had all kinds of funny responses to questions they asked her. But one of them was special. It turned out Cortana had an easter egg that triggered the return of the infamous Clippy. If you asked Cortana “Do you like Clippy?”, an animated Clippy would show up instead of the usual Cortana logo.


I wondered how this Clippy easter egg was triggered. So I fired up Fiddler and set it up so I could spy on my Windows Phone’s HTTP communication. It turned out that when you ask Cortana that question about Clippy, she will initiate a HTTP request to Bing’s servers. Bing responds with a bunch of HTML that seems to consist of two parts. One part is some plain HTML that Cortana uses to render the actual response in her window. So if you ask Cortana about the weather, Bing will return a piece of HTML that renders into a nice weather table. This is a very flexible solution, because Microsoft can add new Cortana response types by only changing some server-side code. Some day the following little Windows Weekly in-joke I crafted may even become reality 😉

The second part of the response I found more interesting. It consisted of JavaScript and SSML, which are used to control Cortana’s vocal response. When I combed through the JavaScript, I noticed it contained the following DIV-tag:

<div data-emot="clippy1">

The accompanying JavaScript seems to use the ‘data-emot’ attribute’s value to control Cortana’s emotion setting:


So there it was. I had found the piece of code that triggered the Clippy easter egg animation! I was quickly able to verify this, by using Fiddler’s AutoResponder feature. I set it up so arbitrary Cortana searches would get a response that contained the ‘clippy1‘ emotion. I posted a small video of this feat some days ago on YouTube:

One thing the above also showed, was that the Clippy animation had to reside on the Phone itself. I checked all HTTP traffic and it didn’t seem to contain the Clippy animation. I wondered if there were more easter eggs / emotion values I could use. Some further HTTP-sniffing showed that normal Cortana responses used the ‘CALM’ emotion. So I tried ‘SAD‘, ‘ANGRY‘, etc. Unfortunately those didn’t seem to have any effect. I also tried ‘clippy2‘, ‘clippy3‘, ‘bob1‘, ‘billgates1‘, etc. But they all didn’t seem to work.

Then I tried a different approach. I knew the new Windows Phone 8.1 Developer SDK emulator images included a working Cortana. And because Windows Phone 8.1 is just Windows 8.1 and the emulator images are basically Hyper-V images, it’s possible to mount their VHD’s and get access to their file system (you DID know that, right?). After searching the file system for a while I found Cortana’s image resources were stored in a couple of DLL’s located in the \Windows\System32 directory of the phone’s main partition; their filenames matching the pattern ‘PersonaAssets*.dll‘ (i.e. PersonaAssets720x1280.dll)


Looking at the resource list, I immediately noticed that ‘clippy1‘ didn’t seem to be the only easter egg. The file also contained a ‘CLIPPYRETIRED1‘ resource. I went back to Fiddler and changed the AutoResponder so that it would return ‘clippyretired1‘ instead of ‘clippy1‘. And when I performed a Cortana search I was greeted by an animation of a seemingly older and retired Clippy, wearing a pair of reading glasses! Brilliant 🙂

Retired Clippy

Excited by this, I extracted all resources from the file and inspected them with a HEX editor. I noticed it were basically animated GIF images that had a little XML header stuck to the front. Using the HEX editor I stripped the XML header from the file, so they could be viewed as normal animated GIFs.

Cortana Persona HEX

And that was when I first met all of Cortana’s current personas! Let me introduce them to the rest of the world:


I’m not sure if all of the resources extracted above can be used as an actual Cortana emotion (i.e. replace the default Cortana logo). I tried a couple of them and those all worked. It looks like the two Clippy animations are the only real Easter Eggs currently available. However, one of the resources is named ‘CIRCLE_SIRI1′, which must be a direct reference to Apple’s Siri.

Cortana Siri

Now that I’ve found these personas, I think it’s time to find the accompanying Cortana questions. What could you ask her that might trigger the retired Clippy animation. What question will trigger the Siri animation?

I think the fun has only just begun… 😉

Update: After I published this article, some members of the Cortana team have reached out to me! Marcus Ash, a Group Program Manager for Cortana, sent me the following tweet. Pretty cool!

I’m looking forward to future Cortana updates!

Update 2: Looking for Cortana’s sounds? They’re WAVE resources, located in \\MainOS\Windows\System32\SpeechUXRes.dll on the phone. Use your favorite resource extractor to get them 🙂

Cortana Sounds

The Return of the Search Application Topology Component Health State Error

Today I was prepping a SharePoint 2013 VM and noticed the Search Service Administration page in Central Admin didn’t display the Search Application Topology health indicators. Instead it showed an error I hadn’t seen in a while:

“Unable to retrieve topology component health states. This may be because the admin component is not up and running”

I had encountered this error several times during the SharePoint 2013 preview phase. Then it turned out to be a genuine bug and fixing it required the installation of several hotfixes as detailed in the SharePoint Server 2013 Known Issues article. However, on this particular VM I had installed the RTM versions of Windows Server 2012, SQL Server 2012 and SharePoint 2013. I also had installed the latest updates and the SharePoint 2013 Prerequisites Installer should have taken care of other missing stuff. So this bug shouldn’t have happened again.

Just to be sure I tried installing the four hotfix packages, but (of course) this failed as they were either already installed or not relevant anymore. Now, I was pretty sure I had seen fully working Search Application Topology health indicators the night before (yeah, we SharePointies are nightly creatures). So what could have changed?

Then I remembered I had performed a little tweak to minimize SharePoint 2013’s memory usage. As you may know SharePoint 2013 has some very high memory requirements, especially around Search. Specifically the Search Engine’s NodeRunner.exe processes are memory hungry beasts. A SharePoint 2013 server configured for Search typically has five such NodeRunner.exe instances running, each consuming hundreds of megabytes of memory. For production environments this may be just fine, but on my development and demo VM’s I like them a little less greedy.

From earlier blog posts I learned it is possible to limit the amount of memory a NodeRunner.exe instance uses, by adjusting the memoryLimitMegabytes parameter in the C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\NodeRunner.exe.config XML file. In fact, in the SharePoint 2013 Preview version the NodeRunners suffered from a memory leak and adjusting this config was a quick hack to fix this leak. One of the last things I did last night was change this value from 0 (unlimited) to 100 MB. However, I hadn’t actually come around to testing the outcome of this change (yes, I do need some sleep).

<nodeRunnerSettings memoryLimitMegabytes="100" />

So the first thing I did was starting SysInternals Process Explorer (which I always use instead of the built-in Task Manager) and searched for the NodeRunner.exe processes.

To my surprise there weren’t any…

The NodeRunner.exe instances are spawned by SharePoint’s Search Host Controller Service, so the first thing I did was go to the “Manage Services on Server” page in Central Admin and restart that service. Thanks to Process Explorer’s handy ‘Difference Highlighting’ feature it was clearly visible that several NodeRunner instances got started, but almost immediately stopped running.

To get a little more information I started SysInternals Process Monitor (another indispensable tool) and logged the system while restarting the Search Host Controller Service. From Process Monitor’s ‘Process Tree’ dialog it was clear (too) many NodeRunner instances were started, but somehow weren’t able to survive for long.

Process Explorer also showed me the NodeRunner processes were started using the following command line parameter:

"C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\NodeRunner.exe" --noderoot "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\9E4952\IndexComponent1" --addfrom "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\9E4952\IndexComponent1\Configuration\Local\Node.ini" --tracelog "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\9E4952\IndexComponent1\Logs\NodeRunner.log"

From the bold piece above I learned NodeRunners keep a trace log. When I checked that file, the cause for the failing of the NodeRunners became pretty clear:

Caught exception in node activator: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Microsoft.Ceres.CoreServices.Node.NodeActivator.ActivateNode(IDictionary`2 configuration)
--- End of inner exception stack trace ---

Since I figured this must be due to me changing NodeRunner’s memory limits, I reset the NodeRunner’s config file (hey, always keep a backup!) and again restarted the Search Host Controller Service. This time the NodeRunner instances continued to run just fine. So it became clear to me there apparently is some minimum amount of memory a NodeRunner instance needs to be able to run and in my case it needed more than 100 MB.

With the NodeRunner instances running just fine, the Topology Health indicators also worked again, showing my farm’s Search stuff was in perfect health. I experimented with the NodeRunner memory limits some more and finally settled for a 250 MB limit. Not as small as I’d liked them to be, but still a significant decrease when compared to their original memory footprint.

I do want to make clear that even with this 250 MB limit I experienced some NodeRunner crashes. The general advice is to NOT change the NodeRunner memory limit configuration. And NEVER EVER do this in a production environment!

Hopefully this blog post helps out other people who might stumble upon this problem.

The iPad Backup that took a long time…

This saturday I decided to create a full backup of my iPad. To cut a long story short: it took a long time. A very long time. A very very very very long time. In total the backup my iPad 16GB, which was filled with about 7GB of data, took a shocking:

18 hours, 22 minutes and 19 seconds!

Unbelievable. If you Google with Bing you can find a lot of users experiencing similar long backup times. As it turns out Apple has devised a backup mechanism, whereby backup duration is heavily dependent on the number of files it has to backup. Each file apparently takes several seconds to process, so backing up several thousand files can easily take hours.

I just had to investigate which apps caused these huge backup times. Analysis of the directory where iTunes had placed my backup showed that it basically contains pairs of two types of files: *.mdinfo and *.mddata. As these extensions imply, the *.mddata file contains the actual backup data and the related *.mdinfo file contains some meta data about that data file.

The *.mdinfo files are binary, but when opened with a text editor like NotePad they contain some readable elements. One of the more interesting things in this file is a text string beginning with the word "AppDomain". This looks like a unique identifier for an iPad app. I noticed that many files have the same AppDomain identifier in them, which leads to the conclusion that they all belong to the same app.

Based on this analysis I came up with the following scheme:

  1. Collect the AppDomain identifier for every *.mdinfo file in the iPad backup directory.
  2. Group all these files based on their AppDomain identifier.
  3. Sort these groupings, based on the total file count.

I wrote a quick C# console application for this and below are the results. First a little pie chart:

iPad App File Count

As you can see from the pie chart there are two apps that have claimed a really large number of files:

The winner (or better: loser) is the Wired app, the iPad version of Wired Magazine. It somehow needs 4947 files to operate. For the record: I installed the Wired app and did an in-app purchase of the issue featuring Toy Story 3. I didn’t buy any other issue.

The runner-up is the app, which is an app that shows news content from the popular Dutch news website It’s a fantastic app, but I really haven’t got the faintest clue why it would need 3596 files to operate, since it should fetch its content live from the website.

The third place actually surprised me. For these files no AppDomain identifier could be identified. They did contain the string "MediaDomain", but I didn’t investigate in its meaning. However, since this group of files finished third, I’ve definitively needed to include them in my list.

Below you’ll find the full breakdown. Deleting this list’s top five apps from my iPad dramatically decreased backup times.

So, in the end I guess both app developers and Apple can learn something from this. App developers need to make sure their apps only use a limited number of files. And Apple needs to seriously fix their stupid backup mechanism.

AppDomain File Count
AppDomain-com.condenet.wiredmag 4947
AppDomain-nl.sanomadigital.nuhd 3596
*Unknown* 824
AppDomain-nl.veronica.veronicamagazine 659
AppDomain-se.illusionlabs.labyrinth2 504
AppDomain-com.nytimes.editorschoice 367
AppDomain-com.linkedin.LinkedIn 338
AppDomain-com.outfit7.talkingtomipad 314
AppDomain-com.vsallaccess.vsaaipad 240
AppDomain-com.tunaverse.iPadCinq 199
AppDomain-com.chillingo.cuttheropehd 154
AppDomain-com.isignz.aanbieding 121 113 86
AppDomain-org.pogi.dp 77
AppDomain-com.NewspaperDirect.PRiphone 73
AppDomain-com.atebits.Tweetie2 70
AppDomain-se.illusionlabs.touchgrind 64
AppDomain-com.concretesoftware.acestrafficpackipad 43
AppDomain-com.vitotechnology.StarWalkiPad 36 30 24
AppDomain-com.dawikihoodcom 21 20
AppDomain-BibleReader 19
AppDomain-com.ikmultimediaus.amplitubepadfull 19
AppDomain-com.flipboard.flipboard 18 18
AppDomain-com.firemint.flightcontrolipad 17
AppDomain-com.davidme.soundpaper 16
AppDomain-com.goodiware.GoodReaderIPad 16
AppDomain-com.spotify.client 15
AppDomain-com.trautvetter.atomicbrowser 15
AppDomain-com.mobilepioneers.tvgidslite 13
AppDomain-com.smule.magicpiano 13
AppDomain-wefwef 13
AppDomain-com.chillingo.angrybirdsipad 12
AppDomain-com.evernote.iPhone.Evernote 12 11 9 9 9
AppDomain-com.mobilepond.ipadphotogene 9
AppDomain-pocketinformantiPad 9
AppDomain-osfoora.hd 8
AppDomain-ch.graphicarteservice.newspaper.nrc 7
AppDomain-com.com2us.sliceit 7 7
AppDomain-com.fotopedia.heritage 7
AppDomain-com.layar.Layar 7
AppDomain-com.tmsoft.FartCushion 7
AppDomain-com.truphone.ipad 7
AppDomain-com.ebay.core.ipad 6
AppDomain-com.marcoarment.instapaperpro 6
AppDomain-com.teamviewer.teamviewerhd 6
AppDomain-com.ted.TED 6
AppDomain-com.aemotion.magworld 5
AppDomain-com.mythpeople.SparkleHD 5
AppDomain-com.natenai.glowhockey4ipad 5
AppDomain-com.sqrt3innovations.calculatorpad 5
AppDomain-nl.hyves.hyves 5
AppDomain-nl.telegraaf.krant.ipad 5
AppDomain-nl.vpro.3voor12.luisterpaal 5 4
AppDomain-ca.jubilo.ipadflickr 4
AppDomain-com.bryandev.GearedHD 4
AppDomain-com.Buienradar.BuienradarHD 4 4
AppDomain-com.getdropbox.Dropbox 4
AppDomain-com.idealbinary.rumpel 4 4
AppDomain-com.jellyfilledstudios.AmzngDiceHD 4
AppDomain-com.laya.osiris.wikipanion.ipad 4
AppDomain-com.tarsin.Dilbert 4
AppDomain-com.thinkfun.rushhour 4
AppDomain-com.wolframalpha.wolframalpha 4
AppDomain-org.pogi.tigmatching 4 3
AppDomain-com.europeandirectories.dtgipad 3
AppDomain-com.handypadsoft.deviceinfo 3
AppDomain-com.johnmoff.PocketPondHD 3
AppDomain-com.lucasarts.miseipad 3
AppDomain-com.vitotechnology.SolarWalk 3
AppDomain-com.wireddevelopments.rocketeeripad 3
AppDomain-de.meteogroup.WeatherProHD 3
AppDomain-fr.febo.multipong 3
AppDomain-com.appbc.touchable 2
AppDomain-nl.mangrove.Nozzman 2
AppDomain-com.tiogadesigns.bigcalculatorfree 1

Googling with Bing

I laughed out loud when I read this quote from Scott Hanselman’s latest blog post:

I googled with Bing

How will Microsoft ever win this…

A Little Fun

I remember reading this funny text several years ago. I just stumbled on it again and decided to archive it here. I think it’s pretty funny 🙂

Woman are like computers, because:

  1. No one but their creator understands their internal logic.
  2. The native language they use to communicate with other computers is incomprehensible to everyone else.
  3. Even the smallest mistakes are stored in long term memory for possible later review.
  4. As soon as you make a commitment to one, you find yourself spending half your paycheck on accessories for it.

Men are like computers, because:

  1. In order to do anything with them, you have to turn them on.
  2. They have a lot of data but still can’t think for themselves.
  3. They are supposed to help you solve problems, but half the time they ARE the problem.
  4. As soon as you commit to one, you realize that if you had waited a little longer you could have gotten a better model.

SharePoint Frustrations #1: The undocumented “IncludeTimeValue” CAML attribute

I’m planning on doing some posts about frustrating things I have encountered (and still do!) during my SharePoint development efforts. Here’s the first one:

Last year while working on a MOSS 2007 project for one of our customers I stumbled on what I thought was a bug in SharePoint 2007. I had created a custom list that was filled with Electronic Program Guide (EPG) information for the streaming video media that that site contained. I then created a Webpart that used ASP.Net AJAX to continually show the actual EPG information below the video stream.

In order to obtain the EPG items I used a CAML query to query the EPG list. The list was simply a custom list containing amongst others a column of type DateTime that was called "ProgramEnd". As the name suggests it contained the time the program ended. I then used the following code to create a query that was supposed to obtain the currently broadcasted item and all future items.

   1: const string EPG_QUERY_TEMPLATE = @"
   2: <Where>
   3:  <Geq>
   4:   <FieldRef Name='ProgramEnd' />
   5:   <Value Type='DateTime'>{0}</Value>
   6:  </Geq>
   7: </Where>"; 
   9: SPQuery query = new SPQuery();
  10: query.Query = String.Format(EPG_QUERY_TEMPLATE,
  11:     SPUtility.CreateISO8601DateTimeFromSystemDateTime(DateTime.Now)); 
  12: // ...

To my surprise this query always returned too many items. Further investigation showed that time part of the query seemed to be ignored completely, so it returned the items as if no time was specified! So instead of getting the current item and all future items it returned all items broadcasted for that day.

After wasting a lot of time debugging, using the U2U Caml Query Builder and looking for an answer / solution on the internet I gave up and wrote a quick hack around it, which fortunately wasn’t very difficult. It would just run the query and run the results through some additional code that checked the ProgramEnd DateTime field and filter out the wrong items, like this:

   1: SPListItemCollection results = ...; // The results from the query mentioned above
   2: // Trim the results to only include the current and future items
   3: List<SPListItem> trimmedItems = new List<SPListItem>();
   4: foreach (SPListItem result in results)
   5: {
   6:     DateTime programEnd = (DateTime)result["ProgramEnd"];
   7:     if (programEnd >= DateTime.Now)
   8:         trimmedItems.Add(result);
   9: } 

Fortunately this worked just fine and the customer was happy. I was not… 😦

Today while surfing the Net I stumbled on this entry in the MSDN forums, from which I learned it wasn’t a bug, but that you need to included the "IncludeTimeValue" attribute to the CAML query, like this:

   1: <Where>
   2:  <Eq>
   3:   <FieldRef Name='programEnd' />
   4:   <Value Type='DateTime' IncludeTimeValue='TRUE'>
   5:    2008-07-30T12:00:00Z
   6:   </Value>
   7:  </Eq>
   8: </Where> 

If only I had known it was this simple… What bothers me is that this little, but very important attribute seems to be totally undocumented. I couldn’t find any information about it in the WSS / SharePoint SDKs.

Ofcourse once I knew what to look for I found some other blogs and forums mentioning this issue. It turns out the UCSharp blog had already blogged about this way back in October 2007, only a few months after I searched for it. Even Karine Bosch, U2U’s "CAML Girl" and author of the famous U2U CAML Query Builder, says she only recently found out about this. Fortunately she has included support for the "IncludeTimeValue" attribute in her latest version of the U2U Caml Query Builder, which I know a lot of SharePoint developers use to construct and test their CAML queries.

I also noticed that someone called puneetspeed has added some Community Content to the online SharePoint SDK’s SPQuery docs explaining this issue. So hopefully Microsoft will add information about the "IncludeTimeValue" attribute to the official SDK text in the near future.

No More *BEEP*

I might have happened to you too: make some error in a Windows virtual machine and your system will *BEEP* out loud. Not some nice and fancy WAV/MP3 sample, but a raw *BEEP* coming straight from your system’s motherboard. This *BEEP* does not respect your speaker volume and mute settings. And it will probably irritate most people who sit near you.

Fortunately getting the beep to shut up forever is relatively simple. Here’s how to do it:

  1. Open a Command Prompt window (in Vista make sure you open it using the “Run as administrator” option).
  2. Use the following commands:
    • To stop the Windows Beep Service:
      net stop beep
    • To make sure it never gets started again:
      sc config beep start= disabled

That should take care of them BEEPs 🙂

Cool e-book: The Web Part Infrastructure Uncovered

The Web Part Infrastructure Uncovered coverA while ago I did some proofreading for Teun Duynstee‘s e-book The Web Part Infrastructure Uncovered. I found it a very good read. In fact I think it is one of the most thorough books available on the subject, even though it only has 135 pages. This e-book has been available for some months now, but since I just recently started blogging I thought I might spend a blog entry to it. With SharePoint 2007 now using the .NET Framework 2.0 Web Part technology as one of its core technologies, knowledge about this framework has become more and more important.

Teun sells his book online through The price is €14.00 (ca. $17) for the printed book (+ shipping & handling) or €9.00 for the downloaded PDF (ca. $11) which is peanuts considering the book’s useful content.

I recommend this e-book to every .NET developer that’s working with or interested in Web Parts. You can check out some sample chapters here and here.

Cool Surface video

Today I stumbled on the blog of the Microsoft Surface team. Although there are only a few entries as of yet, they did put up a pretty cool video showing the ScatterView control from the Surface SDK. Amazing what you can do with only a few lines of XAML code and data binding!
Video: The Microsoft Surface SDK In Action