Blog Archives
Hacking Cortana: meet *all* of Cortana’s personas!
Last week after Microsoft had released the Windows Phone 8.1 update there was a lot of buzz around Cortana, Windows Phone’s brandnew personal digital assistant. People found out Cortana had all kinds of funny responses to questions they asked her. But one of them was special. It turned out Cortana had an easter egg that triggered the return of the infamous Clippy. If you asked Cortana “Do you like Clippy?”, an animated Clippy would show up instead of the usual Cortana logo.
I wondered how this Clippy easter egg was triggered. So I fired up Fiddler and set it up so I could spy on my Windows Phone’s HTTP communication. It turned out that when you ask Cortana that question about Clippy, she will initiate a HTTP request to Bing’s servers. Bing responds with a bunch of HTML that seems to consist of two parts. One part is some plain HTML that Cortana uses to render the actual response in her window. So if you ask Cortana about the weather, Bing will return a piece of HTML that renders into a nice weather table. This is a very flexible solution, because Microsoft can add new Cortana response types by only changing some server-side code. Some day the following little Windows Weekly in-joke I crafted may even become reality 😉
The second part of the response I found more interesting. It consisted of JavaScript and SSML, which are used to control Cortana’s vocal response. When I combed through the JavaScript, I noticed it contained the following DIV-tag:
<div data-emot="clippy1">
The accompanying JavaScript seems to use the ‘data-emot’ attribute’s value to control Cortana’s emotion setting:
t.SetEmotionBySelector("div[data-emot]")
So there it was. I had found the piece of code that triggered the Clippy easter egg animation! I was quickly able to verify this, by using Fiddler’s AutoResponder feature. I set it up so arbitrary Cortana searches would get a response that contained the ‘clippy1‘ emotion. I posted a small video of this feat some days ago on YouTube:
One thing the above also showed, was that the Clippy animation had to reside on the Phone itself. I checked all HTTP traffic and it didn’t seem to contain the Clippy animation. I wondered if there were more easter eggs / emotion values I could use. Some further HTTP-sniffing showed that normal Cortana responses used the ‘CALM’ emotion. So I tried ‘SAD‘, ‘ANGRY‘, etc. Unfortunately those didn’t seem to have any effect. I also tried ‘clippy2‘, ‘clippy3‘, ‘bob1‘, ‘billgates1‘, etc. But they all didn’t seem to work.
Then I tried a different approach. I knew the new Windows Phone 8.1 Developer SDK emulator images included a working Cortana. And because Windows Phone 8.1 is just Windows 8.1 and the emulator images are basically Hyper-V images, it’s possible to mount their VHD’s and get access to their file system (you DID know that, right?). After searching the file system for a while I found Cortana’s image resources were stored in a couple of DLL’s located in the \Windows\System32 directory of the phone’s main partition; their filenames matching the pattern ‘PersonaAssets*.dll‘ (i.e. PersonaAssets720x1280.dll)
Looking at the resource list, I immediately noticed that ‘clippy1‘ didn’t seem to be the only easter egg. The file also contained a ‘CLIPPYRETIRED1‘ resource. I went back to Fiddler and changed the AutoResponder so that it would return ‘clippyretired1‘ instead of ‘clippy1‘. And when I performed a Cortana search I was greeted by an animation of a seemingly older and retired Clippy, wearing a pair of reading glasses! Brilliant 🙂
Excited by this, I extracted all resources from the file and inspected them with a HEX editor. I noticed it were basically animated GIF images that had a little XML header stuck to the front. Using the HEX editor I stripped the XML header from the file, so they could be viewed as normal animated GIFs.
And that was when I first met all of Cortana’s current personas! Let me introduce them to the rest of the world:
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
||
I’m not sure if all of the resources extracted above can be used as an actual Cortana emotion (i.e. replace the default Cortana logo). I tried a couple of them and those all worked. It looks like the two Clippy animations are the only real Easter Eggs currently available. However, one of the resources is named ‘CIRCLE_SIRI1′, which must be a direct reference to Apple’s Siri.
Now that I’ve found these personas, I think it’s time to find the accompanying Cortana questions. What could you ask her that might trigger the retired Clippy animation. What question will trigger the Siri animation?
I think the fun has only just begun… 😉
Update: After I published this article, some members of the Cortana team have reached out to me! Marcus Ash, a Group Program Manager for Cortana, sent me the following tweet. Pretty cool!
I’m looking forward to future Cortana updates!
Update 2: Looking for Cortana’s sounds? They’re WAVE resources, located in \\MainOS\Windows\System32\SpeechUXRes.dll on the phone. Use your favorite resource extractor to get them 🙂
The new HTC 7 Mini or: How to fake WP Bench
I am so evil… Whoahahahahaha…
Earlier today I sent out the following tweet:
LMAO reading today's #wp7 news. I'll explain later... #wp7nl
So I fired up this old blog of mine to tell you why I just had to laugh:
Last night I was playing with the latest version of the popular Windows Phone 7 benchmarking app WP Bench, made by Robert Varga. Some time ago Robert added the ability to upload your benchmark results to his server and compare them with others.
However, WP Bench does not only publish the benchmarking results. It also publishes information about the phone models and the Windows Phone OS versions that are used. And these two lists have become an important source for people to scout for new Windows Phone models and OS versions. In fact, every new device/OS version that is listed in those stats is sure to get some headlines at popular news sites like WMPowerUser.com.
That’s when my evil plan starting taking shape. What if I could alter some of the data that WP Bench uploaded to its server and pretend I was using some new, unknown (and fake) Windows Phone model? Surely WP Bench had built in some mechanism that would prevent this, wouldn’t they? But I just had to try. Maybe it was possible to generate some buzz amongst the Windows Phone community and even get featured on some news sites…
First thing to do was to actually inspect what data and in what format is being sent back to the server. So I quickly hooked my Windows Phone to Fiddler, the popular HTTP debugging proxy. I started WP Bench on my phone and ran a simple speed test. I then uploaded the results to the WP Bench server.
The Fiddler Web Sessions list showed that benchmark results are uploaded to the server by using a single HTTP GET request that has all relevant result data specified as query string parameters, like this (I’ve removed my phone’s unique device ID from the URL):
http://www.computemark.com/wpb/index.php?deviceID=...&deviceManufacturer=HTC&deviceName=7+Mini+T86861&
os=Microsoft+Windows+CE+7.11.1131&publishSpeed=true&cpu1=3%2c67&
cpu2=3%2c67&data1=14%2c98&data2=7%2c26&gpuFPS=18&speedScore=58%2c3
What was even more interesting was that there didn’t seem to be any verification mechanism in place, to make sure this data wasn’t tampered with. No verification hashes or any of that kind. So this meant I could easily tamper with such a request and pass my own, fake, data.
I activated Fiddler’s Automatic Breakpoints feature, so it would intercept each request and allow me to edit it, before it was passed on to the WP Bench server. I again ran the WP Bench speed test and uploaded the results. This triggered the Fiddler breakpoint and I made the following changes to the request:
- I slightly changed the deviceID parameter, so the results wouldn’t be related to results I had previously uploaded using the same device.
- I changed the deviceName parameter from ‘7 Trophy T8686’ to ‘7 Mini T86861’.
The fun part of this was that I came up with the name ‘Mini’, because just days ago HTC had announced a new and very large Windows Phone device, called the ‘Titan’. I really didn’t know that HTC had actually once shipped a device called the ‘HD Mini’, which ran Windows Mobile 6.5. So I guess by accident this ‘Mini’ name added more credibility to my fake model name. - I changed the OS version from ‘7.10.7712’ to ‘7.11.1131’, hoping that a previously unseen change from 7.10 to 7.11 would generate some buzz. Also, the 1131 part refers to the birthday of someone very close to me 🙂
I didn’t change any of the performance scores, but of course could have easily done so. After that I allowed Fiddler to send the altered request to WP Bench’s server and went to bed.
Fiddler screenshot of the altered WP Bench request
So today I visited WMPowerUser.com and sure enough there it was: a nice new article titled ‘HTC 7 Mini running OS 7.11.1131 shows up in WP Bench’. It talked about the fact that WP Bench stats showed this mysterious new model, the HTC 7 Mini, complete with a totally new OS version in the 7.11.xxxx range. There was talk about the fact that this could be the first evidence of Windows Phone Apollo running on a handset. It was also mentioned by many other news sites and of course there was some buzz on Twitter.
WMPowerUser article
So that’s how easy it was to create some fake Windows Phone news. Just to be clear: as far as I know the HTC 7 Mini does NOT exist. Also there is no Windows Phone OS version 7.11.1131 (yet…). It was all made up by me. Whoahahahahaha…
Lastly, as a suggestion to WP Bench’s creator Robert Varga, I would advice him to add some sort of protection scheme, so that uploaded WP Bench results cannot easily be tampered with. I also wouldn’t list a new device/model in the WP Bench stats, unless it has been seen more than once, preferably with result uploads coming from a variety of IP addresses. Remember it took only one little request to get my fake phone to show up in the WP Bench stats.
Update: It appears Surur from WMPowerUser.com wasn’t happy about this and has banned me from posting comments to WMPowerUser. This is a JOKE, guys! Why so serious… !?
Update 2: WMPowerUser.com today has posted an article about another new phone. This one is probably legit, but notice the screenshot that accompanies the article. There is a device called ‘NA NA’ in the list. I know for sure that device was already present before I conducted my little experiment. So I have a strong feeling I wasn’t the first to submit fake data to WP Bench…
A device called 'NA NA'!?
LeonZandman.com 