Hacking Cortana: meet *all* of Cortana’s personas!
Last week after Microsoft had released the Windows Phone 8.1 update there was a lot of buzz around Cortana, Windows Phone’s brandnew personal digital assistant. People found out Cortana had all kinds of funny responses to questions they asked her. But one of them was special. It turned out Cortana had an easter egg that triggered the return of the infamous Clippy. If you asked Cortana “Do you like Clippy?”, an animated Clippy would show up instead of the usual Cortana logo.
I wondered how this Clippy easter egg was triggered. So I fired up Fiddler and set it up so I could spy on my Windows Phone’s HTTP communication. It turned out that when you ask Cortana that question about Clippy, she will initiate a HTTP request to Bing’s servers. Bing responds with a bunch of HTML that seems to consist of two parts. One part is some plain HTML that Cortana uses to render the actual response in her window. So if you ask Cortana about the weather, Bing will return a piece of HTML that renders into a nice weather table. This is a very flexible solution, because Microsoft can add new Cortana response types by only changing some server-side code. Some day the following little Windows Weekly in-joke I crafted may even become reality 😉
So there it was. I had found the piece of code that triggered the Clippy easter egg animation! I was quickly able to verify this, by using Fiddler’s AutoResponder feature. I set it up so arbitrary Cortana searches would get a response that contained the ‘
clippy1‘ emotion. I posted a small video of this feat some days ago on YouTube:
One thing the above also showed, was that the Clippy animation had to reside on the Phone itself. I checked all HTTP traffic and it didn’t seem to contain the Clippy animation. I wondered if there were more easter eggs / emotion values I could use. Some further HTTP-sniffing showed that normal Cortana responses used the ‘CALM’ emotion. So I tried ‘
ANGRY‘, etc. Unfortunately those didn’t seem to have any effect. I also tried ‘
billgates1‘, etc. But they all didn’t seem to work.
Then I tried a different approach. I knew the new Windows Phone 8.1 Developer SDK emulator images included a working Cortana. And because Windows Phone 8.1 is just Windows 8.1 and the emulator images are basically Hyper-V images, it’s possible to mount their VHD’s and get access to their file system (you DID know that, right?). After searching the file system for a while I found Cortana’s image resources were stored in a couple of DLL’s located in the
\Windows\System32 directory of the phone’s main partition; their filenames matching the pattern ‘
Looking at the resource list, I immediately noticed that ‘
clippy1‘ didn’t seem to be the only easter egg. The file also contained a ‘
CLIPPYRETIRED1‘ resource. I went back to Fiddler and changed the AutoResponder so that it would return ‘
clippyretired1‘ instead of ‘
clippy1‘. And when I performed a Cortana search I was greeted by an animation of a seemingly older and retired Clippy, wearing a pair of reading glasses! Brilliant 🙂
Excited by this, I extracted all resources from the file and inspected them with a HEX editor. I noticed it were basically animated GIF images that had a little XML header stuck to the front. Using the HEX editor I stripped the XML header from the file, so they could be viewed as normal animated GIFs.
And that was when I first met all of Cortana’s current personas! Let me introduce them to the rest of the world:
I’m not sure if all of the resources extracted above can be used as an actual Cortana emotion (i.e. replace the default Cortana logo). I tried a couple of them and those all worked. It looks like the two Clippy animations are the only real Easter Eggs currently available. However, one of the resources is named ‘CIRCLE_SIRI1′, which must be a direct reference to Apple’s Siri.
Now that I’ve found these personas, I think it’s time to find the accompanying Cortana questions. What could you ask her that might trigger the retired Clippy animation. What question will trigger the Siri animation?
I think the fun has only just begun… 😉
Update: After I published this article, some members of the Cortana team have reached out to me! Marcus Ash, a Group Program Manager for Cortana, sent me the following tweet. Pretty cool!
I’m looking forward to future Cortana updates!
Update 2: Looking for Cortana’s sounds? They’re WAVE resources, located in
\\MainOS\Windows\System32\SpeechUXRes.dll on the phone. Use your favorite resource extractor to get them 🙂
Posted on April 22, 2014, in Uncategorized, windows phone and tagged cortana, fiddler, hacking. Bookmark the permalink. 25 Comments.
Reblogged this on Karthikeyan.
Reblogged this on Exploring Windows Phone.
Ask her, “Where is clippy now?” She’ll respond with living his sunset years playing some game, I forget what exactly.
Reblogged this on /666/.
This is incredible! Well done!
Is there anyways we could have a HD download of these Gifs? Awsome work by the way! 😀
Do you know clippy. Gets the sunset years responce also
About Cortana Windows Desktop, I modifed the Chrome Extension “Big2Google” to redirect the query to SARAH (http://blog.encausse.net/2015/07/30/quand-cortana-parle-a-sarah/) SARAH do Home Automation with voice control (http://sarah.encausse.net)
– Can I send back some HTML to Cortana ?
– Can I hack Cortana request before it trigger bing request ?
I see in Windows folder
– there is come kind of HTML “templates”.
– And also voices for each language (In french seems to be Julie)
– And compiled Grammar (i assume they contains “hey cortana” but I don’t know how to override it
I look into regitry but there is no HTTP declaration about request performed
So for Nof it’s Hey Cortana => Chrome => SARAH => Action + SARAH answer
Trying to find a way to contact you directly. Please respond.
Pingback: Cortana Hack Reveals All The Possible Animations Included For It In WP8.1 | WMPoweruser
Pingback: Windows Phone Cortana Animations | Windows Phone Portal
Pingback: | Technoholism.com
Pingback: [WP 8.1] Les animations cachées de Cortana –
Pingback: [动图]除了回形针 Cortana还能装扮成其他的样子 | 极分享-专注于分享最新的科技资讯|关注我们，做科技达人
Pingback: Cortana reveals all her Avatars. Nokia updates App Folder for Windows Phone 8.1. | WinCentral
Pingback: Windows Phone 8.1 Cortana 彩蛋形象全揭露 | 我爱互联网
Pingback: Meet all of Cortana’s current personas and behind the scenes research into the digital PA : My Nokia Blog
Pingback: Se filtran los posibles avatares de Cortana
Pingback: New and *undocumented* URI Schemes for Windows Phone 8.1 | LeonZandman.com
Pingback: Problems using the ‘sharedUserCertificates’ capability in the Windows Phone 8.1 Developer Preview | LeonZandman.com
Pingback: Cortana Hack Reveals All The Possible Animations Included For It In WP8.1 | APPXAP