Monthly Archives: April 2014
As you may know I’ve been busy lately inspecting the Windows Phone 8.1 internals. Last week I discovered some new Cortana personas. Now I’ve uncovered some new and, as far as I know, currently undocumented URI schemes. These can be used to launch certain apps and System Settings pages directly from within your own app using the
Here are the ones I’ve uncovered so far:
||Launches the Usage tab of the Battery Saver app.|
||Launches Wallet app.|
||Launches the Photos + Camera Settings app.|
||Launches the Network Profile Update app.|
||Launches the NFC Settings app.1|
||Launches the Notifications + Actions Settings app.|
||Launches the NFC Settings app.1|
||Launches a SIM Applications related app. UICC appears to stand for Universal Integrated Circuit Card.2|
||Launches the Workplace Settings app.|
1 The URI scheme names seems to imply they lead to more specific pages within the NFC Settings app. However, during my testing they both opened the main NFC Settings page. This could be due to the fact that the phone I tested this on wasn’t fully in service (i.e. no SIM card inserted, no NFC trusted apps, etc.)
2 I didn’t insert a SIM card into my the Windows Phone 8.1 phone I was testing this with, so all I got was a page containing a ‘SIM applications aren’t available right now’ message.
Disclaimer: As these URI Schemes haven’t (yet) been publicly documented by Microsoft, you probably aren’t supposed to use them. They could have been provided for internal/carrier/OEM usage or maybe they will be documented at a later date. In short: no guarantees! In fact, using these may even cause your app to be rejected from the Store.
I find I’m really liking the new Windows Phone 8.1 ‘Project my Screen’ feature. This feature enabled the projection of the handset to a secondary screen via wireless Miracast (newer handsets only, as it requires special hardware) or a wired USB connection. This is great for presentations and demos. It even works in the opposite direction: you can remote control your phone by interacting with the projection using your mouse or your touch screen! Also great is that it has support for displaying multiple touch points, so your audience can actually see where you touch your device. Great stuff!
If you haven’t already, I suggest you give the feature a try yourself. Microsoft’s Cliff Simpkins has done a great writeup on his blog on getting this thing to work.
In order to use the ‘Project my Screen’ feature on your Windows PC, you have to download and install the official ‘Project my Screen’ app. By default the app shows a pretty boring image of a nondescript Windows Phone, much like the Windows Phone SDK emulators do. Fortunately Microsoft included the ability to customize the appearance of the app, so it may suit our own needs. Here’s a short video that shows a quick customization I made using the application:
As you can see I’m using a custom background image featuring the Cortana logo. Also the projected phone’s skin matches my real phone (a yellow Nokia Lumia 920). Let me tell you how I did that:
If you look inside the ProjectMyScreenApp’s installation folder, which by default is located at
C:\Program Files (x86)\ProjectMyScreenApp, you’ll notice there’s a file called
config.xml. This is a XML configuration file that can be used to alter the appearance of the app. By default it’s not used by the app, though. To use it you need to explicitly launch the application with the configuration file passed in as a parameter. To complicate things a little further: in its original state the configuration file doesn’t work. It refers to images that don’t exist in the installation folder (they do exist, but are embedded as resources inside the application’s executable). So you HAVE to alter the
config.xml (or create your own file based on it) to get it to work.
So, how do we configure this thingy? Fortunately the
config.xml file itself contains instructions on how to use it. The basic structure of the file looks like this:
<?xml version="1.0" encoding="utf-8"?> <Config> <PhoneConfigurations> <PhoneConfiguration> <Background /> <PhoneImage /> <VideoOutput /> <!-- <TouchDots /> --> </PhoneConfiguration> <PhoneConfiguration> . . . </PhoneConfiguration> </PhoneConfigurations> </Config>
config.xml file contains
PhoneConfiguration elements for five different resolutions: WXGA (768×1280), HD1080 (1080×1920), HD720 (720×1280), WVGA (480×800) and FWVGA (480×854 with the new on-screen soft buttons). I suppose you can add additional resolutions as devices are released, but I’m currently not able to test this. The application will start up with a default configuration (which you can define; see below). When you connect an actual device and allow screen projection the application will switch to the configuration that matches the connected device’s resolution.
PhoneConfiguration element has three attributes. The
resolutionY values are used to specify the resolution that this configuration belongs to. The
default attribute is used to specify whether this is the default configuration to use when no device is connected.
<PhoneConfiguration resolutionX="768" resolutionY="1280" default="true">
phoneConfiguration section we can configure three settings: the background image, the phone’s skin and the size and placement of the projection. Actually, there is a fourth element,
touchDots, that’s present in the file, but I couldn’t get it to work. I suspect it should be used for configuring the appearance of the dots that appear when you touch your device, but it didn’t seem to respond to changes I made to it.
Let’s go into detail about each element:
Background element controls the application’s background. The entire app background will be filled with the specified color. On top of that you can place an image that will also be scaled to fit. Note that using a background image is optional. If you only want to use the solid background color, you still have to add the
imageRelPath attribute, but just give it an empty (“”) value.
<Background color="0x00000000" imageRelPath="Cortana.png" />
PhoneImage element you can control the appearance of the on-screen phone (i.e. the skin). The
imageRelPath attribute specifies the path to the image to use. For my demo video I used one of the official Nokia Lumia 920 press images that I downloaded from the Nokia Press media library. I cropped the image, so it didn’t have any border, and saved it as a PNG. You need to specify the path to the image relative to the ProjectMyScreenApp’s executable file. The easiest way to do this is by adding your image file to the application’s installation folder, so you only need to supply the filename without any additional path information.
scaleFactor attribute specifies the scale factor of the phone image as a fraction of the background image’s width or height, whichever is more constrained. If no background image was specified, the app window’s dimensions are used instead. I used a value of 0.9, so the phone is nice and large, but doesn’t go edge to edge.
centerOffsetY values you can position the phone image relative to the background image. I’ve used this to place my phone on the right side of the screen, so it doesn’t obscure the Cortana logo of my background image. Interestingly, if you didn’t supply a background image, these values are ignored and the phone is always displayed in the center of the application’s window.
For devices that have physical buttons we can configure the location of the Back, Windows and Search buttons. For each button you need to supply the coordinates of its bounding box. This way the application can determine which button to press when you click on the skin image with your mouse/touch screen. The
searchButton attributes accept a value that consists of four numbers that represent the left, top, right and bottom coordinates of the hardware back, Windows and search buttons.
Here’s my complete
<PhoneImage imageRelPath="Lumia920-2.png" scaleFactor="0.9" centerOffsetX="350" centerOffsetY="0" backButton="128,2357,400,2600" windowsButton="610,2357,900,2600" searchButton="1160,2357,1370,2600" />
Next up we need to specify the location of the video output using the
videoOutput element. Using the
topLeftOffsetY you specify the top left coordinates of the video image. Unfortunately you cannot specify the width of the video window. Instead you have to do some math and calculate the weight value, which is defined as being ‘the width of the video output as a fraction of the phone image’s width’. So we need determine the width (in pixels) of the phone’s image display as it appears in the skin image (i.e. the bounding box where you want to projection to appear) and divide that by the total width of the skin image. For my Lumia image that results in the following formula: 1273 / 1535 = 0.8293159609120521.
Here it is:
<VideoOutput weight="0.8293159609120521" topLeftOffsetX="125" topLeftOffsetY="233" />
So the complete
phoneConfiguration section for my Lumia 920 skin looks like this:
<PhoneConfiguration resolutionX="768" resolutionY="1280" default="true"> <Background color="0x00000000" imageRelPath="Cortana.png" /> <PhoneImage imageRelPath="Lumia920-2.png" scaleFactor="0.9" centerOffsetX="350" centerOffsetY="0" backButton="128,2357,400,2600" windowsButton="610,2357,900,2600" searchButton="1160,2357,1370,2600" /> <VideoOutput weight="0.8293159609120521" topLeftOffsetX="125" topLeftOffsetY="233" /> </PhoneConfiguration>
Now launch the ProjectMyScreenApp using the
config.xml file as an input parameter. One way to do this is starting it from a Command Prompt window like this:
C:\Program Files (x86)\ProjectMyScreenApp>ProjectMyScreenApp.exe Config.xml
You can also create a Shortcut with the parameter passed in. The ProjectMyScreenApp should start in full-screen mode. You can press the
[ESC] key to switch to windowed mode.
You can download my sample Lumia 920 files here. Just unzip the files into the ProjectMyPhoneApp’s installation folder (consider making a backup of the original
config.xml file, just in case). As a bonus I’ve extracted the default phone images that are contained within the ProjectMyPhoneApp executable and added them too.
Please note that if you connect using a device that has a different resolution than my Lumia 920, you won’t get to see the custom stuff I mentioned above. Instead the ProjectMyScreenApp will use one of the other phone configurations and I didn’t bother changing those.
Maybe someday someone will create a nice tool that assists in creating the
config.xml and/or the image files, just like Geert vd Cruijsen has done in the past with his Windows Phone 7 Emulator Skin Switcher application. Or maybe we can build a repository of compatible skin images. I’m even willing to host/link to them from here.
In any case: have fun! You know I am 😉
Last week after Microsoft had released the Windows Phone 8.1 update there was a lot of buzz around Cortana, Windows Phone’s brandnew personal digital assistant. People found out Cortana had all kinds of funny responses to questions they asked her. But one of them was special. It turned out Cortana had an easter egg that triggered the return of the infamous Clippy. If you asked Cortana “Do you like Clippy?”, an animated Clippy would show up instead of the usual Cortana logo.
I wondered how this Clippy easter egg was triggered. So I fired up Fiddler and set it up so I could spy on my Windows Phone’s HTTP communication. It turned out that when you ask Cortana that question about Clippy, she will initiate a HTTP request to Bing’s servers. Bing responds with a bunch of HTML that seems to consist of two parts. One part is some plain HTML that Cortana uses to render the actual response in her window. So if you ask Cortana about the weather, Bing will return a piece of HTML that renders into a nice weather table. This is a very flexible solution, because Microsoft can add new Cortana response types by only changing some server-side code. Some day the following little Windows Weekly in-joke I crafted may even become reality 😉
So there it was. I had found the piece of code that triggered the Clippy easter egg animation! I was quickly able to verify this, by using Fiddler’s AutoResponder feature. I set it up so arbitrary Cortana searches would get a response that contained the ‘
clippy1‘ emotion. I posted a small video of this feat some days ago on YouTube:
One thing the above also showed, was that the Clippy animation had to reside on the Phone itself. I checked all HTTP traffic and it didn’t seem to contain the Clippy animation. I wondered if there were more easter eggs / emotion values I could use. Some further HTTP-sniffing showed that normal Cortana responses used the ‘CALM’ emotion. So I tried ‘
ANGRY‘, etc. Unfortunately those didn’t seem to have any effect. I also tried ‘
billgates1‘, etc. But they all didn’t seem to work.
Then I tried a different approach. I knew the new Windows Phone 8.1 Developer SDK emulator images included a working Cortana. And because Windows Phone 8.1 is just Windows 8.1 and the emulator images are basically Hyper-V images, it’s possible to mount their VHD’s and get access to their file system (you DID know that, right?). After searching the file system for a while I found Cortana’s image resources were stored in a couple of DLL’s located in the
\Windows\System32 directory of the phone’s main partition; their filenames matching the pattern ‘
Looking at the resource list, I immediately noticed that ‘
clippy1‘ didn’t seem to be the only easter egg. The file also contained a ‘
CLIPPYRETIRED1‘ resource. I went back to Fiddler and changed the AutoResponder so that it would return ‘
clippyretired1‘ instead of ‘
clippy1‘. And when I performed a Cortana search I was greeted by an animation of a seemingly older and retired Clippy, wearing a pair of reading glasses! Brilliant 🙂
Excited by this, I extracted all resources from the file and inspected them with a HEX editor. I noticed it were basically animated GIF images that had a little XML header stuck to the front. Using the HEX editor I stripped the XML header from the file, so they could be viewed as normal animated GIFs.
And that was when I first met all of Cortana’s current personas! Let me introduce them to the rest of the world:
I’m not sure if all of the resources extracted above can be used as an actual Cortana emotion (i.e. replace the default Cortana logo). I tried a couple of them and those all worked. It looks like the two Clippy animations are the only real Easter Eggs currently available. However, one of the resources is named ‘CIRCLE_SIRI1′, which must be a direct reference to Apple’s Siri.
Now that I’ve found these personas, I think it’s time to find the accompanying Cortana questions. What could you ask her that might trigger the retired Clippy animation. What question will trigger the Siri animation?
I think the fun has only just begun… 😉
Update: After I published this article, some members of the Cortana team have reached out to me! Marcus Ash, a Group Program Manager for Cortana, sent me the following tweet. Pretty cool!
I’m looking forward to future Cortana updates!
Update 2: Looking for Cortana’s sounds? They’re WAVE resources, located in
\\MainOS\Windows\System32\SpeechUXRes.dll on the phone. Use your favorite resource extractor to get them 🙂