Categories
technical

Mobile Ad Serving

I’ve seen a lot of AdMob [admob.com] ads in iPhone applications over the past two years. But recently I downloaded two free — i.e. ad supported — applications that appear to use a Google ad server (of course Google purchased AdMob but I don’t think it’s just a re-branding I think it’s a different service.)

I noticed this for two reasons. First; for the past few years I have been working in the mobile advertising space, so this stuff stands out. Second; the ads were odd.

Why odd? Well, as I mentioned these were ads in iPhone applications. Keep that in mind and take a look at the ads:

Google mobile ads banner for SingTel Mobile SME services
Google mobile ads banner for Android
Google mobile ads banner for Nokia 5530 Apps Store
Google mobile ads banner for Nokia E63 Apps Store

The issue, I think, with these ads is they are all for competitors. I can excuse the SingTel ad, I’m a Starhub customer but this is an add for enterprise services and the ad was served while I was on a WiFi network, so there is not much chance that Google could have determined my operator.

However, the ad for Android and the two ads for the Ovi store — which sells applications that work on Nokia handsets are not useful to me as a consumer and most likely a waste of the advertisers money. The odds that I am going to patronize either of these services from my iPhone is next to zero.

And there is little excuse for this. When I was working on requirements for a mobile advertising system the product team was adamant that it include basic relevance filtering. Now relevance filtering is complex and the simple business rule “the advertisement should be relevant and useful to the consumer” actually breaks down to a lot of technical requirements. The technical requirements of significance here are:

Requirement 1
The system shall attempt to retrieve the User-Agent header from the HTTP request. The header should be used to reduce the pool of relevant banners by removing banners that:
  1. have a User-Agent whitelist that does not include the User-Agent retrieved from the HTTP request
  2. have a User-Agent blacklist that does include the User-Agent retrieved from the HTTP request
Requirement 2
They system shall allow buyers to construct a whitelist or a blacklist of User-Agent strings which for specific campaigns or banners in a campaign.
Requirement 3
User-Agent lists (white- or black-) should be constructed of strings entered by buyers by selecting full User-Agents or pre-coded regular expression from a list or entering an arbitrary regular expression.

(Obviously there are a lot more details and other requirements that need to be clarified before you can actually implement this.) Here’s a use case that would prevent the issue of the Ovi banners being shown to me on my iPhone.

Use Case:
Setting up a campaign level whitelist, to filter out non-Nokia handsets.
Pre-Condition:
The buyer has created a campaign
The buyer selects Relevance Filtering from the interface
Post-Condition:
A new filter is added to the campaign level whitelist for the selected campaign
Scenario:
  1. The Buyer selects Create/Edit Whitelist from the Relevance Filtering menu.
  2. The System loads the Relevance Whitelist interface
  3. The Buyer selects the Filter Campaign checkbox
  4. The Buyer clicks on the Add Filter button
  5. The Buyer selects the Manual Filter filtering method
  6. The Buyer enters *nokia* in the Manual Filter textbox and clicks the Case Insensitive checkbox
  7. The Buyer clicks on the Test Filter button
  8. The System displays a list of all matching User-Agent Strings, highlighting the match(s)
  9. The Buyer clicks on Confirm Filter button
  10. The System adds the filter to the campaign level whitelist in the Database for the selected campaign

So now the Buyer (in this case Nokia or an agent acting on Nokia’s behalf to setup the ad campaign) has created a campaign level whitelist (i.e. all banners in the campaign will be filtered by this whitelist) which includes a filter of *nokia* that is case insensitive. This means that, based on the requirements enumerated above, that no ad request that includes a User-Agent string that does not includes the word nokia in it will be served any banner from this campaign. The effect? Ovi store ads will only be shown to users who are using Nokia handsets (or who’s requesting user agent does not include a User-Agent string or is using an incorrect User-Agent string that includes the word nokia.

Lets look at two use cases for requesting an ad. One where the requesting handset is Nokia N95 and one where it is an iPhone 3GS.

Use Case:
A Apple iPhone 3GS makes an Ad Request from an application running the Ad Server SDK.
Pre-Condition:
The application is using the provided Ad Server SDK
The application makes an ad request
Post-Condition:
A banner is served
Scenario:
  1. The application sends a well-formed HTTP GET Request to the Ad Request Handler URL including an Ad Request payload and the Device User-Agent Header (Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.11 (KHTML, like Gecko) Version/3.1.1 Mobile/7A238j Safari/525.20)
  2. The Ad Request Handler Thread retrieves the Ad Request Payload and the HTTP User-Agent Header constructs an Ad Request, sets it’s state to New and pushes the Ad Request onto the Inbound Ad Request Queue
  3. The Ad Request Handler Thread registers with the Ad Dispatcher to wait for it’s Ad.
  4. The Ad Request Processing Thread pops the next Ad Request off of the Inbound Ad Request Queue
  5. The Ad Request Processing Thread checks the Ad Request state
  6. The Ad Request Processing Thread finds the Ad Request state is New
  7. The Ad Request Processing Thread pushes all ads in its Recycle Ad Queue onto the Active Ad Queue
  8. The Ad Request Processing Thread sets the Ad Request state to Unfulfilled
  9. The Ad Request Processing Thread pops the first add off the Active Ad Queue
  10. The Ad Request Processing Thread checks the selected ad for White- and Black- lists
  11. The Ad Request Processing Thread finds an active Campaign Level User-Agent Whitelist
  12. The Ad Request Processing Thread attempts to match each string in the Campaign Level User-Agent Whitelist against the Device User-Agent String in the Ad Request
  13. The Ad Request Processing Thread finds no match for the string *nokia* (case insensitive) in the Device User-Agent String;Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.11 (KHTML, like Gecko) Version/3.1.1 Mobile/7A238j Safari/525.20
  14. The Ad Request Processing Thread rejects the ad for this request placing it on the Recycle Ad Queue
  15. The Ad Request Processing Thread pushes the Ad Request back on the Inbound Ad Request Queue
  16. The Ad Request Processing Thread monitors the Ad Request Queue
  17. The Ad Request Processing Thread pops the next Ad Request off of the Inbound Ad Request Queue
  18. The Ad Request Processing Thread checks the Ad Request state
  19. The Ad Request Processing Thread finds the Ad Request state is Unfulfilled
  20. The Ad Request Processing Thread pops the first add off the Active Ad Queue
  21. The Ad Request Processing Thread checks the selected ad for White- and Black- lists
  22. The Ad Request Processing Thread finds no active White- or Black- lists for the ad
  23. The Ad Request Processing Thread selects the Ad for this Ad Request attaches the Ad to the Ad Request and updates the Ad Request state to Pending
  24. The Ad Request Processing Thread pushes the Ad Request onto the Outbound Ad Request Queue
  25. The Ad Request Processing Thread pushes all ads in its Recycle Ad Queue onto the Active Ad Queue
  26. The Ad Request Processing Thread Monitors the Ad Request Queue
  27. The Ad Dispatcher pops the Ad Request off the Outbound Ad Request Queue
  28. The Ad Dispatcher reads the Ad Handler Thread Id from the Ad Request and passes the Ad Request to the corresponding Ad Handler Thread
  29. The Ad Dispatcher Monitors the Outbound Ad Request Queue
  30. The Ad Handler Thread parses the Ad Request and retrieves the ad
  31. The Ad Handler Thread constructs an HTTP Response with the Ad Response Payload including the Ad
  32. The Ad Handle Thread sends the HTTP Response to the requesting application

A few explanations/notes:

The looping nature of the Ad Request Processing Thread makes it hard to write an efficient and clear Use Case but I’m not getting paid to do this (any more) so I’m not going to spend the time to do the work to make it clearer.

This use case leaves out details of filters other than the Whilelist filter, such as Logged In/Logged Out Ad Queue selection, Ad Unit size, etc. for simplicity.

There would be multiple Ad Request Processors/Threads running, so when the first thread that retrieved the Ad Request and found the request was in the New state it pushed all the ads in it’s internal Recycle Ad Queue back onto the Active Ad Queue because those banners still need to be served and may be suitable for this Ad Request. The second time the Ad Request Processing Thread finds the Ad Request in the Unfulfilled state it does not empty it’s Recycle Ad Queue to avoid endlessly looping over the same unsuitable ad. (Writing this I think I need to think more about this—there could be a condition where a thread only ever sees Ad Requests in the Unfulfilled state and would therefore never empty its Recycle Ad Queue…)

I’ve included a lot of stuff that would actually go to other use cases and just be referenced as Include X Use Case, again I hope this makes it clearer (at least to the techies) what is happening.

Anyway… Here is what would happen when an Nokia N95 made a request:

Use Case:
A Nokia N95 makes an Ad Request from an application running the Ad Server SDK.
Pre-Condition:
The application is using the provided Ad Server SDK
The application makes an ad request
Post-Condition:
A banner is served
Scenario:
  1. The application sends a well-formed HTTP GET Request to the Ad Request Handler URL including an Ad Request payload and the Device User-Agent Header (Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaN95/10.0.018; Profile/MIDP-2.0 Configuration/CLDC-1.1 ) AppleWebKit/413 (KHTML, like Gecko) Safari/413)
  2. The Ad Request Handler Thread retrieves the Ad Request Payload and the HTTP User-Agent Header constructs an Ad Request, sets it’s state to New and pushes the Ad Request onto the Inbound Ad Request Queue
  3. The Ad Request Handler Thread registers with the Ad Dispatcher to wait for it’s Ad.
  4. The Ad Request Processing Thread pops the next Ad Request off of the Inbound Ad Request Queue
  5. The Ad Request Processing Thread checks the Ad Request state
  6. The Ad Request Processing Thread finds the Ad Request state is New
  7. The Ad Request Processing Thread pushes all ads in its Recycle Ad Queue onto the Active Ad Queue
  8. The Ad Request Processing Thread sets the Ad Request state to Unfulfilled
  9. The Ad Request Processing Thread pops the first add off the Active Ad Queue
  10. The Ad Request Processing Thread checks the selected ad for White- and Black- lists
  11. The Ad Request Processing Thread finds an active Campaign Level User-Agent Whitelist
  12. The Ad Request Processing Thread attempts to match each string in the Campaign Level User-Agent Whitelist against the Device User-Agent String in the Ad Request
  13. The Ad Request Processing Thread finds a match for the string *nokia* (case insensitive) in the Device User-Agent String;Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaN95/10.0.018; Profile/MIDP-2.0 Configuration/CLDC-1.1 ) AppleWebKit/413 (KHTML, like Gecko) Safari/413
  14. The Ad Request Processing Thread selects the Ad for this Ad Request attaches the Ad to the Ad Request and updates the Ad Request state to Pending
  15. The Ad Request Processing Thread pushes the Ad Request onto the Outbound Ad Request Queue
  16. The Ad Request Processing Thread pushes all ads in its Recycle Ad Queue onto the Active Ad Queue
  17. The Ad Request Processing Thread Monitors the Inbound Ad Request Queue
  18. The Ad Dispatcher pops the Ad Request off the Outbound Ad Request Queue
  19. The Ad Dispatcher reads the Ad Handler Thread Id from the Ad Request and passes the Ad Request to the corresponding Ad Handler Thread
  20. The Ad Dispatcher Monitors the Outbound Ad Request Queue
  21. The Ad Handler Thread parses the Ad Request and retrieves the ad
  22. The Ad Handler Thread constructs an HTTP Response with the Ad Response Payload including the Ad
  23. The Ad Handle Thread sends the HTTP Response to the requesting application

This type of White- and Black- list filtering would prevent me from seeing ads for the Ovi store on my iPhone (assuming the people provisioning the ads or campaigns used, and used correctly, the filtering options mentioned above — but that’s a business problem not a technical one.)

The ad serving space is complicated but have been surprised to see a number of examples like this on my phone over the past few days. In fact most of the ads I receive in the applications that use the Google system over the AdMob system seem to fall into this category. I have notices a lack of relevant ads in general for Singapore in the Google Web system. On some sites I see very relevant ads; global companies doing brand building or selling online. But on a lot of sites, and I mean a lot I see the same ads for the same service—a Singapore specific company so at least the location filtering is working. Or I see public service ads (invariably for Kiva [kiva.org].) So either there are not enough relevant ads for Singapore or these local guys are spending a lot of money to spam everyone.

The people at Google are a lot smarter than me, so I wonder if there is something I am missing in all this? Were my product people wrong? Are the users who are creating the campaigns not using some feature of the Google system that would filter these ads from me? Am I completely nuts? Do I have too much time on my hands?

Categories
technical

Concerning Color Algorithms

Long ago I worked on an interface for a reporting system that displayed transaction volume data as a large table; days on one axis and hours on the other.  To improve the readability of this densely packed data we wrote a coloring algorithm to change the background color of the table cell.  The algorithm colored each cell in relation to every other cell using the minimum and maximum values of the displayed data.  The effect was quite good.  Users could instantly see patterns in the data without having to read the actual number in the cells.  This data display, though simple was one of the main selling points of the system.  End-users like the sexy packaging, the unseen technology that actually makes the system a wonder is only of interest to the developers—unless of course it does not work.

I spent a lot of time improving the UI of the system as a project once, and when my CTO asked me what I was going one day I said “optimizing the color algorithm”—that was a mistake.  I didn’t sleep for the next 6 months as I was assigned the new, high priority, has-to-be-done-now project.  For a new technology, that I had no idea how it worked.  It was fun, but that’s a story for another day. The reason I raise this here is that I never finished the intended work on the color algorithm. One of the major goals that I never got to was to fix the color range. The algorithm as implemented ended up with colors that transitioned from purple to yellow. I wanted to improve the algorithm to output colors that transitioned from red through yellow to green. Never got done.

Recently I was working on a project that required some visual UI work. One of the elements was a bar chart that displayed volume over time. So I used the same-old, same-old color algorithm (in PHP):

/*
* getColor()
* $max := the highest value in the set to be colored.
* $min := the lowest value in the set to be colored.
* $c := the current item's value.
*
* Returns a string; the HEX representation of an RGB color.
*/
function getColor( $max, $min, $c) {
   $j=0;
   $r=0;
   $g=0;
   $b=0;

   $j = $max != $min ? ( $c - $min ) * 20 / ( $max - $min) : 0;
   $j = $j > 20 ? 20 : $j;

   $r = 5 + $j;
   $r = $r > 15 ? 15 : $r;

   $g = 5 + $j/2;
   $g = $g > 15 ? 15 : $g;

   $b = 10 - $j/2;
   $b = $b > 0 ? 0 : $b;

   return sprintf( "%01x%01x%01x%01x%01x%01x", $r,$r,$g,0,$b,0 );
}
Old getColor output

Simple right? Well using that you end up with this:

So I started scratching my itch to fix the colors.

My first thought was to just ask a question over at StackOverflow [stackoverflow.com]. As I was typing in my question the system suggested existing questions… and since it’s an SO faux pas to repeat questions I started looking at them. It didn’t take long before I found two questions that seemed to be pointing me in the right direction: 1. Generate colors between red and green for a power meter [stackoverflow.com] and 2. Color scaling function [stackoverflow.com].

Reading the answers to those questions it took about 30 minutes to do this:

/*
* getColor()
*
* $max := the highest value in the set to be colored.
* $min := the lowest value in the set to be colored.
* $c := the current item's value.
*
* Returns a string; the RGB representation of a color (eg. "rgb(255, 255, 255)").
*/
function getColor( $max, $min, $c) {
   $h = (.33 / ($max - $min)) * ($c - $min);
   $s = .8;
   $v = 1;

   $rgb = hsv2rgb($h, $s, $v);

   return(sprintf("rgb(%d,%d,%d)", $rgb['r'], $rgb['g'], $rgb['b']));
}

/*
* hsv2rgb()
*
* $h := the hue, normalized to 0-1.
* $s := the saturation, normalized to 0-1.
* $v := the value, normalized to 0-1.
*
* Returns an associative array containing the red ('r'), green ('g') and blue ('b') values in the range 0-255.
*/
function hsv2rgb ($h, $s, $v){
   $rgb = array();

   if($s == 0){
      $r = $g = $b = $v * 255;
   } else {
      $var_h = $h * 6;
      $var_i = floor( $var_h );
      $var_1 = $v * ( 1 - $s );
      $var_2 = $v * ( 1 - $s * ( $var_h - $var_i ) );
      $var_3 = $v * ( 1 - $s * (1 - ( $var_h - $var_i ) ) );

      if ($var_i == 0) { $var_r = $v ; $var_g = $var_3 ; $var_b = $var_1 ; }
      else if ($var_i == 1) { $var_r = $var_2 ; $var_g = $v ; $var_b = $var_1 ; }
      else if ($var_i == 2) { $var_r = $var_1 ; $var_g = $v ; $var_b = $var_3 ; }
      else if ($var_i == 3) { $var_r = $var_1 ; $var_g = $var_2 ; $var_b = $v ; }
      else if ($var_i == 4) { $var_r = $var_3 ; $var_g = $var_1 ; $var_b = $v ; }
      else { $var_r = $v ; $var_g = $var_1 ; $var_b = $var_2 ; }

      $r = $var_r * 255;
      $g = $var_g * 255;
      $b = $var_b * 255;
   }

   $rgb['r'] = $r;
   $rgb['g'] = $g;
   $rgb['b'] = $b;

   return $rgb;
}

And…. voila:

New getColor output

That’s great! Now maybe I’ll play with the hue bounds, the saturation and the value, but it looks fairly good on my first try.

Moral of the story: this is what StackOverflow is for; getting an answer to a question without even asking it… Because you are not re-inventing the wheel, someone had your problem before, and maybe someone found the answer already. SO, thanks to Paul Dixon [stackoverflow.com] (also here [blog.dixo.net]) and ΤΖΩΤΖΙΟΥ [stackoverflow.com] for their answers!