The Crazy World of Web Advertising

auction-hammerLately you might have noticed that while you are browsing the web that you are experiencing a big delay in loading some web pages. A web page will pop up on your browser, but then it will sit for a while before it lets you navigate the page. The delay can last for many seconds and always feels longer than it is.

These delays are due to web advertisers. Web sites contain two kinds of advertising. There are embedded ads that a web site owner puts onto their site and lock. Embedded ads cannot be accessed or changed by an outside party and are integrated into the web page. But there are also remnant ads, which are ads that fit into blank spaces left for that purpose by the website owner. It is the process of putting ads into these remnant spaces that is causing the delay in loading pages.

There are a lot of companies that sell advertising into the remnant ad space including: Google (Doubleclick), Yahoo, Amazon, Facebook, AOL, AppNexus, Openx, Adroll, RightMedia, and dECN. The whole process is fascinating and is not much talked about outside of the advertising world.

These companies compete to put their ads into the remnant spaces. Each of these companies sell internet advertising and the remnant ads are where they are able to place most of their ad inventory. I always assumed that the web advertisers made deals with the popular web sites to place ads on those pages. But it doesn’t happen that way at all.

When somebody puts a remnant ad space on a web site it is open real estate and any advertiser that gets access to that web site gets to put advertising into the slots. So all of these advertisers play a game of real estate grabbing to try to be the first company to get to the remnant ad space.

Some advertisers dominate this market because they are associated with a web service that has lot of eyeballs. For example, if you go to a web page through the Google search engine or through another Google site like YouTube, then Doubleclick is going to grab the opportunity to place the ads. This is why the Google search engine is so lucrative. They charge companies to get a top ranking from searches, and they make more money placing ads on those sites that contain remnant ad space.

But the same is true for other large companies. If you go to a web page from Facebook or Yahoo or AOL then those companies are first in line to grab the remnant ad spaces. So a web site will allow any ad companies depending upon how the customer came to be at that site. As an aside, this is why it’s big news when some web service says they are changing browsers, because it means a big shift of dollars between these advertisers.

But web pages don’t run slow because of the primary advertiser on the web page. If Doubleclick gets to a web site and has enough ads in its inventory to fill all of the ad spaces the process is fast and nearly imperceptible to the end-user. The delays happen when the primary advertiser doesn’t have enough ads to fill all of the remnant ads on a given page.

Ad companies will only place ads that they have pre-sold. Since there are billions of requests per day to fill remnant ad spaces there will be many times when an advertising company doesn’t have enough ad inventory to fill a given page at a given moment. At that point, the first advertiser in line will fill the slots they can, and then they will send the ad to what is called the ad exchange. This is a consortium of all of the web advertisers.

At the ad exchange the remaining open remnant ad spaces are offered at auction to the highest bidder. Money is exchanged from these auctions through a process called the ad trading desk. So let’s say in this example that OpenX purchases the open slots on the web site in question from the ad auction. That gives them temporary control of the web site and they place ads in the remaining slots.

But if OpenX doesn’t have enough ads to fill all of the remaining slots, it goes back to the ad exchange again and there is another auction. Web pages can get stuck in this process and keep going back for auction, and that is when you will see a really big delay before you can navigate the page. But even when there are only a few advertisers involved the delay can be a few seconds.

There is one ugly part to this process in that the prices in the ad exchange can get really low, as in having four or five zeroes in front of the price. These low prices have lured the guys who push malware to the ad exchanges. These are the same guys who try to spread malware through spam. They will buy really cheap ads and then attach their malware to the remnant ad space. This is really insidious because an end user doesn’t have to click on the ad to get the malware – just loading the web site is enough to give them the malware. At the end of 2014 Cisco identified this ad malware, which is now called malvertising, as the second biggest source of malware in the web ecosystem.

Who Owns Internet Ad Space?

advertise-hereGoogle made a very interesting announcement a few weeks ago that led me to find out more about the ad space on web sites. Google announced that for $2 per month they would block all ads on web sites for a customer as long as they browse through the Chrome browser.

I find this fascinating because it means that Google thinks that they have the ability to block an ad, even when they are not the one to have placed the ad in the first place. Google sells a lot of ads, and so it makes sense that they can block ads that they have placed on a web page. But when they say they can block all ads it also means that they think they have the ability to block ads placed by somebody else.

Just to be clear about what I mean by ads, look at this web page. At the top is a banner ad. At the top right of the story is an ad. And across the bottom of the article are four ads. After loading this web site multiple times I noticed that the ads changed.

It turns out that there are two kinds of ads on a web page. There are fixed ads and remnant ads. Fixed ads are placed there by the web site owner or somebody they partner with to advertise for them. Fixed ads embedded into the web page and can only be accessed by the website owner. The other kind of ads are called remnant ads. These are coded in such a way as to be available to outsiders, and anybody that has access to a website before it reaches a customer can change what is in the remnant ad space.

And as you would expect, these remnant ad spaces get changed all of the time. There are a lot of companies that sell advertising into the remnant ad space including Google (DoubleClick), Yahoo, Amazon, Facebook, AOL, AppNexus, Openx, Adroll, RightMedia and dECN. It was very easy for me to spot remnant ads in the recent election season, because I swear that every web page I looked at here in Florida had a political ad for Rick Scott who was running for reelection as Governor. So somebody was being paid in Florida to put those ads onto Florida computers.

The first question this raised for me is: who owns this ad space? The web page example is from the TechCrunch web site. TechCrunch chose to make the ads open to the public and I assume they gets revenues from at least some of the parties that use that space, which is their motivation to use remnant ad space. Google thinks they have a right to go in and block whatever is on the remnant ad space on that page, so they are sure that it is theirs to grab. I know that some of the larger ISPS like cable companies are also in the advertising business, through partners, and I wouldn’t be surprised if it was Comcast that gave me all of the Rick Scott ads.

I was shown a recent legal opinion by one of the companies that advertises in the remnant space who was gracious enough to share it with me as long as I don’t publish it. The opinion says basically that nobody owns the remnant ad space. The legal opinion says that the act of a web site owner in making this available to the public means just that, and it can be used by anybody who somehow has access to the website before it reaches a customer. That generally is going to mean some company who is part of the chain between a web site and the customer. Obviously the web site owner can hire somebody to place ads in the remnant space. If you reach the web site through a browser then the browser owner can place the ad in there. If you get to a web site through a link on another web site like Yahoo News then they can place ads there. And your ISP also would have access to this ad space.

I really like the Google product that blocks ads. I think there are plenty of customers who would love to avoid all of those ads. Further, blocking ads means a faster Internet experience for a customer. I know there are web sites I go to that have multiple videos automatically running that seems like an extravagant use of my bandwidth. I have a 50 Mbps Internet connection and there are still web sites that load very slowly due to all of the extra videos that have been layered into the ad spaces. I also learned that remnant ads are one of the most common sources today of adware and malware and I will talk about that more in tomorrow’s blog.

Primer on Internet Cookies

Am I a brownie? Am I a cookie? I'm so conflicted!

Am I a brownie? Am I a cookie? I’m so conflicted! (Photo credit: Wikipedia)

I spent some time recently educating myself on how Internet cookies work and thought I would share what I learned. I think that most people assume that cookies are largely malicious and that is not necessarily the case. But there are certainly types of malware on the Internet that can do every bad thing to you that you can imagine.

Plain cookies by themselves are pretty simple. People think of cookies as programs that are put on to your computer to spy on you. But cookies are nothing like that. A good definition of a cookie is a piece of text that a server puts onto a computer. For example, a cookie allows a web site to store something on your computer that it can retrieve later if you come back to the same web site.

Basic cookies perform a number of simple tasks that make your web browsing experience more enjoyable. For example, a cookie might record your preferences for a given website so that it doesn’t have to ask you about yourself every time you visit. A good example is when you visit a weather site and it gives you the weather for your zip code. Having a cookie from that site stops you from having to type in your zip code every time you want to check the weather. And if you want to track the weather in a dozen locations it will remember all of them.

The basic tool used by cookies is referred to as a name value pairs. Most cookies from website assign you a user ID and that is the first half of the name value pair. This lets them know who you are (actually, only which computer you are on) no matter how many times you come back to the site. The second half of the name value pair is then whatever data they want to store on your machine to prepare for your next visit, such as the zip code for the weather website.

If you have ever deleted all of the cookies from your computer you know the hassle you have after that. All of a sudden the web pages for your bank, credit cards, music site, games, and whatever else you routinely use don’t know who you are any more. Those sites were probably using cookies in the basic value pair mode, using them only to store your preferences.

You have the ability to control these kinds of cookies if you want. There is a setting in most web browsers that will allow you to get notified and make a choice every time a web site asks for a pair value from you. Try this and you will quickly abandon it because it will slow your web browsing to a crawl.

Of course, basic cookies can create annoyance at the user level. For example, my wife and I share a computer sometimes and I don’t really want to log automatically onto her Facebook or Amazon page. In fact, one day I bought a Kindle book without noticing it was her account and the book downloaded to her Kindle instead of mine. And I use multiple computers and thus I often get a different experience from a given web site based upon which computer I am using.

So why the big controversy over cookies? Unfortunately there are now companies that take cookies to the next level, and these are the ones that bother people. For example, DoubleClick is used by a lot of web shopping sites and it allows your user ID and data to be used across multiple web sites. If you buy something on the web you obviously have to tell whoever you are buying from all about you – your name, address, credit card number etc. But with cookies that can create an identify across multiple we sites the companies you buy from can start putting together profiles about you and they can see all of the different sites you are visiting that are within the DoubleClick universe. And this is all done without you knowing it.

And of course, there are all sorts of malware on the Internet that do all of the bad things that people associate with cookies. There are all sorts of adware, sneakware, keyloggers, browser hijackers, trojan horses and worms that gather all sorts of information about you. But basic cookies are not a threat to you. A cookie is a data file and it cannot gather other information on your computer. That requires one of the forms of malware. However, cookies that share information across multiple web sites can gather a lot more information about you than you might want to share. It’s a tricky world on the Internet and it’s really easy for people to forget that somebody else is at the other end of every keystroke you make.