But we might have reached the point in the life of the web where the search engine will decline in value. This is because search engines rely on public content and an ever-increasing amount of our collective information is now being created for and stored in the dark web. This term refers to networks which use the public internet but which require specific software to access.
The best example of the dark web is social media sites like Facebook, Twitter and LinkedIn. There are huge amounts of content now created for and stored inside of these platforms. Facebook is the best example of this. There are now many businesses that no longer have a web site but which instead have a Facebook page. There are many organizations that do the same and that communicate with members only through a Facebook group. Every social media site has something similar. For instance, there are now thousands of articles that are written for and published within LinkedIn.
But the dark web includes a lot more than just social media. Corporations and trade associations now routinely keep information hidden from the general public by requiring password access to read whatever they have published. You can understand the motivation behind this – a trade association might have more luck recruiting members if membership includes access to unique content.
But corporations also hide a lot of content that used to be public. For example, cable companies like Comcast require a customer to enter a valid address before showing them current products and prices. Those prices used to be openly published, but the act of asking for an address now ‘hides’ this information from search engines. Comcast certainly wants this iformation hidden since they don’t offer the same prices everywhere.
While the information behind corporate and trade association web sites is already unavailable to search engines, for now the information behind most social media sites is still searchable. But there is nothing that requires it to stay that way. Google and Facebook are now engaged in a fierce battle to win web advertising and there is nothing to stop Facebook from flicking a switch and hiding all of its content from Google search. To do so would instantly devalue Google since businesses listed only within Facebook would disappear from the search engine results.
It almost seems inevitable that this day will come, and probably not long from now. Both Google and Facebook have requirements from stockholders to continue to grow and both businesses are fueled largely by advertising revenue. It’s hard to think at some point that Facebook won’t deliberately try to gain an edge in this battle.
But the consequences of the dark web to all of us an ever-increasing lack of information. I remember spending evenings in the early days of the web just browsing the web for interesting content. It seems like every college and high school pushed huge amounts of interesting content onto the web – because in those early days that’s what the web was all about. All of the content from textbooks and homework assignments were on the open web for anybody to read. But there is no longer any major motivation to push information to the web. And many schools and universities are now behind a dark web wall as well.
The web has shifted massively towards entertainment and content is no longer king. Most of the early content-heavy web sites have died since they’ve either been pulled down or are no longer maintained. And so every day more content is removed from the web, and every day search engines become a little less valuable to the human race.
We may already be in a world where there is more useful data on the dark web than on the open one. Facebook and a few others could push the search engines onto the road to decline. History has already shown us that search engines can come and go. Excite was one of the first web companies that sold for billions, but the company that bought it, @Home went bankrupt. There is nothing to say that the Google search engine or any other is something that we can always rely on. And in fact, they may just fade away someday due to irrelevance.