Skip to Main Content

Deep Web

The Deep Web Library Guide discusses why the Deep Web exists and what it contains. The Guide also provides tools for searching the Deep Web, along with resources for further information.

Welcome

Welcome to the Deep Web Library Guide. This guide will help you understand what the Deep Web is, why it is so difficult to search, and the efforts being made to uncover the information located within it.

Defining the Deep Web

The Deep Web consists of all information on the World Wide Web which cannot be located by general-purpose search engines (Google, Yahoo, Bing etc.).  There's a large amount of information which even powerful engines such as these cannot find.  

Sometimes you'll hear the term "Dark Web." These terms are sometimes used interchangeably, but the deep web and the dark web are not the same thing, as this helpful article from BrightPlanet explains: Clearing up Confusion: Deep Web vs. Dark Web

There is nothing inherently special about Deep Web pages.  The Deep Web is largely the product of limitations in web crawler technology.   As technology changes, so does the Deep Web.

Web Page vs. Web Site

A web page is often defined as a document on the World Wide Web with a single, unique web address.  A web site is a group of web pages created by an individual, group or organization.  Pages within a web site link to one another and are usually related by topic.  One web site may contain many web pages.

Example: Saint Louis Community College has a web site.  The opening page of the web site is http://www.stlcc.edu.

Within this site are numerous individual web pages, such as the STLCC Bookstores page: 

Other Important Terms

Synonyms for Deep Web:

  • Hidden Web
  • Invisible Web
  • Secret Web

Synonyms for Web Crawler:

  • (Web) Spider
  • (Web) Bot

Open Web, Visible Web, Surface Web: The portion of the Web which is accessible to general search engines.  This part of the Web is free of charge, available for use by anyone. However, note that there are free sites within the Deep Web as well, which for one reason or another are inaccessible to web crawlers.

Dark Web: part of the deep web, used for secure transactions and (often) criminal activity. 

How Big Is the Deep Web?

Estimating the size of the Deep Web is a difficult task, given the sheer number of web pages involved, and the hidden nature of those pages. However, BrightPlanet, a "deep web intelligence company" notes this:

"...it is important to tap into the rich resources existing in the Deep Web. The last time an extensive study was completed estimating the size of the Deep Web was in 2001 — a time when the internet consisted of only approximately three million different domains. The 2001 study revealed that at that time the Deep Web was approximately 400-500 times the size of the Surface Web.

Today’s internet is significantly bigger with an estimated 555 million domains, each containing thousands or millions of unique web pages. As the web continues to grow, so too will the Deep Web and the value attained from Deep Web content."

Source: Pederson, Steve. "Understanding the Deep Web in 10 Minutes." BrightPlanet, Mar. 2013. Web. 18 Sept. 2013.

 

Iceberg symbolizing Deep Web

St. Louis Community College Libraries

Florissant Valley Campus Library
3400 Pershall Rd.
Ferguson, MO 63135-1408
Phone: 314-513-4514

Forest Park Campus Library
5600 Oakland
St. Louis, MO 63110-1316
Phone: 314-644-9210

Meramec Campus Library
11333 Big Bend Road
St. Louis, MO 63122-5720
Phone: 314-984-7797

Wildwood Campus Library
2645 Generations Drive
Wildwood, MO 63040-1168
Phone: 636-422-2000