CHAPTER ONE

Search: Reflecting Consciousness and Connecting Commerce

The term search engine optimization (SEO) is a bit of a misnomer. People who work in the SEO field do not optimize search engines; they optimize web pages so that they are accessible and appealing to both search engines and people. Therefore, a more appropriate label would be “search result optimization,” because the immediate goal of SEO is to improve the rankings of your web pages in search results. Notice we didn’t say “the ultimate goal,” because ranking higher in search results is not an end unto itself; it is only a means for increasing traffic to your site, which—if it is optimized for converting visitors to customers—will help your business be more successful.

Why is this such a critical field? Quite simply, search has become integrated into the fabric of our society. More than 7.5 billion Google searches are performed every day, which equates to more than 85,000 queries per second (with users typically expecting that responses to their search queries will be returned in less than a second). Further, it is estimated that more than half of all traffic across the web comes from organic (nonpaid) search, which means that for many businesses, SEO is the most important digital marketing investment.

Through the power of search, we’re often able to find whatever we want in just a minute or two, or even a few seconds. People can use search to conduct many of their research activities and shopping, banking, and social transactions online—something that has changed the way our global population lives and interacts. As a result, it’s critical for owners of websites to increase their visibility in search engine results as much as they can. Obtaining the desired prime search result real estate is not a simple matter, but it is one that this book aims to deconstruct and demystify as we examine, explain, and explore the ever-changing art of search engine optimization.

Is This Book for You?

The short answer? Yes, if you’re someone whose work touches a website. For the advanced SEO, this book is filled with material that will help you develop your SEO skills even further. For beginners or intermediates, or for people who work in peripheral fields or roles (such as product managers, marketers, graphic designers, web developers, or other roles that have anything to do with creating, managing, or maintaining a website), this book will help you learn to apply SEO to your work on a day-to-day basis.

Most SEO books are short and focus only on the technical aspects. Understanding these aspects is critical, as failure on this front can prevent your pages from being accessible to Google or from ranking as well as they should, regardless of how great the content is on your site. Unfortunately, most content management systems (CMSs) and ecommerce platforms are not naturally optimized for SEO, and the same is true of most JavaScript frameworks and static site generators (if you’re using one of these). This fact is what makes technical SEO such a critical area of expertise, and it’s one that demands a large amount of time and energy to successfully execute.

Ultimately, though, technical SEO is just table stakes, and its main benefit is that it enables your pages to compete for rankings. It’s the quality and depth of your content and your overall reputation and visibility online (including the links that your site earns) that drives how highly you rank. For these reasons, SEO also has a major strategic component.

These aspects of SEO require you to think holistically and creatively. You have to research trends and personas for the industry or market that your site is designed for, analyze the sites and companies that compete with it, develop excellent content and try to get high-value sites to link to it, approach your topics and keywords (search terms and phrases) from a variety of different search perspectives and contexts, document your work, and measure your progress, all while communicating effectively with various managers, engineers, designers, and administrators who control different aspects of the website. You’ll learn more about the strategic aspects of SEO in Chapter 4.

SEO Myths Versus Reality

There are few industries or professions that suffer more from myths and misconceptions than SEO. Here are some of the more common ones:

Myth: SEO is just a few secret tricks.

Reality: SEO is a complex, multifaceted, iterative process, and it takes time and effort to fully realize its potential. About the only time you’ll reliably see a substantial “quick win” in legitimate SEO is when you fix a technical issue that prevents search engines from properly accessing your website content. There may be some lesser-known tools or techniques, but there aren’t really any secrets per se. And if there were, then over time, search engines would find them and fix them, and you’d most likely lose any rankings you’d gained by exploiting them.

Furthermore, optimizing for search engines does not mean cheating or manipulating them. Optimizing a web page is a lot like optimizing a career resume or a profile on a dating site; the goal is to enable your site to be found quickly and easily by the right people, without sabotaging yourself with lies or misrepresentations.

Myth: All SEO is spammy and unethical.

Reality: There are some unethical SEO practices out there that seek to exploit loopholes in search engine algorithms, intentionally violate rules or guidelines published by search engines, or circumvent systems designed to detect or prevent the manipulation of organic search rankings. We don’t teach such “black hat” tactics in this book; in fact, we specifically implore you to avoid them. Like pretty much everything else worth doing in life, you will get the most benefit out of SEO efforts that focus on mastering the fundamentals instead of exploiting sketchy shortcuts.

Myth: SEO that is not spammy and unethical is a waste of time and money.

Reality: The black hat stuff sometimes does get a page to the top of the search results, but only for a short period of time before it gets banned from the index. When unethical SEO practices are detected, search engines are quick to punish sites that employ them. There are no legitimate SEO tactics that will instantly rank a new page highly for a high-volume search term; it takes time, effort, and money to compete for the top 10 spots. It’s a lot like bodybuilding: steroids may yield big gains very quickly, but at what cost?

Myth: You need to be a web developer or an IT expert to do SEO.

Reality: If you’re not tech-savvy, there are SEO tools (which are covered in Chapter 4) that can help you discover and diagnose technical SEO issues. You can then rely on those who are responsible for building or maintaining your site to make the necessary changes. You’ll benefit from being familiar with HTML, but you likely won’t have to do any coding or Unix command-line wizardry.

Myth: SEO is only for ecommerce sites or huge corporations.

Reality: If you want to be found via web search, then you will benefit from search engine optimization. Your site doesn’t have to directly sell anything. Do you have a mailing list that you want people to sign up for? Do you want to collect qualified sales leads through a web form or survey? Perhaps you have a content-centric site such as a blog that generates revenue from ads or affiliate links, or from paid subscriptions; or maybe you want to promote your book, movie, or band. If more traffic would help you in any way, then you’ll benefit from SEO.

Myth: Optimizing my site means I’ll have to pay for a complete redesign, or use a different content management system, or pay for upgraded web hosting.

Reality: In some cases, you may need to make changes to your site or web host in order to solve technical SEO problems. There are many different options, though, and if you will benefit from increased traffic, then it’s probably worth the effort and expense. There is no universal rule of SEO that says you must use Apache or WordPress or anything like that.

Myth: SEO requires me to be active on social media and schmooze with influencers.

Reality: You don’t necessarily have to do anything like that if it doesn’t make sense for your organization. It certainly will help you obtain more incoming links (this is explained in Chapter 10), but if your site is search friendly and you have great content, then you’ll eventually get some links without having to solicit for them—it’ll just take more time and may not yield the same level of improvement in rank.

Myth: Buying Google Ads (or hosting a Google Ads banner on my site) is cheaper than SEO and works just as well.

Reality: Paid search will certainly yield more visitors than doing nothing, but successful SEO programs usually deliver much more traffic with a better return on investment (ROI) than paid campaigns, and will not directly influence organic search rankings. You could pay for ads for your best “link-worthy” content (this is covered in Chapter 10) and hope that some of the people who click the ads will then share the links; this could indirectly increase your search rankings for some keywords, but there is no direct correlation to buying a Google ad and ranking higher in organic results.

If paying for search ads in Google or Bing is working for you and there is a decent ROI, then great—keep doing it! But why would you not put in the effort to improve your organic rankings as well? You don’t have to pay every time someone clicks on a high-ranking organic result, and as a result, SEO frequently provides a higher ROI than paid search campaigns.

SEO does not have to be expensive. Certain scenarios are, of course, inherently costly, such as if you are competing against huge corporations for very difficult keywords, are attempting to rehabilitate a site that has been banned for previous black hat shenanigans, or have tens of thousands of pages with messy HTML. Even if you run into a cost barrier like one of these, though, you can still find creative alternatives or commit to making smaller incremental improvements over a longer period.

Hosting Google AdSense units on your pages won’t impact how these pages rank in organic results either. If your goal is to get indexed, then you’re much better off submitting an XML sitemap to Google Search Console (this is covered in Chapter 6) instead. If your site is a content site, then Google AdSense may be worthwhile from a revenue standpoint, but it may show ads from your competitors. And if your goal is to get conversions (sales, sign-ups, leads), then why would you want to show ads for them?

NOTE

Google Ads is Google’s parent company Alphabet Inc.’s largest source of revenue. In general, paid search placement is not within the scope of SEO. However, Google Ads data can be useful for keyword research purposes and for A/B testing of organic landing pages, and many SEO professionals use Google Ads data, including conversion and user engagement data, to assist in SEO strategy development.

The Mission of Search Engines

While those on the internet are free to use any of the many available search engines to find what they are seeking, Google remains the dominant player worldwide, with over 90% market share (see Figure 1-1). Nonetheless, the burden is on Google (and other search engines) to provide a satisfying search experience. For the most part, search engines aim to accomplish this by presenting the most relevant results and delivering them as fast as possible, as users will return to the search engine they believe will return the results they want in the least amount of time.

To this end, search engines invest a tremendous amount of time, energy, and capital in improving their speed and relevance. This includes performing extensive studies of user responses to their search results, continuous testing and experimentation, analysis of user behavior within the search results (discussed later in this chapter), and application of advanced machine learning techniques to tune their search algorithms.

Search engines such as Google generate revenue primarily through paid advertising. The great majority of this revenue comes from a pay-per-click (or cost-per-click) model, in which the advertisers pay only for the number of users who click on their ads. Because the search engines’ success depends so greatly on the relevance of their search results, manipulations of search engine rankings that result in nonrelevant results (generally referred to as spam) are dealt with very seriously.

Each major search engine employs teams of people who focus solely on finding and eliminating spam from their search results (these are referred to as “webspam” teams). Larger search engines such as Google additionally apply dynamic algorithms that detect and deal with poor-quality content and/or spam automatically. These efforts to fight spam matter to SEO professionals because they need to be careful that the tactics they employ will not be considered spammy by the search engines.

Figure 1-1 shows the global market share for search engines as of June 2023, according to Statcounter. As you can see, Google is the dominant search engine on the web worldwide, with a 92.5% worldwide market share. Bing comes in a distant second, with a 3.1% share.

Figure 1-1. Search engine market share (source: Statcounter)

However, in some markets, Google is not dominant. In China, for instance, Baidu is the leading search engine, and Yandex is the leading search engine in Russia. The fact remains, however, that in most world markets, a heavy focus on Google is a smart strategy for SEO.

Goals of Searching: The User’s Perspective

The basic goal of a search engine user is to obtain information relevant to a specific set of search terms entered into a search box, also known as a query. A searcher may formulate the query as a question, but the vast majority of searches are performed by users simply entering word combinations—leaving the search engines to do the work of determining a query’s “intent.” One of the most important elements of building an SEO strategy for a website is thus developing a thorough understanding of the psychology of your target audience and how they use words and concepts to obtain information about the services and/or products you provide. Once you understand how the average search engine user—and, more specifically, your target audience—utilizes query-based search engines, you can more effectively reach and retain those users.

Search engine usage has evolved over the years, but the primary principles of conducting a search remain largely unchanged. Most search engine use includes the following stages:

  1. Users experience the need for information. They may be looking for information on a specific website, and they will search for that website (a navigational query); they might want to learn something (an informational query); or they might want to buy something (a transactional query). We will discuss the challenge of determining the user’s intent in more detail in the following section.

  2. Users formulate that need using a string of words and phrases (i.e., search terms), comprising the query. Data provided to us by seoClarity in March 2021 (see Figure 1-2) showed that 58.8% of user search queries are one to three words long, though as users become more web savvy, they may use longer queries to generate more specific results more quickly.

    Figure 1-2. Search query lengths (source: seoClarity)

  3. Users execute the query, check the results, and, if they seek additional information, try a refined query.

When this process results in the satisfactory completion of a task, the user, search engine, and site providing the information or result have a positive experience.

Determining User Intent: A Challenge for Search Marketers and Search Engines

Good marketers are empathetic, and smart SEO practitioners share with search engines the goal of providing searchers with results that are relevant to their queries. Therefore, a crucial element of building an online marketing strategy around SEO and organic (nonpaid) search visibility is understanding your audience and how they think about, discuss, and search for your service, product, and brand.

Search engine marketers need to be aware that search engines are tools—resources driven by intent toward a content destination. Using the search box is fundamentally different from entering a URL into the browser’s address bar, clicking on a bookmark, or clicking a link to go to a website. Searches are performed with intent: the user wants to find specific information, rather than just land on it by happenstance. Search is also different from browsing or clicking various links on a web page.

This section provides an examination of the different types of search queries and their categories, characteristics, and processes.

Informational Queries

Informational searches involve an incredibly broad range of queries. Consider the many types of information people might look for: local weather, driving directions, a celebrity’s recent interview, disease symptoms, self-help information, how to train for a specific type of career…the possibilities are as endless as the human capacity for thought. Informational searches are primarily nontransaction-oriented (although they can include researching information about a product or service); the information itself is the goal, and in many cases no interaction beyond clicking and reading is required for the searcher’s query to be satisfied. Figure 1-4 shows an example of an informational query.

Informational queries are often lower converting but good for building brand and attracting links. Here is how you can evaluate whether a query is worth pursuing:

Opportunities

Provide searchers who are already aware of your brand with positive impressions of your site, information, company, and so on; attract inbound links; receive attention from journalists/researchers; potentially convert users to sign up or purchase.

Average traffic value

The searcher may not be ready to make a purchase, or may not even have long-term purchase intent, so the value tends to be “medium” at best. However, many of these searchers will later perform a more refined search using more specific search terms, which represents an opportunity to capture mindshare with those potential customers. For example, informational queries that are focused on researching commercial products or services can have high value.

Figure 1-4. An informational query

Transactional Queries

Transactional searches don’t necessarily have to involve a credit card or immediate financial transaction. Other examples might include creating a Pinterest account, signing up for a free trial account at DomainTools, or finding a good local Japanese restaurant for dinner tonight. Figure 1-5 shows an example of a transactional query.

These queries tend to be the highest converting. You can use the following two criteria to evaluate the value of a specific query to you are:

Opportunities

Achieve a transaction (financial or other).

Average traffic value

Very high. Transactions resulting from these queries may not be immediate, and it’s up to the site receiving the related traffic to provide enough value to the user to convert them on their site or to make enough of an impression that the user comes back and converts later.

=
Figure 1-5. A transactional query

Local Queries

As the name implies, local searches relate to users seeking information as regards a specific location, such as where they currently are, or a location that they reference in the query. Examples include looking for directions to the nearest park, a place to buy a slice of pizza, or the closest movie theater (Figure 1-6). Local queries are not an intent in the same way that navigational, informational, and transactional queries are, but represent a subclass that cuts across all types of queries. Many local queries are transactional (though they differ as they relate to actions or transactions that will occur in person), but you can have navigational or informational local queries as well.

Figure 1-6. A local query

Consider the following when evaluating the value of ranking for a local query:

Opportunities

Drive foot traffic based on the proximity of the searcher. Offers a strong potential to achieve a transaction (financial or other).

Average traffic value

Very high. When users search for something near them, the probability that they are interested in direct interaction, and possibly a near-term transaction, is high. We can see that in the way Google has tailored its search engine results pages (SERPs) for local queries to meet this demand.

Searcher Intent

When you are building keyword research charts for clients or for your own sites, it can be incredibly valuable to determine the intent of each of your primary keywords. Table 1-1 shows some examples.

Table 1-1. Examples of query types
Term Queries Intent Monetary value per visitor
Beijing Airport 5,400 Navigational Low
Hotels in Xi’an 110 Informational Mid
7-Day China tour package 30 Transactional High
Sichuan jellyfish recipe 53 Informational Low

Hopefully, this data can help you to think carefully about how to serve different kinds of searchers based on their individual intents, and how to concentrate your efforts in the best possible areas. This type of analysis can help you determine where to concentrate content and links, as well as where to place ads.

Although informational queries are less likely to immediately convert into sales, this does not mean you should forgo pursuing rankings on these queries; getting your informative content in front of users seeking information can be incredibly valuable and can turn users into potential customers. As you can see in Figure 1-7, data from Conductor shows that users who find useful informational content on your site are 131% more likely to come to you to make a related purchase at a later date. They may also decide to share your information with others via their own websites, or through social media engagement—an indirect but potentially more valuable result than converting the single user into a paying customer.

One problem in search is that when most searchers formulate their search queries, their input is limited to just a handful of words (per Figure 1-2, 78.8% of queries consist of one to four words). Because most people don’t have a keen understanding of how search engines work, they often provide queries that are too general or that are presented in a way that does not provide the search engine (or the marketer) with what it needs to determine, with 100% accuracy 100% of the time, their specific intent.

Some search engine users may not have a specific intent behind a query beyond curiosity about a currently trending topic or a general subject area. While this can make it challenging for a search engine to deliver relevant results, it poses a great opportunity for the digital marketer to capture the mind of someone who may not know exactly what they are looking for, but who is interested in the variety of results the search engine delivers in response to a general query. These types of queries are important to most businesses because they often get the brand and site on the searcher’s radar, which initiates the process of building trust with the user. Over time, the user will move on to more specific searches that are more transactional or navigational in nature.

If, for instance, companies buying pay-per-click (PPC) search ads bought only the high-converting navigational and transactional terms and left the informational ones to competitors, they would lose market share to those competitors. Over the course of several days, a searcher may start with digital cameras, hone in on Olympus OMD, and then ultimately buy from the store that showed up in their search for digital cameras and pointed them in the direction of the Olympus OMD model.

Figure 1-7. How informational content impacts user trust (source: Conductor)

To illustrate further, consider the case where a user searches for the phrase Ford Focus. They likely have numerous considerations in mind when searching, even though they only use those two words in the query. Figure 1-8 gives an idea of what the range of those considerations might be.

As we can see, the user’s needs may have many layers. They may be specifically interested in a hatchback, a sedan, an electric car, or one of many specific model numbers. If they’re buying a used car, they may want to specify the year or approximate mileage of the car. The user may also care about the car having aluminum wheels, Spotify, a roof rack, front and rear seat warmers, and various other options.

Figure 1-8. The pyramid of user needs

Research published by Think with Google generalizes this concept, referring to it as the “messy middle.” As Figure 1-9 shows, this is the gap between the trigger that causes the user to take action and the actual purchase.

Figure 1-9. Typical user journey to a purchase

The exploration/evaluation part of this journey is highly complex and differs for every user. Whatever desires/needs users bring to this process, it’s incumbent on the website to try its level best to meet them in order to earn the conversion.

Given the general nature of how query sessions start, though, determining intent is extremely difficult, and it can result in searches being performed where the user does not find what they want—even after multiple tries. Research from the American Customer Satisfaction Index (ACSI) found that during the most recent reporting period, 79% of Google users and 71% of Bing users were satisfied with their experiences. Figure 1-10 shows the ACSI satisfaction scores for Google from 2002 through 2020.

)
Figure 1-10. User satisfaction with Google over time (source: Statista)

While 79% satisfaction is an amazing accomplishment given the complexity of building a search engine, this study still showed that over 20% of users were not satisfied with Google’s search results. These numbers could reflect users’ dissatisfaction with the number of ads that increasingly infiltrate the SERPs.

The important takeaway here is that in all instances, determining searcher intent remains a challenge; and when the searcher’s intent is broad, there is ample opportunity to leverage your content assets with SEO.

As an SEO practitioner, you should be aware that some of the visitors you attract to your site may have arrived for the wrong reasons (i.e., they were really looking for something else), and these visitors are not likely to help you achieve your digital marketing goals. Part of your task in performing SEO is to maintain a high level of relevance in the content placed on the pages you manage, to help minimize this level of waste—while still attempting to maximize your overall presence in the SERPs and gain brand exposure.

How Search Engines Drive Ecommerce

People make use of search engines for a wide variety of purposes, with some of the most popular being to research, locate, and buy products. Digital Commerce 360 estimates that the total value of US ecommerce sales in 2022 reached $1.03 trillion. Statista forecasts that worldwide ecommerce retail sales will reach $7.4 trillion by 2025, as shown in Figure 1-14.

Figure 1-14. Statista online retail forecast to 2025 (source: Statista)

It is important to note that search and offline behavior have a heavy degree of interaction, with search playing a growing role in driving offline sales. Figure 1-15 shows data from a May 2019 study by BrightEdge which found that 27% of the traffic across their client base came from paid search. Note that this dataset is drawn from BrightEdge’s customer mix of over 1,700 global customers, including 57 of the Fortune 100.

Driving traffic to your ecommerce site isn’t just about driving conversions of every visitor. As shown in Figure 1-16, visitors that come to your site from a search engine may be at any stage of the customer journey. This is why ecommerce sites should consider creating content for each and every stage—capturing visitors when they are in the initial stages of discovery and research can significantly increase your chances of making the sale when they are ready to buy.

Figure 1-15. Sources of traffic to BrightEdge customer sites (source: BrightEdge)
Figure 1-16. Search delivers traffic across the customer journey

Types of Search Traffic

By now, you should be convinced that you want your site content to be prominently displayed within SERPs. However, data shows that you may not want to be #1 in the paid search results, because the cost incurred to gain the top position in a PPC campaign can reduce the total net margin of your campaign. As shown in Figure 1-17, London-based Hallam Internet published data in 2019 that suggests that the #3 and #4 ad positions may offer the highest ROI.

Figure 1-17. Position 1 in search ads may not be the most profitable

Of course, many advertisers may seek the #1 position in paid search results, as it offers benefits including branding and maximizing market share. For example, if an advertiser has a really solid backend on their website and is able to make money when they are in the top position, they may well choose to pursue it. Nonetheless, the data from Hallam suggests that, due to the lower ROI, there are many organizations for which being #1 in paid search does not make sense.

Search Traffic by Device Type

The majority of traffic to websites today comes from mobile devices. In 2021, while working at Perficient, Eric Enge (one of this book’s authors) conducted a comprehensive study of mobile versus desktop traffic that showed that over 60% of all traffic in the US and nearly 70% globally comes from mobile devices, with tablets accounting for a further 3% (as shown in Figure 1-18).

Figure 1-18. Mobile versus desktop traffic, US and global views (source: Perficient)

This does not mean that desktop search has become unimportant; indeed, it delivers the most total time on site (also known as “dwell time”) from visitors in aggregate, and nearly the same number of page views as mobile devices. Figure 1-19, also from the Perficient study, shows the aggregated (across all visitors) total time on site for desktop versus mobile users.

Figure 1-19. Mobile versus desktop aggregate time on site, US and global views (source: Perficient)

Google has long been aware of the growing importance of mobile devices, and first announced an algorithm update to focus on these devices in April 2015. This announcement was dubbed “Mobilegeddon” by the industry; many expected the update to cause a dramatic upheaval of the search landscape. In reality, it’s almost never going to be in Google’s interests to completely disrupt the existing search results in a dramatic fashion because, as we discussed earlier in this chapter, user satisfaction with Google has stayed at roughly the 80% level every year since 2002. This is reflected in how Google handled its switch to mobile-first indexing, with the initial announcement in November 2016 stating:

Today, most people are searching on Google using a mobile device. However, our ranking systems still typically look at the desktop version of a page’s content to evaluate its relevance to the user. This can cause issues when the mobile page has less content than the desktop page because our algorithms are not evaluating the actual page that is seen by a mobile searcher.

Nearly everyone in the industry expected this switch to unfold far more quickly than it did. The reason that it didn’t is that Google has to perform extensive amounts of testing on any change it makes to how it indexes and ranks content in order to minimize unintended negative impacts on search results. Due to the scale of search, this is a highly involved and cumbersome process.

It was not until March 2020—over three years later—that Google announced a target date of September 2020 to make to mobile-first indexing universal. Making such a change is incredibly complex, however, and in November 2022 Google had still not 100% switched over. Nonetheless, the great majority of sites are indexed mobile-first. Unless your site has unusually difficult issues, it is likely already being indexed that way. From an SEO perspective, this means that Google crawls the mobile version of your site and analyzes its structure and content to determine the types of queries for which the site is relevant. This means that the majority of your SEO focus needs to be on the mobile version of your site.

The fact that most visitors to your site are going to come from mobile devices is of critical importance. Among other things, it means that website design should start with mobile functionality, design, and layout. Any other approach is likely to result in a mobile site that is not as optimal as it could be. It also means that for most queries, you should be studying the structure and format of the mobile search results from Google. Figure 1-20 shows sample search results for the query digital cameras.

In the first three screens of results on mobile devices, the ranking opportunities are in Google Shopping, People Also Ask boxes, and local search. A core part of your SEO strategy is to develop an understanding of the search landscape at this level of detail, as it can directly impact the search terms you choose to target.

Targeting informational queries is quite different, as shown in Figure 1-21. Here you still see Google Shopping at the top of the results, but the next two screens are filled with access to informational content. As a result, the nature of the ranking opportunities is not the same.

Figure 1-20. Mobile SERPs for the query “digital cameras”
Figure 1-21. Mobile SERPs for the query “history of prague”

More on the Makeup of SERPs

In 2019, Perficient published a comprehensive study by coauthor Eric Enge of how Google’s search features—any content that falls outside the basic list of links—impact click-through rate (CTR), based on data for both desktop and mobile searches, as well as for branded and unbranded searches. Figure 1-22 shows the average CTR (percentage of clicks received) by Google SERP position for both branded and nonbranded queries. As you can see, the disparity is significant.

Figure 1-22. Search results CTR by ranking position

Google search results are rich in many different kinds of search features. Figure 1-23 shows the frequency of different search features within the Google SERPs as of December 2019.

Figure 1-23. Popularity of search features

Each of these features creates different opportunities for placement in the search results and impacts the potential CTR you might experience. The reason that CTR is impacted is that users respond to different visual elements and their eyes get drawn to images and parts of the page that look different.

Nearly two decades ago, research firms Enquiro, EyeTools, and Did-It conducted heat map testing with search engine users that produced fascinating results related to what users see and focus on when engaged in search activity. Figure 1-24 depicts a heat map showing a test performed on Google; the graphic indicates that users spent the most amount of time focusing their eyes in the upper-left area, where shading is the darkest. This has historically been referred to in search marketing as the “Golden Triangle.” Note that while this research was done many years ago, it still teaches us about how users react to search results.

Figure 1-24. Eye-tracking results, 2005

That said, the search landscape has changed dramatically since 2005 and has become increasingly more complex, evolving toward results that are media-rich and mobile-centric. As a result, the appearance of the search results is nowhere near as consistent as it used to be, which causes users to have a much less consistent approach to how their eyes scan the SERPs. As shown in Figure 1-25, more recent research performed by Nielsen Norman Group shows that users follow a pinball-like path through the page.

Figure 1-25. How users view search results today (source: Nielsen Norman Group)

Even though richer content sprinkled throughout search results pages has altered users’ eye-tracking and click patterns, the general dynamic of users viewing and clicking upon listings at the top of search results most frequently, with each subsequent listing receiving less attention and fewer clicks, has been supported by multiple studies over time.

These types of studies illustrate the importance of the layout of the search engine results pages. And, as the eye-tracking research demonstrates, as search results continue to evolve, users’ search and engagement patterns will follow suit. There will be more items on the page for searchers to focus on, more ways for searchers to remember and access the search listings, and more interactive, location-based delivery methods and results layouts—which will continue to change as search environments and platforms continue to evolve.

NOTE

Over the past few years, Google has introduced “continuous scroll” across mobile and desktop search, where listings are no longer limited to 10 or so per page, but instead more listings are loaded as one scrolls downward. It is not clear as yet what impact this change has had upon searcher behavior.

The Role of AI and Machine Learning

By now you’ve got the idea that user behavior is highly complex, and the challenge of meeting users’ varied needs with a search engine is enormous. As the seoClarity data in Figure 1-2 showed, nearly 80% of all search queries consist of 4 words or fewer. These short phrases are all that a search engine gets in order to determine what results to return in the SERPs. In addition, March 2021 data from SEO tool provider Ahrefs shows that nearly 95% of all search queries are searched on 10 times per month or fewer. That does not provide search engines with a lot of prior history in order to model what the user wants.

Google deploys a large array of resources to try to meet these challenges, including a growing number of machine learning (ML) algorithms such as RankBrain, BERT, SpamBrain, and MUM (we’ll talk more about some of these in later chapters). You can expect to see Google continuing to roll out new ML algorithms—used to supplement the human-generated algorithms that it has developed over the course of decades, which are also undergoing continuous improvements—on an ongoing basis. This brings some unique challenges, as the nature of how these algorithms work is opaque even to those who create them. As a result, testing them and validating that they work as expected is incredibly complex. We’ll return to many of these topics later in the book.

Using Generative AI for Content Generation

In late 2022, OpenAI released a generative AI model called ChatGPT. ChatGPT quickly drew major media and industry attention because of its ability to provide detailed natural language responses to relatively complex queries. Soon after, both Microsoft and Google announced the release of new conversational AI services. Bing Chat, which leverages ChatGPT as its underlying technology, was released in February 2023, and Google launched its generative AI solution, called Bard, in March 2023. At around the same time OpenAI updated the GPT-3 platform (the underlying engine that drives ChatGPT) to GPT-4, resulting in significant improvements in the quality of ChatGPT results. ChatGPT can respond to queries such as:

  • Write a 600-word article on the life of Albert Einstein.

  • Create a detailed outline for an article on quantum mechanics.

  • Suggest titles for eight articles on the American Depression.

  • Read the following content and provide five potential title tags (then append the content to the prompt).

  • Create the Schema.org markup required for the following content (then append the content to the prompt).

  • Debug this block of Python code (then append the Python code to the prompt).

  • Create a new Python script to do X (where X is a description of the task).

An initial review of the output for queries like these shows impressive results (we’ll show some examples in Chapter 2, where we discuss generative AI further). We’ll talk more about global concerns about the future of this technology in Chapters 2 and 15, but some of the additional issues specific to ChatGPT’s (and Bing Chat’s and Bard’s) ability to help you create your own content are summarized here:

  • Prone to including overtly incorrect information in its responses

  • Prone to showing bias (towards ethnicity, gender, religion, nationality, and other areas)

  • Frequently omits information that may be considered materially important in a response to the query

  • Doesn’t provide insights

  • Bias toward being neutral (when you may want to take a position)

In March 2023, coauthor Eric Enge did a study designed to see which of the generative AI solutions performed the best. ChatGPT scored the highest for responses that were free of overt inaccuracies, with a score of 100% accuracy across tested queries 81.5% of the time. Figure 1-26 shows more detail on these results.

However, a separate test of 105 different queries exposed various limitations with types of queries that ChatGPT doesn’t handle well, such as:

  • Product pricing

  • Directions

  • Weather (and other real-time topics)

  • News

  • Sports scores

  • Your Money or Your Life (YMYL) topics

This study also showed that Bing Chat and Google Bard have significant problems with accuracy and completeness too, but these platforms don’t share the problems with other types of content (such as product pricing, directions, YMYL topics, etc.).

Generative AI study results
Figure 1-26. Generative AI study results

Despite these concerns, ChatGPT, Bing Chat, and Bard represent a huge step forward in language generation, and their capabilities are impressive. As mentioned at the beginning of this section, generative AI solutions can already be used to provide responses to a huge range of queries, and as these algorithms continue to be improved, their use will only grow. In recognition of this, Google has updated its guidance on using AI for content generation. The blog post “Google Search’s Guidance About AI-Generated Content” includes a section called “How automation can create helpful content” that acknowledges that AI has been used to help generate sports scores, weather forecasts, and transcripts, and provides other guidance on when it’s OK to use AI to generate content.

As these algorithms evolve, AI will increasingly be able to produce better answers and higher-quality content. We therefore encourage you to experiment with AI, but to be careful to not get too far out in front of its capabilities. You still need to have your own subject matter experts owning the responsibility for all of your content (no matter how it’s initially generated) and your own brand voice, positioning, and priorities.

SEO as a Career

The authors of this book have been in the SEO industry for a long time and have founded and sold successful SEO consultancy businesses. In the modern web-first world, though, it has become more common for companies to have one or more full-time SEO employees, or at least employees whose responsibilities include SEO.

SEO does not align perfectly with any traditional business divisions or departments, but since its primary goal is to increase conversions (from visitors into customers, clients, list subscribers, and so on), it is nominally within the realm of marketing. You don’t need to have a strong background in marketing to succeed in SEO; however, you must be able to communicate well with designers, product managers, and marketing managers. They will explain the organization’s goals and how their projects contribute to them, and you will inform them of changes that must be made to enhance the site’s effectiveness.

Similarly, while the World Wide Web is inherently technical in nature, you don’t need to be a software engineer or system administrator in order to be an effective SEO. There are some technological concepts that you must understand at a high level, such as the fact that web pages are files stored on web servers or are dynamically generated by web applications and are rendered in web browsers and (hopefully!) cataloged by search engines. You must be able to communicate effectively with system administrators, web developers, IT managers, and database administrators so that low-level technical issues can be resolved.

Optimizing for search engines also means optimizing for the people who visit your site; your pages must appeal to visitors without sacrificing search engine friendliness. You don’t have to be a web designer or UI engineer, but you do need to be able to help people in those roles to understand the impact that their decisions may have on search traffic, links from other sites, and conversion rates. It is often the case that some piece of web technology is nice from a user perspective, but unintentionally hinders or blocks search engines; conversely, if you design a site only with search engines in mind, then visitors won’t find it very appealing.

Lastly, you will probably (or eventually) need to have some skill in dealing with decision-makers. A successful SEO will have a great deal of organizational influence without needing to have direct control over site design or web hosting. Depending on your role and the company or client you’re working for, you may have to collaborate with several decision-makers, such as the business owner, CEO, VP of marketing, director of engineering, or a product manager or IT manager. Some of these people may view SEO as a one-off enhancement, like a rebranding or a style refresh. Regardless of your role, your first task on any project is to make it clear that SEO is a process, not an event; it requires a long-term commitment to monitoring a site’s search rankings and indexing, and regular reviews of keyword lists and marketing plans.

If your boss or client is not versed in SEO best practices, they may expect unrealistic (immediate) results. You must convince them that traffic grows like a healthy child—gradually most of the time, but occasionally in spurts—and will be the result of an accumulation of incremental improvements to various projects and services associated with the website. Since it’s rare for those to all be under one person’s control, the better you are at integrating with existing business projects, product development processes, and budget planning sessions, the more success you’ll have as a professional SEO. There is more information on this topic in Chapter 4.

Conclusion

Search is an integral part of the fabric of global society. The way people learn, work, share, play, shop, research, socialize, and interact has changed forever, and organizations, causes, brands, charities, individuals—almost all entities—need to view their internet presence as a core priority. Leveraging search engines and search functionality within all platforms is essential to generate exposure and facilitate engagement. This book will cover in detail how search, and therefore SEO, is at the center of the web ecosystem, and thus how it can play a major role in your success within the ever-evolving digital economy.

Get The Art of SEO, 4th Edition now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial