What’s the Story?

SEO is a “table-stakes” part of digital marketing, yet companies can find that that often results in a degree of complacency, with SEO strategies and operational procedures stagnating as digital commerce and marketing move on. For retailers, that can mean that companies are leaving money on the table. The Covid-driven surge in e-commerce and, in some cases, tightening of marketing budgets mean now may be an opportune moment for retailers to take a fresh look at SEO for retail.

In this report, part of our Innovator Intelligence series, we analyze the key industry trends driving the need for SEO reconsideration retail and discuss the importance and benefits of organic search. We profile Botify, an enterprise SEO platform provider that can help retailers uncover SEO opportunities to derive real business value.

This report is sponsored by Botify.

Why It Matters

Web searches are the starting point of millions of transactions per day, and our estimates suggest that includes almost half a billion dollars of US retail sales in 2020. Around 35% of online retail sales are generated from organic search, according to a multicountry study by marketing firm Wolfgang Digital. In turn, we estimate that organic search drove $226 billion in US online discretionary retail sales in 2020.

That figure excludes any offline sales supported by organic search, and so SEO, during the shopping process. In practice, consumers’ discretionary purchases follow a complex, cross-channel pattern, combining discovery, research and transactions offline and online. We estimate that organic search was the starting point for around $262 billion of offline discretionary retail sales in 2020—sales made in-store but researched online.

That makes a total of $488 billion in organic search-impacted retail sales last year (and that discretionary total excludes nondiscretionary retail sales such as food and beverages, drugs and household FMCGs)—yet, for many companies, SEO is left on autopilot for months or years on end.

In addition, many businesses assess their performance in blunt “share of voice” terms—i.e., their company’s visibility. Relatively few quantify with precision the value of SEO to their top line, through more sophisticated attribution modeling.

Three Reasons Why Retailers Must Revisit SEO

SEO has long been recognized as a valuable and essential part of a retailer’s marketing mix, and we discuss the underlying benefits of SEO in a later section. Changes in the total online offering, shifts in consumer tracking and accelerated e-commerce adoption mean that retailers need to revisit and update their SEO strategies—it is no longer about one-time “set-and-forget” strategies based on content teams shoehorning key words into pages.

1. More To Crawl—and Not All Is Crawled

The proliferation of online content and the use of complex web languages such as JavaScript are making search engines’ jobs incrementally more complex. Search engines cannot consistently index everything on the web. Search engines have limited resources for crawl budget (the amount of time and computing power a search engine can afford to use indexing a site). This means that content on sites that are large, difficult to crawl or render, or both, is often ignored and, as a result, will not be found in search results by potential customers.

Website owners with a “set-and-forget” approach may frequently find that search engines do not crawl or do not index the entirety of their sites. According to Botify, only half of an enterprise site’s web pages are being found by search engines, on average. Those missed web pages are not indexed, meaning that they lose the chance to rank in search engine results, benefit from organic traffic and generate revenue. SEO can maximize visibility.

2. Tracking Is in Flux

Tracking-based marketing is in flux with major uncertainties over its prospects: Google’s Chrome browser will not support third-party tracking cookies from 2022, joining Firefox and Safari in preventing third parties from tracking individual users across websites. Chrome will switch to profile-based targeting, although how that compares to individual tracking is yet to be seen. Meanwhile, paid search costs are tending to rise and, by their nature, are unpredictable adding to a volatile context.

Due to its organic nature, SEO may increasingly be viewed as a “safe” marketing investment to tracking-based marketing such as ad retargeting and paid search.

3. Accelerated E-Commerce Adoption

The Covid-19 pandemic prompted a surge in online retailing, and Coresight Research predicts that a meaningful amount of that uplift will remain even as the health crisis subsides (see Figure 1). We estimate that US online retail sales increased by $188.5 billion, or 33.5%, in 2020, and we expect total online sales to rise again (albeit more moderately) in 2021.

SEO is a crucial tool for retailers to fight for their share of those dollars. In 2021, we expect organic search to be the starting point for $232 billion of US online retail sales, up from $226 billion in 2020.

---

Figure 1. US E-Commerce as % of Total Retail Sales [caption id="attachment_128012" align="aligncenter" width="700"] Source: US Census Bureau/Coresight Research[/caption]  

Three Reasons Why SEO Is an Essential Component of Retail Marketing

Given organic search’s scale in driving traffic, SEO is a fundamental part of most companies’ marketing. We estimate that around half of inbound traffic to all websites (excluding direct traffic) is via organic search, putting it ahead of rival referrers such as paid search and social media.

Scale aside, organic search has a number of inherent benefits. We present three of these below.

1. Organic Traffic Is Sustainable and Cost-Effective in the Long Term

Paid traffic is expensive, with ad costs on the biggest platforms such as Facebook and Google becoming increasingly expensive. It also tends to be a variable cost—the more paid traffic a company gets, the more it costs them—and such variable costs have long been a drag on profitability in the e-commerce space, as it inhibits the scale advantages that companies would enjoy with fixed costs.

In addition, as noted above, tracking-based marketing has an uncertain future. SEO is evergreen. Investing in overall site health, page speed and ensuring the most critical content is being crawled, rendered and indexed drive organic traffic, yielding long-lasting results and a higher ROI (return on investment) on a more sustained basis.

2. Organic Search Attracts Engaged Visitors

Organic search takes a full-funnel approach, capturing shoppers from initial discovery to purchase intent. In each of those stages, consumers are active participants, rather than passive recipients of marketing messages. Organic search funnels interested consumers to retailers’ websites, and SEO supports companies seeking to capture those opportunities. The opportunities are not just in sales but in brand marketing, with SEO-driven content strategies creating a halo effect of greater brand credibility and relevance (see later discussion of L’Oréal for more on this).

Companies can most effectively drive relevant traffic when they learn to understand consumers’ search intent—in terms of the language they use to communicate, response to the content and successful triggers to visit a website. This can help businesses deliver enriching content to influence each consumer’s purchase decisions: The better the quality of content, the faster the conversion.

3. Organic Traffic Serves the Cross-Channel Shopper

Search, and so SEO, is not funneling shoppers to websites just for online purchases. More than ever, consumers are embarking on cross-channel journeys that span a multitude of online-offline interactions: Mobile-equipped consumers are searching across their journey for information—such as on product choices from home, on stores and retailers while in transit, and on in-depth product and price comparisons while in a brick-and-mortar store.

Shoppers’ regular search engine, whether on their desktop browser or smartphone browser, will be the default starting point for a very substantial share of this information seeking. Businesses must be mindful that their SEO strategies respond to the end-to-end search needs on these journeys.

Case Studies: Changing Approaches to Digital Advertising

Changes made amid the Covid-19 crisis in 2020 have supported structural shifts in marketing spend at some major brands.

This year, Airbnb has signaled a major shift away from paid performance marketing such as search engine marketing (SEM) and display ads in favor of a brand-driven approach. During the pandemic crisis in 2020, Airbnb discovered that it could slash its paid digital marketing with a disproportionate retention of traffic. “What the pandemic showed is we can take marketing down to zero and still have 95% of the same traffic as the year before. We’re not going to forget that lesson,” CEO and Co-Founder Brian Chesky told CNBC. During the final quarter of 2020, more than 90% of Airbnb’s traffic was direct or unpaid, Chesky told analysts on an earnings call in February 2021; “We think that will continue in the future,” he said.

L’Oréal is another major name that cut its marketing spend during the depths of the pandemic crisis and learnt that it could achieve still-strong traffic by leaning more heavily on SEO. In a fireside chat event with Botify, L’Oréal’s Chief Marketing and Digital Officer Elise Ducret said, “We were forced to cut some of our media spend, and it has become clear that SEO has regained its first place as a performance channel for us and become more important than ever.” Ducret also pointed to the credibility associated with organic search, in contrast to the fast but “artificial” nature of paid marketing: “The other key benefit with organic search is the credibility it provides to our consumers… Paid is more straightforward and direct in the sense that we buy keywords and we appear in searches as a result. With SEO, in comparison, we develop a content strategy, become subject matter leaders and ensure we are capturing the interest of our consumers in a more authentic way.”

Market Snapshot: SEO Software

The SEO software market is set to grow by $740 million at a CAGR of over 15% between 2020 and 2024, according to market research company Technavio. SEO market growth is being primarily driven by an upsurge in Internet penetration worldwide—with increased use of online search engines—as well as increased adoption of mobile devices, better infrastructure and the rollout of technological advancements such as 4G and 5G networks. Changing consumer demographics and demand also impact the market trajectory of SEO software, such as the recent shift to online shopping.

Technavio predicts that North America, which was the largest SEO software market in 2019, will remain the largest market during 2020–2024 and account for around 42% of the market’s overall growth.

Company Overview: Botify

SEO is here to stay, but the modern web and increased online shopping activity has made it more complex. SEO teams will need to adapt at scale, but they continue to struggle with implementation and understanding the full impact of organic search. Despite the fact that around 80,000 searches are conducted every second and typically one-third to one-half of a website’s traffic comes from organic search, SEO has remained significantly underinvested in and underutilized, labeled with adjectives such as “slow” “opaque” and “unmeasurable.”

Botify is an enterprise software company that was founded in 2012. The company built the interface and methodology that connects the largest, most complex sites with all the major search engines, enabling them to drive more traffic and revenue from organic search. Botify’s machine-learning (ML) algorithm recommends the actions that will drive the most significant impact and automates time-consuming SEO processes.

What Problem Does Botify Solve?

Enterprise websites are often large, but due to the proliferation of content on the web, limited crawl budget resources and the use of complex web languages such as JavaScript, search engines need help to discover all of the relevant content. As we noted earlier, search engines discover only half of a site’s web pages on average, according to Botify. That equates to lost organic traffic and lost revenue.

Botify’s methodology starts with the technical SEO foundation of crawling, rendering and indexing to ensure that all critical pages can be found, and then takes into account ranking and conversion. It helps companies to gain visibility into every section of the website and pinpoint growth opportunities. The methodology also helps teams working in silos to understand how tactics implemented in one part of the website affect other parts—ensuring the whole team in the organization is on the same page in terms of the SEO efforts. In this way, Botify’s methodology helps organizations such as e-commerce companies, in which teams predominantly work in silos, to have a unified SEO approach.

Methodology Overview

Botify’s full-funnel methodology, as depicted and discussed below, helps turn organic search into an organization’s top marketing channel.

[caption id="attachment_128013" align="aligncenter" width="500"] Source: Botify[/caption]  

• Crawl: Before any content can rank and be found by searchers, search engines have to find it. Botify helps ensure important pages are crawled.
• Render: The modern web runs on JavaScript, so once search engines find a company’s web pages, they need to render them. Botify helps to find where search engines might be missing this content.
• Index: Once search engines find the company’s web pages and render their content, they can be added to the index. Botify helps to optimize pages for crawling and rendering, maximizing the amount of content in the index.
• Rank: Indexed content can be found by searchers. Botify shows the real ways in which consumers are searching so that companies can create intent-matching content that ranks highly.
• Convert: Finally, after content is crawled, rendered, indexed and ranked, it is positioned to be clicked on by searchers who want to buy what companies offer. Botify not only helps drive this revenue but also ties it back to SEO efforts.

A full-funnel methodology such as Botify’s requires the ability to analyze and cross-index a variety of first-party data inside a single unified data model. This is critical for making decisions related to a company’s website.

Solution Overview and Key Benefits

Companies need data from each and every phase of the search process, from crawling to conversions, to understand what is working and where there is scope for improvement. Botify’s platform offerings help companies achieve this (see Figure 2).

---

Figure 2. Botify Platform Offerings

Botify Analytics	Botify Analytics provides a comprehensive website data map and insights from each stage of the organic search process by unifying all SEO data, uncovering what search engines are missing, and analyzing 1,000+ metrics from every phase of the search, from crawling to conversions. This gives SEO full-funnel insights about the websites they manage. Having a unified SEO suite helps companies to visualize which issues affect traffic and pull in the right people to work around the problems—avoiding delays in approvals for making changes to SEO. It also gives insight into each issue’s impact, giving more visibility to stakeholders. Botify’s analytics tools include the following: SiteCrawler—Personalizes crawl to meet the strategic needs and evaluate websites' structure and content at higher crawling speeds without limiting the crawl budget. E-commerce sites having millions of pages face the challenge of their important site pages not getting crawled. With personalized crawl, e-commerce sites can remove duplicate pages, optimize the remaining site pages, group them, and analyze their crawling behavior, thereby ensuring important site pages get crawled and optimize the crawl budget. LogAnalyzer— Provides total crawl accuracy to see whether the search engine is crawling the digital assets or not, and if yes, how is it crawling. The tool is suitable for site migrations, making substantial site changes, seeing how search engines adjust to changes, and catching any issues before traffic takes a hit. E-commerce sites wanting to change their digital strategy and site content based on their customers’ keyword searches due to seasonality shifts can benefit from this tool. RealKeywords—Understands what, when and where people search based on actual queries, impressions, position and clicks. The solution helps e-commerce sites for geotargeting their content and provide a tailored experience to international visitors by identifying their top keywords and steering the marketing efforts to cater to their geographical preferences. EngagementAnalytics—Ensures that SEO efforts accomplish what they are supposed to and connects conversion and revenue data to SEO actions to demonstrate the ROI
Botify Intelligence	Botify Intelligence helps SEO protect existing traffic, suggests actions that help grow traffic and prioritizes SEO tasks to get the greatest impact. Botify Intelligence tools include the following: AlertPanel—Mitigates risk through alerts and quick fixes to mitigate any traffic or revenue losses. E-commerce sites can identify which issues affect traffic and pull in the right people in the organization as quickly as possible to address those issues. ActionBoard—Helps to show opportunities to improve the site based on ML-powered analysis
Botify Activation	Botify Activation turns the data and insights into actions and maximizes resources by automating critical SEO tasks to help SEO teams maximize their impact. Botify Activation tools include the following: SpeedWorkers—Delivers pages quickly to search engines by mitigating the cost of JavaScript, enabling faster crawling and driving traffic and revenue through organic search FastIndex—Automatically generates sitemaps to make crawling easier and more reliable

Source: Botify/Coresight Research

How Do Clients Benefit from Botify?

Botify’s solutions can help e-commerce retailers unlock value from their website pages by improving core functions around organic search. By optimizing e-commerce web pages and content, Botify’s SEO solution enables e-commerce brands to understand their organic search performance, identify areas of growth and drive greater revenue from the organic search channel.

Below, we explore the areas in which clients can benefit from Botify’s solutions, based on real-life case studies:

• Pulling sitewide actionable insights in a timely manner. Before deploying Botify, one organization used SEO tools on an ad hoc basis to analyze their website. This process would take several days or weeks and require significant computer processing power to complete. With Botify, website analysis was completed in a significantly reduced timeframe (less than one day), and it provided deeper, insightful recommendations for website changes.
• Communicating the importance of SEO to non-SEO stakeholders. SEO teams often face challenges in educating their organization on the value of organic search and SEO. Organizations that have deployed Botify’s solutions have reported that it was much easier to communicate and visualize the value of SEO to senior executives, developers and other non-SEO stakeholders in the organization.
• Improving customer experiences on a website. Better website navigation and reliable internal site linking provide better experiences for visitors.

• Growing organic traffic by double digits following site optimization. Organizations using Botify protect their existing organic traffic while finding incremental ways to increase traffic and revenue. By ensuring that the most important pages were indexed and ranking well in the search engine results pages (SERPs), these businesses could reach a larger audience searching for their products.
• Retaining organic traffic that they would have lost. Search engine algorithm changes and ongoing website updates can erode SERP ranking performance and the resulting organic search traffic. Organizations that used Botify received recommended fixes to their website infrastructure and content that helped preserve 99% of their organic search traffic.

Case Study

The following case study illustrates how Botify helped a Finland-based home-improvement retailer with site migration/optimization while protecting and growing its organic traffic and revenue successfully. The retailer had about 500,000 product URLs and 16 years of website growth at the time of migration. The challenge was to overcome any critical business issues likely to arise during migration and ensure they did not affect organic traffic and revenue.

Botify analyzed the entire site to get a complete picture of every web page and understand what was working and what needed improvement. This analysis helped Botify identify the following:

• 90% of the company’s pages were not search engine compliant, so their important URLs were not visible in search engines.
• The company was wasting crawl budgets on unnecessary URLs as search engines were spending time on duplicate pages, missing the important ones.
• Some of the strategic pages were hidden deep on the site, making it inaccessible to search engines and customers alike.

Botify mapped out and tested about 300+ redirects to preserve organic traffic. The retailer successfully completed the site structure with better access to search engines for its top revenue-generating pages. the company experienced a seamless transfer of organic traffic and growth despite reducing its web pages from 500,000 to 100,000.

Competitive Advantage

Botify’s SEO solution helps reduce the complexity of SEO implementation while  offering more automated and performance-based marketing tools that evaluate each step of the conversion process and optimize the weakest parts. Typical SEO approaches do not do this with organic search, often not considering the full organic search process. For example, Google often ignores 50% of a site’s content. Botify’s platform ingests, cross-indexes and analyzes the full stack of site performance data.

In addition, typical SEO solutions share insights on website performance in terms of how they rank, but the analysis is not prescriptive enough to advise on how to improve organic search performance with actionable insights.

Botify’s Crawls Venn Diagram (see an example in the image below) enables clients to see, at a high level, the ratio of their web pages that are crawled by Google and Botify. The two circles represent pages in the client’s site architecture (crawled by Botify only), pages outside the site architecture (crawled by Google only) and pages crawled by both Google and Botify.

[caption id="attachment_128015" align="aligncenter" width="550"] Source: Botify[/caption]  

In order to protect incoming organic search traffic, which accounts for at least 50% of total website traffic, marketers need to be proactive about optimizing their site. Botify supplies search marketers with a holistic set of tools to aid in each stage of the SEO process.

Botify claims to provide 10% growth in organic traffic, 99% reduction in organic traffic loss, 60% less time spent on analysis, improved site experience for customers and higher ROI.

---

Figure 3. Botify: Competitive Advantage Matrix

Positioning	Opportunities
Botify puts organic search at the core of the digital strategy and unlocks more value out of it by making it the most profitable customer acquisition channel. Botify focuses on the technical SEO foundation of crawling, rendering and indexing stages of organic search, as well as ranking and conversion. Botify prevents search engines from missing a page, getting indexed and ranked, thereby driving organic traffic and generating revenue out of these pages.	Coresight Research’s latest weekly US consumer survey states that consumers are now shifting their spending on most retail categories towards online channels. This has forced many players in the retail sector to switch to online, especially those operating in the offline space.
Growth Drivers	Unique Attributes
The consumer shift towards online requires retailers to invest and reinvent digital experiences and also understand their effectiveness since this has become the core for businesses. Many organizations thus adopt online and digital platforms and add SEO capabilities to their businesses. Some companies are also planning to vertically integrate from other related segments making the SEO software market highly competitive. We expect these growing online businesses to create tremendous opportunities for the SEO software market.	Botify’s full-funnel methodology and unified data model ensure the broadest view of a company’s website. Botify’s ML algorithm recommends the actions that will drive the greatest impact and automates time-consuming SEO processes. Botify organizes e-commerce data by product category, making it easier to view specific segment’s/page type’s ranking keywords, content quality and more. Botify crawls and renders JavaScript at 100 URLs per second to enable quick identification of slow-loading resources and content rendering. Botify provides actionable insights in near-real time. With Botify, website analysis includes prescriptive actions and recommendations for website changes.

Source: Coresight Research

What We Think

Organic search takes a full-funnel approach, connecting brands and retailers to shoppers at every stage of their shopping journey, from discovery to purchase. It ultimately drives hundreds of billions of dollars of retail sales in the US.

However, often, companies are running SEO strategies devised and implemented several years previously, even as the online world has shifted. Companies may be missing out on their share of online and offline revenues due to this lack of responsiveness, together with misplaced confidence that their sites are being indexed more fully than they actually are.

As with all other forms of marketing, SEO strategies should be kept under active review and assessed in terms of ROI.

Recommendations for Retailers/Brands

• A starting point to evaluating a company’s current strategy is understanding the value of organic search, and so SEO, to that company in quantitative terms. Generalities over “share of voice” do not provide the kind of quantification that attribution modeling does.
• Retailers should not “set and forget” their SEO strategy. Changes in tracking capabilities, web-page volumes and consumer behaviors mean that a one-time strategy implementation will have decreasing relevance and impact.
• Retailers should consider who in the company should be leading SEO strategy and ask whether that strategy needs to be about more than content teams funneling keywords into web pages.