What’s the Story?

Computer vision is an enabling technology that offers numerous benefits for brands and retailers, including ways to reduce friction, improve the customer experience, automate activities within the store and enhance operations.

In this report, part of our RetailTech series, we provide an overview of the technology and discuss its applications in retail, as well as key technology providers.

Why It Matters

Computer vision offers several benefits for retailers and consumers. We believe that the technology can reinvent retail, enabling ubiquitous visual shopping, where a consumer can take a photo of an item on the street, on a digital screen (such as computer, mobile device or television) or in a magazine and then locate and purchase the item online or in a physical store.

The technology is steadily becoming cheaper and more powerful due to advances in computing power and the capabilities of software, fueled by artificial intelligence (AI) including deep learning, which uses software to identify objects. Since computer vision is software-based, potential users of the technology do not have to make heavy investments in computer hardware—apart from cameras, which are commodities. Computer-vision technology promises to enable a new generation of retail business models that are visually based.

Computer Vision: A Deep Dive

Applications in Retail

Computer vision technology offers numerous functions and benefits to retail, including enabling payments, visual search and personalized product recommendations, as well as tracking inventory, enabling unstaffed stores and more. We summarize the applications of computer vision in retail in Figure 1 and explore selected examples in further detail below.

---

Figure 1. Applications of Computer Vision in Retail

[caption id="attachment_125266" align="aligncenter" width="550"] Source: Coresight Research[/caption]  

Visual Search and Personalized Product Recommendations

We believe that the power of visual search offers opportunities for retailers to spark a new level of consumer enthusiasm for shopping. Visual search enables consumers to search for an identical (or highly similar) item on a retailer’s website or app, based on a photo of an item captured in real life or from a fashion magazine, for example. The item can then be purchased in a physical store or online.

We envision several areas for the deployment of visual search technology:

1. On a brand’s website
2. On a retailer’s website
3. In search engines
4. In virtual shopping malls
5. In tablets or kiosks inside a physical store

Once an item is captured visually, software can match the attributes—such as brand, color, pattern or style—to tags that can be combined with consumer preferences to make future recommendations of similar or complementary items. The image below shows a demonstration of visual search from innovator Syte.ai.

[caption id="attachment_125267" align="aligncenter" width="200"] Visual search
Syte.ai[/caption]  

Recent developments in visual search include the following:

• Google released several search-based features, including those designed to help people shop for cars and clothing in October 2020. Its Google Lens image-recognition tool enables users of the Google app or Chrome browser find products in online stores, including offering product recommendations.
• Microsoft recently announced upgrades to its Bing search engine to make it more visually immersive, including the ability to click on an image, find similar-looking items and then be directed to a retailer’s website when clicking on the item.
• Innovator Syte raised $40 million in October 2020 to accelerate its global expansion and advance its virtual discovery engine.

Apparel Sizing

Improper fit of clothing is a leading motivation for consumers to make returns, which are expensive for retailers. Several innovators are using computer vision to tackle this issue. Solutions typically leverage smartphone cameras to take body measurements, which are then correlated with collected data in order to determine the correct size, taking into consideration the brand’s sizing and consumer preferences for a tight or loose fit.

For example, innovator 3DLOOK’s Mobile Tailor solution uses two consumer-generated smartphone photos to generate a digital 3D rendering of a body that incorporates up to 70 measurements.

Recent developments in apparel sizing include the following:

• Snap acquired Fit Analytics in March 2021, which works with 18,000 retailers and brands globally, including Calvin Klein, Hugo Boss, The North Face and Puma, to help shoppers online find the correct sizes for apparel and footwear.
• 3DLOOK announced that it raised $6.5 million in Series A funding in March 2021, which will be used to expand its US leadership team and establish new R&D (research and development) labs in the US and Western Europe.
• Amazon Fashion launched a service called Made for You in December 2020, which analyzes two photos submitted by smartphone in addition to consumer height and weight data to determine the optimal sizing for made-to-measure, customized T-shirts.

Facial Recognition

Although facial-recognition technology has become commonplace—such as in smartphone security processes or residential security systems—it offers promise for new applications in retail, including the following:

• Determining the identity of consumers in stores for the purpose of identifying customers and particularly, VIP customers
• Determining identities to flag known shoplifters and troublemakers
• Eliminating the check-in process in unstaffed stores
• Detecting a consumer’s emotions, which can be used to generate insights about effective customer service
• Confirming identify in order to make payment

Like many new technologies, particularly those using AI, the use of facial-recognition technology generates several moral and ethical questions, particularly around privacy. While identifying consumers’ faces offers numerous opportunities for enhancing customer service, it does involve the loss of privacy and anonymity and can easily be misused.

Despite the power of the technology, privacy issues will likely prevent the use of facial recognition for identity verification in many countries. New York City passed an ordinance in March 2021 that requires businesses that collect, store, or share biometric information from customers to post a sign near all customer entrances that discloses the practice.

Walmart had tested the technology to identify shoplifters but dropped the test due to privacy concerns. The company had also experimented with computer-vision software for detecting customers’ emotions, but this was likely not deployed for the same reason. There is quite a bit of friction in the signing-in process in traditional and unstaffed stores that could be removed with facial recognition and it could possibly be implemented if consumers opt-in to agree to its use.

Recent developments in facial recognition include the following:

• Russian retailer X5 began rolling out contactless biometric payment solutions in 150 locations branded as Perekrestok, with the goal of expanding to 3,000 stores by the end of 2021.
• Romanian PayByFace announced a commercially available, GDPR-compliant, biometric facial authentication payment system in Europe in December 2020.
• Amazon began deploying its Amazon One technology in September 2020, which uses computer-vision technology to identify consumers from their palms.

Inventory Tracking

Computer-vision systems for tracking inventory in real time in physical stores come in both fixed and mobile configurations.

• Fixed cameras. Cameras in physical stores can be positioned in the ceiling, such as Sensormatic demonstrated in its booth at the NRF Big Show in 2020, or positioned in an array underneath the shelf, such as offered by innovator Trax, as depicted below. Arrays of cameras are also used to track consumer purchases in unstaffed stores (discussed further below).

[caption id="attachment_125269" align="aligncenter" width="500"] Shelf-edge camera
Source: Trax Retail[/caption]  

• Mobile cameras (i.e., robots). There are several vendors that have developed autonomous robots that use computer vision to scan inventory on store shelves, count items and determine that items are priced and/or are positioned correctly, following retailers’ planograms. Zebra Technologies offers a shelf-scanning robot platform as part of its SmartSight retail inventory-intelligence platform. Brain Corp. also offers shelf-scanning robots.

Bossa Nova Robotics offers fixed cameras and drone-mounted cameras, which can be employed in combination. The technology company had supplied shelf-scanning robots to Walmart for five years, but the retailer ended the relationship in 2020.

[caption id="attachment_125270" align="aligncenter" width="500"] SmartSight robotic solution
Source: Zebra Technologies[/caption]  

Recent developments in inventory tracking include the following:

• In March 2021, Trax, in partnership with European crowd-sourcing company Roamler, announced a platform that leverages a crowd-based workforce to capture images of retail shelves.
• In September 2020, Everseen announced its selection by Kroger to deploy its Visual AI technology to reduce friction from errors during the self-checkout experience.

Unstaffed Stores

Unstaffed stores use computer vision for two different purposes: (1) to identify which items the shopper has picked up and plans to take out of the store; and (2) to maintain a real-time account of store inventory. Such stores employ “sensor fusion,” a combination of video cameras and weight sensors.

Companies developing cashierless-store technology include AiFi, Grabango, Standard Cognition, Trigo and Zippin.

Amazon has enhanced its technology to support larger stores than the initial convenience-store format, opening its first Amazon Go Grocery in February 2020. The company has offered its “Just Walk Out” technology to other retailers and has signed up its first customer: the Hudson Nonstop store at Dallas Love Field Airport.

[caption id="attachment_125271" align="aligncenter" width="500"] Ceiling in Amazon Go store
Source: Coresight Research[/caption]  

To read related content, see our previous report on the accelerated development of the unstaffed retail business in China due to Covid-19, and our Retail Reimagined report on contact-light retail.

Recent developments in unstaffed stores include the following:

• Dutch Wundermart announced the opening of two unstaffed “hotel stores” in Belgium in March 2021. The company previously announced that all existing stores would be equipped with computer vision technology from AiFi.
• Standard Cognition announced that it raised $150 million in Series C funding in February 2021, with $25 million from SK Networks, a trading and retail arm of South Korea's third-largest conglomerate SK Group. Standard Cognition’s technology is operational in 27 stores in US cities including Los Angeles, San Francisco and New York City, and the company plans to expand to more than 50,000 unstaffed store locations globally in the next five years. The unmanned store solution utilizes AI to provide quick and convenient transaction services.

Collecting In-Store Data

Computer-vision systems can be used to collect a broader variety of data inside a physical store—in addition to items purchased—such as on consumer shopping behavior, which retailers can use to mitigate losses and generate other valuable insights. Computer-vision systems are able to collect several types of data for analysis:

• Consumer dwell time—Identifying areas within the store that were visited by consumers and recording the time spent in these areas, identifying traffic patterns and developing heat maps
• Consumer activity—Characterizing consumer behavior and identifying loiterers or potential shoplifters that are “casing” the store
• Items picked up—Identifying items picked up by shoppers, taken into a changing room and left there, for example, indicating issues with garment fit
• Activity outside the store—Measuring traffic in the store parking lot via cameras, to determine store peak times
• Store occupancy—Measuring in-store occupancy to determine peak times and enforce maximum store-occupancy levels amid pandemic-led safety restrictions

Other Applications in the Store, Mall or Warehouse

Computer vision is a fundamental technology in autonomous robotics, including shelf-scanning robots as mentioned above. Robots can also perform many of the menial, repetitive tasks around a store or mall.

• Store cleaning: Walmart has used floor-scrubbing robots from Brain Corp. to clean floors while stores are closed. Xenex LightStrike robots use UV light to disinfect hotels and offices.
• Hazard identification: Shelf-scanning robots from Badger Technology (which were named “Marty” when deployed in Giant stores) identify hazards such as spills, shrink wrap, produce, coffee beans and other items.
• Mall security: Robots are being deployed in malls, casinos, hospitals and other large buildings to collect data on license plates, provide facial recognition scans and detect the presence of nearby mobile devices. The ROAMEO (Rugged Observation Assistance Mobile Electronic Officer) by Robotic Assistance Devices is an autonomous robot designed for outdoor security applications.
• Enforcing social distancing: Robots can monitor social distancing in the current pandemic-impacted environment, and the Cruzr in South Korea checks school children’s temperatures and reminds them to wear their masks.
• Greeting and fetching: Robots have been used to greet arriving customers in stores and banks in Japan, such as the Pepper robot by SoftBank Robotics. The LoweBot guides customers to items in Lowe’s stores. There are also robots such as those from Fetch Robotics, which autonomously bring items to workers in warehouses.

[caption id="attachment_125272" align="aligncenter" width="200"] Shelf-scanning, hazard-identifying robot
Source: Badger Technologies[/caption]  

The Computer-Vision Market and Key Technology Providers

The computer-vision market in the US (which is roughly valued in the low-single-billion dollars) is dominated by industrial applications. We believe that retail represents less than 5% of this market, since retail applications are in their infancy. Retailers and other users make use of computer-vision applications, which use AI tools and run on AI platforms. These platforms can achieve high performance through the use of specialized AI chips, and computer-vision systems need cameras for obtaining images. The retail computer-vision value chain is illustrated in Figure 2.

---

Figure 2. Computer Vision in Retail: Value Chain [caption id="attachment_125273" align="aligncenter" width="550"] Source: Coresight Research[/caption]  

Large Companies

US companies offering computer-vision software include Alphabet, Ambarella, CEVA, Cognex, Facebook, IBM, Intel, Leidos, National Instruments and Novanta. Global suppliers include Gemalto, NEC and Panasonic.

Major cloud software and service vendors such as Amazon AWS, Google Cloud and Microsoft Azure also offer APIs (application program interfaces) for image recognition.

Users of facial-recognition hardware and software include Apple and Alphabet (through its Android smartphone operating system).

China is making a push toward leadership in AI in general, and facial-recognition technology is more commonplace there. Technology companies possessing advanced computer-vision solutions include Alibaba, JD.com, Megvii and Tencent.

Innovators

We present selected innovators in computer vision in Figure 3.

---

Figure 3. Selected Computer-Vision Innovators [caption id="attachment_125274" align="aligncenter" width="550"] Source: Coresight Research[/caption]  

Computer Vision as Part of the AI Tech Family

Computer vision aims to extract information from digital image and makes use of knowledge from the disciplines of biology, psychology, mathematics, physics and computer science.

Computer vision leverages the following general functions:

• Image classification
• Object detection
• Object tracking
• Object geometry
• Image segmentation
• Image generation

The technology is just one application of AI that offers benefits to retail (see Figure 4).

---

Figure 4. Applications of AI [caption id="attachment_125275" align="aligncenter" width="550"] Source: Coresight Research[/caption]  

Neural networks are one technology for implementing image recognition using deep learning. The networks consist of a network of nodes, in which each seeks to mimic the function of cells in the human brain. Each node performs a mathematical function on its inputs, which is transmitted to its output. The network can be “taught” by observing the output value and adjusting the parameters on the nodes until the network provides the desired output for a given input. The term “deep” implies multiple hidden layers residing between the input layer and output layer. The output value needs to exceed some threshold value to represent an accurate recognition.

---

Figure 5. Deep Neural Network [caption id="attachment_125276" align="aligncenter" width="550"] Source: Coresight Research[/caption]  

What We Think

Retailers need to consider implementing visual search and other applications of computer vision in order to keep the interest of visually oriented consumers in a crowded landscape and also compete with other visual platforms such as livestreaming and shoppable video platforms.

Implications for Brands/Retailers

• Visual search enables the “holy grail” of consumers finding products they have photographed with their smartphones in a magazine, in a store or on the street.
• Visual search offers the opportunity for retailers to make product recommendations—of similar or private-label products.
• Computer-vision technology enables brands and retailers to gain intelligence from analyzing competitive product images to determine pricing, trends and product ideas from competitors’ websites and social media.

Implications for Real Estate Firms

• Visual search enables product discovery and recommendations within entire malls.
• Computer vision is a key technology for inventory tracking, store-traffic analysis and loss prevention.
• Autonomous robots ensure safety (such as social distancing), store or mall security and cleanliness.

Implications for Technology Vendors

• Multiple emerging areas along the computer vision and AI value chain offer opportunities for emerging vendors.
• While the fundamental technologies have been developed, there are a still numerous opportunities to develop consumer apps that leverage computer vision.
• Ever-increasing computing power and the capabilities of AI software continue to make these tools more powerful and create brand-new business opportunities.