Your web browser is out of date. Update your browser for more security, speed and the best experience on this site.

Update your browser
CapTech Home Page

News June 22, 2017

Accuracy of Six Leading Image Recognition Technologies Assessed by New CapTech Study


Study revealed performance variability in services provided by Google Cloud Vision, Microsoft Azure Computer Vision, Amazon Rekognition, Clarifai, CloudSight, and IBM Watson

Richmond, VA – June 22, 2017 – Businesses and government agencies are increasingly eager to adopt Artificial Intelligence (AI) solutions, including image recognition technology, to incorporate a variety of game-changing opportunities into their ways of operating. Spurred by growing client needs, CapTech, a national IT management consulting firm, recently conducted an independent study to assess the accuracy of six leading image recognition services and today released the full study and infographic detailing their strengths and shortcomings.

A recent Forrester Research report, "Artificial Intelligence: A CIO's Guide to AI's Promises and Perils," discussed the transformative nature of AI and how its deployment will help companies create significant value. The Forrester research shows that more than 55% of enterprises are investing in AI and image recognition in combination with machine learning as a leading area of focus.

According to CapTech, image recognition services promise to drive new capabilities and efficiencies while transforming customer service across diverse industries. The marketers of these services pitch them for a wide range of imaginative uses, such as e-commerce, security screenings, visual search, identity logins, and preventing adult or violent content from making it onto social media.

"Providers of image recognition software services are offering rising sophistication of functionality. However, current services have limitations that users in business and government should be aware of," said Jack Cox, a software engineer, systems architect, and Fellow at CapTech. "Our study evolved from the need to help our clients develop strategies to leverage image recognition services and integrate the technology with new or existing systems to get the greatest value from these services and achieve a competitive advantage.

"We found that no one service has a clear lead across all tested functions. With no one-size-fits-all solution available on the market today, we recommend that organizations looking to adopt image recognition be prepared to use multiple vendors in order to accomplish their strategic goals and objectives. In addition, because of the rapid pace of change in this industry, we recommend that integration of existing systems with image recognition services be architected to provide maximum flexibility so that organizations can switch vendors as needed and adapt to the rapidly changing image recognition landscape," said Mr. Cox.

CapTech presented each service with the same set of approximately 4,800 images, distorting many to recreate real-world conditions such as blurring, overexposing or underexposing, positioning the images at odd angles, and otherwise recreating real-world conditions. CapTech evaluated the services in nine distinct areas of function including facial detection, facial recognition, mood analysis, text recognition, logo recognition, branded product identification, adult content detection, item classification, and item recognition. Based on the results of the accuracy tests, CapTech concluded that it is unlikely that any one service by itself will meet all the image recognition needs of a business or government agency. Each service excelled in some functions but came up short in others. Based on the study results, CapTech recommends that organizations planning to use image recognition services consider the following tips:

    • Plan on using more than one service. Design your image recognition system to allow applications to use the best service for the specific task needed.
    • Plan on architecting the systems so that it is easy to change vendors. The orchestration layer should be designed to enable this.
    • Consider training your own image recognition engine to support your unique business case. An engine trained on a specific type of image will significantly outperform general-purpose engines.
    • Be prepared to handle fuzzy and noisy responses (i.e., incorrect answers). Your systems will need to apply some intelligence of their own to correctly interpret and respond to answers that include probabilities of correctness.
    • Develop a use model to estimate costs and determine which vendor or vendors will provide the lowest cost for the functions needed based on your unique business case.

    The full study, Image Recognition Services: Searching for Value Amid Hype, is available on CapTech's website. The study's findings are also available in an infographic Help or Hype? An Unbiased Image Recognition Services Vendor Assessment.

    Overall Accuracy Assessment

    We found that no one service has a clear lead across all functions tested. With no one-size-fits-all solution available on the market today, we recommend that organizations looking to adopt image recognition be prepared to use multiple vendors to accomplish their strategic goals.

    Overall Accuracy Assessment


    Contributing Researchers:

    Jack Cox, Fellow
    Jack Cox is a software developer, systems architect, and a Fellow at CapTech Ventures, Inc. where he is responsible for the firm's mobile and device software practice. Recently, he has been working with Fortune 500 companies to develop mobile and IoT apps, and define strategies for adopting cutting-edge technologies. Jack is a frequent speaker at professional conferences where he opines on all things technological.

    Chris Heinz, Senior Consultant
    Chris Heinz is a Software Engineer in CapTech's Technology Solutions practice area. He has developed and architected embedded systems, Java-based web service applications, and Android mobile apps. Chris is passionate about delivering the best possible software solution utilizing proven patterns and techniques.

    Kevin Vaughan, Manager
    Kevin Vaughan has developed software for 15 years and has focused on mobile development and cloud platform services for the past 7 years. He has delivered solutions across many industries including gaming, education, e-commerce, and supply chain management. His contributions to open source communities and Q&A sites such as StackOverflow have reached hundreds of thousands of other developers.