How Tribe AI Built a Model on GCP That Increased Security Questionnaire Auditor Efficiency by 55%

Tribe

A software leader that provides end-to-end compliance and audit management for modern companies sought to leverage ML and NLP applications to automate a highly manual process typically required for establishing compliance practices. 

Challenge

The company had reached a standstill on their AI-driven security product when the accuracy results plateaued around 70%. The internal team needed an NLP specialist to optimize the existing model, but did not have the in-house expertise. 

Tribe was brought in to evaluate the company’s approach, optimize their existing model, and investigate alternative models to increase the speed at which their audit team can deliver to their customers (“audit efficiency”). 

Team

Tribe started by assembling a team with deep NLP experience, including:

  • Alex, a former Salesforce ML engineer with deep expertise in NLP and building production-ready NLP and recommender systems 
  • Haggai, an ML Ops engineer who led the prediction team at Cruise and built tools to automate ML ops at Airbnb
  • Saurabh, an experienced head of product who focused on technical product integrations at Zapier and built the Gsuite Marketplace and Gsuite add-ons product at Google

Project

The team started by evaluating the model design, including:

  • Model approach – the current model was based on STS, but the team investigated whether Q&A would perform better
  • Model choice – evaluating several models against benchmarks like inference time
  • Training data – with a focus on how to reduce training time. 

In addition, the team uncovered several existing issues in model workflow, which were increasing the effort in iterating and maintaining models. The Tribe team developed a two-phase, iterative approach to improving the accuracy of the core model and updating the ML infrastructure that would improve model workflow. 

Solution

In ~2 months, using advanced natural language processing and working with a very limited data set, the Tribe team was able to increase the accuracy of the company’s core model by 15%.  This included both selection of an alternative approach, developing an NLP Q&A model, and training the model. 

The increase in accuracy greatly increased the throughput of the audit team, with team efficiency increasing by 55%.

In addition to the model improvements, Tribe AI also built the foundation of a ML infrastructure that would allow the team to rapidly draw in new data, retrain modes, and deploy at scale.

Related Case Studies

Case Study

Taking a Data-Driven Approach to Women's Fertility with Rita

Case Study

How Fantasmo is using machine learning to make GPS obsolete

Case Study

Kettle uses machine learning to balance risk in a changing climate

Case Study

How Togal AI Built the World's Fastest Estimation Software on AWS

Case Study

Togal.ai powers the construction industry into the age of machine learning

Case Study

How Wingspan built a machine learning roadmap with Tribe AI

Case Study

Insurance Company Uses ML to Optimize Pricing

Case Study

Tribe AI & Venture Labs: Accelerating Startups with Tailored AI Expertise

Case Study

Building a Proprietary Investment Engine Using Public Data for a Top PE Firm

Case Study

How Tribe Helped Reservoir Bring Finance Infrastructure to NFT Trading

Case Study

How Nota Built a Roadmap for AI-enabled Journalism with Help from Tribe

Case Study

GenAI Solutions: How Bright Transformed Workforce Training with Tribe AI

Case Study

How Tribe AI Shaped Truebit’s AI Strategy

Get started with Tribe

Companies

Find the right AI experts for you

Talent

Join the top AI talent network

Close
Tribe