Recently, Tribe co-founder Jaclyn Rice Nelson got together with some of our favorite ML and data leaders at the Scale TransformX conference to dig into the question: why do businesses fail at machine learning? And, more importantly, to define how companies can build higher-performing teams and operationalize ML projects to drive success.
In this panel, you'll hear from:
- Cassie Kozyrkov. Chief Decision Scientist @ Google Cloud
- Drew Conway. Head of Strategic Data Science @ Two Sigma
- Deepna Devkar. Vice President, Data Science & Engineering @ CNN
- Jaclyn Rice Nelson. Co-founder @ Tribe AI
Read on for a recap of the main ideas from the conversation or watch the recording on demand.
10 way to succeed at ML according to the data superstars
1. Be clear on what kind of ML you need from the beginning.
“Don’t confuse the research side of ML with the applied side of ML. You’ll hire the wrong experts and solve the wrong problem.”
Cassie Kozyrkov
Companies need to ask themselves: are we building newer and better algorithms? Or are we innovating in what we use these algorithms for? Are there algorithms out there that already meet our needs? Don’t build something bespoke when the tools you need already exist.
For more on this, check out Cassie’s post on a tale of two machine learnings.
2. Don’t jump to the technical solution.
“You have to define the problem before you go to the tools and the solution. Then let the experts you hire define what to build out in order to solve that problem.”
Deepna Devkar
Within research and applied ML, you have to define the business problem first. Too many companies go immediately to the technical solutions and say, “We’re looking to build a recommendation engine” or “We're looking to build a fraud detection system.” These are solutions, not problems. And so, they go out and hire someone who’s an expert at building those technical capabilities without ever realizing that was never the best solution to the business problem they’re trying to solve.
3. Don’t assume ML is the solution.
“Data and AI can be fetishized. It’s really painful to watch people misuse it. At best you’re wasting a lot of time, at worst you’re creating new problems.”
Cassie Kozyrkov
Don’t just assume ML is the solution. This is why the scoping phase of any project is vital. You need to make sure you have a technical team in place that can deeply understand your problem and identify the right tools to solve it in the right way. And, yes, sometimes that’s not ML at all. Simple analytics or exploratory data analysis often serve the business need, but at the very least, they are a prerequisite to identifying the appropriate ML models.
4. Design with the end user in mind.
“If you don’t figure out the most minimally invasive way to present information to the end user in a way that integrates seamlessly into the process, the cost of change is almost always too high.”
Drew Conway
It’s not enough to understand the business problem. It’s not even enough to solve it with ML if you don’t take the actual user behavior and existing processes into account. The question business leaders need to be asking themselves is: how do you actually get people to implement the change in behavior that the data has identified as a need? You can’t just parachute into an organization with new technology and expect everyone to adopt it.
5. Always have someone in the room who understands what’s worth doing.
“Data scientists don’t always have the skill of understanding what’s worth doing and what success looks like. You need someone in the room who does.”
Cassie Kozyrkov
If you hire a data scientist, you have no guarantee this person is going to understand the business or the politics of decision making in your company. Or how the market or prioritization works. You have to find the person who has that skill and let them kick off the projects. Do as much analytics as you want. But everything is analytics until proven otherwise. If you want to do ML or AI you need someone with a deep understanding of what’s worth doing.
6. Train your data scientists to build business acumen.
“Make them understand the ways that they’ll be judged as a data scientist. That it’s about how they were able to solve the problem for that business and how effective that business was at adopting their solution. Not by the depth of their neural network or the complexity of the tools they used.”
Drew Conway
“When you’re building data products you need a holistic view into the business," Deepna agrees. "That means someone who can understand the business problem, the user requirements, and define success metrics. Then they need to actually do the technical work and circle back on the delivery portion of it to stay with users until they’ve adopted the tool.”
So how do we train the next generation of data scientists to be the kind of hybrid talents that are crucial to making ML work for businesses? Or, as Jaclyn asked during the panel, “How do we get more Deepna, Cassie, and Drews?”
According to our panel, the key is about asking data scientists the right questions at every step in their journey – pushing them to think outside the technical toolbox. You have to seek out data scientists who want to work this way – because not everyone does. Then give them agency. “Let’s train a new breed of thinker,” Cassie said, “the decision-maker who has the skills to make data science teams successful.”
For more strategies around this topic, hear from R. Martin Chavez, former CIO and CFO of Goldman Sachs, on how he paired AI experts with domain subject-matter experts for maximum effect.
7. When it comes to building teams, skills are more important than job titles.
“There’s a lot of pressure for people in these nebulously defined job roles to claim that they can do it all while quietly dying of imposter syndrome.”
Cassie Kozyrkov
Data scientist can mean a hundred different things at a hundred different companies. And there are too many job descriptions out there that are so broad they might as well be for the “everything in data scientist.” We can’t expect any single human to be and do it all. (If you’re wondering what flavor of data professional you need, check out Cassie’s essay on the topic.)
We need to better define what skills are needed to get the job done. And, conversely, what skills the people on your team actually have, versus what their job title is.
This changes your approach to hiring by defining teams holistically based on skills rather than a checklist of prior experience. It frees you to have a wider requirement for what it means to be creative with data when hiring and also to know what specialists you need to bring in for distinct projects from outside your organization. “We often work with clients that have their own in-house ML team, but might need a specialist to unblock a problem or do the technical scoping for a new project,” Jaclyn said. “Businesses need to think of new ways of hiring that give them the skills they need when they need them.”
8. Invest in counterparts to your technical talent.
“We found, early on at Tribe, that staffing a PM on every project made a huge difference. It allowed our engineers to focus on what they do best while keeping the project oriented around strategic business goals.”
Jaclyn Rice Nelson
As Deepna says: “Product managers are the unsung heroes of ML projects.” PMs make technology packageable and user friendly for a business, which is ultimately what makes a project successful. By pairing technical talent with a product manager, you’re making sure everyone on the team is focused on the areas where they can add the most value.
This is doubly true for machine learning. ML is experimental in nature, so you have to be comfortable with a certain amount of uncertainty going into any project. And sometimes the outcome is measured in the quality of learning and the insights you get into your business rather than a finished product.
“We’ve found that it’s vital to have someone who can put everything into a framework that a non-technical audience can use to make decisions and understand opportunities,” says Jaclyn. “At Tribe, that’s the role of an ML product manager, but other businesses may have a different way to build that bridge.”
9. Focus on your data early.
“Building the right team means: defining the right problem, focusing on the data early, and investing the time to find the right skills.”
Deepna Devkar
Deepna shared earlier the importance of defining the right business problem. In addition to that, few things are as defining in terms of what makes a good problem for ML than the quality and availability of data. This is also a stage where organizations should explore partners like Scale AI, which can help companies deliver value from AI investments faster with better data.
“Starting projects with a deep dive into that data to figure out whether it’s even possible to get the answers you want is going to set you up for success,” Deepna says. “I also cannot emphasize enough the strength of diverse teams – diverse people with diverse backgrounds and training. Oftentimes the job descriptions you see across companies are too generalized or too specific. Investing the time upfront in hiring for the right skills pays huge dividends later on.”
10. Get excited about the boring stuff.
“The reality is the things that drive impact are often boring. Their novelty is in how existing off-the-shelf tools are stitched together, not in the creation of new tools. Chasing the sci-fi brand of novelty for its own sake can sound cool but it’s best to focus on value.”
Cassie Kozyrkov
There was this feeling early on that in order to get people excited about ML we had to make it futuristic, so they’d care. And then there were the people who claimed AI was a fad. But the reality, we’ve all seen, is that the things that move the needle for a business don’t usually sound like the plot of a sci-fi movie. They’re things like data driven pricing, reducing compute costs, or named entity resolution so you can get better answers from your data. They seem boring and obvious in hindsight, but the implications for a business can be enormous.
Want more? You can read the full transcript from the panel.
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
