Machine Learning Discovery: 4 Weeks to Decide How to Apply ML in Your Business

At Deployr, our mission is to advise, guide, and support companies of all types and sizes on the path to putting machine learning models into production and becoming data-driven. But of course, not every company has the resources to effectively apply these tools, and not every problem should be solved with these technologies.

That’s why it’s crucial to be able to analyze, in a short timeframe, whether applying machine learning is feasible for solving a specific business problem. And we emphasize “in a short timeframe”, because delays in this assessment can have ramifications that impact an organization’s revenue.

As we’ll see, some challenges can be solved with machine learning far more effectively than with traditional tools, but it requires a set of preconditions and dedicated effort to apply it. And to determine whether those conditions are met, we’ve designed what we call the Discovery process: 4 weeks to decide how to apply ML in your business. In this post, we’ll tell you what it’s all about.

Risky Business

Before we start talking about the methodology itself, let’s briefly look at the risks of not doing a quick feasibility check on an ML project. Some of them are:

• High financial cost: Investing a substantial amount of money in acquiring tools and resources for an ML project that later turns out to be unfeasible or ineffective doesn’t just represent lost money; it also eats into the budget available for other key business initiatives.
• Wasted time: Time is money, and if we don’t quickly validate the viability of an ML project, we could waste months of work heading in a direction that yields no positive results, delaying the company’s ability to innovate and adapt rapidly to the market.
• Lost competitive advantage: In today’s business world, technology is key and not moving quickly puts you at a competitive disadvantage. If the market effectively adopts machine learning to streamline processes and make smarter decisions, your company risks falling behind.
• Reputational risk: A failed ML project can have a negative impact on the company’s reputation. If we announce an initiative that then fails to meet expectations, we could lose the trust of customers, investors, and key partners, affecting our image and credibility.
• Team demotivation: Technology teams are the engine of this data-oriented transformation. If they invest time and effort in an ML project that fails due to insufficient validation, they could become discouraged and lose motivation for future initiatives, or even lose key members who decide to join a competitor.
• Investing in the wrong tools and technologies: Acquiring tools and technologies without proper evaluation leads to investing in solutions that don’t align with real business needs. While they may seem promising initially, it’s essential to do thorough research to ensure they’re appropriate and effective for the challenges your company faces.

These are just some examples that can obviously generate even bigger problems down the line or compound each other in a vicious cycle that often proves catastrophic for smaller companies or those with less financial runway. In contrast, quick and effective validation of feasibility gives us the opposite: cost efficiency, faster decision-making, competitive advantage, greater adaptability, focus on key objectives, product quality, and accelerated return on investment.

It’s worth noting that everything mentioned above applies to any software development project, but in data science’s case, there’s an additional layer of complexity since we’re working with statistics and often uncertain outcomes. So, is it possible to apply a feasibility evaluation process to this type of project?

Fortunately, the answer is: yes! And here’s how we do it.

We Walk the Discovery Path with You

The Discovery process has a very concrete objective: validate in four weeks whether or not it’s possible to apply machine learning to solve a specific business problem. To do this, each week is oriented toward understanding a different aspect: the project itself and the company, its resources, and the industry it operates in.

When designing this methodology, we decided to break the Discovery month into the following four parts:

• Week 1: Problem definition and data collection.
• Week 2: Exploratory analysis and data preprocessing.
• Week 3: Preliminary model selection and training.
• Week 4: Final evaluation and results presentation.

Let’s look in more detail at what happens in each of these stages.

Week 1

In this first week, as mentioned, we focus on problem definition and data collection. This means first having meetings with stakeholders to understand how the business works and what problem they want to solve. We typically aim to get familiar with how the industry works, who the users are, and what the pain points are for both users and the business.

And perhaps even more importantly, from this work comes one of the fundamental outputs of this stage: identifying the key metrics they want to improve. Companies don’t operate in the abstract. They have KPIs that, when optimized, result in tangible gains for the business.

Some common examples: reducing wait times for customer service, improving customer satisfaction scores, reducing delinquency rates. Every ML project must be conceived, planned, and developed with the north star of improving those KPIs that translate into financial and competitive advantages.

In addition to these first two business-oriented aspects of week one, we focus on generating a dataset for analysis and exploration, which serves as our starting point for the following three weeks.

Week 2

Week two is focused on exploratory data analysis, where we take the data we accessed in the first week and dive deep into it to try to understand it and find trends, relationships, and insights that give us clues about how it can help with the problem we want to solve.

We also aim to connect that data, which is an abstract representation of reality, with reality itself to see how it relates to the business. Ultimately, what we’re looking for is to tell a story through that data that helps us crystallize a comprehensive view of the landscape in front of us.

To that end, this step produces many visualizations of the data: charts that condense information in an easy-to-understand way and facilitate insight extraction and storytelling, both for technical teams and organizational decision-makers.

Week 3

At this stage, we’ve already understood the problem to solve and which data is relevant, and we’re in a position to begin the modeling itself. This is when we identify the type of problem, and consequently the type of model, we’ll use: classification, regression, or clustering.

The data scientists handling the training are responsible for defining the metric that will evaluate model performance. This doesn’t refer to the business KPIs we discussed earlier, but to mathematical measurements of our model’s precision (accuracy, precision, ROC/AUC…). This point is far from trivial, because choosing the right metric is fundamental to the success or failure of this type of project: using the wrong one can lead us to evaluate a project as viable when it isn’t, or vice versa.

From this week comes what’s known as a POC (proof of concept): an initial model that isn’t necessarily perfectly optimized but serves as a baseline for future comparisons and as raw material for what happens in week four.

Week 4

In the final stage of our discovery process, the team takes the model trained in the previous phase and applies some optimizations for its final evaluation and presentation of results to the business.

We also focus on model explainability: it’s not enough to say “metric X gave value Y”. We want to find coherence and a real-world correlation in it. This can be done through feature importance techniques, which aim to numerically express the weight of each variable for the model. This point is very important because in many cases the client, who is the domain expert, can validate it through their opinion and experience-based vision to understand if our results make sense.

Finally, a presentation is prepared for the client with all our results and learnings from the Discovery phase, along with recommendations for next steps based on the achievements and the project’s viability. It’s a feedback and exchange session that serves as the closing of these four weeks of hard work.

Next Steps

Those were four intense weeks! Ideally, the client walks away with knowledge and information that helps them in their day-to-day work. That said, the process doesn’t end here. After this month, there are many things that can continue to be developed:

• Model deployment: A trained model is just a file on the data scientist’s computer. To truly add value day-to-day, what’s known as deployment must happen: building the software that allows users to use the model easily and quickly.
• A/B testing: Beyond the metrics we used to evaluate the model, there are additional methods to ensure its quality and real-world impact. This is done through techniques like A/B testing, where statistical hypotheses are tested to make sure the model is truly quality and functions as a benchmark for any future development.
• Monitoring and retraining: The metric we use to evaluate the model is neither immutable nor permanent. Changes in data distribution over time can cause performance to degrade, which is why in many cases it’s extremely important to implement monitoring mechanisms and, eventually, model retraining with more recent data.
• Continuous model improvement: This can involve choosing better or newer algorithms as they emerge, more exhaustive feature selection work, optimization processes, incorporating complementary metrics, and a wide variety of other options, until finding the configuration that best fits each business’s needs.

These are just some examples of possible next steps. Every client is unique and every problem to solve requires analysis, dedication, and commitment, so there are no universally valid answers. But at the end of the Discovery month, each client should at least have a clear picture, better understanding of possible solutions, and a concrete roadmap of improvements and next steps.

Want to learn more? Check out our Business Webinars playlist, geared toward companies looking to get the most value from their data.