When are machine learning models not enough?
AI companies like CrowdANALYTIX give their customers the best AI models for their needs, and that means offering accuracy levels of at least 94-96%. In AI development, the majority of models fail—at CrowdANALYTIX, close to 90% of the models our scientists create never get used by a client because they are not accurate or precise enough. This is one reason why CrowdANALYTIX relies on crowdsourcing to create models: if dozens of data scientists work on the same problem, they have a much better chance of eventually producing an accurate, usable model in good time, whereas a single scientist would take longer and would risk bringing in biases.
So if an ML model is not able to achieve the high level precision required for production models, CrowdANALYTIX solves this problem in one of two ways. First: we leverage our crowdsourced network to build multiple models using different approaches, and ensemble their outcomes. This method has helped us successfully achieve a 2-4% increase in accuracy in production models, and works for the majority of use cases.
However, there are still some instances, known as “edge cases,” situations that are not accounted for in the original machine learning model, that are not solved by this method. Not all of these instances of inaccuracy require “going back to the drawing board” and creating more models. Some “weak models” must be discarded, but in other cases a model is simply not independently capable of achieving the high precision or accuracy expected and required by most production-grade models for client use at a company like CrowdANALYTIX.
Why might this be? Most production models that we produce are for use cases like Product Classification, Product Attribute Extraction, or Information Extraction Automation: retail applications that help retailers and distributors maintain accurate, up-to-date catalogs for themselves and for customers shopping online. These models extract and classify information from a broad range of sources: PDFs, complex images, unstructured text, even audio and video files. This complexity and range of possibility means that it can be difficult to achieve the expected 92-95% level of accuracy in real-world application by using machine learning models alone. These models simply cannot be shown enough examples or programmed with enough data to reach the desired precision.
This is where a combination of machine learning and Regex models can raise accuracy and precision.
How ensemble models can solve the accuracy problem
Machine learning models sometimes need to be complemented with what we call Regex models, sequences that define search patterns. The combination of machine learning and Regex models can raise accuracy and precision by covering “edge cases.” These ensemble approaches produce more robust, more generalized models that are improvements over single models, because they combine the output of multiple single models. In a way, an ensemble model raises the potential for success just as a crowdsourced approach to model creation does: by widening the field.
Let’s look at one clear example of an edge case. CrowdANALYTIX often deals with product specification PDF documents, which contain information needing to be extracted and added to product ecommerce catalogs. During extraction, it’s likely that the Optical Character Recognition program might miss some text, misidentify a special character, misspell a word, or eliminate a word altogether, leading to an incorrect attribute being added to the catalog. This is an edge case, and it can be handled by Regex in the pipeline without re-training the entire model for an edge case, or by Regex scripts that are run before final predictions are made. Regex helps validate output and ensure that results are as accurate as possible.
Providing predictions with high-level precision is key for the common retail-specific use cases mentioned above, but also in general for any machine learning application that will be used in the real world with the goal of minimal human intervention and minimal maintenance. The combination of ensemble models with Regex helps achieve and maintain high precision and accuracy without the need for expensive and time-consuming adjustment.
