Can Address Validation automation issues be addressed?

This is the first in a series of blog posts where we’ll share our thoughts on the challenges Providers are facing on their inter-provider business automation journey. This time we’ll focus on Address Validation. Providers have struggled with it for a long time so you may have already encountered some of the problems we present.  In this blog, we will shed some light on several challenges you should be prepared for when working on Address Validation APIs during your MEF LSO Sonata/Cantata adoption journey. We’ll also discuss a few solutions that might help make this trip a bit smoother.

The first challenge is the fact that MEF’s LSO Sonata APIs require that Address Validation be synchronous. This means in practice that the process must be fully automated where the responses to incoming API requests are immediate.

The second challenge is that the majority of Buyers interfacing with Sellers expect a high rate of successful address validation calls. The biggest players require the rate to be as high as above 90%.

This leads as to the third challenge which is somehow connected with the previous two. Even though many operators are relying on well-structured address types using distinct fields for each part of the address, both buyer and seller need to use the same or very similar address structure so they can effectively query and match addresses. Even subtle differences here might cause difficulties in match. It becomes even more challenging to speak the “same address language” when we go abroad. It’s not only the structures that can vary but the importance of different address elements or the meaning of what a good enough match is can vary from one country to another, or even from one Seller to another.

For example, the postal code is extremely accurate in Singapore where each building has its own postal code.  As a result, you can confirm serviceability for that building based purely on the postal code. That’s the exact opposite in numerous other developed countries (we’ve seen numerous examples from the EU, the US, and Canada), especially where postal codes were changing over time. In some cases, the same building in a Seller’s coverage area might “virtually” have a few postal codes attached to it and as a result, an Address Validation request between a Buyer and Seller might receive false-negatives unless the Buyer asks for the “proper” postal code. As you can imagine, it would be difficult for Buyer to find what “proper” means but in a digital API-driven inter-partner ecosystem, he shouldn’t even have to worry about it.

The list of possible areas where problems may arise includes:

• Different languages, character sets, diacritic symbols, acronyms, or even text-directions… Difficult choices start when asking about place names like: Avenue/Avnue/Ave, Parkways/Pkwys/Pkwy or about Street/Strt/Str/St… Without normalization the risk of false-negatives increases in case not using “proper” acronym, letter or symbol.
• Data pollution as operators’ address repositories are rarely well maintained. Their data is often affected by long history of human errors during input, typos, manual and automated corrections e.g. following street rename/postal code updates, different systems’ migrations and of course other systems unsupervised API-driven interactions. That’s why they naturally struggle even more to have uniform, well-structured and clean address dictionaries they can trust. The lower the quality of the data in the database, the harder it is to find a match.

With the above in mind, you probably get a feel for how complex it might be to automate address validation between providers. What are the options then?

Some service providers use third-party normalization and cleaning service (from ESRI, EGON, Google etc.) to help make their address data uniform. Both Buyers and Sellers also use these services to normalize and clean outgoing and incoming addresses that are passed across an Address Validation API. As a result, when a Seller tries to match an address to an equally clean and structured address in its database there is a higher probability of achieving a match when one exists. Based on some Customer testimonies we’ve found the use of such third-party services to be highly efficient, at least for some developed markets.

Other providers try to build their own standardization service which involves building their own uniform address database …

However, they soon encounter problems when they try to find a match using full-text search only. Full-text search cannot deal with cases where there are typos or incorrect postal codes. In fact, there are a large number of search use cases that need complex rules and algorithms to be implemented. These are pretty complex and costly to both deliver & maintain, and the problem can be even worse when you take country and language differences into account.

Another complication arises for providers who have evolved over time by merging multiple business units and / or through acquisitions. They can end up with multiple disparate address database with different formats and levels of quality. They lack a single, uniform & centralized address database they can refer to and trust. This problem is even worse when a single provider is front-ending Address Validation requests for a number of subsidiaries and/or third-party supplier partners in different countries. One solution to this problem is to load the coverage of each partner provider into a single uniform database and run all Address Validation requests against it.

If a Seller fails to implement Address Validation correctly it can result in false-positive matches which can greatly impact their reputation with customers or false-negatives which means they lose out on potential orders from customers. Both can lead to customers diverting business to other suppliers whose systems are more accurate and reliable.

A number of our customers have been successful in using third-party Address Validation services as part of the solution. One of our customers, a large US Tier 1 Buyer who uses the MEF LSO Sonata Address Validation API to send address validation requests to its supplier partners normalizes and pre-verifies those addresses using ESRI before they are sent. This helps to improve the success rate of address matching. Another Tier 2 customer, a Seller based in Italy uses the EGON address normalization service to normalize addresses it receives from its customers, EGON is the standard used by all providers in Italy.

These third-party services are effective as they enable providers to improve their address normalization but they have some limitations, so they are not the entire solution. For example, they don’t provide alternatives, something which is supported by MEF LSO Sonata Address Validation APIs (where you can provide a list of addresses in the coverage and highlight one as “Best match”). Lack of alternatives support in Address Validation can mean potential false-negatives, so again some opportunities may be missed.

These are just some of the problems we have encountered. We are actively working with our customers and industry bodies such as MEF to identify solutions.

Some takeaways:

• Automation of inter-partner business depends heavily on Buyer and Seller being able to understand the same address where services are to be delivered
• The way addresses are managed varies by operators and countries which makes them often difficult to match
• Address Validation is an essential step within the MEF LSO Sonata inter-provider process
• Address normalization using third-party addressing services is one way to increase the matching rate, especially when done by both sides of the interaction
• While normalization helps to clean up the addresses, there’s still a lot of complexity involved when trying to match requests against internal database(s)

Tune-in for the next blog posts where we’ll describe other challenges and share some insights on how you might tackle them…