Entity Resolution with Open Source Zingg

Real world data contains multiple records belonging to the same entity. These records can be in single or multiple systems and they have variations across fields which makes it hard to combine them together, especially with growing data volumes. This hurts analytics - establishing lifetime value, loyalty programs or marketing channels is impossible when the base customer data is not linked. No AI algorithm for segmentation can produce the right results when there are multiple copies of the same customer lurking in the data.
In this talk, we present Zingg(github.com/zin..., an open-source framework for entity resolution based on Spark and Machine Learning. Zingg resolves customers, organizations, suppliers, and other entities through an active learning framework. I will cover the motivation behind Zingg, the design of its core algorithms, and dive into using Zingg in different scenarios.