Calculating Product and Customer Sophistication on a Large Transactional Dataset

The market basket transactions observed at microscale (each individual product bought by each individual customer at each store visit) over a large population for a long time, offer a detailed picture of customers' shopping activity. Given the high cardinality of such a detailed dataset, data mining techniques have been developed to let the hidden knowledge emerge from it. In this technical report, we propose to use the system of all customer-product connections as a whole. We create a framework able to exploit the characteristics of the customer-product matrix and we test it on a unique transaction database, recording the micro-purchases of a million customers observed for several years at the stores of the top national supermarket retailer. We propose it as a novel analytic paradigm for market basket analysis, a paradigm that is challenging both conceptually, given the high complexity of the structures we build, and computationally, given the scale of the data it needs to analyze