Analysis of relationship between entities is at the heart of data mining problems. There are many metrics used for association mining like support, confidence, lift, mutual information etc. However many of these measures provide conflicting results about the interestingness of the association. Therefore it becomes very important to understand what works for an application.
Data mining is the process of discovering previously unknown interesting relationships which can be used to increase customer engagement, boost the sales etc. A fundamental step in pattern mining of transactional datasets is the extraction of frequent and interesting itemsets - a set of entities connected by the frequently occurring relationships between them. For instance, identifying from housing purchase data the correlations to age and income groups they belong to can lead to explainable relationships between the different data points which in turn leads to knowledge discovery. This kind of analysis is often confounded by the presence of spurious correlations and data sparsity especially in e-commerce where much of the traffic is often directed to a small percentage of the catalog.
One very common and interesting example of interesting rule mining would be the discovery of strong link between diapers and beers from transactional data of walmart. This association was definitely not intuitive because you would expect customers to buy other baby products along with diapers. It was then suspected that these purchases were made by fathers baby sitting on weekends. Even Though this was found to be a strong correlation we must not forget that correlation doesn’t imply causation. A strong correlation is an interesting insight from user behaviour which can be fed back to increase customer interaction.
The goal of this talk is to set the stage for mining of associations followed by understanding problems of mining interesting associations in retail domain and looking at evolution of interestingness measure . We will pay special attention to low-traffic situations so that we can mine for interesting patterns without requiring large amount of data to support it. One way in which we achieve this is to build higher-level abstractions for entities such as the brand of the product rather than the product itself. In addition to this domain knowledge and transitive connections can be used to help with the cold-start problem.
Brief outline of the talk -
Interestingness Associations
Motivation - Market basket analysis
Problems of association mining in retail domain
Association rule generation
Evaluation of rules
Real life example
Transitive/Indirect rule mining
Key takeaways