If you use ChatGPT-4 (the paid version, at roughly US$20/month) did you know you can upload a dataset and ask the AI to do Exploratory Data Analysis and draw some graphs for you? Before we start, we need to make sure we activate the ADA (Advanced Data Analysis) feature in…

Mortality tables serve as a foundation for computing vital metrics like life expectancy and the median age of death. Proficiency in utilising mortality tables holds significant value across diverse business sectors. Data scientists working in financial planning, risk assessment, and insurance-related endeavors can leverage this knowledge to determine life insurance…

A step-by-step approach to finding valuable insights — Introduction So you’ve been tasked to set up a marketing A/B test and don’t have a lot of time to figure things out. Here is a quick start guide on how to do it when the main metric to improve is a proportion: click-through rate, conversion rate, open rate, reply rate… 1. Define your null hypothesis and choose your significance level …

Often, we work with datasets containing variables representing rates, which we use to make comparisons between groups or as factors in our models. For those comparisons to be meaningful or more accurate, we need to correct for possible confounding effects in those rates, and use adjusted rates instead. What is…

An alternative approach used among others by Moody’s to rate companies — Bankruptcy prediction has been a very active field of research for many years. Important papers include Edward Altman’s 1968 Financial Ratios, Discriminant Analysis and the Prediction of Corporate Bankruptcy which gave birth to his famous Z-score, still used today, and James Ohlson’s 1980 Financial Ratios and the Probabilistic Prediction of…

RSA was named after Rivest, Shamir and Adleman, from the Massachusetts Institute of Technology (MIT). It is an exponential cryptosystem…

When we are dealing with high dimensional datasets, we can run into issues with clustering methods. Feature selection is a well-known technique for supervised learning but a lot less for unsupervised learning (like clustering) methods. Here we’ll develop a relatively simple greedy algorithm to perform variable selection on the Europe…

Previously, we have seen a very simple implementation of K-means and a method to choose the number of clusters. In some instances, we also want to avoid having empty clusters, or clusters of very different sizes. Here comes constrained optimisation to the rescue. The algorithm is based on a paper…

There are two popular methods: The elbow method The silhouette method We’ll focus on the silhouette method in this article. The silhouette method proceeds as follows: For every single data point i in the dataset we calculate: a(i) = the average distance from that point to…