## xicor for Confusion Matrices

We have found that for 2 by 2 confusion matrices (a common summary relating the relation between categorical variables) the expected value of the xicor coefficient of correlation specializes into the re-normalized square of the determinant! One can summarize how a 0/1 variable x relates to a 0/1 variable y […]

## Exploring the XI Correlation Coefficient

Nina Zumel Recently, we’ve been reading about a new correlation coefficient, $$\xi$$ (“xi”), which was introduced by Professor Sourav Chatterjee in his paper, “A New Coefficient of Correlation”. The $$\xi$$ coefficient has the following properties: If $$y$$ is a function of $$x$$, then $$\xi$$ goes to 1 asymptotically as $$n$$ […]

## How to Read Sourav Chatterjee’s Basic XICOR Definition

Introduction Professor Sourav Chatterjee recently published a new coefficient of correlation called XICOR (refs: JASA, R package, Arxiv, Hacker News, and a Python package (different author)). The basic formula (in the tie-free case) is: Take X and Y as n-vectors of observations of random variable. Compute the ranks r(i) of […]

## Effect Sizes Say More Than Standard Scores

I’d like to write a bit about measuring effect sizes and Cohen’s d. Introduction For our note let’s settle on a single simple example problem. We have two samples of real numbers a_1, …, a_n and b_1, …, b_n. All the a_i are mutually exchangeable or generated by an independent […]

## Data Science Bite: What is Statistics?

Statistics is the science of relating summaries of observable samples to the unobserved summaries of the populations they are drawn from. I try to explain that with an example in this video. (link)

## When Profitable Betting Systems are not Possible

I felt a bit guilty explaining a Kelly/Thorp style card betting system without discussing why these ideas don’t work on fair coin games. So I have “writeup for engineers” on the martingale theory of such games. This has example code, so one could try to come up with a betting […]

## Kelly Thorp Betting

I demonstrate a Kelly/Thorp betting system for the simple card game of guessing if the next card from a standard deck is red or black. I have a video of the play here. And a derivation of the betting strategy in R is here. A derivation of the proof you […]

## What is a Good Test Set Size?

Introduction Teaching basic data science, machine learning, and statistics is great due to the questions. Students ask brilliant questions, as they see what holes are present in your presentation and scaffolding. The students are not yet conditioned to ask only what you feel is easy to answer or present. They […]

## Bilingual Data Science

I’d like to share a new talk on bilingual data science. It is limited to R and Python, so it is a bit of a “we play all kinds of music, both Country and Western.” It has what I feel is a really neat example how I used Jetbrains Intellij […]