gmGiuseppe Muschetta

OverFit

Fit the rule, not the noise.

An interactive simulator of binary or multiclass classification by a decision tree. Cut the board to train the tree on the points you see (blue, orange and green are different classes), then test it on points it has never seen, and answer the question every machine learning model faces: does it generalize, or does it overfit or underfit?

Level 1 of 6 · Pattern and noise

Does it generalize?

Click a grid line inside a region to cut; V or H keeps the cut vertical or horizontal. Hover over a point to see its path through the tree. Drag along the grid to cut a region. Tap a point to see its path.

Your decision tree, growing live with every cut

Experiments

Nine experiments on one question: does the tree generalize? Each one looks at it from a different side: noise, a small real group, a tree too simple for its border, little data, heavy noise, three classes, a greedy library to beat, pruning, and noise with three colors. A level is solved when your tree is wrong only on the flipped training points, as the rule the points came from is, and does as well as the rule on new data. The game can judge this because it knows the rule; a real project only has the score. ★★★ on the first test, ★★ on the second, ★ later, or after Show the rule or the slider was used.

    How a decision tree works

    The board and the diagram are the same model. Here is what each part of the simulation means in machine learning. Each card has a small board to try.

    One question at a time

    A decision tree classifies a point with yes/no questions, each about one feature: is x ≤ 4.5? Then is y ≤ 6.5? The answers lead from the root to a leaf, and the leaf gives the class. On the board, every cut is a question and every region is a leaf.

    A leaf votes

    A leaf predicts the most common class of its training points; a training point of another class in it is a mistake, the small × on the board. Here a tie keeps the color of the region it was cut from, a rule of this game; libraries break ties their own way.

    Generalization

    A model is trained on some data and judged on data it has never seen. It generalizes well when it does well on that new data too, about as well as on its training data: it learned the pattern, not the points. A tree that memorizes the points scores high on training and lower on new data (overfitting); a tree too simple scores low on both (underfitting). Here the training points are the ones you cut around; the new data is one point in every empty cell, from the same rule with the same share of noise, hidden until you test.

    Overfitting

    Keep cutting and a tree can make every region hold a single class, the Full tree of this page: 100% on the data it has seen. It has memorized the noise along with the pattern, and the small regions cut around flipped points are wrong on most new points in them. A large gap between training and new data is the sign of overfitting: the tree generalizes worse than a simpler one would.

    Underfitting

    Stop too early and a tree is too simple for the pattern: some of its regions mix the two classes, and they are wrong on the points it has seen and on new ones alike. Both scores stay low. A small gap between them does not mean the tree generalizes well: the tree has learned too little (high bias). Each cut the pattern needs raises both scores. This board has no noise, so here more cuts never hurt.

    Greedy growth

    CART, the algorithm behind scikit-learn's trees, grows top-down: in each region it takes the cut that lowers the Gini impurity the most, averaged over the two sides by size (Gini: the chance that two points drawn at random, with replacement, have different classes). The slider under the curve grows a tree this way, best cut first.

    Stopping in time

    By default scikit-learn grows a tree until every leaf is pure: the Full tree of this page, 100% on training and worse on new data. To keep it from memorizing, you stop it early (a maximum depth, a minimum number of points per leaf, a maximum number of leaves) or grow it in full and prune it (cost-complexity pruning, ccp_alpha). These settings are chosen by cross-validation or on a validation set; the test set is opened once, at the end.

    Reference: L. Breiman, J. Friedman, R. Olshen, C. Stone, Classification and Regression Trees, 1984 (CART and cost-complexity pruning).

    How it's built

    New data you can trust

    The new data of a level is fixed by a seed: one point per empty cell, with the board's own share of noise. The same tree always gets the same score, so every result can be reproduced and compared.

    Every experiment, tested

    The test suite checks every level and twenty boards made from seeds: the rule explains every point but the flipped ones, accuracy on new data peaks at the rule's own cuts and only falls after, and the full tree clearly generalizes worse. A board that fails a check is not used.

    For every player

    Mouse, touch and keyboard; shapes as well as colors; light and dark themes; and less motion for those who ask their system for it.

    Made from scratch

    No frameworks, no libraries: plain JavaScript, the board on canvas, the tree and the curve in SVG, sounds made on the spot with the Web Audio API. The colors and fonts are the site's own.

    The trees of this page are binary, one yes/no question per cut, as in scikit-learn. For my thesis I built a different kind, with native categorical attributes and multiway splits: Native Categorical Multiway-Split Decision Trees →