Mrthfx

I get the point of a validation and training set, but the importance of a test set doesn't click for me.

Let's say you train a model and you try your best to avoid overfitting by testing your model on the validation set.

After you've decided you have a model your proud of, you do a final sanity check on the test set, let's say the performance is trash. Are you really going to start all over? What decision making does it inform? In my workplace, the way timelines are structured, there's no time to start over.

The point of a test set is to give you a final, unbiased performance measure of your entire model building process. This includes all modelling decisions in your pipeline, so any preprocessing, algorithm selection, feature engineering, feature selection, hyper parameter tuning and how you trained your model in general (5 fold? Bootstrapping? etc.). All of these decisions can lead to overfitting; for instance, selecting a set of hyperparameters that are coincidentally optimal for a particular validation set but not for the general population. If we have no test set you would not be able to identify this and would potentially be reporting highly optimistic scores.

Also, because the above modelling pipeline can get very complex, the possibility of leaking data and overfitting becomes very high. If you tune to your validation set, how will you know if your entire modelling process is not leaking data (and therefore overfitting?)

You bring up a good point; of course if we see that the test set score is poor then we will probably go back and tweak again. Thus, this just demotes the test set into a validation one if you use it too many times as you now run into the possibility of overfitting the test set (see almost every Kaggle competition). However, through repeated test set evaluation (train the model, then test it, then repeat with a different partioning) you will at least get a gauge on how variable your model is to help mitigate this problem. The amount of times you repeat will depend on how much the test set scores vary and how much uncertainty you are willing to accept (also time constraints).

In my opinion, in the business setting you should always make time to properly test your model. The dangers of overfitting are way too high and even worse; you would not even know it. If the test set scores end up being "trash" then at least you know the model is trash and you don't use it and/or you change your approach. This is way better than thinking the model is fantastic based off non rigorous validation and then having the model fail in production. The scientific method is there for a reason right?

I like your question, it is somewhat philosophical in nature.

We know that a test set should not affect the model, otherwise it acts as a validation set. Therefore, even if there is enough time, if we act on a bad test result and change the model, the test set becomes a validation set, although, it is not as involved as a validation set that is used for early stopping or parameter tuning.

In other words, a test set must be useless just the way you have described it! The moment it is useful, it becomes a validation set. Although, to be more precise, a test set is not THAT useless because it probably lowers your (and your boss's) expectation about the later performance of the model in production, so lower risk of heart failure there.

As an example, in a Kaggle competition, the final set is a "test set" since it does not affect the submitted models, however as soon as the final leaderboard is announced, that test set becomes a validation set; e.g., it affects which algorithms we later choose, i.e. those of top competitors.

In summary, it seems that most of the time we are using less-involved validation sets to double check more-involved validation sets.

P.S.: as of writing this answer, @aranglol came up with similar notes and examples :) (+1)