The Early Stopping Problem Where Bayesian Optimization Cannot Be Applied
Recommended Post: 【Philosophy】 Philosophy Table of Contents
When to stop. When is the optimal choice.
We ponder when to stop exploring and make a decision in important life decisions. This issue can be explained through the Bayesian Optimization approach. In general, Bayesian optimization is based on Gaussian processes and assumes that the objective function follows a Gaussian distribution to find the optimal choice. The secretary problem well describes it mathematically.
The secretary problem is a fascinating puzzle that gained popularity among mathematicians in the 1940s and 50s. Here's the situation: you need to hire one secretary, and there are N candidates lined up. You interview them one by one and must decide immediately whether to accept or reject each person. Once rejected, a candidate cannot be reconsidered, and once someone is hired, you can't see any of the remaining applicants. So, as the hiring manager, what’s the best strategy to select the most qualified candidate? Surprisingly, there’s a clear answer. Assuming the number of applicants is large enough, the optimal approach is to automatically reject roughly the first 37% of candidates. For example, if there are 100 applicants, you reject the first 36. Then, among the remaining candidates, you hire the first person who’s better than anyone you’ve seen so far. This strategy gives you about a 36.78% chance of picking the very best candidate — the probability converges to 1/e. A beautiful bit of math magic.
However, the real world is an uncertain and chaotic system, so this assumption may not hold. Additionally, lacking domain knowledge about the real world makes it challenging to apply Bayesian optimization with other assumed distributions. Then, how should the Early Stopping problem be approached in the real world?
The following practical strategies were diagnosed:
Strategy 1. Utilizing instincts, intuition, attraction, and impulses
○ Don’t just judge with your head; use intuitive information you feel with your whole body.
○ If it’s a choice you truly like and are proud of, even if you fail, you can get back up and eventually succeed.
Strategy 2. Not fearing failure and trying again
○ Maintain an attitude of finding ways to try again instead of feeling discouraged.
○ The attitude of “buying more time to try more” is important and connects with Strategy 4.
○ Jeff Bezos
○
When it’s tough, will you give up? Or, will you be relentless?
Strategy 3. Not limiting choices to a binary of 0 and 1
○ Not all choices are as clear-cut as black and white.
○ Understanding these intermediate choices helps promote flexible thinking. However, be cautious not to make unethical choices.
Strategy 4. Admissions are luck, but applications are a choice
○ While outcomes cannot be controlled, how much effort we put into trying is our choice. To increase opportunities, one must actively strive.
Strategy 5. Embracing and enjoying uncertainty and instability
○ Being impatient can lead to failure even when success was possible.
○ It’s important to maintain a flexible and relaxed attitude in uncertain situations.
Strategy 6. Assigning lower weights to confusing problems from the start
○ Life is full of countless problems. Adjusting importance levels yourself and spending less energy on confusing issues is one approach.
Input: 2025.01.04 18:00