Видео

Thanks for your comment. I am glad it was helpful!

Same family. To keep the way we measure the epsilon consistent across rounds, it makes sense to use the same weak learner across rounds.

@@meanxai Okay. Thank you!

Typically, the sampled subset is the same size as the original data set and contains repeated data points. However, if your original data set is too large, you can generate smaller subsets. This is called "boosting-by-filtering". In this case, you need to consider a lower bound on the sample size that the model has to use in to guarantee that the final hypothesis has error smaller than epsilon. The lower bound is presented in theorem 3.2 of Yoav Freund's 1995 paper. Unfortunately, I haven't figured this out.

@@meanxai Oh okay. Thanks, will check that out!

This is what I got from NotebookLM ### Understanding Lambda, Gamma, and the Derivation of 'm' The sub-sample size 'm' plays a pivotal role in boosting algorithms. Here's an explanation of lambda (λ), gamma (γ), and how the lower bound for 'm' is derived: * **Lambda (λ) represents the reliability parameter.** This signifies the desired probability with which the final hypothesis outputted by the boosting algorithm should achieve the target accuracy (ε). In simpler terms, it's the confidence level that the final hypothesis will have the desired error rate. * **Gamma (γ) is a measure of the weak learner's advantage over random guessing.** The boosting algorithm utilizes a weak learning algorithm, denoted as **WeakLearn**, that is assumed to perform slightly better than random guessing. Gamma quantifies this advantage. * The sources provide a detailed analysis of a boosting algorithm that employs a "majority-vote game" analogy to illustrate its functionality. This game involves two players: a "chooser" who selects a weighting scheme for a set of points and an "adversary" who attempts to maximize the "loss" based on the chooser's strategy. * The derivation of the lower bound for 'm' stems from the requirement that **the hypotheses generated by WeakLearn should have a high probability of having an error smaller than 1/2 - γ.** This condition ensures that the weak learner is indeed performing better than random chance. * The sources state that the lower bound for 'm' is calculated using a formula that incorporates ε, λ, and other factors related to the weak learner's performance. This formula ensures that the sample size is sufficient to achieve the desired accuracy (ε) with the specified reliability (λ). * **The sources provide a specific formula for calculating 'm':** **m' >= (2/γ^2) * ln (m/(λ * ε)).** This formula guarantees that the probability of the final hypothesis having an error larger than ε is at most λ. * The choice of 'm' directly influences the number of training examples needed and the computational complexity of the boosting algorithm. A larger 'm' typically leads to higher accuracy but increases computational cost. In summary, lambda and gamma are essential parameters in determining the sub-sample size 'm', which plays a crucial role in the performance of boosting algorithms. The lower bound for 'm' is derived to ensure that the weak learner's performance surpasses random guessing and the desired accuracy and reliability are achieved. ### Selecting Gamma and Lambda Yes, **gamma (γ) and lambda (λ) are parameters that are typically chosen by the user.** There are factors that can help guide the selection of these parameters: * **Desired Accuracy (ε):** The user's desired accuracy for the final hypothesis plays a significant role in choosing gamma and lambda. A higher desired accuracy might necessitate a smaller gamma, indicating a requirement for a weak learner with a greater advantage over random guessing. This, in turn, might influence the choice of lambda to maintain a balance between accuracy and reliability. * **Computational Constraints:** The choice of gamma and lambda indirectly affects the computational complexity of the boosting algorithm through its influence on the sub-sample size 'm'. Users need to consider the available computational resources. A smaller gamma or a higher lambda might lead to a larger 'm', potentially increasing the computational cost. * **Nature of the Weak Learner:** The characteristics of the weak learner employed in the boosting algorithm can provide insights into selecting appropriate values for gamma and lambda. If the weak learner is known to have a significant advantage over random guessing, a larger gamma might be suitable. Conversely, a weaker learner might require a smaller gamma. * **Trade-off between Accuracy and Reliability:** The selection of gamma and lambda involves a trade-off between the desired accuracy and the reliability with which that accuracy should be achieved. A smaller gamma generally leads to better accuracy but might require a smaller lambda to maintain the desired reliability, potentially increasing the computational cost. The sources provided do not offer specific guidelines or formulas for choosing gamma and lambda. These parameters depend on the specific application, the characteristics of the weak learner, and the user's priorities regarding accuracy, reliability, and computational resources. **It's important to note that the information beyond what is stated in the sources about choosing gamma and lambda is not from the sources and may need to be independently verified.**

@@KushJuvekar-j3f Thanks for the useful information. It helps me a lot to understand the lower bound of m.

I totally agree with you. As in your example, dot product similarity does not make intuitive sense. We cannot say that dot product similarity is better than cosine similarity. However, we also cannot say that the latter is always better than the former. Cosine similarity only cares about the angle difference, while the dot product cares about both the angle and the magnitude. In deep learning, the magnitude of a vector may actually contain information we are interested in, so there is no need to remove it. Dot product similarity is said to be especially useful for high-dimensional vectors.

Thanks for your comment.

Thanks for your comment.

Sorry, the slides (PDF) are not public.

Sorry, I am not familiar with PyTorch.

Thanks for your comment.

The code can be found at github.com/meanxai/deep_learning Thanks.

@@meanxai thank you for sharing but I typed the whole 🤣 and I also use CiFAR10 dataset but the val_accuracy wasn't good. Like 52 only the training was more than 90

@@zoraizelya3975 I also got 52% accuracy, which is too low. I think this is because our highway network consists of basic feedforward networks, which are not suitable for challenging image datasets. Thanks for your comment.

The formulas used in this video, such as score, gain, came from XGBoost. More details on those formulas can be found here: [MXML-11-04] XGBoost (4/9) ruclips.net/video/ud7kJv5csxw/видео.html [MXML-11-05] XGBoost (5/9) ruclips.net/video/e_1TJD8tHgE/видео.html Thanks.

I didn't quite understand your question. Could you explain it a bit more?

Thanks for your comment.

Really? The sound is fine, tested on Microsoft Edge, Google Chrome, and Samsung tablet. Please let me know whether you are still facing the problem.

Thank you for your interest in my videos, but I can only make 2 videos a week at the moment. I don't think I can make more than that.

OK. From the next video onwards, I will try to provide more links if we need additional reading material.

@@meanxai Thanks sir