I love the interview, I published a paper on finding these subnetwork by iteratively pruning and sysnthsising connections as the network "needs" them... tried to. I loved hearing form Jonathan talk about how impractical it really is to find these subnetworks. I managed to find that different strategies for discovering these networks can converge to similar subnetworks so maybe there is an avenue to find commonality in subnetworks, this however doesn't show a way to discover more that there might be a trend.
Super cool, thanks for sharing Alastair! Maybe semantic search through graph-representations of the subnetwork structure to find the common patterns? Maybe a GNN could take the net as input and classify the sub-network with enough training data? Not sure haha, but sounds like a fun/useful area of research!
Are there any competitors to Mosaic ? Looks like they have packaged a bunch of the techniques that improve DL training that have been published and make it easy to integrate into pyTorch.
Thank you for the comment -- I've also been back and forth between PyTorch and Keras over the years haha! I am working on a video explaining how to use the Data Augmentations in Composer such that you end up with numpy arrays easy to use in Keras. I hope you find that useful, not sure I can figure out how to adapt the model surgery / model augmentation methods to Keras, but will be trying haha.
@@connor-shorten I usually avoid pointless debates tf/pt since it's all just a matter of personal preference, although I can exhibit every single argument why it's keras obviously more efficient and practical. please Connor do that since I listened chief-researcher's interview with you, then I come excitedly to this video, got ready to open everything in colab and then I saw pytorch and immediately got blinded by the code. I'm very interested im this new tool/framework but I seriously can't waste my time on translation. Thanks for your effort!
Really interesting interview with Jonathan, looking forward to that coming out!
Thanks Tim, really enjoyed your recent podcast with Zak!
I love the interview, I published a paper on finding these subnetwork by iteratively pruning and sysnthsising connections as the network "needs" them... tried to. I loved hearing form Jonathan talk about how impractical it really is to find these subnetworks. I managed to find that different strategies for discovering these networks can converge to similar subnetworks so maybe there is an avenue to find commonality in subnetworks, this however doesn't show a way to discover more that there might be a trend.
Super cool, thanks for sharing Alastair! Maybe semantic search through graph-representations of the subnetwork structure to find the common patterns? Maybe a GNN could take the net as input and classify the sub-network with enough training data? Not sure haha, but sounds like a fun/useful area of research!
Are there any competitors to Mosaic ? Looks like they have packaged a bunch of the techniques that improve DL training that have been published and make it easy to integrate into pyTorch.
Very nice , WIll this library will be aligned to TensorFlow as well ?
Eran
too bad it's not written in keras, so much looping in pytorch, so many unnecessary lines
Thank you for the comment -- I've also been back and forth between PyTorch and Keras over the years haha! I am working on a video explaining how to use the Data Augmentations in Composer such that you end up with numpy arrays easy to use in Keras. I hope you find that useful, not sure I can figure out how to adapt the model surgery / model augmentation methods to Keras, but will be trying haha.
@@connor-shorten I usually avoid pointless debates tf/pt since it's all just a matter of personal preference, although I can exhibit every single argument why it's keras obviously more efficient and practical. please Connor do that since I listened chief-researcher's interview with you, then I come excitedly to this video, got ready to open everything in colab and then I saw pytorch and immediately got blinded by the code. I'm very interested im this new tool/framework but I seriously can't waste my time on translation. Thanks for your effort!