I developed a theory about 15 years ago that neurons actually function more like mathematical group permutations than anything else. Their goal is to organize and group action potentials from synapses (putting similar action potentials closer together in time prior to integration). I actually developed C code to show how they can be used to solve a 50 node 2D Hamiltonian path problem extremely fast (in C it was solved almost instantly). The behavior of individual neuron dendritic growth was modeled by a genetic algorithm. I never published my research, so anyone is welcome to try and reproduce this.
Neuroscience PhD student here. Having followed this topic, although from a distance, since I am more into molecular neurobiology, I must say you did a very good job of explaining this very interesting topic. I have been in lectures from some of the researchers you included and I have to say I understood way more things from you than I did before. Thank you!
I have a PhD from Berkeley in neuro & engineering, currently work in ML, and you break this stuff down really well. I love the digested summary of the paper equating a neuron to a small CNN. Thanks, and keep it up man
i envy you so much. I never had your opportunities but i wanted to do ai research as a programmerin the 80s. It was learning the most basic parts of neuroscience that showes me (and others) how we wpuld achieve it. Unfortunately it also helped me understand how we needed huge leaps in processing power to get there. It is amazing seeing real ai coming to fruition in my lifetime
I feel like the insight that each neuron is essentially like a small CNN function would actually be a useful intuition for better neural hardware accelerations if we could keep the numbers of large distances connected between neurons down. Much like how software design to be multithreaded is better for multicore CPUs.
Always looking forward to your videos, thank you for your hard work! A few questions (maybe answered in the papers, I haven't read them yet): * One neuron is great, how much do we know about the next level, a net of several neurons? IIRC, the number of different neurotransmitters is in many thousands? How does it change the compute? Does it mean we need to create a 5-8 layer nn for every MODE a neuron finds itself in or smth? * Dendritic spines. Neuroplasticity. Has it been incorporated into the simulation models yet? * On a tangential, have you looked at Michael Levin's research? Starting to look like neurons are but a special case of bioelectric signaling.
1. I'm not sure if the number of neurotransmitter types would number in the thousands. For context, serotonin, one of the most complex neurotransmitters (i.e. more diverse receptor class) has about 40 types of receptor types, which around 7 major subclasses. Acetylecholine, which I believe is the most complicated, has upwards of 200 receptor types IIRC. Importantly, a specific neuron doesn't have to express all receptor types for all neurotransmitters, and are likely specialized for their specific role in the circuit, i.e. what specific neurotransmitters they receive. Thus, some neurons may be much more computationally complex than others, at least in terms of receptor subtypes / neurotransmitters received. To simplify this to a 5-8 layer nn is not trivial, as as having a 8 layer temporal CNN for each node in your network is not computationally feasible. 2. I am sure that dendritic spines have been incorporated into simulation models, you may want to look into the dendritic neuron model and active dendrite research in general. As far as neuroplasticity, I would argue that backprop implements a version of plasticity, i.e. plasticity = weight changes. If you are talking about spike timing dependent plasticity (STDP) or behavioral time scale plascity (BTSP), I would assume there are also models which incorporate them.
Wow, thank you very much, Matvey! 1) The number of different neurotransmitters is actually much lower than that - less than a hundred. It is true, that there are any subtypes of receptors to a single neurotransmitter, which differ in kinetics (how fast they open), sensitivity, ion selectivity, etc. It is not clear how to incorporate such diversity into network models, so, usually people just interconnect neurons with different synapses, explicitly defined by maximum conductances (either excitatory or inhibitory) and time constants. It’s worth saying, however, that certain compounds, such as serotonin, dopamine, norepinephrine can act more globally (as neuromodulators), affecting a large population of neurons simultaneously and modulating their excitability. There are certainly models that incorporate neuromodulation, but again, in a very simplified description (e.g. introducing an additional current, which reflects the activation of extrasynaptic receptors). This approach with describing a single neuron with a neural network is very new and I don’t know any papers that model a whole circuit of neurons in this way, so I’m not sure how this could be done. Great question! To quote the authors of the paper on this one: “If indeed one cortical neuron is equivalent to a multilayered DNN, then what are the implications for the cortical microcircuit? Is that circuit merely a deeper classical DNN composed of simple ‘‘point neurons’’? A key difference between the classical DNN and a cortical circuit composed of deep neurons is that, in the latter case, synaptic plasticity can take place mainly in the synaptic (input) layer of the analogous DNN for a single cortical neuron, whereas the weights of its hidden layers are fixed (and dedicated to represent the I/O function of that single cortical neuron)”. 2. Dendritic spines and plasticity (e.g. STDP, BTSP) are definitely accounted for in many realistic network models. However, such simulations are done the “usual way”, by representing neurons with capacitors and resistors and adding plasticity-specific equations to synapses (e.g. increment the weight whenever the 2 neurons is a synaptic pair is co-activated. Or something like that). Since Beniaguev et al. simulated input-output transformations of a single neuron, I don’t think they modeled any plasticity (the neuron just responded to random inputs). When in the future people begin to make use of the DNNs to model networks of neurons, than synaptic plasticity will surely be accounted for (see the quote from above). 3. I’m aware of Michael Levin’s work, but haven’t looked at it in detail just yet ;) It’s true that bioelectricity is really important, not just in neurons, but also in regulating embryonic development, IIRC.
This is crazy-interesting stuff. The whole thing reminds me of the development of the concept of atoms. Atoms were supposedly meant to be the indivisible building blocks of matter from which you could build anything. The greek concept was adapted to what we now call...well.... atoms. The problem is that atoms are not elementary building blocks and as it turns out they are complex systems made of multiple more fundamental particles that can be arranged in many many ways to build many atoms in very different states and configurations. So the real greek concept of "atom" should be applied to fundamental particles of the standard model, and what we call atoms should be renamed. The same goes for neurons. In AI a "neuron" is just the smallest conceivable operational unit that you can use to build more complicated logic systems. Historically we thought that biological neurons were just that, the smallest indivisible element of logic (a simple lightbulb, a binary switch), but then, like when we splitted the "atom", we discovered that in fact a biological neuron is really equivalent to an ensamble of more simple logic gates. So, the actual "atoms of logic", the neurons in neural networks, should not be confused with neurons in biology.
@@ArtemKirsanov By the way, your explanation of the perceptron was one of the best I've ever seen. Clear, straight to the point and esthetically pleasing. Congratulations for the awesome chanel.
In general, all science and discovery is fractal-shaped. The picture is never complete because the more you zoom in the more details there are and you realize everything you saw before zooming was just an approximation.
I’m a fine arts doctorate with only 6 or so credit hours of formal education in science post-high school. You were able to make this topic at least comprehensible for me. Thanks for the engaging format!
In the problem of approximation of the XOR-function, the choice of the activation function not step-func but П-shaped allows us to solve that by just one single neuron. Then the input space will be divided into 3 parts (- + -). The choice of activation function in ANN plays a significant role. In the SIREN family of models, periodic sin cos functions are generally used. And it allows you to encode the input vectors in hierarchy of scales at once
This is interesting! I've never heard about SIREN networks before (or any model that uses periodic activation functions for that matter). I was really impressed by their results presented in this video ruclips.net/video/Q2fLWGBeaiI/видео.html Thank you for pointing it out!
There is a small record of work done at Los Alamos from the late 80’s through the 90s using “radial basis functions”, which are a specific kind of non-monotone activation functions, in adaptive learning NNs. Although not precisely what was used, an erf or ndf is an example of what was used, as opposed to the square wave you mention, but all the same in one sense. Specifically they were useful for both reduction of layers in NN for a given learning task and also, obviously, for logical non-monotonicity such as classically found in the XOR-perceptron problem. But in addition, it seemed and still seems to me that since with real neurons, if you continue to drive them past threshold, the outputs saturate and decline as a function of input values, we find non monotonic activation in a natural way, built in. And it’s computationally useful as well as “nonlinear” and all that stuff. So radial basis functions, periodic ones, even varying ones perhaps, are important and extend basic computational power. It takes some of the network modeling and moves in “inside” that’s true enough, for good or bad. But it potentially extends what is possible to do on a given “budget”. It doesn’t help that much with other things that remain problematic for brute-force ML or DL. Interesting times.
Very interesting observation. Seems like this could be modeled with an appropriate basis function. At first blush, kind of reminds me of spherical harmonics transfer functions in computer graphics I'm a computer scientist, so grabbing something that I have seen before. Very interesting stuff, thank you.
@@Simulera Hah, excellent info about these radial basis functions. I had just typed about spherical harmonic transfer functions and then read your comment. Definitely mind is ticking over. Thanks for sharing this golden nugget.
This content is exceptional! As an AI engineer with limited biology training, I previously felt so overwhelmed by the complex terminology and extraneous information in the realm of neuroscience. Then struck gold by discovering your channel. Thank you!
Awesome content. It's always amazing to see someone that can explain advanced topics in terms that even who are not familiar with the field can understand. Fascinating stuff.
Fascinating. Especially the idea that neurons are sensitive to the activation order of signals along its dendrites. I remember seeing a talk long ago where the speaker was discussing what aspects of the brain's (or the neurons') configuration that are actually coded for in our genes, and argued that (a) almost none of the neuron-to-neuron connections or connectivity strengths are controlled (as you may have thought they were, if the brain was a giant network of threshold gates), but (b) great care is taken to control, across a large variation of different neuron cell types, precisely where along the dendrite each nerve cell type connects and in what manner (which makes no sense in a threshold gate model). That seems rather more on-point with the model you're describing!
The all of the animations on this are amazing. I love them. They make neurons look so colourful and exciting, and its really engaging. I'm still a beginner at neuroscience but I really like both topics of neural networks in machine learning and computational neuroscience, so this video is perfect for me. I will definitely rewatch it.
As an undergraduate who is currently studying to go into computational neuroscience, I feel privileged that such a high quality channel exists for a relatively niche subject like this. Keep up the good work!
Superb video! I'm a first year Data Science and Neuroscience uni student (aged 38) and am very interested in the fusion of these 2 fields. Will come back to this video again to share some thoughts later. Anyway, even if all the functionality of the human biological neural networks and neurons isn't currently fully implemented in artificial neural networks, it doesn't mean that it won't be in the near future. Also, as you point out here, the same functionality is also achievable by deep neural nets, it's just a matter of efficiency.
Wait, your doing a university course that integrates both data science and neuroscience togather? and if it is or isn't, its still cool that your doing both.
@@sergeantoreo8062 The uni that I'm at specializes in double degrees like that. There's data science and physics for example, or psychology and neuroscience, literature and music, and so on.
@@ConnoisseurOfExistence such a lucky guy... i wish i could do that. I'm from Brazil and i'm also super interested in such topic, although we don't have that much of investment in these areas, especially in cities in the middle of nothing. I hope you have a great time while in this course.
@@marvin.marciano Thanks. Why don't you apply in the capital, or another country? You speak English. I'm not studying in my own country at the moment...
This was fantastic - easy to follow and amazing production quality. I was wondering how you do your animations, but I see you have a video for that too! I want to go through all of them!
I loved this! It gives a very optimistic outlook to that artificial neural networks can at the very least simulate our own neurons if they need it to solve a problem. It's important to notice that while a deep neural network can approximate a single biological neuron with a lot of layers and parameters, this only provides an upper bound on the total amount of parameters you'd need to approximate multiple biological neurons! Not only may there be a more efficient architecture the authors haven't tried, but it may be that as artificial neural networks scale to become larger they become more efficient at handling the equivalent of an individual biological neuron. This because the overall behaviour of a system may be simpler to model by exploiting correlations and limiting behavior, just like a glass of water with respect to all of the H2O molecules composing it.
Yes its very likely that your second explanation is at play. Grasping the function of a neuron with active dendrites would be much easier when the dimension of the computational manifold is scaled up.
Neurons don't get put together like a "neural net". They start out as whole undifferentiated cells and then progressively specialize into neurons. Neural nets are put together like machines - from combination of separate parts, components, functions, etc. according to a plan. In other words, although neural nets can simulate some abstract aspect of neurons that might interest humans, you can't really understand neurons using neural nets. For example, neural nets don't socialize. They don't generate projections and try to make it in neuron society, etc. It's kinda like studying humans using speech recognition software that's incapable of going out and making friendships, looking for work on its own, trying to find a spouse, etc.
@@revimfadli4666 Is this activity planned by a researcher or something that a bunch of math functions just decided to do on their own? Math functions performing other's will don't do anything spontaneously for their own purposes. They are completely parasitic on human purpose.
I'm ... NOT a scientist by profession or choice. But I AM fascinated by the digital/analog crossover... started with audio projects and a basic understanding of PWM... then realizing electricity/chemistry/physics/biology/neurology are merging at a level. This video to me was super helpful. It also has me contemplating how the paper mentioned essentially modeled a tiny brain microphone/recording booth, with digital playback and analog capture. It's like amplifier modeling. It's very very very difficult to capture all the possible inputs, but if you get enough of them, the capture and modeling is hard, but it can be sufficiently accurate to be useful, and then needs very little computational power after the model has been created. It's super intriguing and makes me wonder what would happen plugging the analog input/outputs of the dendrite/some would look like plotted as audio signals into some sort of modeler. Probably worse, more translation and compression happening etc. The fact that the neuron basically is a combination of an analog and digital signal is amazing to me. It does pulse width, essentially square wave when the specific voltage activates the dendrite spikes, and then duration and magnitude, and possibly- time-- I'm no interested in if there is a sort of standard "period" for each input/output. It was mentioned there are some differences, but is there a universal cap on what that is? Does a specific input change the effective period of responses? IE, would stress shorten the periods thus the perception of things happening faster? What is our internal "clock" speed. We know it isn't fixed at a top level, but on an individual cell, is it set? Are there a few that are defined based on the chemistry at work, and depending on the channels/ions/resistance etc? So we have individual inputs for location, a comparator for location which acts almost as an input it seems, that is part of a summed input of what order/how long. Then on each input we have duration and magnitude (x/y) and additionally, on dendrites themselves we have the NMDA acting as an additional square wave input? What about the periods of that input? Would different specific voltage inputs result in a response that is different consistently in period? Hey guys- I HATE school. But I might have to go back to school lol
Former Big Tech AI engineer & Researcher. Love what you have to say here. A learned a lot about the neuroscience perspective of what we do. Thanks for sharing!
But if a neuron sits at a negative value in its resting state, surely its the other way round. Voltage gets closer to 0 so increases but potential decreases?
@@tedarcher9120 But regardless of using negative or positive voltages to talk about it, a depolarisation would always refer to a reduction in potential across the membrane. Plus the voltage value falls out of an equation which uses ion concentrations to calculate it, so that's why its consistently referred to as negative inside the cell and positive outside.
Great video, It's cool that single neurons can act as their own mini deep neural networks. I think that it is worth clarifying that even though the biological neurons are not equivalent to perceptron neurons in function, all this does is move the perceptron to a sub part of the neuron. Meaning that the fundamental principle is the same (ie. a network of sub networks is still a single network on the whole)
I was having a hard time understanding how a neuron could be phase sensitive from your video on theta rhythm, but the "xor" neuron kinda makes that make sense. How narrow can the window of sensitivity be?
I would definitely love to learn neuroscience in entirety by you. Thanks for the info. Always was fascinated by the workings of the brain but found it tought to wrap my brain around it, no pun intended. But your explanations make it clear and gives me a visual representation of the workings that i need. Thanks brother♥️ Looking forward for more.
I just found this channel out, and watched a few video's fully and I'm absolutely hooked and subbed directly, keep it up man! - The way you explain it is very easy to understand, even for outsiders!
Spatial and temporal summation allows for LTP and LTD in order to control the plasticity of the neural network. That’s why you need NMDAR which are voltage gated dependent on AMPAR (the non linear system). As a neuroscience undergrad, I love your videos 🎉
Great video as usual! Could you explain the neural backpropagation in a future video? Is it similar to the backprop in artificial neural networks? Or maybe to the Forward-Forward algorithm for adjusting weights proposed recently by Hinton?
a quote I remember from somewhere is that if you can think of two ways neurons could be doing things, the brain is using both and a third or more that you didn't think of. though of course that only works for things that are biologically possible.
one point of interest is that you can approximate gradient backprop with predictive coding, and forward-forward doesn't seem to preclude predictive coding.
As someone who teaches a bit of neuroscience at university level, I thought this video was superb from a didactic point of view and contained a lot of interesting content. You covered the basics of electrical excitability and action potentials in a compact way but also very clearly. I was prepared to wince at sodium ions "flooding" in but you used the less misleading word "rushes". Students tend to be led into erroneously thinking that the ion concentrations change massively during an AP when this isn't the case (especially for large axons like the squid giant axon). Anyway, I really like your style and content and have clicked Like and Subscribe.
18:51 does it imply that a more accurate model of how neurons work would be actually a base 3 or ternary numeral system? It’s not 1 or 0, but 0, 1, 2 states equivalent to under current threshold, over current threshold, over saturation threshold 🧠
I have a question: to the extent where the single neuron can perform XOR operations, can't it be argued that it is simply checking two thresholds instead of one? With both the lower bound and upper bound known it doesn't necessarily need to be multi-layer
Well, technically yes, but to perform this check of the two thresholds you'll need at least 2 layers, if we are talking about conventional perceptrons, so in that sense it's multilayered (in this terminology "multi"=2 :D )
Great video. Now we can encode neural nets using a few new novel signal processing nodes !! Should make neural nets much faster and less power hungry..
Great! Thank you, now I have a good enough video material to throw it in people who are lacking understanding biology moment. :) Especially for giving context of working of neurons nicely in one place. Some fancy neuroglia will not fit in this, but it's already great. I learned biology in school as well to understand, but not as well to explain something for others 🥰
What the hell is wrong with the YT algo??? This video is amazing! It should have minimum 200k views ... This is a slap in the face of all nerds dedicated to STEM fields... Great work YT promoting great twrking videos! Great work!
has reduced function and resulting in summing inputs because of leaky gates -Amplify voltage -Backpropogation : adjusts weights of input? NMDA gates (Neurotransmitter + adequate depolarization) (adjusts synapse connection) Dendritic Calcium Action Potential predictiion and logical results
There's one more important property - temporal sensitivity. I did my diploma in hardware spiking neural nets (on a state-of-the-art Xilinx programmable chip) and this realisation was a wow moment. Basically, any digital system has a clock speed limitation. Natural neurons have much slower propagation speed, but activation is dependant on constant balance of race conditions + no two signals are coming at the exact same moment + basic quantum effects act as a gate (if one ion membrane changes its state from a pulse, concurring pulse sources will be denied and "rerouted"/lost). In the process of digital learning, resolution for these race conditions quickly becomes insufficient, signals are starting to clash, when they are coming in a single clock step. Biological systems, on the other hand, have physical time resolution, which is effectively infinite. No GHz clock can compete with this biological mechanics (only ASIC non-clockable implementation can utiliize this process). BTW, these effects manifest themselves with some neurotoxins: when receptors of a particular kind slow down, the whole system can be knocked out, or become very unstable
@@odettali1000 I meant really basic discrete/quantized effects: such as in "Voltage-gated ion channels". Basically, you can't strip more electrons than the molecule has available for ionization. And you can't stick additional electron into non-ionized molecule. This electron will be "diverted" to go somewhere else
So glad to have discovered your channel, thank you for this brilliant and beautifully made video! Any thoughts on connections between what you've talked about and valence encoding by neurons?
Great Video Artem!, Loved it! One thing has been bothering me tho, at 18:25 , corresponding to the third red spike (A) at the dendrite, shouldn't there be a spike in the soma as well? kindly let me know if this isn't the case, because I am kind of having a hard time making sense of it otherwise. Thanks
This is so cool! I am becoming very interested in artificial neural networks and I always thought that the initial “simplified” model of the neuron was more or less correct
Your channel is awesome, it's purely gold !! and not only for neuroscientist also for everyone. Thanks for sharing all of your knowledge with that easily and great animations. Cheers from Mexico :)
Sure, I believe the picture with activation functions is from here medium.com/@shrutijadon/survey-on-activation-functions-for-deep-learning-9689331ba092 And the one with network architectures is by Asimov Institute: www.asimovinstitute.org/neural-network-zoo/
20:40 you have work to do in the DNN literature. DNNs can fit to any random data. And It is a only model, if additionally it doesn't learn like a neuron under new experience, that network is useless for ANY kind of neuronal theory.
I must say I am very impressed with your content and how you represent it. May I ask what software you use for your animations? I want to start a youtube channel myself, also with more abstract information, although in another field. I think those animations would greatly benefit my viewers. So any pointers would be greatly appreciated.
Thank you! I use mostly Adobe After Effects in combination with Python (matplotlib / manim modules) for creating mathematical animations. In this video, for 3D scenes with neurons I used Blender and a custom-written Python script to run a biophysical simulation in the NEURON software and bring the morphology and animation data into Blender
Great video, Artem Not so long ago I found an overly increasing interest in machine learning and neuroscience and I love finding such great channels. Keep up the good work )
Can you do a video about dendritic clustering, i saw a talk about a model neuron called the “clusteron” where the dendritic branches cluster synapses along the branches, curious how this works and what it does for the neuron
Amazing video Artem, synthetic and clear explanations with impressive animations. As someone doing research on Deep Learning, I got super inspired by this video and I look forward to the future of our field !
Well, it's kind of obvious if you just think about it for 2 seconds. Our "perceptron" is actually a single timed event which is more closely resembling the synapse to dendrite communication, using a master propagation chain as an offset, and the soma is another "perceptron" that details the chain into the axon....but everything is a timed sequence (as noted by the pulses, since channels need to recharge). All of this can be simulated using 2 layers across many "phases" (a phase is a deterioration of a signal as it fades across the channels, giving you the semblance of timing). When you have completed that, then you can realize that new neurons actually train themselves against already existing neuron clusters in order to output similar signals in order to get "the answer" faster in time, allowing the replacement of the slower one (this is how your memory changes slightly as you get older, possibly even losing the memory altogether). Each section of the brain has a different "job" to do, and when you solve that, you create the singularity....which is actually pretty easy. Don't worry, mine doesn't want to kill humans or anything.
I would think this would have profound implications for the design of NN hardware actually, e.g. DOJO etc. The node requires an extra parameter (or more) and needs to allow inputs to be processed not only with weightings but XOR. Is it good enough to just comingle the nodes and allow the multilayers or attention to resolve this? My guess is that biology discovered XOR for a good reason, so following this precedent may lead to much different architecture of NN transformers/diffusers and affect the hardware assumptions. and who knows, maybe this would also get us one step closer. 20:19
Fantastic video as always, thank you for your great work! Question: You mentioned that for the experiments in the last paper, if the NMDA-channels were neglected single-layer networks were sufficient to reproduce the functionality of the neuron. How is this possible? Have the channels responsible for XOR-functionality/capacity been thrown away with the NMDA-channel?
Even with the current achievements in AI this year we still have a long way to go to approximate our own neurons. Artificial networks a still build pretty linearly with information propagating instantly in one direction. Not to mention that they are still have to be pretrained for a long period of time. Real neurons on the other hand can be connected in loops of diverse lengths. Their signals can arrive in any order. They include complicated timing mechanisms and can learn as they go.
2:00 It goes way back 1907 lapicque paper describes the summation with weights with a threshold. Current NNs are 118 years old theory, updated thousands of times and still not right.
It’s interesting to see a more mathematician-centric POV of how neurons are like neural networks, when from a neurobiologist’s perspective, it’s already obvious that real brain cells individually and collectively are already immensely powerful in a computational sense. So much so, it’s weird to see it phrased in this video that it’s the neuronal cells being “exonerated” relative to computational neural networks, when in reality it should be the other way around.
Hi, I didn't get the part that you said about backpropagating action potential. You said those Na channels on the "dendrite" causes bAp.! Is it so?! They can be open due to synaptic inputs and generate the action potential if they are strong enough and later this can cause generation of bAp.
Is the dendrite selection only the filter of voltage to make a XOR function, or does it also do a combination of low and high band filtering on frequency. Just trying to imagine if the dendrite always transmits the voltage in the proper band or that it is even more selective and only propagetes when the receiving pulses follow the right pattern? If so I could imagine that a dendrite would be able to translate a firing pattern received from another neuron into a different signal inside its own cell, and then the soma with its axion can transmit a totally different firing pattern making this simple neuron a huge signal translator. Modern neural networks don't take into account the time domain, which in my opinion is where we loose the link with biology.
What about so called "pulse train" patterns? Or is what you are discussing just a better understanding of what leads to that behavior, and the 'train' is more an obsolete term?
Our brain initially constructs physical resistive rails that a and b must travel along with a certain voltage or their actions will cancel each other out, creating XOR function? Trying to figure this out with only a GED in hand is tricky, but I'm trying!
It's funny how the XOR gate is called the only "nonlinear" gate here because of non linear separability, while it is the only binary gate (among the usual ones) that is linear from an algebraic perspective (seeing bits as a vector space over F2). We can even use this to prove that with only XOR and NOT, you cannot build a circuit computing AND: XOR and NOT can only build linear functions, but AND is not linear.
So I take it this means neuron can do not only addition, multiplication, via a kind of logical AND, OR, NOT, but also sigmoid, relu, thresholds, and some functions involving sequences. Which ones? Not square root I feel this is hard. Logarithm? Yes that. Anything else?
I am curious now, what is the right neural network setup to correctly simulate a single neuron? Ok something about seven layers or so, but how are they connected? Did I miss it?
Thanks! Mostly, yes. Although if a single axon has several terminals connected to the same dendrite and if the release of neurotransmitter is a stochastic process (which means that there is a certain probability
![The moment we stopped understanding AI [AlexNet]](/img/1.gif)
Keep exploring at brilliant.org/ArtemKirsanov/
Get started for free, and hurry-the first 200 people get 20% off an annual premium subscription.
Hey, I love your unique work😁👍, may i ask you if would you like someday explain the basics of synapse plasticity in biological neuron networks?
I developed a theory about 15 years ago that neurons actually function more like mathematical group permutations than anything else. Their goal is to organize and group action potentials from synapses (putting similar action potentials closer together in time prior to integration). I actually developed C code to show how they can be used to solve a 50 node 2D Hamiltonian path problem extremely fast (in C it was solved almost instantly). The behavior of individual neuron dendritic growth was modeled by a genetic algorithm. I never published my research, so anyone is welcome to try and reproduce this.
Like the video.♥️♥️
Can anyone tell me what software has been used for the animations?🙏🏻🙏🏻
On the interesting note:
We don’t just have static weights, we also have fast weights, they’re dynamic and decay fast
Neuroscience PhD student here. Having followed this topic, although from a distance, since I am more into molecular neurobiology, I must say you did a very good job of explaining this very interesting topic. I have been in lectures from some of the researchers you included and I have to say I understood way more things from you than I did before. Thank you!
Wow, thank you so much! It's really nice to hear that!
researchers are not teachers i dont know why universities haven’t understood that yet.
@@zakiyo6109 especially so at top universities. They're mostly hired for the funding and citations they can attract rather than teaching ability...
Sparse distributed representation (thousand brains theory) from Numenta is more accurate since it’s based on actual biological data.
I saw another video here on yt where one was talking about a study showing consciousness is just a download. Pretty sure in
I have a PhD from Berkeley in neuro & engineering, currently work in ML, and you break this stuff down really well. I love the digested summary of the paper equating a neuron to a small CNN. Thanks, and keep it up man
i envy you so much. I never had your opportunities but i wanted to do ai research as a programmerin the 80s. It was learning the most basic parts of neuroscience that showes me (and others) how we wpuld achieve it. Unfortunately it also helped me understand how we needed huge leaps in processing power to get there. It is amazing seeing real ai coming to fruition in my lifetime
Thank you!
I feel like the insight that each neuron is essentially like a small CNN function would actually be a useful intuition for better neural hardware accelerations if we could keep the numbers of large distances connected between neurons down. Much like how software design to be multithreaded is better for multicore CPUs.
Always looking forward to your videos, thank you for your hard work!
A few questions (maybe answered in the papers, I haven't read them yet):
* One neuron is great, how much do we know about the next level, a net of several neurons? IIRC, the number of different neurotransmitters is in many thousands? How does it change the compute? Does it mean we need to create a 5-8 layer nn for every MODE a neuron finds itself in or smth?
* Dendritic spines. Neuroplasticity. Has it been incorporated into the simulation models yet?
* On a tangential, have you looked at Michael Levin's research? Starting to look like neurons are but a special case of bioelectric signaling.
1. I'm not sure if the number of neurotransmitter types would number in the thousands. For context, serotonin, one of the most complex neurotransmitters (i.e. more diverse receptor class) has about 40 types of receptor types, which around 7 major subclasses. Acetylecholine, which I believe is the most complicated, has upwards of 200 receptor types IIRC. Importantly, a specific neuron doesn't have to express all receptor types for all neurotransmitters, and are likely specialized for their specific role in the circuit, i.e. what specific neurotransmitters they receive. Thus, some neurons may be much more computationally complex than others, at least in terms of receptor subtypes / neurotransmitters received. To simplify this to a 5-8 layer nn is not trivial, as as having a 8 layer temporal CNN for each node in your network is not computationally feasible.
2. I am sure that dendritic spines have been incorporated into simulation models, you may want to look into the dendritic neuron model and active dendrite research in general. As far as neuroplasticity, I would argue that backprop implements a version of plasticity, i.e. plasticity = weight changes. If you are talking about spike timing dependent plasticity (STDP) or behavioral time scale plascity (BTSP), I would assume there are also models which incorporate them.
Wow, thank you very much, Matvey!
1) The number of different neurotransmitters is actually much lower than that - less than a hundred. It is true, that there are any subtypes of receptors to a single neurotransmitter, which differ in kinetics (how fast they open), sensitivity, ion selectivity, etc. It is not clear how to incorporate such diversity into network models, so, usually people just interconnect neurons with different synapses, explicitly defined by maximum conductances (either excitatory or inhibitory) and time constants. It’s worth saying, however, that certain compounds, such as serotonin, dopamine, norepinephrine can act more globally (as neuromodulators), affecting a large population of neurons simultaneously and modulating their excitability. There are certainly models that incorporate neuromodulation, but again, in a very simplified description (e.g. introducing an additional current, which reflects the activation of extrasynaptic receptors).
This approach with describing a single neuron with a neural network is very new and I don’t know any papers that model a whole circuit of neurons in this way, so I’m not sure how this could be done. Great question!
To quote the authors of the paper on this one: “If indeed one cortical neuron is equivalent to a multilayered DNN, then what are the implications for the cortical microcircuit? Is that circuit merely a deeper classical DNN composed of simple ‘‘point neurons’’? A key difference between the classical DNN and a cortical circuit composed of deep neurons is that, in the latter case, synaptic plasticity can take place mainly in the synaptic (input) layer of the analogous DNN for a single cortical neuron, whereas the weights of its hidden layers are fixed (and dedicated to represent the I/O function of that single cortical neuron)”.
2. Dendritic spines and plasticity (e.g. STDP, BTSP) are definitely accounted for in many realistic network models. However, such simulations are done the “usual way”, by representing neurons with capacitors and resistors and adding plasticity-specific equations to synapses (e.g. increment the weight whenever the 2 neurons is a synaptic pair is co-activated. Or something like that).
Since Beniaguev et al. simulated input-output transformations of a single neuron, I don’t think they modeled any plasticity (the neuron just responded to random inputs). When in the future people begin to make use of the DNNs to model networks of neurons, than synaptic plasticity will surely be accounted for (see the quote from above).
3. I’m aware of Michael Levin’s work, but haven’t looked at it in detail just yet ;)
It’s true that bioelectricity is really important, not just in neurons, but also in regulating embryonic development, IIRC.
This is crazy-interesting stuff. The whole thing reminds me of the development of the concept of atoms. Atoms were supposedly meant to be the indivisible building blocks of matter from which you could build anything. The greek concept was adapted to what we now call...well.... atoms. The problem is that atoms are not elementary building blocks and as it turns out they are complex systems made of multiple more fundamental particles that can be arranged in many many ways to build many atoms in very different states and configurations. So the real greek concept of "atom" should be applied to fundamental particles of the standard model, and what we call atoms should be renamed.
The same goes for neurons. In AI a "neuron" is just the smallest conceivable operational unit that you can use to build more complicated logic systems. Historically we thought that biological neurons were just that, the smallest indivisible element of logic (a simple lightbulb, a binary switch), but then, like when we splitted the "atom", we discovered that in fact a biological neuron is really equivalent to an ensamble of more simple logic gates. So, the actual "atoms of logic", the neurons in neural networks, should not be confused with neurons in biology.
Wow, this is actually fascinating insight!! I've never thought about it
@@ArtemKirsanov By the way, your explanation of the perceptron was one of the best I've ever seen. Clear, straight to the point and esthetically pleasing. Congratulations for the awesome chanel.
We’re computers made of computers
In general, all science and discovery is fractal-shaped. The picture is never complete because the more you zoom in the more details there are and you realize everything you saw before zooming was just an approximation.
Is it possible that atoms can be infinitely divided into small "things" ?
I’m a fine arts doctorate with only 6 or so credit hours of formal education in science post-high school. You were able to make this topic at least comprehensible for me. Thanks for the engaging format!
Thank you!
I'm glad you enjoyed it :)
What job are you looking for
@@your_-_mom petridish
In the problem of approximation of the XOR-function, the choice of the activation function not step-func but П-shaped allows us to solve that by just one single neuron. Then the input space will be divided into 3 parts (- + -). The choice of activation function in ANN plays a significant role. In the SIREN family of models, periodic sin cos functions are generally used. And it allows you to encode the input vectors in hierarchy of scales at once
This is interesting! I've never heard about SIREN networks before (or any model that uses periodic activation functions for that matter). I was really impressed by their results presented in this video ruclips.net/video/Q2fLWGBeaiI/видео.html
Thank you for pointing it out!
There is a small record of work done at Los Alamos from the late 80’s through the 90s using “radial basis functions”, which are a specific kind of non-monotone activation functions, in adaptive learning NNs. Although not precisely what was used, an erf or ndf is an example of what was used, as opposed to the square wave you mention, but all the same in one sense. Specifically they were useful for both reduction of layers in NN for a given learning task and also, obviously, for logical non-monotonicity such as classically found in the XOR-perceptron problem. But in addition, it seemed and still seems to me that since with real neurons, if you continue to drive them past threshold, the outputs saturate and decline as a function of input values, we find non monotonic activation in a natural way, built in. And it’s computationally useful as well as “nonlinear” and all that stuff. So radial basis functions, periodic ones, even varying ones perhaps, are important and extend basic computational power. It takes some of the network modeling and moves in “inside” that’s true enough, for good or bad. But it potentially extends what is possible to do on a given “budget”. It doesn’t help that much with other things that remain problematic for brute-force ML or DL. Interesting times.
i wonder if that would reduce the amount of layers required in that model to approximate the neuron
Very interesting observation. Seems like this could be modeled with an appropriate basis function. At first blush, kind of reminds me of spherical harmonics transfer functions in computer graphics I'm a computer scientist, so grabbing something that I have seen before. Very interesting stuff, thank you.
@@Simulera Hah, excellent info about these radial basis functions. I had just typed about spherical harmonic transfer functions and then read your comment. Definitely mind is ticking over. Thanks for sharing this golden nugget.
This content is exceptional! As an AI engineer with limited biology training, I previously felt so overwhelmed by the complex terminology and extraneous information in the realm of neuroscience. Then struck gold by discovering your channel. Thank you!
Awesome video thank you for that!
Awesome content. It's always amazing to see someone that can explain advanced topics in terms that even who are not familiar with the field can understand. Fascinating stuff.
While I didn’t understand most of it I subscribed just because I can appreciate the value he is providing for free with this content.
Fascinating. Especially the idea that neurons are sensitive to the activation order of signals along its dendrites.
I remember seeing a talk long ago where the speaker was discussing what aspects of the brain's (or the neurons') configuration that are actually coded for in our genes, and argued that (a) almost none of the neuron-to-neuron connections or connectivity strengths are controlled (as you may have thought they were, if the brain was a giant network of threshold gates), but (b) great care is taken to control, across a large variation of different neuron cell types, precisely where along the dendrite each nerve cell type connects and in what manner (which makes no sense in a threshold gate model).
That seems rather more on-point with the model you're describing!
The all of the animations on this are amazing. I love them. They make neurons look so colourful and exciting, and its really engaging. I'm still a beginner at neuroscience but I really like both topics of neural networks in machine learning and computational neuroscience, so this video is perfect for me. I will definitely rewatch it.
As an undergraduate who is currently studying to go into computational neuroscience, I feel privileged that such a high quality channel exists for a relatively niche subject like this. Keep up the good work!
Superb video! I'm a first year Data Science and Neuroscience uni student (aged 38) and am very interested in the fusion of these 2 fields. Will come back to this video again to share some thoughts later. Anyway, even if all the functionality of the human biological neural networks and neurons isn't currently fully implemented in artificial neural networks, it doesn't mean that it won't be in the near future. Also, as you point out here, the same functionality is also achievable by deep neural nets, it's just a matter of efficiency.
Wait, your doing a university course that integrates both data science and neuroscience togather? and if it is or isn't, its still cool that your doing both.
@@sergeantoreo8062 The uni that I'm at specializes in double degrees like that. There's data science and physics for example, or psychology and neuroscience, literature and music, and so on.
@@ConnoisseurOfExistence such a lucky guy... i wish i could do that. I'm from Brazil and i'm also super interested in such topic, although we don't have that much of investment in these areas, especially in cities in the middle of nothing. I hope you have a great time while in this course.
@@marvin.marciano Thanks. Why don't you apply in the capital, or another country? You speak English. I'm not studying in my own country at the moment...
@@ConnoisseurOfExistence maybe when I get older, it turns out to be more viable than it is now. I'm currently at computer science graduation
This was fantastic - easy to follow and amazing production quality. I was wondering how you do your animations, but I see you have a video for that too! I want to go through all of them!
This is so interesting! I love how it combines and compares neuroscience and computer science. The production quality is so high
I loved this! It gives a very optimistic outlook to that artificial neural networks can at the very least simulate our own neurons if they need it to solve a problem.
It's important to notice that while a deep neural network can approximate a single biological neuron with a lot of layers and parameters, this only provides an upper bound on the total amount of parameters you'd need to approximate multiple biological neurons!
Not only may there be a more efficient architecture the authors haven't tried, but it may be that as artificial neural networks scale to become larger they become more efficient at handling the equivalent of an individual biological neuron. This because the overall behaviour of a system may be simpler to model by exploiting correlations and limiting behavior, just like a glass of water with respect to all of the H2O molecules composing it.
Sounds like what Sandler, Kirsch, and Schmidhuber have tried to do with promising results
Yes its very likely that your second explanation is at play. Grasping the function of a neuron with active dendrites would be much easier when the dimension of the computational manifold is scaled up.
Neurons don't get put together like a "neural net". They start out as whole undifferentiated cells and then progressively specialize into neurons. Neural nets are put together like machines - from combination of separate parts, components, functions, etc. according to a plan. In other words, although neural nets can simulate some abstract aspect of neurons that might interest humans, you can't really understand neurons using neural nets. For example, neural nets don't socialize. They don't generate projections and try to make it in neuron society, etc. It's kinda like studying humans using speech recognition software that's incapable of going out and making friendships, looking for work on its own, trying to find a spouse, etc.
@@StanislavMudrets would graph convolution or attention be enough socialisation?
@@revimfadli4666 Is this activity planned by a researcher or something that a bunch of math functions just decided to do on their own? Math functions performing other's will don't do anything spontaneously for their own purposes. They are completely parasitic on human purpose.
I'm ... NOT a scientist by profession or choice. But I AM fascinated by the digital/analog crossover... started with audio projects and a basic understanding of PWM... then realizing electricity/chemistry/physics/biology/neurology are merging at a level. This video to me was super helpful. It also has me contemplating how the paper mentioned essentially modeled a tiny brain microphone/recording booth, with digital playback and analog capture. It's like amplifier modeling. It's very very very difficult to capture all the possible inputs, but if you get enough of them, the capture and modeling is hard, but it can be sufficiently accurate to be useful, and then needs very little computational power after the model has been created. It's super intriguing and makes me wonder what would happen plugging the analog input/outputs of the dendrite/some would look like plotted as audio signals into some sort of modeler. Probably worse, more translation and compression happening etc. The fact that the neuron basically is a combination of an analog and digital signal is amazing to me. It does pulse width, essentially square wave when the specific voltage activates the dendrite spikes, and then duration and magnitude, and possibly- time-- I'm no interested in if there is a sort of standard "period" for each input/output. It was mentioned there are some differences, but is there a universal cap on what that is? Does a specific input change the effective period of responses? IE, would stress shorten the periods thus the perception of things happening faster? What is our internal "clock" speed. We know it isn't fixed at a top level, but on an individual cell, is it set? Are there a few that are defined based on the chemistry at work, and depending on the channels/ions/resistance etc?
So we have individual inputs for location, a comparator for location which acts almost as an input it seems, that is part of a summed input of what order/how long. Then on each input we have duration and magnitude (x/y) and additionally, on dendrites themselves we have the NMDA acting as an additional square wave input? What about the periods of that input? Would different specific voltage inputs result in a response that is different consistently in period?
Hey guys- I HATE school. But I might have to go back to school lol
Your videos are incredibly well made and are always a special joy to watch.
Thank you for your high quality content, please keep it up :)
Thank you!
I just cannot explain the fascination, love, and inspiration that this video delivers.
Former Big Tech AI engineer & Researcher. Love what you have to say here. A learned a lot about the neuroscience perspective of what we do. Thanks for sharing!
Depolarisation is decreasing voltage but increasing potential. Slight correction by a physicist there ;)
But if a neuron sits at a negative value in its resting state, surely its the other way round. Voltage gets closer to 0 so increases but potential decreases?
@@stafan102938 voltage is just a potential difference so it is a matter of labeling. It makes more sense to use it in absolute terms in this case
@@tedarcher9120 But regardless of using negative or positive voltages to talk about it, a depolarisation would always refer to a reduction in potential across the membrane. Plus the voltage value falls out of an equation which uses ion concentrations to calculate it, so that's why its consistently referred to as negative inside the cell and positive outside.
@@tedarcher9120 Maybe I've just the terminology backwards in my understanding of it all though
@@tedarcher9120 Going to blame my school physics teacher for giving cop out answers to explaining voltage and potential
that was a really great video! I was engaged every second of it, please keep uploading these topics!
Great video, It's cool that single neurons can act as their own mini deep neural networks. I think that it is worth clarifying that even though the biological neurons are not equivalent to perceptron neurons in function, all this does is move the perceptron to a sub part of the neuron. Meaning that the fundamental principle is the same (ie. a network of sub networks is still a single network on the whole)
This channel is so incredibly underrated. Fantastic stuff. Really great.
Amazing. Biology really is incredible in its exploration of forms.
I was having a hard time understanding how a neuron could be phase sensitive from your video on theta rhythm, but the "xor" neuron kinda makes that make sense. How narrow can the window of sensitivity be?
arbitrarily so.
I would definitely love to learn neuroscience in entirety by you. Thanks for the info. Always was fascinated by the workings of the brain but found it tought to wrap my brain around it, no pun intended. But your explanations make it clear and gives me a visual representation of the workings that i need. Thanks brother♥️ Looking forward for more.
I just found this channel out, and watched a few video's fully and I'm absolutely hooked and subbed directly, keep it up man! - The way you explain it is very easy to understand, even for outsiders!
Spatial and temporal summation allows for LTP and LTD in order to control the plasticity of the neural network. That’s why you need NMDAR which are voltage gated dependent on AMPAR (the non linear system). As a neuroscience undergrad, I love your videos 🎉
@Shimmy Shai u r gay
Great video as usual!
Could you explain the neural backpropagation in a future video?
Is it similar to the backprop in artificial neural networks? Or maybe to the Forward-Forward algorithm for adjusting weights proposed recently by Hinton?
I think the correct answer is "we don't know". If you figure out how actual neurons learn on a large scale you are a hot candidate for a Nobel Prize.
a quote I remember from somewhere is that if you can think of two ways neurons could be doing things, the brain is using both and a third or more that you didn't think of. though of course that only works for things that are biologically possible.
one point of interest is that you can approximate gradient backprop with predictive coding, and forward-forward doesn't seem to preclude predictive coding.
Neurons change weights or strength via long-term potentiation LTP and long-term depression LTD
@@doppelrutsch9540 one thing is for sure, they don't do a bunch of global tensor matrix multiplication to do back prop, haha
As someone who teaches a bit of neuroscience at university level, I thought this video was superb from a didactic point of view and contained a lot of interesting content. You covered the basics of electrical excitability and action potentials in a compact way but also very clearly. I was prepared to wince at sodium ions "flooding" in but you used the less misleading word "rushes". Students tend to be led into erroneously thinking that the ion concentrations change massively during an AP when this isn't the case (especially for large axons like the squid giant axon). Anyway, I really like your style and content and have clicked Like and Subscribe.
18:51 does it imply that a more accurate model of how neurons work would be actually a base 3 or ternary numeral system? It’s not 1 or 0, but 0, 1, 2 states equivalent to under current threshold, over current threshold, over saturation threshold 🧠
Absolutely brilliant video!
No doubt. You have created the best RUclips channel. Top quality in every regard. Cheers!
Once again, stellar communication and animation work!
I have a question: to the extent where the single neuron can perform XOR operations, can't it be argued that it is simply checking two thresholds instead of one? With both the lower bound and upper bound known it doesn't necessarily need to be multi-layer
Well, technically yes, but to perform this check of the two thresholds you'll need at least 2 layers, if we are talking about conventional perceptrons, so in that sense it's multilayered (in this terminology "multi"=2 :D )
thats is a second layer
@@ArtemKirsanov Ahh makes sense! Tysm!
Amazing. Best channel on neural networks on YT by FAR!
The visual are hypnotizing of nerves and all that stuff
A good choice for subscribing!!!!
Your animations are crazy. Keep up the great work!
What a great subject you have chosen to work on!
Great video. Now we can encode neural nets using a few new novel signal processing nodes !! Should make neural nets much faster and less power hungry..
Great! Thank you, now I have a good enough video material to throw it in people who are lacking understanding biology moment. :)
Especially for giving context of working of neurons nicely in one place. Some fancy neuroglia will not fit in this, but it's already great.
I learned biology in school as well to understand, but not as well to explain something for others
🥰
Amazing! learned about the XOR nural function for the first time, very intersting
Amazing you broke it down so simple for someone like me to undestand. I love this channel!!
bro, those explanations are so cool. I will definitely watch more of your yt videos. Just curious, where are you located?
4:36 - actually ions and molecules do permeate barrier spontaneously. Depends on lots of variables.
Useful stuff, I'm gonna relate some of these concepts to my paper that I'm writing. 😁 Thank you, beautiful video.
Thank you.this is the exact information i've been looking for so long.
What the hell is wrong with the YT algo??? This video is amazing! It should have minimum 200k views ... This is a slap in the face of all nerds dedicated to STEM fields... Great work YT promoting great twrking videos! Great work!
Thank you so much!
has reduced function and resulting in summing inputs because of leaky gates
-Amplify voltage
-Backpropogation : adjusts weights of input?
NMDA gates (Neurotransmitter + adequate depolarization) (adjusts synapse connection)
Dendritic Calcium Action Potential
predictiion and logical results
You're so good, man! Been trying to find a vid like this for a while
WAAAAAAAAAAAAAH- Your channel is literally top notch!! Thank s a lot for the content!
There's one more important property - temporal sensitivity. I did my diploma in hardware spiking neural nets (on a state-of-the-art Xilinx programmable chip) and this realisation was a wow moment.
Basically, any digital system has a clock speed limitation. Natural neurons have much slower propagation speed, but activation is dependant on constant balance of race conditions + no two signals are coming at the exact same moment + basic quantum effects act as a gate (if one ion membrane changes its state from a pulse, concurring pulse sources will be denied and "rerouted"/lost). In the process of digital learning, resolution for these race conditions quickly becomes insufficient, signals are starting to clash, when they are coming in a single clock step. Biological systems, on the other hand, have physical time resolution, which is effectively infinite. No GHz clock can compete with this biological mechanics (only ASIC non-clockable implementation can utiliize this process).
BTW, these effects manifest themselves with some neurotoxins: when receptors of a particular kind slow down, the whole system can be knocked out, or become very unstable
thanks for the addition, would you have any papers explaining these basic quantum effects and the way they act as a gate?
@@odettali1000 I meant really basic discrete/quantized effects: such as in "Voltage-gated ion channels".
Basically, you can't strip more electrons than the molecule has available for ionization. And you can't stick additional electron into non-ionized molecule. This electron will be "diverted" to go somewhere else
Great video! Love the effort you put into the graphics. Keep it up!
So glad to have discovered your channel, thank you for this brilliant and beautifully made video! Any thoughts on connections between what you've talked about and valence encoding by neurons?
Great Video Artem!, Loved it! One thing has been bothering me tho, at 18:25 , corresponding to the third red spike (A) at the dendrite, shouldn't there be a spike in the soma as well? kindly let me know if this isn't the case, because I am kind of having a hard time making sense of it otherwise. Thanks
This is so fascinating! Thank you for your awesome videos.
Thanks!
This is so cool! I am becoming very interested in artificial neural networks and I always thought that the initial “simplified” model of the neuron was more or less correct
Офигенные анимации!!! Невероятная визуализация, очень помогает понять. Где делали?
Спасибо!
Базовые анимации в Adobe After Effects, а активность нейронов - связка NEURON (для просчета самих симуляций) + Python + Blender
Your channel is awesome, it's purely gold !! and not only for neuroscientist also for everyone. Thanks for sharing all of your knowledge with that easily and great animations. Cheers from Mexico :)
02:59 Is there somewhere I can find those graphs at full resolution?
Sure, I believe the picture with activation functions is from here medium.com/@shrutijadon/survey-on-activation-functions-for-deep-learning-9689331ba092
And the one with network architectures is by Asimov Institute: www.asimovinstitute.org/neural-network-zoo/
@@ArtemKirsanov Thanx :)
I wonder what can be said about transformers abilities to model dendritic nonlinearity
Great stuff, keep these videos coming!
Very good, thank you. I hadn't heard of this interesting property so appreciate you sharing it. A lot to think about.
20:40 you have work to do in the DNN literature.
DNNs can fit to any random data.
And It is a only model, if additionally it doesn't learn like a neuron under new experience, that network is useless for ANY kind of neuronal theory.
I must say I am very impressed with your content and how you represent it. May I ask what software you use for your animations? I want to start a youtube channel myself, also with more abstract information, although in another field. I think those animations would greatly benefit my viewers. So any pointers would be greatly appreciated.
Thank you!
I use mostly Adobe After Effects in combination with Python (matplotlib / manim modules) for creating mathematical animations.
In this video, for 3D scenes with neurons I used Blender and a custom-written Python script to run a biophysical simulation in the NEURON software and bring the morphology and animation data into Blender
Great video, Artem
Not so long ago I found an overly increasing interest in machine learning and neuroscience and I love finding such great channels. Keep up the good work )
Thank you! :)
Can you do a video about dendritic clustering, i saw a talk about a model neuron called the “clusteron” where the dendritic branches cluster synapses along the branches, curious how this works and what it does for the neuron
Very interesting video! Was pleasure to listen to and to watch
Dude, this is brilliant! Nice! Definitely subscribed.
Amazing video Artem, synthetic and clear explanations with impressive animations. As someone doing research on Deep Learning, I got super inspired by this video and I look forward to the future of our field !
Thank you!
Well, it's kind of obvious if you just think about it for 2 seconds. Our "perceptron" is actually a single timed event which is more closely resembling the synapse to dendrite communication, using a master propagation chain as an offset, and the soma is another "perceptron" that details the chain into the axon....but everything is a timed sequence (as noted by the pulses, since channels need to recharge). All of this can be simulated using 2 layers across many "phases" (a phase is a deterioration of a signal as it fades across the channels, giving you the semblance of timing). When you have completed that, then you can realize that new neurons actually train themselves against already existing neuron clusters in order to output similar signals in order to get "the answer" faster in time, allowing the replacement of the slower one (this is how your memory changes slightly as you get older, possibly even losing the memory altogether). Each section of the brain has a different "job" to do, and when you solve that, you create the singularity....which is actually pretty easy. Don't worry, mine doesn't want to kill humans or anything.
Thank you for giving the references.
Love your channel dude
I would think this would have profound implications for the design of NN hardware actually, e.g. DOJO etc. The node requires an extra parameter (or more) and needs to allow inputs to be processed not only with weightings but XOR. Is it good enough to just comingle the nodes and allow the multilayers or attention to resolve this? My guess is that biology discovered XOR for a good reason, so following this precedent may lead to much different architecture of NN transformers/diffusers and affect the hardware assumptions. and who knows, maybe this would also get us one step closer. 20:19
Fantastic video as always, thank you for your great work!
Question: You mentioned that for the experiments in the last paper, if the NMDA-channels were neglected single-layer networks were sufficient to reproduce the functionality of the neuron. How is this possible? Have the channels responsible for XOR-functionality/capacity been thrown away with the NMDA-channel?
Even with the current achievements in AI this year we still have a long way to go to approximate our own neurons. Artificial networks a still build pretty linearly with information propagating instantly in one direction. Not to mention that they are still have to be pretrained for a long period of time. Real neurons on the other hand can be connected in loops of diverse lengths. Their signals can arrive in any order. They include complicated timing mechanisms and can learn as they go.
2:00 It goes way back 1907 lapicque paper describes the summation with weights with a threshold. Current NNs are 118 years old theory, updated thousands of times and still not right.
Hey, great vid btw very informative. How do you create the graphics for your videos?
You're video is absolutely amazing!!!
Thank you!
this was just the video I was looking for
It’s interesting to see a more mathematician-centric POV of how neurons are like neural networks, when from a neurobiologist’s perspective, it’s already obvious that real brain cells individually and collectively are already immensely powerful in a computational sense. So much so, it’s weird to see it phrased in this video that it’s the neuronal cells being “exonerated” relative to computational neural networks, when in reality it should be the other way around.
Wow so not only the connections of the cells but proteins themselves assist in the process. Great!
Hi, I didn't get the part that you said about backpropagating action potential. You said those Na channels on the "dendrite" causes bAp.! Is it so?! They can be open due to synaptic inputs and generate the action potential if they are strong enough and later this can cause generation of bAp.
Is the dendrite selection only the filter of voltage to make a XOR function, or does it also do a combination of low and high band filtering on frequency. Just trying to imagine if the dendrite always transmits the voltage in the proper band or that it is even more selective and only propagetes when the receiving pulses follow the right pattern? If so I could imagine that a dendrite would be able to translate a firing pattern received from another neuron into a different signal inside its own cell, and then the soma with its axion can transmit a totally different firing pattern making this simple neuron a huge signal translator. Modern neural networks don't take into account the time domain, which in my opinion is where we loose the link with biology.
Interesting stuff thanks to the wonderful landscape
Amazing video, thank you for your effort!
your channel is awesome! thank you! this will be huge
Thanks!
Impressive content! Thank you very much!
What about so called "pulse train" patterns? Or is what you are discussing just a better understanding of what leads to that behavior, and the 'train' is more an obsolete term?
your channel is outstanding , keep on❤❤
Our brain initially constructs physical resistive rails that a and b must travel along with a certain voltage or their actions will cancel each other out, creating XOR function? Trying to figure this out with only a GED in hand is tricky, but I'm trying!
Thank you for explaining this so well!
It's funny how the XOR gate is called the only "nonlinear" gate here because of non linear separability, while it is the only binary gate (among the usual ones) that is linear from an algebraic perspective (seeing bits as a vector space over F2). We can even use this to prove that with only XOR and NOT, you cannot build a circuit computing AND: XOR and NOT can only build linear functions, but AND is not linear.
So I take it this means neuron can do not only addition, multiplication, via a kind of logical AND, OR, NOT, but also sigmoid, relu, thresholds, and some functions involving sequences. Which ones? Not square root I feel this is hard. Logarithm? Yes that. Anything else?
Wow this video is great. It answered all the questions I recently had, as I am not in AI research. Cool!
Neurons are quite capable.
I am curious now, what is the right neural network setup to correctly simulate a single neuron? Ok something about seven layers or so, but how are they connected? Did I miss it?
it is a temporal convolutional neural net
Amazing video. I have a question, individual dendrites execute XOR operations using several axons connected to that dendrite as input?
Thanks! Mostly, yes. Although if a single axon has several terminals connected to the same dendrite and if the release of neurotransmitter is a stochastic process (which means that there is a certain probability
@@ArtemKirsanov Thanks! so, if a dendrite execute a XOR with several inputs (axons), it would activate when only one of them is active right?
@@joamarticorena9018 Exactly!
11:39 is that the fundamental reason we perceive time?
No. Time just exists because we keep past states of the universe. You could implement it without this feature.