As a math student who is interested in information theory and neural networks, i discovered this gem of a lecture series when i was looking for videos to fall asleep to! In fact i've finished the lectures when i was sleeping :D Now I decided to start it properly and just finished watching this lecture and taking notes. I would love to send David a mail when i finish the course. Thanks for leaving this behind my man, rest in peace.
I am not involved in information technology but this lecturer is making a difficult subject like information theory look so easy. You really must see this...
Thank you, Thank you, Thank you for sharing these excellent video lectures. Dr Mackay is amazing at teaching complicated topics. These lectures are great supplement to his excellent book on information theory which has so many excellent plots and graphs that enables one to visualize information theory. Information theory comes alive in pictures. Thank you for sharing these.
Repetition (redundancy) is dual to variation -- music. Certainty is dual to uncertainty -- the Heisenberg certainty/uncertainty principle. Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics. Randomness (entropy) is dual to order (predictability) -- "Always two there are" -- Yoda. Teleological physics (syntropy) is dual to non teleological physics. Duality: two sides of the same coin.
00:30 Information Theory invented by Claude Shannon to solve communication problems. Fundamental problem: Reliable communication over an unreliable channel. e.g. [Voice->(Air)->Ear], [Antenna->(Vacuum)->Mars Rover], [Self->(magnetized film)->Later Self] 04:00 Received signal is approximately equal to the transmitted signal because of added noise. 05:45 What solutions are there for having received and transmitted signal be the same? Either physical solutions or system solutions. 07:30 Source message -> [Encoder] -> Coded transmission -> [Channel (noise introduced)] -> Received message -> [Decoder] -> Best guess at original source 08:45 Encoder is some system that adds redundancy. Decoder makes use of this known system to try to infer both the source message and n.
It was amazing, Finally I learned Shannon noisy channel theorem: there is exist an encoding and decoding system that could reach to the capacity of channel. so error correcting and detecting course is about to learn these encoding and decoding system. wow Amazing lots of thanks to the teacher
Indeed so. It's so frustrating and irritating when the camera focuses on the lecturer when you really want to look at the slide with plots and equations.
I would use SUDOKUS for communication. Only few numbers have to be transmitted correctly, and the other numbers/information can be restored by the decoder :-D
The problem is that only the sudoku numbers that you transmit count as information: the rest of the numbers are constrained by having to follow the rules of Sudoku, and can't be any numbers you want, thus they can't be information. So you're still having to get 100% data transmission.
Cheering, in these days of advanced educational gizmolgy, to see Professor MacKay making extensive and effective use of a stick of chalk and a polychromatically absorbent surfaced board. {Tyneside, England.]
I'm new to information theory. At 4:24 why is the received signal is "approximately" and not "equal" to the transmitted signal + noise? like what else is there other than the transmitted signal and the noise? Thanks for sharing such helpful lectures.
It gets extremely difficult trying to model everything that affects the signal, but it's relatively easy to add noise to the signal, but not just noise but an assumption that the noise is additive (hence the plus sign), white, and gaussian( normally distributed). As you would know from statistics, a normal distribution has nice properties for estimation. This stuff gets really complex but I guess you get the idea. Note, we can make other assumptions about the characteristics of noise
Received = Transmitted + Noise makes a *lot* of simplifying assumptions! In the real channel, the noise might be multiplied, not added. The transmitted signal might go through weird transformations, like you might receive log(Transmitted), or it might be modeled as a filter which changes different parts of your Transmitted signal in different ways!
he is being sarcastic i think , he finds it silly that distributions are named after people who discovered them while it would have been more logical if they were named after what they represent (which binomial distribution does).
@@vik24oct1991 he doesn't find it silly, he is himself just being silly, because so many things are named after someone he's joking and extending that to binomial
38:19 I am unsure of the 1/2 in the denominator According to my calculations P(r=011) = (1-f)f^2+f(1-f)^2 This is done by the sum rule. The numerator should have 1/2 due to the P(s=1) term My final answer is P(s=1|r=001) = (1-f)/2 Could anybody help me with my concerns?
When you add up the probability of r given s=1 and given s=0, you must include the probability of each of those events. It results as P(r=011) = P(r=011, s=0) + P(r=011, s=1) P(r=011) = P(r=011/s=0) * P(s=0) + P(r=011/s=1) * P(s=1) P(r=011) = (1-f)f^2 * 1/2 + f(1-f)^2 * 1/2
I'm a bit confused. If we're trying to estimate the mean of the Binomial distribution, wouldn't we use the sigma/sqroot(n) formula for the +- bound? In other words, the mean +- the standard deviation of the estimator: 1000 +- 30/sqroot(10000) = 1000 +- 0.3? My statistics is a bit rusty.
Arthur K I believe you are confusing the standard deviation of the sample mean rate with the standard deviation of the mean total count. The standard deviation of the mean rate does indeed drop as 1 / (N)^0.5, while the standard deviation of the total count increases by (N)^0.5. Since we care about the total number of bits flipped, it's about the total count rather than the rate. You can see that multiplying the standard deviation of the rate, 1 / (N)^0.5, with the total number, N, results in a standard deviation of the total ~(N)^0.5. Please let me know if that clarifies.
the fact is the received signal is not identical to sent signal due to corruption and distortion in the signal, in a process , so how much of the the original signal is received ,what would be the measurement in what unit ,,,,,thats why i do drugs ,and dont give a damm !
Profesor McKay is certainly extremely competent in the subject but this really is the most un-intuitive way of introducing information theory. We are given the answer right from the start and work our way backwards to see that it is an effective system, instead of trying to understand the problem and find the adequate solution. We are never trying to understand the nature of the problem but instead made to test the effectiveness of the solution. Typical of classical academic philosophy. Let's make knowledge as boring and abstruse as possible so the riff raff is kept out of our little club!
@@tonewreck1 Thank you. I took a look at it and felt that it described the act of measuring information very thoroughly. Are there any videos you recommend for learning about system solutions for transmission errors?
@@vedantjhawar7553 I am glad you found it helpful. I suggest to see the other videos in this series. Once you get a feel for what information theory is really about, you can go back to MacKay for error correction and all the nitty gritty, he does the details very effectively.
@tonewreck1 For some reason I'm unable to see the reply in which you've suggested the source that @Vedant Jhawar has requested. Could you please share the source again?
This is painfully slow. I'm sure the information is fantastic but you have to watch him write everything he says on the chalkboard. When I got to him telling the class to discuss how much 10% of 10,000 is I couldn't take it anymore. Can anyone suggest a similar lecture that moves more quickly?
Lectors goal ultimatly is to share information and for audience to absorb as much as possible. There are different ways how people can do that and for each it's different. Some people are better at listenening others better absorb visual representation (either looking at chalkboard or writing notes themselves) :)
That's completely understandable - that's what it was like when I went to school. It's just that most lectors now give you the notes themselves so they don't have to take the time to write much and can spend the lecture explaining the material.
p is equal to f, and q is equal to 1 - f. Since we assumed f to be 0.1, 0.1 * (1 - 0.1) = 0.09 thus giving us the variance of 900 after multiplying 10.000 by 0.09
As a math student who is interested in information theory and neural networks, i discovered this gem of a lecture series when i was looking for videos to fall asleep to! In fact i've finished the lectures when i was sleeping :D
Now I decided to start it properly and just finished watching this lecture and taking notes. I would love to send David a mail when i finish the course. Thanks for leaving this behind my man, rest in peace.
RIP prof David, you were, and still a great inspiration to us
ah man sad to hear 😔
Fundamental Problem: Reliable communication over unreliable channel
Binary Symmetric Channel: 9:35
Disk Drive Problem: 11:30
Redundancy: 23:00
Repetition Code: 24:35
Decoding is inference
Inverse Probability: 31:33
Forward Probability: 40:30
Hamming Code: 47:10 (ruclips.net/video/eixCGqdlGxQ/видео.html%29)
Capacity of channel: 58:10
Shannon Noisy Channel Coding Theorem: 58:45
The Weighing Problem 1:00:50
thank you :)
Unreliable channel = Unreliable information
I was saddened to read that David MacKay passed away earlier this year.
what a great teacher.. he will live on :)
He was a very great teacher! Very Articulate!
I was shocked when I read your comment. I'm following his lecture and his book on this course, it is a sad news.
what a great teacher.. he will live on
RIP For great teacher of human beings. His passion and endeavor for tech would remain for descendant
I am not involved in information technology but this lecturer is making a difficult subject like information theory look so easy. You really must see this...
kinda boring though
One of the best lecture of " information theory and coding " I have ever seen....love from India 🇮🇳🇮🇳🇮🇳🇮🇳
How am I just now discovering this lecture series??! This is awesome!
better late than never! glad you are finding it useful
i got it now bro, i am latter than ya ; )
@@anantk2675 I'm later than you, hhhh
@@siweiliu9925 i am later than you
Dr. Mackay is a great explainer. Anyone interested in machine learning and Bayesian statistics can also read his doctoral thesis.
I don't mind the slowness. With some decoding, my brain is receiving a cleaner signal with that extra redundancy.
Bagana also does he have to write EVERYTHING
This may be the longest blackboard I've ever seen
Thank you, Thank you, Thank you for sharing these excellent video lectures. Dr Mackay is amazing at teaching complicated topics. These lectures are great supplement to his excellent book on information theory which has so many excellent plots and graphs that enables one to visualize information theory. Information theory comes alive in pictures. Thank you for sharing these.
Repetition (redundancy) is dual to variation -- music.
Certainty is dual to uncertainty -- the Heisenberg certainty/uncertainty principle.
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics.
Randomness (entropy) is dual to order (predictability) -- "Always two there are" -- Yoda.
Teleological physics (syntropy) is dual to non teleological physics.
Duality: two sides of the same coin.
Thank you for these great lectures your memories will live on ... RIP
what an amazing lecture and great teacher. May you rest in peace. You will not be forgotten.
00:30 Information Theory invented by Claude Shannon to solve communication problems. Fundamental problem: Reliable communication over an unreliable channel. e.g. [Voice->(Air)->Ear], [Antenna->(Vacuum)->Mars Rover], [Self->(magnetized film)->Later Self]
04:00 Received signal is approximately equal to the transmitted signal because of added noise.
05:45 What solutions are there for having received and transmitted signal be the same? Either physical solutions or system solutions.
07:30 Source message -> [Encoder] -> Coded transmission -> [Channel (noise introduced)] -> Received message -> [Decoder] -> Best guess at original source
08:45 Encoder is some system that adds redundancy. Decoder makes use of this known system to try to infer both the source message and n.
It was amazing, Finally I learned Shannon noisy channel theorem: there is exist an encoding and decoding system that could reach to the capacity of channel. so error correcting and detecting course is about to learn these encoding and decoding system. wow
Amazing
lots of thanks to the teacher
This is a great lecture. The design of case problem is really helpful here. Thanks for the lecture.
Thankyou very much prof David MacKay, Your lecture helps me understand information theory. Btw I live in the country where toilet is not so common
You are from where??
So nice to watch a video like this that does not have music.
This is great, thank you! The lecture is far more entertaining than just reading the book.
whihc book is it?
Thank you for a great lecture, looking forward to follow the rest and study the book!
Could anyone explain how is the flip pobability is 10^-13 and and 1% failure is 10^-15. Video reference is at 22:10
Genius camera work
Indeed so. It's so frustrating and irritating when the camera focuses on the lecturer when you really want to look at the slide with plots and equations.
awesome example of teaching style
this really helps. I really wanted to learn information theory. this video series is really easy to understand and awesome.
I looove the lectures! Thank you for the upload!
These lectures are great. Thanks for sharing.
I would use SUDOKUS for communication.
Only few numbers have to be transmitted correctly, and the other numbers/information can be restored by the decoder :-D
The problem is that only the sudoku numbers that you transmit count as information: the rest of the numbers are constrained by having to follow the rules of Sudoku, and can't be any numbers you want, thus they can't be information. So you're still having to get 100% data transmission.
9:50
32:13
13:24
at 46:50, what is rate? i guess it is (1/no of repetitions) but what does it mean in layman terms
Cheering, in these days of advanced educational gizmolgy, to see Professor MacKay making extensive and effective use of a stick of chalk and a polychromatically absorbent surfaced board.
{Tyneside, England.]
Any idea about the final puzzle? My answer is 3. :-)
I like this channel. I'm already teaching this topic for student in Iraq. In Arabic.
3/8 3/8 2/8 cake and siblings
That's the most equal way to share
In the 3rd cake they are most close to equal
4/82/8 2/8 is far behind equal
I'm new to information theory. At 4:24 why is the received signal is "approximately" and not "equal" to the transmitted signal + noise? like what else is there other than the transmitted signal and the noise?
Thanks for sharing such helpful lectures.
It gets extremely difficult trying to model everything that affects the signal, but it's relatively easy to add noise to the signal, but not just noise but an assumption that the noise is additive (hence the plus sign), white, and gaussian( normally distributed). As you would know from statistics, a normal distribution has nice properties for estimation. This stuff gets really complex but I guess you get the idea. Note, we can make other assumptions about the characteristics of noise
Received = Transmitted + Noise makes a *lot* of simplifying assumptions!
In the real channel, the noise might be multiplied, not added. The transmitted signal might go through weird transformations, like you might receive log(Transmitted), or it might be modeled as a filter which changes different parts of your Transmitted signal in different ways!
Ah! thanks, was looking for a good series... this is just the one. A great cliffhanger at the end to be precise... ^_^
I am curious to know how Shannon would have interpretted the internet as part of his theory... noise perhaps
This is fantastic I love it.
the link of getting free book is not working,can anybody tell from where i can get that free book?
Thanks in advance
awesome! thanks for sharing
I'm a bit curious about when these Lectures were recorded. Was it 2003??
Thank you professor
J'ai compris pourquoi répéter 3 fois la même information de suite à mes enfants n' était pas forcément efficace :-)
can someone explain how to solve the homework problems?
towards 56:00 he uses the term "bit error" and "block error" but doesnt define them properly..
hey is there any pre-requisite of it
Thanks for these.
is dis how u mak a plane
2:00
I wonder if he was making a joke when he said "Mr. Binomial" or if he actually meant to say "Bernoulli".
he is being sarcastic i think , he finds it silly that distributions are named after people who discovered them while it would have been more logical if they were named after what they represent (which binomial distribution does).
@@vik24oct1991 he doesn't find it silly, he is himself just being silly, because so many things are named after someone he's joking and extending that to binomial
38:19
I am unsure of the 1/2 in the denominator
According to my calculations P(r=011) = (1-f)f^2+f(1-f)^2
This is done by the sum rule.
The numerator should have 1/2 due to the P(s=1) term
My final answer is P(s=1|r=001) = (1-f)/2
Could anybody help me with my concerns?
When you add up the probability of r given s=1 and given s=0, you must include the probability of each of those events. It results as
P(r=011) = P(r=011, s=0) + P(r=011, s=1)
P(r=011) = P(r=011/s=0) * P(s=0) + P(r=011/s=1) * P(s=1)
P(r=011) = (1-f)f^2 * 1/2 + f(1-f)^2 * 1/2
I'm a bit confused. If we're trying to estimate the mean of the Binomial distribution, wouldn't we use the sigma/sqroot(n) formula for the +- bound? In other words, the mean +- the standard deviation of the estimator: 1000 +- 30/sqroot(10000) = 1000 +- 0.3? My statistics is a bit rusty.
Arthur K thanks for the comment. Which part of the lecture are you referring to?
Hi. I was referring to the discussion at 14:00.
Arthur K I believe you are confusing the standard deviation of the sample mean rate with the standard deviation of the mean total count. The standard deviation of the mean rate does indeed drop as 1 / (N)^0.5, while the standard deviation of the total count increases by (N)^0.5. Since we care about the total number of bits flipped, it's about the total count rather than the rate. You can see that multiplying the standard deviation of the rate, 1 / (N)^0.5, with the total number, N, results in a standard deviation of the total ~(N)^0.5. Please let me know if that clarifies.
Was Mr. Bionomial a joke that fell on dead ears or was it a genuine confusion with Bernoulli?
+omniflection I laughed when I heard it. I take it was just a very dry joke.
+Mark Chimes Hahah. Me too. This is when I clicked the pause and check the comments...
+Mark Chimes Well hello Mr. Chimes :)
Helpful lecture
very great theory
"People need 20 GB drives nowadays" Laughs in Call of Duty
Hi, I just wanted to ask what was meant when at 24:20 when he says that "1 is the same as 5, and if there was a 4, there would be a 0."
1 and 5 are both ofd so assigned 1, and 4 is even so assigned 0.
@@MN-sc9qs Thanks.
I'm fairly new to this. What is a "flip" ?
a flip is to change the bit from a 1 to 0
or a 0 to a 1.
What are the requirements to understand this lecture?
As it says the only piece of Mathematics is binomial distribution 16:08
After listening to this lecture, my IQ went up 20 points!
🎉
K
You know he's a jokster cuz at 1:00:13 the slide says his textbook weighs "~35 lbs".
XD, that's the price. 35 british pounds.
Thanks for the lectures. I am just curious about what happened to lecture 2. It says it was deleted.
Hi Hussain, thanks for catching this. I have reached out to youtube support to find out. Hopefully will be resolved soon.
Hi Hussain, Apparently there was a bug in the youtube system and they deleted it by accident. The video is back online now.
Good lecture except for the highly distracting camera work.
Information theory
I have read the extraordinary book.
im learning this at school... yet I'am here watching a lecture on RUclips ... I dont know why ... I should pay attention in my class I guess.
you're definitely exercising inference
Binomial is more related to probability topics. Bernoulli is about hydraulics.
en.wikipedia.org/wiki/Bernoulli_distribution
rip david
the fact is the received signal is not identical to sent signal due to corruption and distortion in the signal, in a process , so how much of the the original signal is received ,what would be the measurement in what unit ,,,,,thats why i do drugs ,and dont give a damm !
measurement would be in bits and I am not joking
channel immigration
I was able to achieve f=0.1 for this video with x1.5 speed ...
hehe if you re like me, then you have that exam in one week
Profesor McKay is certainly extremely competent in the subject but this really is the most un-intuitive way of introducing information theory. We are given the answer right from the start and work our way backwards to see that it is an effective system, instead of trying to understand the problem and find the adequate solution. We are never trying to understand the nature of the problem but instead made to test the effectiveness of the solution. Typical of classical academic philosophy. Let's make knowledge as boring and abstruse as possible so the riff raff is kept out of our little club!
Do you know of any sources to check out that might teach the content in the method you are talking about? Thank you.
@@vedantjhawar7553 try this ...ruclips.net/video/PtmzfpV6CDE/видео.html
@@tonewreck1 Thank you. I took a look at it and felt that it described the act of measuring information very thoroughly. Are there any videos you recommend for learning about system solutions for transmission errors?
@@vedantjhawar7553 I am glad you found it helpful. I suggest to see the other videos in this series. Once you get a feel for what information theory is really about, you can go back to MacKay for error correction and all the nitty gritty, he does the details very effectively.
@tonewreck1 For some reason I'm unable to see the reply in which you've suggested the source that @Vedant Jhawar has requested. Could you please share the source again?
This is painfully slow. I'm sure the information is fantastic but you have to watch him write everything he says on the chalkboard. When I got to him telling the class to discuss how much 10% of 10,000 is I couldn't take it anymore. Can anyone suggest a similar lecture that moves more quickly?
use the youtube speed feature, set it to 2x, done
Lol, nice, good idea.
Lectors goal ultimatly is to share information and for audience to absorb as much as possible. There are different ways how people can do that and for each it's different. Some people are better at listenening others better absorb visual representation (either looking at chalkboard or writing notes themselves) :)
That's completely understandable - that's what it was like when I went to school. It's just that most lectors now give you the notes themselves so they don't have to take the time to write much and can spend the lecture explaining the material.
Now the question wasn't aimed at the 10% of 10k but, rather at the +- part, i.e. std.dev, IMO. Did you get that correctly as well?
Where in the fock am I?
There is no information here and when will it be realized?Never!!!
In the first example, p is equal to f? p should be 1-f = 0.90
p is equal to f, and q is equal to 1 - f. Since we assumed f to be 0.1, 0.1 * (1 - 0.1) = 0.09 thus giving us the variance of 900 after multiplying 10.000 by 0.09
15:52
34:43
8:22