Troubleshooting the Centroid Acorn and Ether1616 Network | Why won't it connect? | Beelink + Realtek
HTML-код
- Опубликовано: 25 фев 2022
- Today we're troubleshooting why the Beelink computer cannot see the Centroid Ether1616 I/O board in my new CNC controller build. It turned out to be a really subtle (fun) issue. The system is running Centroid CNC12, with an Acorn and Ether1616 I/O Expander. And when we're done, it will all be working.
This appears to be a bug in the Realtek GbE network hardware of the Beelink computer. But there's an easy workaround.
00:22 Intro
01:15 Demonstration
02:53 Extracting the board
04:43 Checking voltages and PINGing
07:35 Sniffing the debug port
13:38 Testing with another computer
15:47 Testing with another network card
17:11 Sniffing the network traffic
18:55 A brief lesson in ARP
21:52 Analyzing the network traces
24:24 Network driver settings
26:33 Fixed!
Tools used in this video:
*This site contains affiliate links for which I may be compensated
EEVBLOG Brymen BM235 Multimeter (Amazon*): amzn.to/2YKFSEk
Saleae Logic Pro 16 Logic Analyzer (Amazon*): amzn.to/3h3kaXN
Internetworking with TCP/IP Volume One (Amazon*): amzn.to/3p4uMKD
Beelink SEI8 Mini PC (Amazon*): amzn.to/35kOMBK
Portable 14" Touch-Screen Monitor (Amazon*): amzn.to/3BCczJg
USB C to Ethernet and USB Hub (Amazon*): amzn.to/3BH4JOM
High Temperature ESD Safe Mat (Amazon*): amzn.to/2Xz0n6D
Raw Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License
creativecommons.org/licenses/b...
"if you don't know what this is, it's called a book. This was used in the before times to encode information on slices of dead tree"
This totally got me
It's seriously shitty legacy tech. Have you tried doing a search through it...?
My computer science lecturer used to (and apparently still does) introduce the concept behind a binary search by ripping a phone book in half until he finds someone’s grandma’s number… then called it in front of the lecture theatre!
It's fine when you've only got to search one book, they create indexes at the back for that... not as good as google but surprisingly effective for the common search terms. It's when you have to search for information in archival book repositories it's a problem.
These things called Libraries you see... except rather than pointing you to a single book the search index for Libraries barely points you to the correct shelf and sometimes people fail to put the books back in the right place anyway.
Checking voltages. Not only would you hear Dave Jones ' voice but you'll also hear Louis Rossmann yelling: "did you check PP3V3_G3_HOT" ? :-)
PP buss. I never opened a MacBook once, but that's lodged deep in my head.
Having worked in networking for the past 20+ years, as soon as James mentioned that this board was using Ethernet and that there were communication issues, I was suspecting that there was a Realtek chip involved. These are very cheap Ethernet chips often found in low-cost consumer hardware and they always seem to have weird issues if you are doing any type of networking more complicated than simple web browsing, and even then they tend to have weird latency and jitter issues. This is why Intel chipsets are the defacto standard for 1G networking (including 10/100 speeds too). Manufacturers like Realtek Ethernet chips due to their cost, users are better off with anything but Realtek, due to the shortcomings inherent to Realtek chips.
Same for me, in early days it was always a Novell 10-base 10 card that worked and the ne200(0) cards that failed in odd ways, now it's realtek that drives me nuts with linux issues where my intel based units work fine.
While I (sysadmin here) agree that the Realtek card are not always stable, and sometimes have weird implementations. The arp broadcast functionality is one of the very base block oh the tcp network stack. I doubt that even them would fail this spectacularly on a simple mask calculation. I’m wondering if the implementation of the Realtek chip on the nuc board itself isn’t at fault. Or some hardware init’ that should have been handled by the bios on boot that wouldn’t happen ? Those Chinese nuc (I used a few for dynamic displays before replacing them with pi’s) sometimes have basic functionality mangled.
@@amahashadow It's more than that with Realtek chips. The don't handle jumbo frames, ARP, 802.1q, udp, promiscuous mode, any offloading (tcp, udp, arp, crc, etc.), multicast, ipv6 anycast, and more.
@@amahashadow I would say in this case that the centroid software sends out a packet that LOOKS like an arp packet, and got flagged as such by the offload processor, but in actuality is not an ARP packet at all. Hence when you turn the offloading off, it ends up coming out the cable as the packet that was intended, rather than being groked by the offload mechanism on the card and munged. The cheap cost of realtek chipsets comes at a price, "rack of testing". There's a bug somewhere - the driver, the hardware? Realtek doesn't care as it works in 99% of applications - good enough, ship it!
It's not just their ethernet, it's also their USB products. They are truly awful!
"...record information on slices of dead tree". Absolutely brilliant!! Priceless!!!
I loved the historical element of the "Book". Great explanation. I'm looking forward to see it work.
I also appreciate the explination of 'how things work' in a layman's manner so even I can understand the process.
Thanks
James, great example of troubleshooting technique, particularly as it shows how to approach a situation where you don't know what is wrong and aren't starting with a system that did work and now doesn't (most troubleshooting charts assume that something used to work and now doesn't).
Back in the neanderthal times before the interweb (early 1980's), when I was the founding VP of the Hershey Apple Core (still member #2, although a lot of things have changed in the group), I helped Member #1 troubleshoot his newly built Apple ][ clone since I dabbled in electronics (wires inside of evacuated glass envelopes and chunks of germanium inside little metal cans), and had the necessary equipment. We weren't able to use the troubleshooting charts, so resorted to a symbolic diagram of the circuitry (schematic?) and poked around with my oscilloscope. When we lost the clock signal, we found two unsoldered resistors, applied iron & solder and Voilá, everything was right with the world.
Many of the details of your adventure were a little too esoteric for me (I was OK up to and including hexadecimal, but then got glassy-eyed). However, the "book" thing really mystified me until I remembered a "motion picture" from the past called "ZARDOZ," where James Bond had discovered an entire building filled with items as you described, falling to dust as you would expect of dead wood. I now realize that these are the things that my parents had a vast collection of, which descended upon me and my siblings and are now stored in unused "guest" rooms.
All my best,
Charlie
You are a naturally talented teacher. Thanks for presenting everything in such clear terms. I would add that the electronic lead screw is one of the best projects I have ever found on the net in thirty years of searching.
Thanks!
I second this for sure! Really easy to follow your thinking to arrive at the solution
I'm a retired Network Engineer and have used WireShark for many years. I was around when all the tools you have been using in this video were not around. WireShark is probably the most valuable tool that I have used. Like all things network, finding the arp offload issue is amazing. Good job. My handle when working was Jedi...
Good job Jedi...
Yeah, I remember the days when we had to lug around the Compaq portable sniffer. I think it was a Compaq. Looked basically like the Dolch, but with light colored plastic.
When Compaq was bought by HP that was a sad day. Having been certified in Compaq and also HP, those older bricks were more trouble than they were worth. Wireshark made it so much easier to troubleshoot weird networking issues.
Stay safe...
I am not very knowledgeable about electronics, I just enjoy your explanations man. You rock!
this is probably one of my favorite videos you have made. I think many of your viewers aren't necessarily interested in a particular subject, but more widely in just learning new things. And as this was an area I happen to know a lot about (unlike machining, which is why I am here to learn) I was thrilled to see how you worked through the problem and used the opportunity to introduce folks to some networking concepts including ARP. Really really great. And it looks like you found a bug in that Realtek driver. Thanks so much for this great video. Would love to see more like this... just figuring things out and us along for the ride!!!
IMO this is the real point of difference for someone like James, and a 'dedicated' CNC (or whatever the topic you follow is), he can be agile, he can cover this weird networking bug today, then next week cut a swan out of a block of steel.
totally agree, I'm generally not interested in the main topics he covers (expensive hardware I will never own), his approach and methodologies is what I enjoy.
This went in a completely different direction than I would have thought. This mix of machining, design, electronics and now even networking stuff is what makes me love your channel.
I have never looked in detail to those network driver properties in Windows but this ARP Offload setting being the problem was a good find!
Your network debugging skills are pretty good. "it's called a book. This was used in the before times ... " -- luvit!! And yeah, 'stupid stuff' like that can cost enormous amounts of time, which is why for many -- just get the one which some other engineers already qualified and follow the cookbook. Whenever you stray off the recommended one, you are going on an adventure, and most DIYers are not going to have anywhere near your network skills. Even so, I appreciate the many comments below from experienced network troubleshooters.
I understood 2% of the words said in this video but watched it in its entirety. Nice content, James
Well, that was a trip. When I saw the logic analyzer come out, I didn't expect this outcome...
James, I think you are a genius at troubleshooting. It doesn't matter if it is a CNC machine, a circuit board, a Cadillac car or a fire truck, you can always figure out what is wrong and how to fix it. I guess that is what you do for a living but you are great at it.
Wait, cadillac car or fire truck? Are there some videos on this channel somewhere I've never seen??
Not sure this video is juicy for the algorithm, but your viewer base certainly appreciates it. As always, learned something new from it.
Yeah, I'm a little shocked at the caliber of comments and questions I'm getting. I was totally expecting to hear "Less talk, moar chips!"
It's amazing how much I enjoyed a video that was mostly in Greek... but troubleshooting is, to a point, troubleshooting and I do understand that language.
Very interesting issue. I was starting to wonder if the netmask was wrong since it was trying to ARP the broadcast address, but an implementation bug makes more sense.
The net mask is set to /24. I did check that.
Amazing - thanks James for a small glimpse into a world I know virtually nothing about….. yet I had no issue following along. 👍👍😎👍👍
Thank you for taking the time to explain each element. I find that troubleshooting definitely forces us to understand what is actually happening. As others have already stated this was a great mix of software and hardware interrogation. I appreciated seeing your logic and technique in the piece by piece approach to locating the cause.
I ran a CNC shop for 8 years (2000-2008). At that time, I had little interest and no time to understand how the all of the elements come together to make such a machine operate. I'm now in my mid 50's I am discovering a great interest understanding how all the pieces fit together. I find myself obsessed with learning all that I can, and working towards a reason to build something that provides me the sense of accomplishment.
I am very interested in learning, understanding and applying how test equipment works. Your logic analyzer seems like an interesting tool. I want to understand that tool and the oscilloscope!
This is an awsome troubleshooting session. I'd never been able to find a simiar bug myself, but seeing you doing so - my compliments!
My degree is in computer science. I do applications-level programming for my job, but my interest has always been systems; kernels and drivers, hardware/software interface, embedded systems, parallel, distributed, real-time computing, etc. I love seeing stuff like this
There seems to be more interest than I expected. Or rather...RUclips has shown this video to more interested people than I expected. :)
@@Clough42 seems like everyone enjoys seeing a big being found
Nice one James… I like how you demonstrated a “binary” cut on the results of your testing and observations. Followed the decision tree down to the last twig.
Long experience. :)
Pretty nice debugging 👍 Reminds me of a piece of network hardware I had once in my hands that repeatedly brought down the complete network. Found out that its broken IP stack replied "yes, this is me" to every ARP request on the network, regardless of the actual IP in the request 😅
That's really interesting. I've never had a reason to hate on Realtek until now. Sounds like I'm in good company.
I've seen that behavior from Unitronics PLCs. A lot of industrial "ethernet" is very broken.
Wow, was that setup to force a proprietary switch or point to point hookup only?
@@seanclark8452 No. It was an industrial RFID reader controller which could be used in any setup. But, to be fair, I think it was pre-production firmware. The issue got fixed after I reported it.
@@RoterFruchtZwerg Oh, that makes sense then. I'll hard code things like that in code under development to make testing a specific section easier. It wouldn't be hard to forget to remove a test mod, especially if there were a bunch and you just missed one.
Wow! What a sneaky little bug.
Great demonstration of the fault finding process. I can't say I've ever seen that particular fault before, although I'm always suspicious of any power saving features on devices that misbehave, and disable any obviously relevant configuration settings. Bitter experience has taught me that there's usually something that doesn't quite go right upon resume, and sometimes on initialisation too. ARP Offload wouldn't necessarily be one of them at first sight, but now we know.
Thanks for taking us on the journey of discovery James.
I smiled when you whipped WireShark on the problem, I've worked with it since it was called Ethereal. I laughed out loud when Doug Comer's book (a method of encoding information on slices of dead trees!) came into play.
WireShark was/is an essential tool in my late career as a network engineer for, among others, Cisco Systems. I used WireShark and tools like it (you can imagine my bosses reaction when, in 1990 I needed to order a Compaq "Lunchbox" and accessories for around thirty-five thousand dollars), if not daily, at least on a weekly basis for thirty years. Comer's book and also "Routing TCP" lived on my desktop.
Anyway, nice catch on the Realtek chip configuration. I'll store it away in my memory for future troubleshooting. ARPing for the broadcast IP was the key. I saw the high-values in the pcap and it made me wonder but I have no experience with that chip set.
Absolutely fascinating! Your "Mini-Snars" are always so easy to follow! BTW, you can use that descriptor if you like!😁
“This is a book.” LOL! Good one, James. Great job of troubleshooting. 👍🏻👍🏻
I am totally impressed with your debug capability, considering you are mainly a cnc machine shop guru. Very good detective work. I wish I had your patience.
The machine shop is a hobby.
@@Clough42 which is even more impressive!
Your explanation of what is a very complex problem is simply amazing.
It was nice to see all the steps in easy to follow order. Keep up the good work!
I just randomly found your channel and wow I'm glad I did. Your professional troubleshooting knowledge is top notch. I will have to dive into some of your other videos. All the best with this cnc project going forward.
Loved all the detail as you chased down the bug. Brings back memories of my former life as a network architect.
You clearly know what you're doing so I'm sure the first thing you tried was blowing into the connectors.
Great video. Its great to have a simple explanation of basic internet protocol. Its great after almost 81 years to learn something new.
That was an excellent step through of a tricky networking issue and a very good explanation of ARP and TCP/IP!
I've been doing IT for over 20 years and that was the most clear and concise explanation I've ever seen, well done!
As others have noted before, and below, your ability to connect the dots in such an effortlessly understandable manner is SO appreciated. Carry on, sir.
Very nicely done with a 'surprise ending' as frosting on the cake! Although I have a four channel scope with decode functions, adequate for most data com issues, your utilization/ demonstration of the Saleae analyzer is going to cost me some money! Keep up the great work!
Excellent video James! It has given me the confidence to jump down the debugging rabbit hole if I need to. 👍🏻
One man's agony is another person's joy have fun Thanks Frank for the video
Troubleshooting involves a pretty universal set of tools and approaches to start with the broad picture and continuously narrow down the investigation until all that is left is the troublemaker.
You are a good troubleshooter!
Thanks for sharing! :)
I always learn something in your videos, but it is like a box of chocolates, you never know what you will learn.
Little glitches like that can be so frustrating. Good that you were able to figure out the issue.
When you brought the book out……that cracked me up.
As someone who works with computers and networking for a living, this video was AWESOME. Great example of systematic troubleshooting plus we get a weird cause as well. Thanks for the great video and you have my subscription :)
I absolutely hate computers and anything to do with them, but I want to say I found your issue, explanation and repair absolutely fascinating !!!!!
18:90 LOL brilliant! Also, Dave Jones has been waiting a decade for a proper diagnostic video opportunity. He needs to get out of his comfort zone and try fabricobling some cnc stuff. So many opportunities! Well done, sir.
James I have been a user of the Acorn Centroid product for a number of years and the company has an active user forum as well as tech support via email for troubleshooting. I am surprised you were not able to resolve the issue with what is available and were compelled to go into such detail to resolve the issue. However, I am sure they will appreciate your feedback on the problem.
I opened a thread in the forum and was immediately assisted by a couple of people. After going through the usual suggestions to turn off IPv6, firewalls, virus scanners, etc. they were stumped. I reported my solution back to the thread and added a link to the solution in the first post.
Enjoyed the info. Retiring soon and will be starting my cnc router build. I'll watching...
Brilliant problem solving! Quite impressive. I especially enjoyed the serial port R/E -- I've never seen that before.
A very accomplished beautiful logical troubleshoot.
Very interesting stuff. I used wire shark one time just to find an IP address of a device I was working on and had no clue what I was looking at. Your explanation from that paper book thing really helped me understand more of what wireshark can do. Thank you.
Great troubleshooting!! Checks all the boxes!!
Good teaching !!! My first time understands TCP IP
Good work on tracking this one down. FYI, the x.x.x..255 would only be a broadcast address if the netmask is properly set to 24 bits. If your netmask is set to 16 bit, bit would be a perfectly reasonable thing to do to ARP for any address that ends with .255, but that is in the middle of the /16 as those are usable addresses. I'm sure you set your netmask correctly, but you didn't show it, so I figured I'd mention it.
Many people have pointed this out. The netmask is indeed /24.
Very cool. Thanks for posting this. Always enjoy (other people's) debug sessions! :-)
Well I confess most of that went over my head but I did learn something and it was interesting to see you troubleshoot the problem. Thanks for sharing.
That was some great and interesting troubleshooting. Definitely shows to be systematic and don't assume anything.
So many levels of wonderful. What an excellent troubleshooting exercise
Excellent video; many thanks for sharing! I'm just starting my second Acorn build, and this is just the sort of thing that would have caught me out if I'd been adding the 1616 board (not needed on the current machine, but may be required on the next build).
i didn't fastforwarded any second of this video .. really good job 🙏
@18:55 "...if you don't know what this is, this is called a book." 🤣🤣🤣 Great video on troubleshooting....
Thought it was going to be a simple issue like a cable but as always you have impressed with going the extra distance to sort and show us too 🙂. Currently building a control box too, my first one and already learning a bit 🙂
Great Video!
Curious if manually adding the MAC address to the ARP cache would have solved the problem (i.e., arp -s ).
Protocol handling in the logic analyzer was really neat to see in your video. I sort of knew it could be done but the last time I used one, I don't think those feature were common. At least not on the entry level equipment I was using.
One of the things I love about your videos is your ability to deep dive into any category along with your willingness to share the journey as your learn and figure it out. You are also one of the few people I know that thinks sewing machines are cool too. After all, it's a tool!!! THANK YOU.
In that case you add it to the software cache. The would problem was that the software stack was not used at all. Instead a broken implementation on the ASIC was used. So your suggestion would probably not change that.
Could still try though
Interesting idea. I did notice that when it is working, the broadcast adress does indeed end up in the cache.
I have 28 + years in IT. I watched the entire episode, and I understand your conclusion. I don't think it entirely a ARP issue. I think it is more of a WOL issue or adapter compatibility issue.
I have used several different small form factor pc's and pi's and have seen driver issues with several of them.
You fixed it but I would careful going forward, you have other driver going forward as Windows Continue to be updated on you NUC.
I loved the Doug Comer book. I reviewed it once and gave an enthusiastic must buy recommendation and noted there was a bug in a sample implementation in it but it was a bug with zero effect. Less than two minutes later I got an email from Doug Comer saying, "What?" I explained the bug which was not following an rfc protocol exactly (but with no ill effect) and he got back to me quickly saying thanks. In the next edition it was fixed, but I didn't get any thanks in the book. Sigh. I could have been famous, lol!
Wow, pretty obscure! Great debugging, *really* appreciate these debug videos!
Great job troubleshooting the problem. I am sure the most of us would never have found it. Keep on keeping on.
An odd bug that could certainly manifest some really odd behavior. There is absolutely no reason it shouldn't know the broadcast address regardless of IP or mask. This is also the exact reason as a professional LAN engineer I continue to stress to our desktop guys that we need to be buying laptop and desktop machines with proper network chipsets in them. The wireless chipsets can be even more wonky, and embedded SCADA systems are even worse. At one point back in the 90s we managed to get a small number 3com NICs that all had the same MAC burnt into them. That was fun LoL. Enjoyed the whole video but the logic analyzer was my favorite, never had the opportunity to use one.
Very interesting troubleshooting!
I plan to purchase a system from Centroid and a as a cost saving measure I plan to purchase a non-Centroid mini pc. The detail is deep and hope I do not have the same problems. However, thanks to you a roadmap is in place.
Very (to me) impressive troubleshooting👍👍👍
I don't know why you don't have a million subs. That was a thorough and detailed account of how to troubleshoot. I use Wireshark from time to time and I was happy to see you use it. Also... this booook you spoke of, I have "a few", they are amazing. There's no boot time. They can't take the site down. Its cool tech. :)
I’ve done my fair share of network analysis. I would not have caught that!
Thanks this was great!
My knowledge of (And usual interest in) electronics could be written in large type on a very small piece of paper. That being said, sir, you are the only one I know who actually makes me think I understand what is happening. All others make me think they are speaking in Vulcan or something. Thank you!
Super video, James. Thanks for publishing this.👍🏻
Well done! Never give up until reach the solution!
Very informative James, thanks for sharing
Cheers
I enjoyed the troubleshooting video. Thanks.
Nice work Watson
You are a teacher. I’ve always been a mechanical guy at heart with family and friends in comp science I’ve always loved the idea of comp sci but never had a place to start so I ordered that book and that will be my place to start, back in the beginning of something. I’ve always cheated my way through programming buy forums copy and paste and calling friends. and I love building cncs. I need to know this stuff!!
This was most interesting,thats why i subscribed,keep it up,great channel.
Damn! This was like watching a murder mystery. *Lieutenant Columbo:* "One more question. There's something that bothers me."
This is the kind of thing that sets my hair on fire. Good on you for having the perseverance to slug through this.
Just where do you think HIS hair went? He just didn't film that part.
Thank you for doing this video. It was fantastic training and informative
I always love your videos. Massive thumbs up
I love the fact that you have the EEVBlog DMM. I have the newer one on my workbench....
Yeah, that looks really nice. Maybe someday.
@@Clough42 - it's a great meter, but I find its a bit overkill for most daily tasks. 99% of the time, I use my older Fluke. I only break out the 121GW when I need one of its advanced features, or if I need all my meters for a 4-meter set up.
Great info James laymans terms for how things work. I have a BCS and ended up liking machining better, gotta love it. :)
Your a natural teacher
Your a wealth of knowledge
Great job troubleshooting! I would have gotten stuck at the ARP packets. I would not have realized that the broadcast query was unnecessary.
It took me a while to realize it was wrong.
James, sometimes you give me a headache! ;-) But it's the good kinda headache like.... AAhhhhh Input!!! NICE! That book discription had me laughing. I am gonna steal that one from ya and use it. Cheers!!
Thats crazy. Congratulations on finding the problem.
Fantastic as always James! Will need to explore using the Ether1616 now… I am not familiar with this device to expand my AcornSide note… and maybe you have this planned… but I’d be interested in a video on how you leverage Fusion with the imported electrical components to design your control panels.
Solid. I still have my 5 book Microsoft Windows TCP/IP networking manuals from around that time. Pity the trees. Good catch on the ARP bug.
I'd never have found that! Probably a good general setting for "budget" boards to disable ARP Offload.
Brilliant episode the best yet I work on complex networks in industry and the transport world and would not have seen this issue as the equipment we deploy is generally high end stuff the NICs in the cheaper end are just that cheap I’m working on my own design and build CNC mill and will buy a 2nd hand quality PC over a new cheap one for the reasons you found here, keep up the great work.
Fun times. I cant even imagine how much fun ill have when i finsh my build...raspbi pi 4 and linux cnc...ive already built the pi box and have linux cnc installed...servos next....great video...nice trouble shooting...
Very nice, excellent explanations.