Thanks for digging in on this; it's been a bit of wild ride with the NanoKVM's software. From covering other Sipeed products, it seems like they hit the same wall many other Chinese hardware startups do-they get a working product, and wrap it up and ship it, expecting people either to be happy or the community to do the rest of the enablement work :( At least with this hardware being open enough, we can flash an alternate community-supported OS image! But I'm still more on board the JetKVM train now, even though it's a bit more expensive.
The price difference is not that big, so it makes sense to buy JetKVM if one is concerned with the quality of the NanoKVM firmware. That being said, I really wish we had more options like the (just-sip-the-power-from-and-offer-nothing-else-)PCIe Nano KVM. Having a device that sits inside the chassis make things much cleaner, rather than having KVM on a rack shelf with wires running through some grommets or PCI slots to the PC.
@@BartomiejSacharski What are options for somebody who does not want to buy any additional HW ? Simplicity is better for economy of energy,security,labor... Linux kernel provides insane level of functionality, so maybe just use what is already available ? Security review is ok, shilling nonsensical hw is scam.
Jeff has looked at a PiKVM-based board which sits in a PCIe slot - www.jeffgeerling.com/blog/2022/blikvm-pcie-puts-computer-your-computer I'd personally love to see a board that actually can speak PCIe x1, using the Linux PCIe endpoint framework
@@apalrdsadventures I've seen that, but at that pricepoint, I would've rather tried Asrock PAUL - at least it would've been easier to return it to store in Poland, rather than having to ship it back to PRC
No! Stop it! Even if the hardware was open, you’re currently paying for hardware AND garbage software, and then expecting other people to come along and fix it. If you won’t provide good software, sell the device without software, at a price that reflects as much. Heck they I can the difference toward the open source devs that actually do good work. And re: “This hardware being open enough” - They literally use undocumented opcodes… Maybe *some* of the hardware is open, but the specific parts needed to make the product work aren’t open.
Would you be interested in making a similar security deep dive for the JetKVM? It is a competing product which has been getting quite popular recently. Thanks!
I already made a review video of the JetKVM, but didn't feel like it needed a security deep-dive. In short: - JetKVM uses bcrypt for password encryption - this is a well-researched password hash based on Blowfish and developed by OpenBSD - JetKVM uses WebRTC for remote access instead of Tailscale and only reaches out when you enroll it from the UI - JetKVM requires SSH key login, and lets you set the keys from the UI (no default password) - JetKVM respects my network DNS servers - JetKVM does not support HTTPS locally, but does use encryption for WebRTC traffic - JetKVM 'hardcoded' NTP also - Jet KVM doesn't support IPv6 (same as NanoKVM) Generally all of the issues with the JetKVM can fall more under 'bug' than 'gaping hole'.
@@apalrdsadventures One other difference between the two is it seems JetKVM's software dev side is responding to feedback with more of a 'ah, yes, we'll get to that', and less of a 'this is how it is, thanks' (which is how many of the responses have gone in the NanoKVM issues...).
The JetKVM is not a "competing product" in the same way that Ferrari does not compete with Kia. The whole point of these devices is to add JUST ENOUGH remote KVM capability, at a price that is actually worth spending. Once you get into the $65+ range, you could have just spent that much to have ACTUAL IPMI on the board with all the bells and whistles.
My tiny attention span was railing against an hour long video. Turned out it was riveting from start to finish. I have the "full" version and struck some of the more obvious "eww!" points you covered (The "not USB" connector; Tailscale; Flakiness of video/input; H.265 not yet implemented; etc.). Thanks for going the extra mile on the network and binary analysis. So much more hair-raising than I thought it would be.
Thank you for your service sir! I seem to have a bad paperweight until someone creates a decent firmware. I am glad to hear that the JetKVM seems to be doing much better. And for those saying: use ILO and such. Not all pc's have that option.
As a sysadmin I found this very interesting, and would like to see more of it :) I have a chineese NAS sitting in my rack (turned off), I might want to capture it's network traffic as well...thank you for the video!
Yeah, i was gunna get a lite or two, but all of these issues is offputting. Ill either get a jetkvm if i need a kvm, or wait for sipeed to fix their shit
Thank you for your amazing review and breakdown. I really like how you took your time (I bet it took a while....) and analyzed every part of it. Especially the last thoughts were very good! In these "times" it's easy to say that China is bad and everything else is great while that's simply not the case. Many western companies cut all the corners they can and use binary blobs. Thank you!
@apalrdsadventures this is some seriously good work, I dont think most people will understand the amount of time and effort you put into the binary reverse engineering. That takes some solid time and effort to map out, get your head wrapped around it, look into custom instructions, etc.
It's a lot easier to do with a library than something like a firmware image, because every function in the library which is exported must have a name and address. The dynamic linker needs this to resolve all of the functions when it loads the library, so you can't strip it out of a library like you can with an application. So most of the functions already have names, and the names tell you a lot about what they do.
would you consider root:root for ssh to be a backdoor or just unsecured? The lib update process also works in a way which would allow a backdoor to be deployed only to specific serial number units
Hmmm. . Although - can I blacklist that MAC for outbound at the router? You had it working with no network so I guess it just shrugs if it can't get out. I use headscale so I guess I can reach from outside over my mesh. Worth a try I guess. Was *mightily* relieved to see your response about JetKVM as I have one of those (hopefully) inbound shortly.
it worked with no network since it had already been updated / library downloaded when it came to me, and it continued to work fine. If you don't have an updated version, then you'd need to update it first then block it.
a good chunk of the reason for its price is that it is using specialized hardware that requires a custom firmware made using the vendor SDK. PiKVM is too heavy for the hardware in NanoKVM and would not be able to use some of the hardware acceleration
PiKVM would work fine if the vendor supported V4L2 instead of a proprietary API. They aren't doing anything that V4L2 wouldn't support - it supports hardware-offloaded encoding, which is exactly what they are doing with the cvitek lib.
44:48 imho these boards are separated because you might want to keep optocouplers pcb inside case while having kvm outside (to reach buttons for example). Supposedly opto-pcb should fit without jumper wires directly to atx front panel connector so keeping these two parts separated seems like pretty good idea
I like NAC in general, but I haven't used PacketFence specifically. I'm more building up to a full solution out of more basic concepts (802.1x, wpa-enterprise, ...)
I have a question: if I use nanoKVM in a "untrusted lan/network" which don't have acces on firewall to WAN and I can connect to it only from one specific subnet in my homelab could i feel quite safe or should i toss this KVM trough a window?
if it has no access at all then it is safe, but if it needs to update, it will fail to download libmaixcam_lib.so from the server and won't work until it does. If it has WAN access via a firewall, it will try to NAT-PMP map a port, and that could be risky if the software doing the port map is sus. It's done by tailscale, which I trust way more than Sipeed, however, Sipped is shipping an older Tailscale binary from Sipeed's CDN instead of from Tailscale directly, which looks bad even if it's just them being bad at packaging. Behind a firewall it should be fine, in theory.
@@apalrdsadventures big thanks for answer. From day-one of using this device I set untrusted network on pfsense with block to WAN an other subnets rules and I see on firewall monitor same shady behavior as You talk about in video. I update it manualy by by switching the networks just for download the updates and then throw it again in jail subnet. Anyway, one more time thank You for professional investigation in this product!
I thought you aren't supposed to just hash passwords client side since the password hash effectively becomes your new password? Aren't you supposed to hash the password either just on the server side or better both client and server side? Or did that recommendation change somehow?
The PCIe card is powered off the PCI bus, it doesn't require you to plug in USB power. You can actually see it power up on the OLED when you plug it into the motherboard in the video before you connect power.
it's really HW dependent. i couldn't enter and operate in bios of dell optiplex 330 while i could remotely drive my dell laptop. you might try touch /boot/BIOS thingy which messes up a bit with HID descriptors
As far as I can tell, all of the protected functions involve the NPU / ai processor on the chip, and aren't even needed for just passing through video. The library is designed to do video analysis in the NPU, and they are using a small subset of that to capture video. The hardware already does HDMI -> CSI
Hell no. It would be good to put KVM on a couple of machines, but it should operate locally, and you decide if you want any outside access, going out to all sorts of locations is an absolute no-no for me. I have pfSense and so I can see all the disturbing traffic these IoT devices generate - I'm thinking of ripping out Alexa, Google, and most of my existing home automation. For now they are being moved to an isolated VLAN. People who don't have a fancy router with VLANs can move them to your Guest WiFi network and isolate that.
Your point is well taken about non-standard use of USB connectors but I have a feeling that they're incredibly cheap given they're so ubiquitous. Since they're trying to target a low price I can understand trying to shave a few cents where you can. The software security, on the other hand, is inexcusable.
PiKVM-based solutions tend to use RJ45 for this. There are 4 pairs, which they directly use for the 4 IO signals, with isolation within the KVM. JetKVM uses an RJ11, which provides power and a serial bus for accessories USB microcontrollers are cheap enough that they could have used one here (and done proper USB 2.0). They chose pretty beefy digital isolators too, they probably could have made a better choice there and it would have been cost-neutral.
@@apalrdsadventures thanks for taking the time to respond. I was trying to give them the benefit of the doubt but that's a fair point. It seems like they just made a bad engineering decision.
On the HTTPS issue: according to the product wiki, it does support HTTPS, but only one or the other--either HTTP or HTTPS, but not both. Upload a cert and key (easy enough via scp, or heck, even paste it in to vi), edit a config file to point to them, and Robert is your father's brother. In theory. That did work with the three-month-old image when I bought my device, but for the last few updates, it doesn't any more; it's http-only regardless of what the config file says. There's been no response to my issue #283 on their repo about this issue, and I lack your skill at digging into the code to sort out why this is happening.
No idea what broke, but the code to start in either http or https is here: github.com/sipeed/NanoKVM/blob/97a9b376c79611ca1cceb8f6df282670bb597379/server/main.go#L43 When Proto == "https", it kicks off a goroutine (asynchronous action) to run the HTTP server, then (not in a goroutine) it launches the HTTPS server. Otherwise, it launches the HTTP server (not in a goroutine). On its face this seems correct to me. They seem to 'commit' to the open-source repo infrequently, and dump commits to basically every single file in the project at once, with completely unrelated changes, so it's super annoying to actually track changes properly. For some probably stupid reason, a commit titled 'Support wifi configuration' renamed the 'protocol' key in the config file to just 'proto'. github.com/sipeed/NanoKVM/commit/5a39562f2d32695933f4e7e86866136236cc9903#diff-19136d80005b994ac9a98d8d280b36fc5e6a5be77cdf68eac6cafc55e029d04c Of course, that single commit touched 66 files and 7000 line-changes. No way that was this change reviewed by anyone in this state. There is of course no justification for changing 66 files to support wifi, or why this touched so many other things in the project unrelated to wifi.
@@apalrdsadventures ...and that was indeed the fix--the first key needs to be `proto:`, not `protocol:`. So with that, it's back to running https again. But not http at the same time, so no redirect or anything.
26:03 oh, one more thing. in risc-v you can trap illegal instructions with SBI so it's possible that looking at the firmware + "sketchy library" might not paint the whole picture. Imagine scenario where 8b 57 traps to "take pointer from a0, validate agaist our-very-custom-challenge-response" and there you have it, rest is glue code and bunch of mumbling to exhaust researcher.
Learning about the ssh login made me immediately losing any trust in that device, the shady shared modules later too. For fun I've exposed one device on a static IPv4 since like last year, however with changed passwords. Just to see if there is any specific attempt to break in.
23:20 these might be valid SG2002's opcodes or even Sipeed's own design embedded into SG2002-based SoC. It's risc-v at the end of the day, there's nothing holding you back from making DEAD BEEF a valid opcode for your design.
42:59 and i disagree again. Including rj11 or, god forbid, propietary connector just to deliver some voltages to bunch of optocouplers not only would increase complexity of design but also would made BOM more expensive. Personally i find it great that they "repurposed" usb a/c connectors because at least i am not tied to special snowflake cable made of unicorns but can take cheapest usb a-c cable and buy longer/ shorter one to my liking.
This reminded me that I was supposed to set up firewall rules for my management VLAN. Thanks Apalrd! As for the firmware/software itself...well, it's definitely less than stellar right now, hopefully in future either Sipeed or "The Community" will make it better. It's still more than I can say about built-in IPMI firmware on server motherboards that technically should be good (and definitely is somewhat better than what Sipeed provides currently), but without some serious rev-eng we can't really tell whether it contains some sneaky stuff.
I feel like Sipeed shipped this thing months before it was ready and is just hoping that 'the community' will fix all of their firmware issues because the hardware is cheap In the IPMI space, many are moving to or already running OpenBMC which is a great improvement on the entirely proprietary solutions
So this a big 'no' from me. The custom instruction is probably nothing more than reading an embedded serial number but it does raise some unsettling questions about the security and 'undocumented features' of RISC-V cores that are being used out there. With regard to the unsigned downloads, it kind of makes me laugh--I've written some software and had to sweat every detail of security best practices, and looking at some of these IoT implementations is just comical.
ARM has done some great engineering talks on how they do fuzzing and coverage testing at a micro-op / RTL level, to prove that all of the security barriers in the instruction set can't be violated by some corner case bug in specific instructions. Since ARM is strictly licensed and they only let their largest partners (architecture licenses) design/modify their cores, this same sort of engineering and testing extends to basically all ARM-based products. In the RISC-V world, we aren't bound by the licensing constraints, so nothing requires a company to produce a compatible implementation other than the desire to work with the existing software and tools. It's even encouraged to add extend it with custom opcodes instead of designing an entirely custom processor for a special purpose, but that of course means we lose the fairly strict guarantees that ARM used to give us. On the flipside, the lack of licensing means we can get entirely open source RTL for an actually good CPU, and the C906 core in this SoC is open source itself.
@@apalrdsadventures Yeah this leads us into the "Reflections on trusting trust" territory. My feeling is that RISC-V and having open source cores available is overall a Good Thing. On the other hand we owe it to ourselves to know what is actually in the silicon.
Since RISC-V is a fixed-length instruction, there can't be more than 4M instructions (32 bits). So, one could design a test suite which exhaustively tries all of them and verifies that the functionality is according to the architecture. However, many of these instructions change behavior based on data, so there are 'hidden' states. Hence, ARM doing their instruction coverage testing at the micro-op level and not instruction level. Overall though I think RISC-V is just newer and these sorts of things will develop eventually. We've already seen people develop tools to find and map hidden instructions in x86 (Sandsifter, for the curious).
but would you like to have a liability in your network? Even when operating in an shielded/"gapped" VLAN? I've not yet received my JetKVM but I hope it's better or at least open source.
@@FuchsHorst sure, that's why VLANs are for. I have handful of smart bulbs and other IoT crap sitting in their internetless vlan and still happily doing their thing.
@@lis6502VLAN is actually for network segmentation, the application and security measurements applied to it are something else I can have thousands of VLANs in my network and, at the same time, apply nothing on them.
Not just chinese stuff handles security this crappily. There was one tool, where I had to enter the sha256 hash of my password and that was it for security. It did not even allow me to set a password hash function.
19:38 and here we have another point of guy not knowing what he's talking about. nanokvm has 128 MB of ram and is 99% based on buildroot BECAUSE IT'S FSCKING IOT xD. it really doesn't need systemd, command-not-found, need-restart, graphical www client and god knows what. Sure you can spin off minimal build of debian but at the end of the day you'll have to throw in some propietary sauce like maixcam driver; for these scenarios Buildroot was invented, to quickly deliver custom, low footprint MVP which 99/100 times is way better and more flexible than your average NAS underlying linux variant.
There's nothing inherently wrong with Buildroot. The comment suggesting Debian was because of how many packaging and version-related problems there are on this thing - for example there's an old version of the Tailscale RV64 client which Sipeed distributes via their CDN and hasn't updated, and these kinds of packaging and versioning issues have been solved over and over by mainstream distros. They also completely rolled their own software updater, which doesn't validate any sort of signature on the update, and those issues have also been completely solved by the packaging systems on mainstream distros with signed packages. Relying on a mainstream package manager (even something simple like Alpine APK) would greatly improve their mess.
@@apalrdsadventures @apalrdsadventures it's an applicance, a monolith having this specific version of that working perfectly with this specific version of the other. Imagine sipeed indeed would package tailscale client separately but it would require newer version of libssl which in turn would break webserver. Packaging is good and really expected way in general-purpose devices like laptops. NanoKVM is just another router/ NAS "heavily relying" on parts of udhcpcd, parts of haveged( which is btw PRNG) but at the end of the day being bunch of random things glued together. Even QNAP NAS, while extensible via "apps" being sort of flatpacks internally still ship their firmware as an "image to be flashed". I'd say even more: having package-oriented approach in embedded world is very uncommon.
Haveged is a Linux daemon which generates randomness from the cpu for /dev/random. It shouldn’t be needed in this environment since its algorithm was merged into the kernel in 5.6 and this is running 5.10 but who knows what old shit they copied from.
As an IP protection measurement, it's implemented in a way which requires the client to download and execute an unsigned binary from a remote server without any sort of validation As an open-source project, it's a binary blob which is using undocumented opcodes to do unknown things.
It can't hash check the downloaded .so because it is patched for each device. Not that this is an excuse... Now that RE'd the thing you could load a copy with serial=0, hash check it, then patch locally.
19:20 ffs, apalrd... come on, this stuff was created to be used LOCALLY for such guys like you and me - wanting to have on the go ipkvm solution as well as "serverize" teir repurposed old PC. Argument about "forcing password change and forcing itbeing strong password" is the one and only redflag i need to call someone "secoority expurt". Like your home tplnik has 32 characters passphrase with 2fa ;d.
@@apalrdsadventures not at all and i prefer it over "it's the same password as we forced you to change including big letter, current yearm random emoji and three special characters". Maybe it's just me but first thing i do to my newest purchase is to change all passwords and exchange my ssh key. Like the good old days where wireless accesspoints came broadcasting open wifis instead of forcing me to remember underside sticker saying that my passphrase is AkslfFapz[Z];f4&6@r. It's as good as default admin/admin and alike subject to be first thing to change at initial setup
Personally i do not understand why are people so interested in these. My "hypervisor" is reachable over lan, no matter if vms are locked or what. Maybe i can see use case for board developers/machine farms where they need to test stuff on real machine. Just make that HDMI and usb a pass thru for nefarious purposes
These devices come into play if for example your hypervisor is having trouble at the boot stage / or has a network misconfiguration. Remote desktop, VNC and the likes wont work there. Of course if it's a proper server it has IPMI built in which is basically the same thing as the NanoKVM - just integrated on board
It is possible to "mirror" lan port on some switches/routers so you can connect wireshark directly to that and see traffic realtime, no need to work with pcap files. Less work for you. I am using that for invisible IDS type of stuff.
@@c1nema1 yeah ast2500 is ridiculous ! you can even flash "bios" with it, for example if you buy old revision of epic board which does not support new cpu yet, very useful. but for homelab for me is any hw kvm overkill.
I use PiKVM for bare metal automation using redfish. I have a cluster of 5 PCs that I switch between a bare metal install of Windows for gaming, or Proxmox running my homelab. I can lose 2 proxmox nodes and still retain quorum. This way I can consolidate my GPU compute power for everything I need it for. I have seperate storage servers and use fast networking, my reboot times are like 2-3 minutes. I control this through Home Assistant with a button to turn on my PC.
GL.Inet, that travel router company which makes them with opensource openwrt is releasing a KVM, they say it will also be opensource. Their products are usually good quality, so worth a try to getting one for review or participating in beta.
![Felix "Unfair" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/Oswujxm2Ag0/mqdefault.jpg)
Thanks for digging in on this; it's been a bit of wild ride with the NanoKVM's software.
From covering other Sipeed products, it seems like they hit the same wall many other Chinese hardware startups do-they get a working product, and wrap it up and ship it, expecting people either to be happy or the community to do the rest of the enablement work :(
At least with this hardware being open enough, we can flash an alternate community-supported OS image! But I'm still more on board the JetKVM train now, even though it's a bit more expensive.
The price difference is not that big, so it makes sense to buy JetKVM if one is concerned with the quality of the NanoKVM firmware.
That being said, I really wish we had more options like the (just-sip-the-power-from-and-offer-nothing-else-)PCIe Nano KVM. Having a device that sits inside the chassis make things much cleaner, rather than having KVM on a rack shelf with wires running through some grommets or PCI slots to the PC.
@@BartomiejSacharski What are options for somebody who does not want to buy any additional HW ? Simplicity is better for economy of energy,security,labor... Linux kernel provides insane level of functionality, so maybe just use what is already available ? Security review is ok, shilling nonsensical hw is scam.
Jeff has looked at a PiKVM-based board which sits in a PCIe slot - www.jeffgeerling.com/blog/2022/blikvm-pcie-puts-computer-your-computer
I'd personally love to see a board that actually can speak PCIe x1, using the Linux PCIe endpoint framework
@@apalrdsadventures I've seen that, but at that pricepoint, I would've rather tried Asrock PAUL - at least it would've been easier to return it to store in Poland, rather than having to ship it back to PRC
No! Stop it! Even if the hardware was open, you’re currently paying for hardware AND garbage software, and then expecting other people to come along and fix it. If you won’t provide good software, sell the device without software, at a price that reflects as much. Heck they I can the difference toward the open source devs that actually do good work.
And re: “This hardware being open enough” - They literally use undocumented opcodes… Maybe *some* of the hardware is open, but the specific parts needed to make the product work aren’t open.
Would you be interested in making a similar security deep dive for the JetKVM? It is a competing product which has been getting quite popular recently. Thanks!
I already made a review video of the JetKVM, but didn't feel like it needed a security deep-dive. In short:
- JetKVM uses bcrypt for password encryption - this is a well-researched password hash based on Blowfish and developed by OpenBSD
- JetKVM uses WebRTC for remote access instead of Tailscale and only reaches out when you enroll it from the UI
- JetKVM requires SSH key login, and lets you set the keys from the UI (no default password)
- JetKVM respects my network DNS servers
- JetKVM does not support HTTPS locally, but does use encryption for WebRTC traffic
- JetKVM 'hardcoded' NTP also
- Jet KVM doesn't support IPv6 (same as NanoKVM)
Generally all of the issues with the JetKVM can fall more under 'bug' than 'gaping hole'.
@@apalrdsadventures One other difference between the two is it seems JetKVM's software dev side is responding to feedback with more of a 'ah, yes, we'll get to that', and less of a 'this is how it is, thanks' (which is how many of the responses have gone in the NanoKVM issues...).
@@JeffGeerling you've probably forgot to end sentence with "my dearest friend hoping for your understanding" :D
can confirm JetKVM devs have been great and very responsive to issues / feedback
The JetKVM is not a "competing product" in the same way that Ferrari does not compete with Kia.
The whole point of these devices is to add JUST ENOUGH remote KVM capability, at a price that is actually worth spending.
Once you get into the $65+ range, you could have just spent that much to have ACTUAL IPMI on the board with all the bells and whistles.
Someone's heart skipped a beat after hearing, 'So, I'm pretty good at binary reverse engineering.'
I doubt it was the NanoKVM people though. If it were they would have done a better job on the architecture in the first place.
Very comprehensive material, even if we disagree in certain parts. Love your content, hope you'll keep on growing and doing THAT deep dives :)
Thanks!
Thank you for covering this product. I was really looking forward to it.
FINALLY someone is doing a basic audit of these things.
My tiny attention span was railing against an hour long video. Turned out it was riveting from start to finish.
I have the "full" version and struck some of the more obvious "eww!" points you covered (The "not USB" connector; Tailscale; Flakiness of video/input; H.265 not yet implemented; etc.).
Thanks for going the extra mile on the network and binary analysis. So much more hair-raising than I thought it would be.
Did you really say, victim computer?
Haha I laughed at that too
Test Subject #234 would be preferred?
Thank you for your service sir! I seem to have a bad paperweight until someone creates a decent firmware.
I am glad to hear that the JetKVM seems to be doing much better.
And for those saying: use ILO and such. Not all pc's have that option.
35:12 This basically a software version of a door that is held by a cheeto
As a sysadmin I found this very interesting, and would like to see more of it :) I have a chineese NAS sitting in my rack (turned off), I might want to capture it's network traffic as well...thank you for the video!
41:27, whats this terminal orogram that you are using to examine the traffic?
Mitmproxy (I am running it in transparent mode)
Had a similar experience of "this CPU peripheral only works with a closed source lib" from V831 camera board
well i'm 1/3 of the video and I'm think i know how is gonna end. Seemed to be a good product, but that backdoor thingy is scary as duck.
Yeah, i was gunna get a lite or two, but all of these issues is offputting. Ill either get a jetkvm if i need a kvm, or wait for sipeed to fix their shit
Thank you for your amazing review and breakdown. I really like how you took your time (I bet it took a while....) and analyzed every part of it. Especially the last thoughts were very good! In these "times" it's easy to say that China is bad and everything else is great while that's simply not the case. Many western companies cut all the corners they can and use binary blobs. Thank you!
Great work. Thank you for covering this
@apalrdsadventures this is some seriously good work, I dont think most people will understand the amount of time and effort you put into the binary reverse engineering. That takes some solid time and effort to map out, get your head wrapped around it, look into custom instructions, etc.
It's a lot easier to do with a library than something like a firmware image, because every function in the library which is exported must have a name and address. The dynamic linker needs this to resolve all of the functions when it loads the library, so you can't strip it out of a library like you can with an application. So most of the functions already have names, and the names tell you a lot about what they do.
I just bought this. Now I have a question, is it just unsecured? Or does it make a backdoor?
would you consider root:root for ssh to be a backdoor or just unsecured?
The lib update process also works in a way which would allow a backdoor to be deployed only to specific serial number units
@@apalrdsadventures thank you for the information. It awnsers my question. It was a good video! ❤
just plug it in and tell us the IP
S tier video name🎉
Hmmm. . Although - can I blacklist that MAC for outbound at the router? You had it working with no network so I guess it just shrugs if it can't get out. I use headscale so I guess I can reach from outside over my mesh. Worth a try I guess. Was *mightily* relieved to see your response about JetKVM as I have one of those (hopefully) inbound shortly.
it worked with no network since it had already been updated / library downloaded when it came to me, and it continued to work fine.
If you don't have an updated version, then you'd need to update it first then block it.
Excellent review, thanks! I guess I'll look for a different solution ...
All they had to do was use Pikvm or another open source OS as their base. Hardware is probably awesome.
a good chunk of the reason for its price is that it is using specialized hardware that requires a custom firmware made using the vendor SDK. PiKVM is too heavy for the hardware in NanoKVM and would not be able to use some of the hardware acceleration
PiKVM would work fine if the vendor supported V4L2 instead of a proprietary API. They aren't doing anything that V4L2 wouldn't support - it supports hardware-offloaded encoding, which is exactly what they are doing with the cvitek lib.
@@apalrdsadventures makes sense. This is why I use a blikvm. Pilvm quality and software at a discount.
Amazing video! Love the jeff geerling shirt.
Most of the shirts I wear are from other creators, you might be able to identify them. I have two from Jeff.
at 33:40 when you plug in the PCI card, you can see a blue LED blinking, so it's using power from the PCI for something?
there is exactly one trace to one PCIe finger pin, and that is 3.3v aux.
44:48 imho these boards are separated because you might want to keep optocouplers pcb inside case while having kvm outside (to reach buttons for example). Supposedly opto-pcb should fit without jumper wires directly to atx front panel connector so keeping these two parts separated seems like pretty good idea
What are your thoughts on Network Access Control (NAC) open source projects like PacketFence?
I like NAC in general, but I haven't used PacketFence specifically. I'm more building up to a full solution out of more basic concepts (802.1x, wpa-enterprise, ...)
I have a question: if I use nanoKVM in a "untrusted lan/network" which don't have acces on firewall to WAN and I can connect to it only from one specific subnet in my homelab could i feel quite safe or should i toss this KVM trough a window?
if it has no access at all then it is safe, but if it needs to update, it will fail to download libmaixcam_lib.so from the server and won't work until it does.
If it has WAN access via a firewall, it will try to NAT-PMP map a port, and that could be risky if the software doing the port map is sus. It's done by tailscale, which I trust way more than Sipeed, however, Sipped is shipping an older Tailscale binary from Sipeed's CDN instead of from Tailscale directly, which looks bad even if it's just them being bad at packaging.
Behind a firewall it should be fine, in theory.
@@apalrdsadventures big thanks for answer. From day-one of using this device I set untrusted network on pfsense with block to WAN an other subnets rules and I see on firewall monitor same shady behavior as You talk about in video. I update it manualy by by switching the networks just for download the updates and then throw it again in jail subnet. Anyway, one more time thank You for professional investigation in this product!
I thought you aren't supposed to just hash passwords client side since the password hash effectively becomes your new password? Aren't you supposed to hash the password either just on the server side or better both client and server side? Or did that recommendation change somehow?
If you have no transport security then you need to hash on both sides. If you do have transport security (TLS), then you only need to do server-side.
As far as I can tell, the trace going to the edge connector is going to... pin 10? Which is weird, because that's standby power.
Super good job man!!! 👏
The PCIe card is powered off the PCI bus, it doesn't require you to plug in USB power. You can actually see it power up on the OLED when you plug it into the motherboard in the video before you connect power.
you need USB for keyboard/mouse anyway
@apalrdsadventures correct. I was just clarifying that the PCI adapter had some functionality to it
Have you been able to have it working with FreeBSD? The mouse an keyboard does not work on my PfSense.
it's really HW dependent. i couldn't enter and operate in bios of dell optiplex 330 while i could remotely drive my dell laptop. you might try touch /boot/BIOS thingy which messes up a bit with HID descriptors
They work on my OPNsense system, FWIW.
Is it possible that the HDMI regulation is why they have to do the custom firmware nonsese? I don't think it's a valid excuse but...
As far as I can tell, all of the protected functions involve the NPU / ai processor on the chip, and aren't even needed for just passing through video. The library is designed to do video analysis in the NPU, and they are using a small subset of that to capture video.
The hardware already does HDMI -> CSI
Hell no. It would be good to put KVM on a couple of machines, but it should operate locally, and you decide if you want any outside access, going out to all sorts of locations is an absolute no-no for me. I have pfSense and so I can see all the disturbing traffic these IoT devices generate - I'm thinking of ripping out Alexa, Google, and most of my existing home automation. For now they are being moved to an isolated VLAN. People who don't have a fancy router with VLANs can move them to your Guest WiFi network and isolate that.
Your point is well taken about non-standard use of USB connectors but I have a feeling that they're incredibly cheap given they're so ubiquitous. Since they're trying to target a low price I can understand trying to shave a few cents where you can. The software security, on the other hand, is inexcusable.
PiKVM-based solutions tend to use RJ45 for this. There are 4 pairs, which they directly use for the 4 IO signals, with isolation within the KVM.
JetKVM uses an RJ11, which provides power and a serial bus for accessories
USB microcontrollers are cheap enough that they could have used one here (and done proper USB 2.0). They chose pretty beefy digital isolators too, they probably could have made a better choice there and it would have been cost-neutral.
@@apalrdsadventures thanks for taking the time to respond. I was trying to give them the benefit of the doubt but that's a fair point. It seems like they just made a bad engineering decision.
10:50 NAT is not a firewall, it can ask for a DNAT, not to remove forward drop/reject rules.
NAT-PMP is used in IPv6 to request open incoming ports as well, it's not strictly for DNAT despite the name
@@apalrdsadventuresThat's PCP, not NAT-PMP.
On the HTTPS issue: according to the product wiki, it does support HTTPS, but only one or the other--either HTTP or HTTPS, but not both. Upload a cert and key (easy enough via scp, or heck, even paste it in to vi), edit a config file to point to them, and Robert is your father's brother. In theory.
That did work with the three-month-old image when I bought my device, but for the last few updates, it doesn't any more; it's http-only regardless of what the config file says. There's been no response to my issue #283 on their repo about this issue, and I lack your skill at digging into the code to sort out why this is happening.
No idea what broke, but the code to start in either http or https is here: github.com/sipeed/NanoKVM/blob/97a9b376c79611ca1cceb8f6df282670bb597379/server/main.go#L43
When Proto == "https", it kicks off a goroutine (asynchronous action) to run the HTTP server, then (not in a goroutine) it launches the HTTPS server. Otherwise, it launches the HTTP server (not in a goroutine). On its face this seems correct to me.
They seem to 'commit' to the open-source repo infrequently, and dump commits to basically every single file in the project at once, with completely unrelated changes, so it's super annoying to actually track changes properly.
For some probably stupid reason, a commit titled 'Support wifi configuration' renamed the 'protocol' key in the config file to just 'proto'. github.com/sipeed/NanoKVM/commit/5a39562f2d32695933f4e7e86866136236cc9903#diff-19136d80005b994ac9a98d8d280b36fc5e6a5be77cdf68eac6cafc55e029d04c
Of course, that single commit touched 66 files and 7000 line-changes. No way that was this change reviewed by anyone in this state. There is of course no justification for changing 66 files to support wifi, or why this touched so many other things in the project unrelated to wifi.
@@apalrdsadventures ...and that was indeed the fix--the first key needs to be `proto:`, not `protocol:`. So with that, it's back to running https again. But not http at the same time, so no redirect or anything.
Can you please do a similar deep dive on jetkvm?
He already did (see comments above)
26:03 oh, one more thing. in risc-v you can trap illegal instructions with SBI so it's possible that looking at the firmware + "sketchy library" might not paint the whole picture. Imagine scenario where 8b 57 traps to "take pointer from a0, validate agaist our-very-custom-challenge-response" and there you have it, rest is glue code and bunch of mumbling to exhaust researcher.
Learning about the ssh login made me immediately losing any trust in that device, the shady shared modules later too. For fun I've exposed one device on a static IPv4 since like last year, however with changed passwords. Just to see if there is any specific attempt to break in.
Just checked, device uptime 37 days but httpd crashed.
23:20 these might be valid SG2002's opcodes or even Sipeed's own design embedded into SG2002-based SoC. It's risc-v at the end of the day, there's nothing holding you back from making DEAD BEEF a valid opcode for your design.
they are certainly valid opcodes for the SG2002, but them being undocumented makes them a security risk for developers/users
42:59 and i disagree again. Including rj11 or, god forbid, propietary connector just to deliver some voltages to bunch of optocouplers not only would increase complexity of design but also would made BOM more expensive. Personally i find it great that they "repurposed" usb a/c connectors because at least i am not tied to special snowflake cable made of unicorns but can take cheapest usb a-c cable and buy longer/ shorter one to my liking.
"Sketchy", says the "expert" that can't understand why a device would ping the network gateway!
Tp-link got a backdoor too 😂
This reminded me that I was supposed to set up firewall rules for my management VLAN. Thanks Apalrd!
As for the firmware/software itself...well, it's definitely less than stellar right now, hopefully in future either Sipeed or "The Community" will make it better.
It's still more than I can say about built-in IPMI firmware on server motherboards that technically should be good (and definitely is somewhat better than what Sipeed provides currently), but without some serious rev-eng we can't really tell whether it contains some sneaky stuff.
I feel like Sipeed shipped this thing months before it was ready and is just hoping that 'the community' will fix all of their firmware issues because the hardware is cheap
In the IPMI space, many are moving to or already running OpenBMC which is a great improvement on the entirely proprietary solutions
So this a big 'no' from me. The custom instruction is probably nothing more than reading an embedded serial number but it does raise some unsettling questions about the security and 'undocumented features' of RISC-V cores that are being used out there. With regard to the unsigned downloads, it kind of makes me laugh--I've written some software and had to sweat every detail of security best practices, and looking at some of these IoT implementations is just comical.
ARM has done some great engineering talks on how they do fuzzing and coverage testing at a micro-op / RTL level, to prove that all of the security barriers in the instruction set can't be violated by some corner case bug in specific instructions. Since ARM is strictly licensed and they only let their largest partners (architecture licenses) design/modify their cores, this same sort of engineering and testing extends to basically all ARM-based products.
In the RISC-V world, we aren't bound by the licensing constraints, so nothing requires a company to produce a compatible implementation other than the desire to work with the existing software and tools. It's even encouraged to add extend it with custom opcodes instead of designing an entirely custom processor for a special purpose, but that of course means we lose the fairly strict guarantees that ARM used to give us.
On the flipside, the lack of licensing means we can get entirely open source RTL for an actually good CPU, and the C906 core in this SoC is open source itself.
@@apalrdsadventures Yeah this leads us into the "Reflections on trusting trust" territory. My feeling is that RISC-V and having open source cores available is overall a Good Thing. On the other hand we owe it to ourselves to know what is actually in the silicon.
Since RISC-V is a fixed-length instruction, there can't be more than 4M instructions (32 bits). So, one could design a test suite which exhaustively tries all of them and verifies that the functionality is according to the architecture. However, many of these instructions change behavior based on data, so there are 'hidden' states. Hence, ARM doing their instruction coverage testing at the micro-op level and not instruction level.
Overall though I think RISC-V is just newer and these sorts of things will develop eventually. We've already seen people develop tools to find and map hidden instructions in x86 (Sandsifter, for the curious).
So sad - great hardware that clearly took some effort to make. And then comes the sketchy stuff.
This think is so cheap that makes me consider it, maybe disconnected from the web.
but would you like to have a liability in your network? Even when operating in an shielded/"gapped" VLAN? I've not yet received my JetKVM but I hope it's better or at least open source.
@@FuchsHorst sure, that's why VLANs are for. I have handful of smart bulbs and other IoT crap sitting in their internetless vlan and still happily doing their thing.
@@lis6502VLAN is actually for network segmentation, the application and security measurements applied to it are something else
I can have thousands of VLANs in my network and, at the same time, apply nothing on them.
havaged seeds system random number generator. Legit in embedded devices which have limited sources of randomness.
yes, it's legit. haveged *should* be in-kernel by 5.10, but maybe it isn't in RISC-V, I'm not entirely sure.
Not just chinese stuff handles security this crappily. There was one tool, where I had to enter the sha256 hash of my password and that was it for security. It did not even allow me to set a password hash function.
sha256 is a secure hash function, it's just not intentionally cpu/memory hard like argon2, so dictionary attacks offline are faster
19:38 and here we have another point of guy not knowing what he's talking about. nanokvm has 128 MB of ram and is 99% based on buildroot BECAUSE IT'S FSCKING IOT xD. it really doesn't need systemd, command-not-found, need-restart, graphical www client and god knows what. Sure you can spin off minimal build of debian but at the end of the day you'll have to throw in some propietary sauce like maixcam driver; for these scenarios Buildroot was invented, to quickly deliver custom, low footprint MVP which 99/100 times is way better and more flexible than your average NAS underlying linux variant.
There's nothing inherently wrong with Buildroot. The comment suggesting Debian was because of how many packaging and version-related problems there are on this thing - for example there's an old version of the Tailscale RV64 client which Sipeed distributes via their CDN and hasn't updated, and these kinds of packaging and versioning issues have been solved over and over by mainstream distros. They also completely rolled their own software updater, which doesn't validate any sort of signature on the update, and those issues have also been completely solved by the packaging systems on mainstream distros with signed packages. Relying on a mainstream package manager (even something simple like Alpine APK) would greatly improve their mess.
@@apalrdsadventures @apalrdsadventures it's an applicance, a monolith having this specific version of that working perfectly with this specific version of the other. Imagine sipeed indeed would package tailscale client separately but it would require newer version of libssl which in turn would break webserver. Packaging is good and really expected way in general-purpose devices like laptops. NanoKVM is just another router/ NAS "heavily relying" on parts of udhcpcd, parts of haveged( which is btw PRNG) but at the end of the day being bunch of random things glued together. Even QNAP NAS, while extensible via "apps" being sort of flatpacks internally still ship their firmware as an "image to be flashed". I'd say even more: having package-oriented approach in embedded world is very uncommon.
Haveged is a Linux daemon which generates randomness from the cpu for /dev/random. It shouldn’t be needed in this environment since its algorithm was merged into the kernel in 5.6 and this is running 5.10 but who knows what old shit they copied from.
28:02 why extremally sketchy? Looks for me like a valid IP protection measurement, rendering given copy of library useless on unauthorized device.
As an IP protection measurement, it's implemented in a way which requires the client to download and execute an unsigned binary from a remote server without any sort of validation
As an open-source project, it's a binary blob which is using undocumented opcodes to do unknown things.
It can't hash check the downloaded .so because it is patched for each device. Not that this is an excuse... Now that RE'd the thing you could load a copy with serial=0, hash check it, then patch locally.
raw dog
19:20 ffs, apalrd... come on, this stuff was created to be used LOCALLY for such guys like you and me - wanting to have on the go ipkvm solution as well as "serverize" teir repurposed old PC. Argument about "forcing password change and forcing itbeing strong password" is the one and only redflag i need to call someone "secoority expurt". Like your home tplnik has 32 characters passphrase with 2fa ;d.
does ssh enabled with root:root not seem like a problem to you?
@@apalrdsadventures not at all and i prefer it over "it's the same password as we forced you to change including big letter, current yearm random emoji and three special characters". Maybe it's just me but first thing i do to my newest purchase is to change all passwords and exchange my ssh key. Like the good old days where wireless accesspoints came broadcasting open wifis instead of forcing me to remember underside sticker saying that my passphrase is AkslfFapz[Z];f4&6@r.
It's as good as default admin/admin and alike subject to be first thing to change at initial setup
@@apalrdsadventures No. If you're exposing this to the internet, then you deserve what you get.
So, basically a 20 minute "review" and then about of 35 minutes of assembler reviews ... 🤔
Personally i do not understand why are people so interested in these. My "hypervisor" is reachable over lan, no matter if vms are locked or what. Maybe i can see use case for board developers/machine farms where they need to test stuff on real machine. Just make that HDMI and usb a pass thru for nefarious purposes
These devices come into play if for example your hypervisor is having trouble at the boot stage / or has a network misconfiguration. Remote desktop, VNC and the likes wont work there. Of course if it's a proper server it has IPMI built in which is basically the same thing as the NanoKVM - just integrated on board
It is possible to "mirror" lan port on some switches/routers so you can connect wireshark directly to that and see traffic realtime, no need to work with pcap files. Less work for you. I am using that for invisible IDS type of stuff.
@@c1nema1 yeah ast2500 is ridiculous ! you can even flash "bios" with it, for example if you buy old revision of epic board which does not support new cpu yet, very useful. but for homelab for me is any hw kvm overkill.
I use PiKVM for bare metal automation using redfish. I have a cluster of 5 PCs that I switch between a bare metal install of Windows for gaming, or Proxmox running my homelab. I can lose 2 proxmox nodes and still retain quorum. This way I can consolidate my GPU compute power for everything I need it for. I have seperate storage servers and use fast networking, my reboot times are like 2-3 minutes. I control this through Home Assistant with a button to turn on my PC.
it might happen once a year but sometimes the hypervisor itself can get down and needs a reboot.
GL.Inet, that travel router company which makes them with opensource openwrt is releasing a KVM, they say it will also be opensource. Their products are usually good quality, so worth a try to getting one for review or participating in beta.
already on it, I guess I'm the kvm review guy now