Depends. Whether the sum of features and specs will justify the price. If you manage to convince the audience that the router as a sum is worth it , then people will justify a small increase in price for a better cpu. It depends if subconsciously we will consider it as high or mid router , thus beginning to compare it with other routers in the market.
i am excited about your router, i prefer lower cost, because very expensive consumer-grade routers have around 2GB of memory, like this Asus ROG Rapture GT-BE98 that has 2 10-Gbit ports, i think 4GB of RAM is allready an overkill, i am going to keep watching your videos. Thank you!
@@davidmcken Wireguard is designed to run fast on CPU, using algos that are best suited for CPU. The flip side of this is that nobody is going to make a hardware accelerator for them because they run so fast on CPU anyway
@5:40 DING! Wrong. The GPL is very clear on this. ANY modification done to the kernel source MUST be made available to anyone to whom you've provided a binary. In short, any kernel you compile, your customers must also be able to compile. HOWEVER, external, 3rd party _modules_ can be 100% closed source. I've been here, it can be messy. And yes, giant manufacturers like Broadcom and NXP don't care about breaking the law, or forcing you to. (And there's some debate about making changes to the kernel simply to facilitate external closed modules. For example, undoing the "taint" caused by non-GPL modules, so one can use a GPL-only exported function.)
Agreed, patches and modification to the kernel are covered by GPL and need to be public. A single binary is all that's needed and even then it's still a poor argument for closed sourced code within this project. If he held NXP to the fire and made them comply with GPL in the first place then I would be more open to using their secret sauce.
@@francoisscala417 The issue is that the module must still hook into the kernel APIs. Nettrack hasn't all the necessary APIs for offloading exposed. There is some work in the progress but so far the HW accelleration possible is very very limited. This kind of patches are mainly in the core part of the kernel and not in the external modules. Then they also patched userspace utilities for offloading, meaning you are running more unknown modified software. That have to hook into the kernel in a different way for offloading. I'm thinking about tcpdump that is not able to see the traffic. NXP probably has a patched version. then there are the various VPNs stacks, the ppp daemon etc. it will become a huge mess very easily.
@@francoisscala417 They *could* release the binary module, and then open source a wrapper to load the binary module in the kernel. However this isn't what they do in most cases - and this project isn't anywhere near big enough to convince NXP to change their ways... This just means that without being transportable to newer kernels and being able to be built without relying on NXP for everything, this entire project is lifetime-limited e-waste. Once you don't want to support it anymore, or it gets too hard to keep porting changes - that's the EOL for your product.
I am concerned that relying on the hardware offload capabilities will spell an early end for this device, for example if a bug in handling certain packet types is found (e.g. with new protocols afterwards) or the limitations that this places on additional features. For example the Ubiquiti EdgeRouters can’t do QoS with the hardware offload enabled - just an example that in general you end up being locked in to the hardware capabilities.
And when that happens, it will fall back to the CPU, like on EdgeRouter. The feature will be available, but not offloaded. In that case, performance will be the same as not having HW Offload in the first place. No HW offload for this kind of CPUs means gigabit speeds at best. It is much better to have it, and in the edge cases where someone needs something that it is not offloaded the performance will be the exact same as if HW offload was never implemented. The amount of CPU power you need for 10Gbps routing depends a lot on the features you need. My router has 4x4.1Ghz Core i5 8th gen cores and 8GB DDR4 RAM, with IPS it can only do 7.5Gbps on BSD (pfSense) and 15.5Gbps on Linux. In this particular scenario, 1.2Ghz vs 1.8Ghz won't make a difference.
This is a good point I didn't think of, I've also made a comment that the hardware offloading has no bearing on the CPU choice - We need to know what changing the CPU WILL affect, not what it won't.
@@tomazzaman That's not how drivers work. For example, the syscalls can change from release to release. Binaries that work with BSD will not work for Linux.
Yeap. See Also: Broadcom's "Open Source" NDK. (it's a huge binary blob with a few lines of shim code.) If they have an NDA and access to SDK _source_ then they can build the driver(s) for whatever kernel versions supported - or they're willing to port to. Sadly, this is the way of the modern world... want performance better than a hamster in a wheel, you'll need hardware packet processing, and no one openly documents that stuff.
@@jfbeam I wonder how hard it would be co convince Linus Torwalds or someone in the Linux kernel community to provide an open source hardware acceleration for networking. Especially if you could pay them from the router's development budget. This would effectively create an open source competition to the weird NDA garbage that you just obtained.
Correct me if I'm wrong but this (5:35) seems like a violation of the GPL to me. I don't think you are allowed to distribute derivative kernels under non compatible (i.e. non copyleft) licenses. That means that the proprietary software can only run in loadable modules and not patches built into the actual kernel binary. I might have misunderstood what you meant here though.
@@EraYaN closed source kernel patches will violate the GPL anyway and by the time someone packs and sells this thing, that is, distributing as in the kernel, the recipient has all the rights in the worled to get the code for both the kernel patches *and* the drivers. To bypass GPL with dkms etc, you need the user to interactively install the needed stuff. If it's automated, it is as distributed, meaning it breaks GPL.
Umm... Only the parts that are written by others need to be attributed to and source made available. Anything you write custom can be proprietary, that's exactly how Cisco, Oracle and tons of others do it. Patches are arguable because someone else wrote the patch to make the modification, thus one could argue that the patch it's self is an original work. (Similar to claiming your artwork isn't original because you used my canvas material to make it, or Crayola's red paint to accent the sunset....)
@@jamess1787 If you change even a single bit in a GPL'd source file, YOU. MUST. PUBLISH. YOUR. CHANGES. Cisco, Oracle, "and tons of others" _do_ publish their changes to GPL'd code. Their products are built from a sizable amount of non-GPL stuff, and they obviously don't publish any of that. (It used to be easier to find those tarballs, but Cisco being Cisco makes you jump through hoops to get anything these days.)
I don't want to wait until you release a bug fix kernel for a security issue that has already been patched months ago in existing distros' kernels. This makes the product a non-starter. If having to wait who knows how long for patched security issues in the kernel is a requirement to have decent performance with low power, I would prefer just to build a mini form factor PC with an AMD 5800X, pay the extra cost of the electricity, and do 10Gb line speed using 50% CPU.
You actually don't need that beefy and power hungry machine, there is a BananaPi R-4 which has great mainline Linux and OpenWRT support (with NAT HW flow offloading, WiFi encryption offloading (WED) working in snapshot and even QoS offloading with TC Flower in the making), already able to route 10Gbps with 0% CPU usage, it costs around $120 on Aliexpress. I would rather support companies and chip makers that care about mainline Linux and provide upstream patches for their SOC features (Mediatek in this example).
@@elektronick88 Yeah I was looking into the BananaPi R4 precisely for all the reasons you cited... unfortunately, there's barely any documentation (most I could find was in Chinese, as *pictures*), no information on how to put the OS together, and it's effectively not even close to a finished product. With my 25 years experience with Linux, I would still take a few weeks to build a usable router out of it. But I suppose I can wait a few months until the whole thing has matured more.
@@RuddODragonFear Not sure what documentation do You need, I have a BPI R3 and i just clone OpenWRT and build firmware for it with things I like (like adblock, tools like tcpdump, container runtimes like Podman etc.), same story for BPI R4, You just have to go through menuconfig and choose what You need. There is also active community willing to help if You have any issues or questions. I agree that there are some things not finished yet or things that require some work and configuration, but hey... this is DIY, not a consumer product, but still I think that the work is worth the effort :)
the problem with closed source, is that lets say, 3 years down the line. what happens when NXP wont update their drivers to work on anything newer than a 3 year old kernel release?
It's not closed source that way. We get the sources as a manufacturers, but can't share them. So in case NXP (which is extremely unlikely) stops working on it, we can pick it up ourselves.
@@tomazzaman that just kicks the problem down the road, how about a test without hw offloading and lets compare the results to make an informed decision, currently we only have the data for 1 side
@@tomazzaman The most likely issue will be the end-of-life for the chip you're using, and thus, newer SDK's dropping support. Once you have the source, you will always have that code. (your license to use it can expire, but that doesn't magically erase it from your drives.)
The Linux kernel see's around 60+ CVE patches a week (2024 Linux Foundation numbers). If you have to manually provide the kernels due to proprietary binary blobs, it takes the customization out of this device no? In fact, there are very interesting scheduling behaviour changes with newer kernels as well.
to be fair, the linux kernel makes a CVE entry for _every_ bug/regression that appears, simply because they can't know if there is any security implications or not. the number of actual CVEs that matter and have any impact is significantly lower than that.
@@commanderguy-rw7tj This isn't totally true. Kernel upstream assigns a CVE for anything that COULD have security implications, not every bug outright. To be fair, a vast majority of bugs in the kernel can be considered a security concern due to the ring level of the relevant code...
Logically, it's a no-brainer to reduce the costs of the cpu and ram, since they're barely being utilized. However, people like extravagance. And, depending on the final price of the product, it may make more sense to keep it as it is. You could also make the cheaper model have those reduced specs while putting the improved specs into the aluminum case model.
I'm sorry, but the third-party closed-source drivers would kill the entire product for me. If I have to rely on a chain of third parties to provide kernels - I might as well just get a proprietary solution to begin with, like a Ubiquiti or MikroTik, that give me an entire polished software suite.
I hope that you're aware of that using closed source patches and drivers in Linux breaks the GPLv2 license under which the Linux kernel is licensed. According to the license, if you do so, the drivers and patches are designated as derivative works and thus licensed under GPLv2. This is the ilcense and thus the law!
Well depends. For example zfs code is licensed under cddl which is incompatible with gpl but binaries are not so you can use zfs with linux. I don't know how those patches are applied here but I would presume they know what they doing to bypass gpl, or maybe not :).
I'd love this to be an awesome router first and foremost, not another do-it-all homeserver. So I think the lower specs are plentiful with the hardware offloading capabilities you demoed.
Go for multiple SKUs. Faster CPU and more RAM should neither require a different PCB layout nor different SW, just different placement. The little extra cost for managing added SKU should be more than offset by larger reached customer base.
Simple routing with the iperf3 example is the easy part. Now check NAT functions and other IP functions. Like DSCP marking/rewriting. Or VXLAN/MPLS. Using the full pipe-line of the SoC to check 10Gbps performance. And in regard to CPU, things like Bird/Quagga to run routing protocols like OSPF/BGP. The more prefixes the more CPU/RAM. Maybe overkill voor CPE devices.
@@tomazzaman How many prefix lists, acl rules, nat entries, etc. are possible in hardware? I remember devices which rebooted when the nat table was full - hello bintec! :)
The number you are showing are impressive, but we are only talking about pure L3 routing, no firewalling. My question to you would be: what CAN'T be offloaded to the CPU and how will it affect the performance? Examples: advanced packet filtering with 1000+ rules, OpenVPN, IDS/IPS, SSL, etc. Are there any kind of encryption/decryption acceleration/offloading built into the CPU such as AES-NI that would benefit those features? In many cases, the CPU sizing for a firewall is directly linked to the resulting throughput of the encrypted tunnels it can deliver (IPSec, OpenVPN, Wireguard, etc.). With that in mind, I don't think that 1.4, 1.6 or 1.8 Ghz will make much of a difference, the 1.8 GHz will be around 30% faster than the 1.4 GHz, this will clearly not double or triple the encrypted tunnel bandwidth. I am curious of seeing the performance of a single OpenVPN tunnel (OpenVPN is single-threaded) on your hardware. For the RAM and CPU, I would go for the most cost-effective option regarding the capacity it provides (biggest bang for the buck), not the cheaper. Which CPU cost less per GHz? What amount of RAM cost less per GB? From what you are saying in the video, the 1.6 GHz CPU with 8 GB of RAM seem to be the most cost-effective combinaison. Choosing a middle ground between the most powerful and the cheapest will probably be attractive to the highest number of early adopters and be commercially more viable initially and accelerate your return on investment. After securing a steady flow of sales, then releasing a lite (less powerful) version and a "beast" version (more powerful) would probably be possible based on which one you get the most requests for.
Not only that, it shows only a single connection with max packet size, what is not a realistic test, that speed with this condition any cpu would have near 0 utilisation (even without any dedicated hardware acceleration). Comparable SoC (not same but comparable by cores and speed and have hardware acceleration made by reputable company that make most SoC used in consumer routees) in comertial product (I will not name company but name starts with A) in real user case it bearly pass 650Mbps speeds even in tests like this it get full 10Gbps speed.
Raw forwarding throughput is cool, but I'm assuming firewall throughput is probably a more practical data point for most of the ethusiasts following this project.
Without offloading I think you could also push that performance using VPP graph nodes in userspace with a AF_XDP socket as the entrypoint. You can use dhcpd and other apps to make it a great home solution.
It depends. Hardware offloading means the CPU and RAM will be free to do "something else". I think it depends on what this means. It would be good to know which kind of configurations, or features break the hardware offload, and what difference does having a slightly better CPU make. Anyway, I think that in most cases, a cheaper CPU with less RAM is a better option. If HW offload is disabled for a feature, is not like 400Mhz CPU clock will amount to a night and day difference in throughput
I've followed this project for a while, while your other claims I'm fine with I have some questions / concerns about the NAT-P / PAT. I get this is a consumer level device but how much memory is this hardware offload module going to have? Looking at the LS1046A block diagram I guess my question centers on the conntrack monitor module (you show the command reading data from it). Whats the max number of connections this module can handle? This would also be a limiting factor for the SPI firewall as well. A less pressing question would be with the L2 bridging is it vlan aware (yes I do see it supports vlans but is the bridging aware of vlan tags?). As much as I would love to say people will pay for the extra CPU speed money talks so while getting yourself established I would not want "too expensive" to be a reason people reject your product while you are trying to get established.
My concern with all this is still security patching and zero day response, especially for 3rd party libraries, OS and SoC drivers etc. Performance is great but I am still not seeing how secure this product's lifecycle will be.
so you suggest providing only a pre-built kernel and without any option for the user to recompile it or upgrade it without losing acceleration? since doesn't seem like you can redistribute the patches?
Go with lower spec device because it will be more affordable and more competitive. It is meant for home usage. Then when you start selling it make upgrade version. Maybe you will ad more ports or add POE+ or something else...
The first packet in a new connection goes through the firewall normally and if it passes, all the subsequent ones from the same connection then bypass it.
Great video again! Just wanted to know : Did you happen to have run some testing without the hardware offloading ? I'm curious about the result. On the price aspect : I think cheaper is the way. But cheaper mean that we rely more on the 'block box' that is the binary and the support that NXP is willing to provide.. If the cheaper variant can still handle some 10G routing capabilities, without hardware offloading, why not !
Thanks for sharing the journey of selecting both hardware and software decision. This is very interesting. Will it have a stateful IPv4 and IPv6 firewall builtin ? What size of the arp table and neighbor table it can support? What will be the size of firewall session table?
I'd be interested in learning if inter-VLAN routing via tagged interfaces requires bypassing the HW offload of the firewall. Most solutions I've used in this space see a huge performance hit once you try to do this, and you usually get encouraged to separate your firewall from your inter-VLAN routing performance when doing so. I have a stupidly overcomplicated network in my house, and right now my core switch is doing inter-VLAN routing, but I'd love to have everything being done on one device.
Seeing how people respond with concerns like bugs, platform deprecation, and so on, I'd say that it would be a good idea to consider making sure that the router is capable of reaching basically the same results in terms of performance with the kernel bypass *disabled*. If you can downsize to 1.4 GHz while keeping CPU utilization under 70-80% on full load, then go for it.
I would like to see 2 models on the market. Those who want the more powerful one and want to shell out for it can, those who want the cheaper version can have that.
Maybe ask the viewers, what else they would run on the router, build a list of those things and depending on time and resource requirements either estimate resource usage or actually test running those services and record usage. Then decide what spec to go with?
I'm not a network guy, but I understand that the traffic component is handled out of CPU and Memory which is very nice. But won't there be use cases where having these ressources will be a must. have? Such as running services for traffic analysis, ads remover, maybe VPN tunnels etc... I'm curious.
VPN tunnels can be offloaded to router hardware, typically. There is also a bunch of types here too. If they can’t and yes that will be the case for wireguard for instance, Traffic analysis is usually handled by a separate process, which can gobble up a mirror of whatever is being routed. If you are taking about IPS that’s a slightly more intense task but just getting metrics on the requests it’s easy. Ads removing, due to the nature of HTTPS is usually done based on origin. That’s easy enough to get a list of firewall policies and feed that into the hardware as an ACL
Hi Tomaz Regular follower your project, I do really like the openness and explanations, I really do appreciate it. Maybe in the future revisions, as this device will be a tinkerers heaven, if you want to even further drop the cost, maybe consider adding NVME or SATA storage option, leave the EMMC optional, and maybe you can revise use SO-DIMM options too, so the customer can decide what Storage + RAM configuration they want to use in the future. I don't know if the CPU will have enough PCI-E lanes, but for a router appliance some kind of SATA based storage would be enough too, if the CPU has controller, or perhaps source a sata controller chip, but I have no idea if that makes sense of a cost perspective and how much complexity it would be to develop and revisit the board. Just chiming in, throwing here some ideas, options, I love your work, keep it up!
Thank you! CPU does indeed come with a SATA controller, but we opted to use those SerDes lanes for Wifi 6/7 instead. Will see how the market responds with this first version and then act accordingly.
I would say that, at least for the initial launch, using the lower-end components to slightly reduce the cost might be the better route, especially if the system is as resource efficient as you're claiming; maybe after getting yourself established you can release a "pro" version of the router with the higher-end components. Also, is there any chance that PoE will be available to any of the ports?
I haven't gotten that far, yet. You can configure what exactly you want to offload, so for tcp dump, you'll have to turn it off, and then back on again once you're done debugging or whatever it was you were doing with tcpdump.
@@tomazzaman would that require a restart of network services/interrupt the network? If yes, i think it's a big problem, if not i think you would need to test the cpu while it's not offloaded to see if you can go for the lower cpu / hw
TCPdump is for dumping the packets that traverse the kernel. If the packet doesn't transverse it, you will not see it. That's why, in high-end routers like Juniper or Cisco, you must tell the ASIC to send the packet through the control pane.
Please do a bandwidth test with slower CPU and less RAM amount (it might take less memory channels to have less RAM and that could really impact bandwidth) to be sure those don't impact hardware packet processing.
The whole "FreeBSD is faster at networking tasks" sounds like the same tired old arguments that that community keeps claiming based on some very old benchmarks and "Netflix is using it so it must be fast". 😄 Nobody has provided any modern benchmarks to prove that this is still the case. On the contrary, the few modern benchmarks that I've seen seem to favor modern Linux. 🙂 VPP will obviously totally smoke kernel networking though, as will hardware offloading. Those hardware offloading engines can definitely be very impressively efficient and performant. 😀
Indeed. FreeBSD was faster in the long long ago. Then Linux vastly improved. But since then, let's say "less enlightened people" have messed with linux networking - eg. removing the route cache - so I can't say either is very effective these days. Hardware routing and switching will ALWAYS be faster than software.
It’s not about benchmarks, it’s about the apis and facilities to explicitly support hw acceleration in their networking stack that linux currently lacks
Whether the A72 cores run at 1.4GHz, 1.6GHz or 1.8GHz is irrelevant. This is because if there is an edge case where the traffic cannot be offloaded, it will just be slow. So yeah, save money on the CPU. As for the RAM, what do we need 8GB for? Most routers ship with 256MiB or 512MiB, some with more. Even if you're doing some amount of BGP, 4 gigs is plenty. But what I want to know more about are the offloading capabilities. It's probably wishful thinking but I want good performance with wireguard and traffic shaping (fq_codel, cake).
I wouldn't consider deploying these routers, since I'd be fully dependent on you for upgrading to future kernel versions. Even decades old industry giants mess up device lifecycles on the regular. Asking for that level of trust in a relatively small operation like this sounds unreasonable to me. On the other hand, if the entire software stack was open source I'd be more than willing to evaluate it for serious use, as the features do sound rather compelling. It would make not blatantly violating the GPL easier as well :^)
Why not have 2-4 mounting options? RAM is drop in replaceable, same with that CPU. 2 extra reels on a pick and place is not that much tho they are going to occupy a few more slots it is feasible since you may free that much space just from resistor and capacitor optimisations. BUT when it comes to logistics there may be problems since you will need to have some way to mark them visually and for the SW. BUT that is simple , you can use a resistor/led to visually indicate what it is or laser engrave a mark or use a sticker/ marker mark . as for the SW knowing what is has a top command should be enough but you can also use a free IO pin to read a pull up/down. Or simply write it in flash/eeprom
I liked your venture due to your open sharing and open source numbers. Been watching since the first videos and plan to buy a finished version. It has been fascinating to see your process. To see you not being able to talk about certain parts due to NDA and closed source software is a turn off. I don’t want some closed source module for this to run properly. I won’t be buying this if it’s using closed source OS software. In addition if you add in software for packet inspection, nat, vpn tunneling, does the offloading still work as well? For a modern router there is more to it than just pure routing of packets.
Tomaž, sorry to sound like a noob, but would this router be a drop in replacement for our Slo fiber internet providers supplied routers like T-2 now that the "fiber everywhere" initiative is progressing nicely. To me the CPU/RAM question boils down to how much extra load is anticipated for non routing activities, otherwise you just end up with hardware waiting as fast as it can. I do like the proposal made by others to have a "basic" and "pro" version if you can tolerate the separate SKU and BOM parts. Do you happen to know if the HW offloading allows for port mirroring for analysis purposes? Future marketing slogan for the device "Routing from Slo is Fast!" [OK, I'll go now. No need to push me into Soča :) ]
I would say the per device unit costs are something to consider. When you are manufacturing 100s or 1000s of units that extra 70 eur per device adds up quickly. So once you have your final design then your challenge is to reduce the cost per unit as much as possible to keep manufacturing costs down and profits up. I have a few thought questions to think about: So I have to wonder at your current cpu/hardware design can you go faster than 10GbE? Is 25, 40, or 100GbE in this hardware’s future where you might need the CPU cores? What impact will full stateful packet inspection have on CPU usage (this is the big question). You can move packet forwarding to asic but how well does the hardware handle layer 3 to layer 7 packet inspection? I would think there would need to be kernel intervention beyond just simple packet forwarding. Before deciding on removing hardware lets see a near final prototype. With that development hardware can you underclock the hardware to see what the impact of 1.0 or 1.4Ghz means to the system’s operations? But from what you have shown there is no need for a 1.6Ghz 4 core cpu. Full packet inspection might change that opinion. On the open source front, I could see two versions (depending on the hardware dev kit licensing). The advanced version of your board design could use the hardware acceleration kit with its licensing fees. And the “basic” version where it does all in kernel routing and packet inspection. The former could use a lesser CPU and ram. The latter one would need the 1.6Ghz and 8GB of ram. Or you can just use the same hardware and the only difference is if you use the close source with the acceleration kit or open source with kernel based packet routing. This isn’t a fully formed plan, only an idea. I realize you have limited hardware for testing, but I wonder what the results would be if you add more senders with the one receiver. The idea is to see if you can flood the 10GbE link until you start to get retrans errors. This would see how the hardware would manage to much data to forward. But all in all your hardware and your design team is doing a great job. You need to find the niche where your hardware has a market.
With regards to cost savings, my question would be are the CPU, RAM, and EMMC storage on sockets, so that we could easily upgrade them afterwards? If we can easily upgrade the device ourselves, then I don’t see a problem with shipping lower speed/capacity/cost components. If you’re going to be like Apple and soldering them all onto the logic board, then I think you’re going to end up needing to provide multiple different versions with different capacities and speeds, much like Apple does. Because that would be our only way to upgrade.
It really depends on how much Physical SFP+ and actual 10G compatible ports you want to have in the final version of your product. If it’s going to be a solution up to 8 ports you can definitely go with a smaller requirement, if you go 16 and up you would need the additional power. You should also consider a upgrade path for PoE++ input and PoE+ Output on Multiple Ports. Today that is a essential feature for a more expensive network appliances. Consider the price of you CPU alone I suspect you solution to end up above 250€ or even close to 400€ for that price point I expect a lot more of a device considering what you can get on the market right now.
Definitely figure out how hard it is to break hardware offload, and what the performance is when you do. But, even assuming the performance is great without hardware offload, I'd still like to see what you can do without the ASK. If you're stuck with the ASK, I would probably just stay within the unifi product line and not bother with fragmenting my network stack.
I mean for comparison, I'm using an off the shelf router with what I believe is a 1.2 GHz CPU and 128 MB of RAM, serving as an edge router for 4 APs and 7 devices actively connected at any given time (give or take). Is the latency good? Absolutely not, but it's held up fine for several years. Pretty much the only issues I've had were some DNS outages (quad9) and some issues with the ISP modem.
@@tomazzamanmaybe hardware packet processing isn't as powerfull in slower CPU (marketing reasons) or it requires full RAM bandwidth. Please also check slower CPU with less RAM (it could mean less memory channels) for hardware accelerated performance.
How is performance altered with IP filtering? One feature of pfsense I use is pfBlockerNG, which blocks IP ranges used by known bad actors. In my current config there are ~16000 ranges being blocked. Is this also completely offloaded? Are there limits to the number of ranges we can block with offload?
I think the main purpose of the router is to perform flawlessly in the workloads you demonstrated in the video. I think therefore that the 1.6GHz or even the 1.4GHz cpu would not compromise the performance of the router. Similarly 8Gb vs 4Gb given one option or the other do not affect significantly the routing performance you should go with the option that makes the product as much competitive in the market as possible. Consideration should be given to what kind of UI and life comfort features you plan to make available in the product. But I don't think a UI, even with heavy graphical sophistication and maybe historical logging would be affected much by lower clock speed CPU or 4Gb of RAM. My 2 cents.
Nice video :) Personally I would strongly prefer to keep the current specs (1.6 GHz CPU and 8 GBs of RAM), but I would buy it even with the lovered specs.
I think it would make sense to keep the hardware modest. 1.5 to 1.6 Ghz would be fine, 2 or 4 GB of RAM would be fine. But I guess the big part of this is would be the shipping feature set.
I already see two version of this router emerging, depending on what traffic can really be offloaded and what software configurations would affect(and disable) hardware offloading, because that is important on a 10G router... A cheaper version is preferred, yes, for the customers and for you guys... And that's probabably your main goal. But a higher end version might make sense too, maybe the faster CPU or something else added, maybe more connectivity options, or more RAM(example). That might be a different product, one that completes a product line. ;)
I would be fine with the lower clock version, but reducing RAM? Not something I think should be done. Maybe for simple routing not needed, but if you want to use the router with additional features on the software side, RAM could become more important. So I rather have the RAM available.
Well... speaking about performance vs price, it would be much easier to make a conclusion by seeing the practical tests and seeing approximate price. Otherwise, pretty much interesting project, good luck with your routers!
This kind of hardware offloading is incredibly cool! One thing that would help with the decision is to see what the CPU utilization looks like at 10Gib without the hardware offloading. I would still prefer a higher-end chip and open-source core if we're talking about a difference between closed-source and open-source of 1 - 10%. But if the difference is more like 30 - 40%, I might lean more towards the closed-source side. Keep in mind, too, that reducing the CPU speed can also have more damaging effects on IPS/IDS. Keep these videos coming; your level of engagement and openness is definitely giving me hope for the product!!
I'm a bit concerned that the packets handled by hardware offloading won't be available in the kernel - will the device support packet inspection or other forms of packet manipulation outside of the dedicated chip?
i could see a market for undecided users that would purchase the router if it's inexpensive enough and would later purchase a top of the line one if it met expectations
At the start you mentioned that FreeBSD is better than Linux for routing, but I've found the opposite to be true. I was testing opnsense vs OpenWrt on an old SFF PC with an Intel i5-9500 and X540-T2 NIC. OpenWrt was able to successfully reach 8Gbps over a single iperf connection over the internet. Opnsense struggled to even hit 3Gbps, with one CPU core pegged at 100% (which makes me think that maybe NAT or conntrack is single threaded... not sure)
My question is, what would 8GB of RAM even be used for if we kept it? Do you reckon we could setup a home assistant server directly on the router? Would that benefit the extra specs that much? Personally, I am just fine with the reduced specs. Specifically, 8GB RAM is overkill in my opinion, even for a top-of-the-line variant.
Aim for €149 retail price, and go with 4GB and 1.4Ghz if you need. With offloading this good, no need to go up in specs, better to create a product first with mass apeal then later launch a more expensive option. If you want to make this more than a one product series, you need mass appeal, like the first EdgeRouter Lite, which really launched Ubiquiti.
Very cool project but for me if it can not be distribute like normal OpenWRT distribution is a deal breaker (not a fork crap staying years behind but true update one). Middle cpu variant is good option.
are the CPUs not footprint compatible? for the memory, couldn't you just design for the full 8GB and either leave half of them unpopulated or use lower density ICs? It seems like you could pretty easily do variants; use the same boards and stencils and just different pick-and-place files. have the lower-end variant (4GB/1.4GHz) be the main product where you allocate all the inventory risk, and make the higher-end variant (8GB/1.6GHz or maybe 1.8GHz) in smaller "limited-edition" runs at whatever quantity you're confident you can sell through quickly. Due to the smaller purchasing quantities for the CPU and ram, the limited high-end variant's cost overheads will be higher, but you can just price it accordingly, and then if demand for the high-end variant is greater than expected, you can increase production, and pocket the reduced overhead as extra margin on your halo product, while still offering something reasonable for the rest of your customers.
As long as it can handle everything it is supposed to, why waste resources? Question is, what you want to ship and what people can do with it. From what I understand this is a router, not a server or NAS. So I’m not sure if we would run other software with it, that would require such extra headroom. From a marketing standpoint and interesting opportunity: Segmentation - if you can show you can build high end first, then maybe you can then later release a less customisable version. You can see this in the 3D Printer space with Bambulab, they released their high end x1c model and then introduced more core feature focused versions that were cheaper.
My thoughts exactly, go with a more powerful version first, to both show we can and also test how the market responds, and then, if there's demand for it, release a "lite" version.
for ram: even 2GB should be more than enough. throw 32 gigs of emmc onto that (mostly just for logging. if it weren't for that, 8 gigs would be enough), and use the cheapest CPU you can get that still handles 10 gbps. would be nice to see the CPU be able to handle at least 500mbps of wireguard/openvpn throughput though.
Are you still planning on this being capable of IDS/IPS inspection? I thought I remembered that was the original plan. I am wondering if all this hardware acceleration and low power goes out the window as soon as you start doing that though? That was a problem on a lot of Ubiquiti earlier gen devices where their 10gb Unifi router went from 10gb throughput to 1~ gb throughput with IPS on because it had to move the traffic to general CPU cores. Newer gen Unifi devices changed that up by using different SoCs to get more IPS speed. This product doesnt have to be any kind of full decrypt/encrypt interception inspection with certificates on end devices sort of thing, but having a standard IPS type system (probably suricata running ET Open ruleset?) with pattern and packet signature matching to detect malware or hack attempts in encrypted traffic would be great to have. I know not everyone is of this opinion, but to me, all firewalls should have that type of traffic inspection as standard now days as a basic stateful firewall just isn't good enough anymore for evolving threats. Id be fine with a 1.4ghz CPU as long as it has the same HW accel blocks in it since the CPU cores dont really matter. Edit: Looking at the Layerscape CPUs, I see that the 1046A does not have a pattern matching HW engine, so I am guessing if you do enable IDS/IPS that the speed will drop a ton like I was thinking. I also see that the next gen LS2084A CPU steps things up a little bit (but for a lot more money) and increases the security engine from 10gb to 20gb, adds a 20gb data compression/decompression engine, and adds on a 10gb pattern matching engine. So this would be the CPU to use in order to get 10gbps IDS/IPS I would think. Sadly it costs $285~, though probably a bit less when buying by the thousand. At that point though IDK if it is even worth using the next gen CPU when you could step up to an even higher end on, the LX2082A that gets you 50gbps HW accel throughput for only an extra $15. lol. Though I dont see the LX2082A having a pattern matching engine like the LS2084A has... So maybe that's why it is not much more for so much more throughput, as it is missing the pattern match part the "lower end" part has.
1.4 GHz / 4GB RAM is probably good enough for a router with HW offloading in my opinion. You can release a more powerfull version at a later point in time. CPU is probably not an issue unless someone wants to run a firewall on it. The 8GB RAM could be useful for running containers to handle DNS, Caching, etc, but even with 4GB RAM it should be fine for basic stuff.
I understand the trade off with binary blobs and proprietary hardware. The biggest problem is security audits and ongoing support. For security, we just have to trust NXP, but I expect there isn't any fully open source options out there with hardware offloading, without going full insanity mode and building a custom RISCV chip and drivers to do it. The bigger issue IMO is ongoing support and maintenance, they are supplying not only kernel patches, but also patches to userspace programs as well. Those patches need to be kept in sync with subsequent releases of kernels / programs. This can be mitigated a bit by going with an LTS kernel but security updates are a certainty and again you are going to be relying on NXP to support that product. I understand all hardware has a shelf life, but given this is a 10 gig router for home / small business users, outside of hardware failures I would expect these to last well past a decade. It honestly highlights a design issue with the linux kernel if it requires patches to both the kernel and userspace programs in order to effectively make use of hardware features.
I'm kind of looking forward to a lower cost model, but if there was a higher cost option, I might pick that just because speed is important for some cases.
Do not go with cheaper only option. Use extra cpu/memory for extra features to justify price increase and this will give you more competitive product in the end. You are not in SOHO market, you're in enthusiast/lower pro segment. Having good IDS/IPS or, I do not know, maybe, graylog-style log ingesting server and viewer might be the reason for somebody to prefer your solution over putting another server or SBC for network management in remote location. For me that is a common problem with any small setup - you always need some box for things like pihole, syslog, ups monitoring or just to log into remotely and do speedtest.
@tomazzaman I've following your updates on this router for quite some time now. I personally believe if anyone if building new hardware in today era, it should be fast enough to handle most network security services/apps. I'm curious to know how many 10gbps streams can it handle simultaneously before it starts to throttle. Second, Will the cpu be able to handle IDS and perhaps IPS?
As for the number of streams, I can't answer that yet, as I've literally just gotten my hands on the ASK so it's something I plan to test (and report here). And as for IDS, from the documentation, it supports up to 32k rules with Snort, so it should still work full speed, but I need to test this too. Stay tuned!
If you guys only decided to go with MTK as I've said many videos ago.... Yea propietary BSPs dont age well, GPL violations etc... as always. I'm not surprised anymore... Man those chips are expensive! What they think they are? Broadcom or Nvidia? Still thinking that MTK Filogic 880 would have been a lot cheaper and more functional for a FOSS router. With Wireless, PPoE and NAPT accelleration available with FOSS drivers. More accellerated protocols is supported by the SOC but those patches are available as FOSS.
I figure it out right away. That's why Windows has an advantage with well supported hardware, and Apple makes hardware and software, so there should be no issues there (as always, I have noticed on older hardware that even they control both hardware and software, they removed the very optimized drivers in favor to something which doesn't perform as good as it was, but it's so old people don't care about it). Drivers are what enable full use of the hardware, offloading all the operations to the ASICs (Application-Specific Integrated Circuits). And the software needs to call the driver functions to enable the hardware processing. The same issue described here with Linux is present on "alternative firmware" for home routers / access points like the awful DD-WRT (and OpenWRT for that matter) and other variants which the developers have reverse engineered the hardware of those routers to provide support, but these reverse engineered drivers are basic most of the times and process everything on the CPU. For those wondering what I'm talking about, think this way: you have a pretty decent GPU on your system, but no connection to the Internet or embedded drivers for that GPU. Windows enables it using the "STANDARD VGA ADAPTER" driver. It can enable a high resolution, SVGA, XVGA and the sorts, but window composition (moving windows, drawing) is bad. And it cannot draw video frames fast enough or accurately that way, simple tasks take up all your CPU cores. That's what's happens when you cannot offload processing to the many auxiliary processors, ASICs and utilize the hardware you own to the fullest. Pretty interesting video.
If you keep asking public opinion on every step on the way, you are never going to complete anything. You have a product, you had a vision, you wanted something good for the world. Make it happen. Stop asking should I go this route, or that route, you need to choose the path for us. As a consumer yourself, you should know that a small fraction of the price is not going to keep us from buying. Make a great product that will cover everything, the sales will take care for themselves. It is okay if I pay for pricey hardware. I just do not want SaaS embedded into the software or do not lock it down, so we can freely load up any solution we desire. I strongly believe you will deliver something unique at the end goal.
I think you go with a cheaper version out of the gate, but know you could create a more performant version for those who want to pay for it. Not everyone needs all the bells and whistles. One question on the 10 gig ports. Can they be used as a WAN or LAN interface? I have 5 gig fiber Internet.
You should not save on RAM unless it's modular. RAM is never enough, and you should leave headroom for stuff to run services on top of the OS. Go for max RAM even if it ups the cost.
I'd prefer that we keep the 1.6Ghz and 8GB configuration. The more RAM, the better, in my opinion. Not to mention that with the larger amount of RAM, it's easier to keep the device going for much longer if you have the RAM to support it, not that it'd likely be needed given the impressive performance here, but still... I say keep it.
I know that this will be a significant overhead for logistics , but cant we have 2 versions? For example i want to have the best of the best no mater the cost (in resonable ranges) , while others might not need a faster cpu or more ram
I liked your venture due to your open sharing and open source numbers. Been watching since the first videos and plan to buy a finished version. It has been fascinating to see your process. To see you not being able to talk about certain parts due to NDA and closed source software is a turn off. I don’t want some closed source module for this to run properly. I won’t be buying this if it’s using closed source OS software.
I'm only slowly entering the world of custom networking gear and not using the ISP provided gear with a couple of the cheapest APs and switches possible, so apologies for the novice questions and random thoughts; Would it make sense to choose a mix of the two, so the 1.4GHz with the 8GB of ram? To me it seems like the 0.2Ghz is much less of a downgrade, than the 50% less memory. I don't know what sort of services would benefit from another 4GB of ram or, perhaps more interestingly, 0.2Ghz faster processor. Would something like WireGuard run noticeably quicker on the 1.8 vs the 1.4Ghz, would the 4GB ram be enough for it? What are some services that you would usually run on a router? Nonetheless I do think the first version should probably be the fastest to get your name out and known as solid, if slightly over built, routers
![GUILTY GEAR -STRIVE- Season Pass 4 Playable Character #1 [Queen Dizzy] Trailer](http://i.ytimg.com/vi/nXDsAgYXXv4/mqdefault.jpg)
Do you prefer a slightly slower version of the CPU if it makes the device cheaper, or do we go all in?
Depends. Whether the sum of features and specs will justify the price. If you manage to convince the audience that the router as a sum is worth it , then people will justify a small increase in price for a better cpu. It depends if subconsciously we will consider it as high or mid router , thus beginning to compare it with other routers in the market.
I'd go for the cheaper one for more possible customers.
I vote keep the cpu the same. means those who want a pure oss build can still get decent performance.
i am excited about your router, i prefer lower cost, because very expensive consumer-grade routers have around 2GB of memory, like this Asus ROG Rapture GT-BE98 that has 2 10-Gbit ports, i think 4GB of RAM is allready an overkill, i am going to keep watching your videos. Thank you!
What about two versions?
Unpublished kernel patches for a published kernel binary sounds like an outright GPL2 violation.
It's a tough one, but I would benchmark Wireguard performance and opt for the cheaper alternative if the numbers are still great.
Big plus for Wireguard!
Wireguard is stateless so I would be surprised if it can't be hardware accelerated once the crypto algorithms are supported.
I don't think he can show any tests that would make NXP look bad. NDA and pre-sales agreements are active so all we get is fluff from here on out.
@@davidmcken Wireguard is designed to run fast on CPU, using algos that are best suited for CPU. The flip side of this is that nobody is going to make a hardware accelerator for them because they run so fast on CPU anyway
Wireguard doesn't run in any hardware acceleration - even on full-fat AMD or Intel CPUs... It's highly unlikely that it can be offloaded.
@5:40 DING! Wrong. The GPL is very clear on this. ANY modification done to the kernel source MUST be made available to anyone to whom you've provided a binary. In short, any kernel you compile, your customers must also be able to compile. HOWEVER, external, 3rd party _modules_ can be 100% closed source. I've been here, it can be messy. And yes, giant manufacturers like Broadcom and NXP don't care about breaking the law, or forcing you to.
(And there's some debate about making changes to the kernel simply to facilitate external closed modules. For example, undoing the "taint" caused by non-GPL modules, so one can use a GPL-only exported function.)
Agreed, patches and modification to the kernel are covered by GPL and need to be public. A single binary is all that's needed and even then it's still a poor argument for closed sourced code within this project. If he held NXP to the fire and made them comply with GPL in the first place then I would be more open to using their secret sauce.
Oh thats a netting that noone should cut, GPL my beloved. ;(
The solution is simple: provide NXP as binary modules
@@francoisscala417 The issue is that the module must still hook into the kernel APIs.
Nettrack hasn't all the necessary APIs for offloading exposed. There is some work in the progress but so far the HW accelleration possible is very very limited.
This kind of patches are mainly in the core part of the kernel and not in the external modules.
Then they also patched userspace utilities for offloading, meaning you are running more unknown modified software.
That have to hook into the kernel in a different way for offloading. I'm thinking about tcpdump that is not able to see the traffic. NXP probably has a patched version.
then there are the various VPNs stacks, the ppp daemon etc. it will become a huge mess very easily.
@@francoisscala417 They *could* release the binary module, and then open source a wrapper to load the binary module in the kernel. However this isn't what they do in most cases - and this project isn't anywhere near big enough to convince NXP to change their ways...
This just means that without being transportable to newer kernels and being able to be built without relying on NXP for everything, this entire project is lifetime-limited e-waste.
Once you don't want to support it anymore, or it gets too hard to keep porting changes - that's the EOL for your product.
I am concerned that relying on the hardware offload capabilities will spell an early end for this device, for example if a bug in handling certain packet types is found (e.g. with new protocols afterwards) or the limitations that this places on additional features. For example the Ubiquiti EdgeRouters can’t do QoS with the hardware offload enabled - just an example that in general you end up being locked in to the hardware capabilities.
there is just tcp/udp so far, everything else is above this.
And when that happens, it will fall back to the CPU, like on EdgeRouter. The feature will be available, but not offloaded. In that case, performance will be the same as not having HW Offload in the first place. No HW offload for this kind of CPUs means gigabit speeds at best. It is much better to have it, and in the edge cases where someone needs something that it is not offloaded the performance will be the exact same as if HW offload was never implemented. The amount of CPU power you need for 10Gbps routing depends a lot on the features you need. My router has 4x4.1Ghz Core i5 8th gen cores and 8GB DDR4 RAM, with IPS it can only do 7.5Gbps on BSD (pfSense) and 15.5Gbps on Linux. In this particular scenario, 1.2Ghz vs 1.8Ghz won't make a difference.
@@andibiront2316yes, so you can't cheap out on cpu and memory. Or the project might fail.
@@MelroyvandenBerg It depends. The performance difference from 200/400Mhz probably is minimal. But we need benchmarks to be sure, I guess.
This is a good point I didn't think of, I've also made a comment that the hardware offloading has no bearing on the CPU choice - We need to know what changing the CPU WILL affect, not what it won't.
A closed source kernel driver from the SoC manufacturer ???
The problem is that you are stuck with a specific Linux kernel release that they support
Not really, no. We get the sources, we just can distribute them, meaning we can patch whichever kernel version we want with them.
@@tomazzaman Hope you have some lawyers onboard, sounds like a GPL violation
@@tomazzaman That's not how drivers work. For example, the syscalls can change from release to release. Binaries that work with BSD will not work for Linux.
Yeap. See Also: Broadcom's "Open Source" NDK. (it's a huge binary blob with a few lines of shim code.) If they have an NDA and access to SDK _source_ then they can build the driver(s) for whatever kernel versions supported - or they're willing to port to. Sadly, this is the way of the modern world... want performance better than a hamster in a wheel, you'll need hardware packet processing, and no one openly documents that stuff.
@@jfbeam I wonder how hard it would be co convince Linus Torwalds or someone in the Linux kernel community to provide an open source hardware acceleration for networking.
Especially if you could pay them from the router's development budget.
This would effectively create an open source competition to the weird NDA garbage that you just obtained.
Correct me if I'm wrong but this (5:35) seems like a violation of the GPL to me. I don't think you are allowed to distribute derivative kernels under non compatible (i.e. non copyleft) licenses. That means that the proprietary software can only run in loadable modules and not patches built into the actual kernel binary. I might have misunderstood what you meant here though.
Yeah it seem alike that will run afoul of the GPL, I presume NXP also knows this and has a proper solution for it. (Modules, dkms etc)
@@EraYaN closed source kernel patches will violate the GPL anyway and by the time someone packs and sells this thing, that is, distributing as in the kernel, the recipient has all the rights in the worled to get the code for both the kernel patches *and* the drivers. To bypass GPL with dkms etc, you need the user to interactively install the needed stuff. If it's automated, it is as distributed, meaning it breaks GPL.
@@roysigurdkarlsbakk3842😮
Umm...
Only the parts that are written by others need to be attributed to and source made available.
Anything you write custom can be proprietary, that's exactly how Cisco, Oracle and tons of others do it. Patches are arguable because someone else wrote the patch to make the modification, thus one could argue that the patch it's self is an original work.
(Similar to claiming your artwork isn't original because you used my canvas material to make it, or Crayola's red paint to accent the sunset....)
@@jamess1787 If you change even a single bit in a GPL'd source file, YOU. MUST. PUBLISH. YOUR. CHANGES. Cisco, Oracle, "and tons of others" _do_ publish their changes to GPL'd code. Their products are built from a sizable amount of non-GPL stuff, and they obviously don't publish any of that.
(It used to be easier to find those tarballs, but Cisco being Cisco makes you jump through hoops to get anything these days.)
These videos are amazing. I’ve never seen someone talk so much and say nothing. Amazing!
what?
I don't want to wait until you release a bug fix kernel for a security issue that has already been patched months ago in existing distros' kernels.
This makes the product a non-starter.
If having to wait who knows how long for patched security issues in the kernel is a requirement to have decent performance with low power, I would prefer just to build a mini form factor PC with an AMD 5800X, pay the extra cost of the electricity, and do 10Gb line speed using 50% CPU.
You actually don't need that beefy and power hungry machine, there is a BananaPi R-4 which has great mainline Linux and OpenWRT support (with NAT HW flow offloading, WiFi encryption offloading (WED) working in snapshot and even QoS offloading with TC Flower in the making), already able to route 10Gbps with 0% CPU usage, it costs around $120 on Aliexpress. I would rather support companies and chip makers that care about mainline Linux and provide upstream patches for their SOC features (Mediatek in this example).
@@elektronick88 Yeah I was looking into the BananaPi R4 precisely for all the reasons you cited... unfortunately, there's barely any documentation (most I could find was in Chinese, as *pictures*), no information on how to put the OS together, and it's effectively not even close to a finished product. With my 25 years experience with Linux, I would still take a few weeks to build a usable router out of it.
But I suppose I can wait a few months until the whole thing has matured more.
@@RuddODragonFear Not sure what documentation do You need, I have a BPI R3 and i just clone OpenWRT and build firmware for it with things I like (like adblock, tools like tcpdump, container runtimes like Podman etc.), same story for BPI R4, You just have to go through menuconfig and choose what You need. There is also active community willing to help if You have any issues or questions. I agree that there are some things not finished yet or things that require some work and configuration, but hey... this is DIY, not a consumer product, but still I think that the work is worth the effort :)
@@RuddODragonFear OpenWrt has support for it, albeit only on snapshot for now. Next major release should have stable support.
@@subrezon Let's hope that comes soon, and then I'll experiment with it.
the problem with closed source, is that lets say, 3 years down the line. what happens when NXP wont update their drivers to work on anything newer than a 3 year old kernel release?
It's not closed source that way. We get the sources as a manufacturers, but can't share them. So in case NXP (which is extremely unlikely) stops working on it, we can pick it up ourselves.
@@tomazzaman that just kicks the problem down the road, how about a test without hw offloading and lets compare the results to make an informed decision, currently we only have the data for 1 side
@@ss-xy2imexactly. I can't make a decision without a benchmark without hw offloading
@@tomazzaman Haha, maintaining and keeping up to date in C and ASSEMBLER in a changing Kernel?
@@tomazzaman The most likely issue will be the end-of-life for the chip you're using, and thus, newer SDK's dropping support. Once you have the source, you will always have that code. (your license to use it can expire, but that doesn't magically erase it from your drives.)
The Linux kernel see's around 60+ CVE patches a week (2024 Linux Foundation numbers). If you have to manually provide the kernels due to proprietary binary blobs, it takes the customization out of this device no? In fact, there are very interesting scheduling behaviour changes with newer kernels as well.
to be fair, the linux kernel makes a CVE entry for _every_ bug/regression that appears, simply because they can't know if there is any security implications or not. the number of actual CVEs that matter and have any impact is significantly lower than that.
@@commanderguy-rw7tj This isn't totally true. Kernel upstream assigns a CVE for anything that COULD have security implications, not every bug outright. To be fair, a vast majority of bugs in the kernel can be considered a security concern due to the ring level of the relevant code...
@@commanderguy-rw7tj The new maintainer had to officially make a presentation regarding the topic due to EU regulation changes.
Logically, it's a no-brainer to reduce the costs of the cpu and ram, since they're barely being utilized. However, people like extravagance. And, depending on the final price of the product, it may make more sense to keep it as it is. You could also make the cheaper model have those reduced specs while putting the improved specs into the aluminum case model.
I just want it to be in a good price/performance ratio so why use a more powerful and expensive chip if it won't affect much performance
if he's paying $72-92 per CPU, price has to be at least $800 either way
are 4GB of RAM still enough if you run a copuple of docker containers and VMs on it?
I'm sorry, but the third-party closed-source drivers would kill the entire product for me. If I have to rely on a chain of third parties to provide kernels - I might as well just get a proprietary solution to begin with, like a Ubiquiti or MikroTik, that give me an entire polished software suite.
I hope that you're aware of that using closed source patches and drivers in Linux breaks the GPLv2 license under which the Linux kernel is licensed. According to the license, if you do so, the drivers and patches are designated as derivative works and thus licensed under GPLv2. This is the ilcense and thus the law!
Doesn't this make the company that he obtained those codes from break the law, not him?
@@hubertnnn It's kind like saying that a drug dealer is innocent because they were just re-selling the heroin - they did not make it originally.
Well depends. For example zfs code is licensed under cddl which is incompatible with gpl but binaries are not so you can use zfs with linux. I don't know how those patches are applied here but I would presume they know what they doing to bypass gpl, or maybe not :).
I'd love this to be an awesome router first and foremost, not another do-it-all homeserver. So I think the lower specs are plentiful with the hardware offloading capabilities you demoed.
Go for multiple SKUs. Faster CPU and more RAM should neither require a different PCB layout nor different SW, just different placement. The little extra cost for managing added SKU should be more than offset by larger reached customer base.
Agreed!
Would be interesting to see the performance with IDS/IPS running. Since ,AFAIK needs a lot of processing power, would determine the specs of the cpu.
Simple routing with the iperf3 example is the easy part. Now check NAT functions and other IP functions. Like DSCP marking/rewriting. Or VXLAN/MPLS. Using the full pipe-line of the SoC to check 10Gbps performance.
And in regard to CPU, things like Bird/Quagga to run routing protocols like OSPF/BGP.
The more prefixes the more CPU/RAM.
Maybe overkill voor CPE devices.
On the way! Or should I say, under way! 💪
@@tomazzaman How many prefix lists, acl rules, nat entries, etc. are possible in hardware? I remember devices which rebooted when the nat table was full - hello bintec! :)
The number you are showing are impressive, but we are only talking about pure L3 routing, no firewalling. My question to you would be: what CAN'T be offloaded to the CPU and how will it affect the performance? Examples: advanced packet filtering with 1000+ rules, OpenVPN, IDS/IPS, SSL, etc. Are there any kind of encryption/decryption acceleration/offloading built into the CPU such as AES-NI that would benefit those features? In many cases, the CPU sizing for a firewall is directly linked to the resulting throughput of the encrypted tunnels it can deliver (IPSec, OpenVPN, Wireguard, etc.). With that in mind, I don't think that 1.4, 1.6 or 1.8 Ghz will make much of a difference, the 1.8 GHz will be around 30% faster than the 1.4 GHz, this will clearly not double or triple the encrypted tunnel bandwidth. I am curious of seeing the performance of a single OpenVPN tunnel (OpenVPN is single-threaded) on your hardware.
For the RAM and CPU, I would go for the most cost-effective option regarding the capacity it provides (biggest bang for the buck), not the cheaper. Which CPU cost less per GHz? What amount of RAM cost less per GB? From what you are saying in the video, the 1.6 GHz CPU with 8 GB of RAM seem to be the most cost-effective combinaison. Choosing a middle ground between the most powerful and the cheapest will probably be attractive to the highest number of early adopters and be commercially more viable initially and accelerate your return on investment. After securing a steady flow of sales, then releasing a lite (less powerful) version and a "beast" version (more powerful) would probably be possible based on which one you get the most requests for.
Not only that, it shows only a single connection with max packet size, what is not a realistic test, that speed with this condition any cpu would have near 0 utilisation (even without any dedicated hardware acceleration). Comparable SoC (not same but comparable by cores and speed and have hardware acceleration made by reputable company that make most SoC used in consumer routees) in comertial product (I will not name company but name starts with A) in real user case it bearly pass 650Mbps speeds even in tests like this it get full 10Gbps speed.
@@viaujoc Exactly, he should test with Trex and use a firewall. Everything else is just hypocrisy.
Raw forwarding throughput is cool, but I'm assuming firewall throughput is probably a more practical data point for most of the ethusiasts following this project.
Without offloading I think you could also push that performance using VPP graph nodes in userspace with a AF_XDP socket as the entrypoint. You can use dhcpd and other apps to make it a great home solution.
You had my curiousity. Now, you have my attention.
Same. And if he manages to provoke our anticipation then this project is a sucess.
It depends. Hardware offloading means the CPU and RAM will be free to do "something else". I think it depends on what this means. It would be good to know which kind of configurations, or features break the hardware offload, and what difference does having a slightly better CPU make. Anyway, I think that in most cases, a cheaper CPU with less RAM is a better option. If HW offload is disabled for a feature, is not like 400Mhz CPU clock will amount to a night and day difference in throughput
I've followed this project for a while, while your other claims I'm fine with I have some questions / concerns about the NAT-P / PAT. I get this is a consumer level device but how much memory is this hardware offload module going to have? Looking at the LS1046A block diagram I guess my question centers on the conntrack monitor module (you show the command reading data from it). Whats the max number of connections this module can handle? This would also be a limiting factor for the SPI firewall as well.
A less pressing question would be with the L2 bridging is it vlan aware (yes I do see it supports vlans but is the bridging aware of vlan tags?).
As much as I would love to say people will pay for the extra CPU speed money talks so while getting yourself established I would not want "too expensive" to be a reason people reject your product while you are trying to get established.
My concern with all this is still security patching and zero day response, especially for 3rd party libraries, OS and SoC drivers etc. Performance is great but I am still not seeing how secure this product's lifecycle will be.
so you suggest providing only a pre-built kernel and without any option for the user to recompile it or upgrade it without losing acceleration?
since doesn't seem like you can redistribute the patches?
same with strongswan patches
if there is a security issue, there would be no way for an end-user to upgrade until it's fixed by NXP?
Valid question, thank you - added on my list for when I talk to NXP, and I'll make a dedicated video to answer all these.
Holly molly!
That is fantastic!
Great job!!!
Go with lower spec device because it will be more affordable and more competitive. It is meant for home usage. Then when you start selling it make upgrade version. Maybe you will ad more ports or add POE+ or something else...
How does the system behave when there are actual firewall rules involved? So when it isn't just fast-path?
The first packet in a new connection goes through the firewall normally and if it passes, all the subsequent ones from the same connection then bypass it.
protip: use `ip -br a` instead of just `ip a`, or if you really want all the info, then use `ip -o a` to format it on "one line" per entry
also "btop" is much better than "htop" imho
btop allows you to e.g. hide all the other panes other than cpu graph, if you want to, for example
Great video again!
Just wanted to know : Did you happen to have run some testing without the hardware offloading ? I'm curious about the result.
On the price aspect : I think cheaper is the way. But cheaper mean that we rely more on the 'block box' that is the binary and the support that NXP is willing to provide.. If the cheaper variant can still handle some 10G routing capabilities, without hardware offloading, why not !
Thanks for sharing the journey of selecting both hardware and software decision. This is very interesting. Will it have a stateful IPv4 and IPv6 firewall builtin ? What size of the arp table and neighbor table it can support? What will be the size of firewall session table?
I'd be interested in learning if inter-VLAN routing via tagged interfaces requires bypassing the HW offload of the firewall. Most solutions I've used in this space see a huge performance hit once you try to do this, and you usually get encouraged to separate your firewall from your inter-VLAN routing performance when doing so. I have a stupidly overcomplicated network in my house, and right now my core switch is doing inter-VLAN routing, but I'd love to have everything being done on one device.
Seeing how people respond with concerns like bugs, platform deprecation, and so on, I'd say that it would be a good idea to consider making sure that the router is capable of reaching basically the same results in terms of performance with the kernel bypass *disabled*. If you can downsize to 1.4 GHz while keeping CPU utilization under 70-80% on full load, then go for it.
I would like to see 2 models on the market. Those who want the more powerful one and want to shell out for it can, those who want the cheaper version can have that.
Maybe ask the viewers, what else they would run on the router, build a list of those things and depending on time and resource requirements either estimate resource usage or actually test running those services and record usage. Then decide what spec to go with?
I'm not a network guy, but I understand that the traffic component is handled out of CPU and Memory which is very nice. But won't there be use cases where having these ressources will be a must. have? Such as running services for traffic analysis, ads remover, maybe VPN tunnels etc... I'm curious.
VPN tunnels can be offloaded to router hardware, typically. There is also a bunch of types here too. If they can’t and yes that will be the case for wireguard for instance,
Traffic analysis is usually handled by a separate process, which can gobble up a mirror of whatever is being routed. If you are taking about IPS that’s a slightly more intense task but just getting metrics on the requests it’s easy.
Ads removing, due to the nature of HTTPS is usually done based on origin. That’s easy enough to get a list of firewall policies and feed that into the hardware as an ACL
Hi Tomaz
Regular follower your project, I do really like the openness and explanations, I really do appreciate it.
Maybe in the future revisions, as this device will be a tinkerers heaven, if you want to even further drop the cost, maybe consider adding NVME or SATA storage option, leave the EMMC optional, and maybe you can revise use SO-DIMM options too, so the customer can decide what Storage + RAM configuration they want to use in the future.
I don't know if the CPU will have enough PCI-E lanes, but for a router appliance some kind of SATA based storage would be enough too, if the CPU has controller, or perhaps source a sata controller chip, but I have no idea if that makes sense of a cost perspective and how much complexity it would be to develop and revisit the board.
Just chiming in, throwing here some ideas, options, I love your work, keep it up!
Thank you! CPU does indeed come with a SATA controller, but we opted to use those SerDes lanes for Wifi 6/7 instead. Will see how the market responds with this first version and then act accordingly.
I would say that, at least for the initial launch, using the lower-end components to slightly reduce the cost might be the better route, especially if the system is as resource efficient as you're claiming; maybe after getting yourself established you can release a "pro" version of the router with the higher-end components.
Also, is there any chance that PoE will be available to any of the ports?
Excuse my ignorance but if hw offloading means you cannot run a tcpdump how would you run a tcpdump given its a network device?
This is a really good question - I (hope) that one can use hw offloading for packet capture aswell?
I haven't gotten that far, yet. You can configure what exactly you want to offload, so for tcp dump, you'll have to turn it off, and then back on again once you're done debugging or whatever it was you were doing with tcpdump.
@@tomazzaman would that require a restart of network services/interrupt the network? If yes, i think it's a big problem, if not i think you would need to test the cpu while it's not offloaded to see if you can go for the lower cpu / hw
TCPdump is for dumping the packets that traverse the kernel. If the packet doesn't transverse it, you will not see it. That's why, in high-end routers like Juniper or Cisco, you must tell the ASIC to send the packet through the control pane.
Please do a bandwidth test with slower CPU and less RAM amount (it might take less memory channels to have less RAM and that could really impact bandwidth) to be sure those don't impact hardware packet processing.
The whole "FreeBSD is faster at networking tasks" sounds like the same tired old arguments that that community keeps claiming based on some very old benchmarks and "Netflix is using it so it must be fast". 😄
Nobody has provided any modern benchmarks to prove that this is still the case. On the contrary, the few modern benchmarks that I've seen seem to favor modern Linux. 🙂 VPP will obviously totally smoke kernel networking though, as will hardware offloading.
Those hardware offloading engines can definitely be very impressively efficient and performant. 😀
Indeed. FreeBSD was faster in the long long ago. Then Linux vastly improved. But since then, let's say "less enlightened people" have messed with linux networking - eg. removing the route cache - so I can't say either is very effective these days. Hardware routing and switching will ALWAYS be faster than software.
It’s not about benchmarks, it’s about the apis and facilities to explicitly support hw acceleration in their networking stack that linux currently lacks
Whether the A72 cores run at 1.4GHz, 1.6GHz or 1.8GHz is irrelevant.
This is because if there is an edge case where the traffic cannot be offloaded, it will just be slow.
So yeah, save money on the CPU.
As for the RAM, what do we need 8GB for? Most routers ship with 256MiB or 512MiB, some with more.
Even if you're doing some amount of BGP, 4 gigs is plenty.
But what I want to know more about are the offloading capabilities.
It's probably wishful thinking but I want good performance with wireguard and traffic shaping (fq_codel, cake).
Networking beast because it leaves it open for more applications
I wouldn't consider deploying these routers, since I'd be fully dependent on you for upgrading to future kernel versions. Even decades old industry giants mess up device lifecycles on the regular. Asking for that level of trust in a relatively small operation like this sounds unreasonable to me. On the other hand, if the entire software stack was open source I'd be more than willing to evaluate it for serious use, as the features do sound rather compelling. It would make not blatantly violating the GPL easier as well :^)
Why not have 2-4 mounting options? RAM is drop in replaceable, same with that CPU. 2 extra reels on a pick and place is not that much tho they are going to occupy a few more slots it is feasible since you may free that much space just from resistor and capacitor optimisations. BUT when it comes to logistics there may be problems since you will need to have some way to mark them visually and for the SW. BUT that is simple , you can use a resistor/led to visually indicate what it is or laser engrave a mark or use a sticker/ marker mark . as for the SW knowing what is has a top command should be enough but you can also use a free IO pin to read a pull up/down. Or simply write it in flash/eeprom
I liked your venture due to your open sharing and open source numbers. Been watching since the first videos and plan to buy a finished version. It has been fascinating to see your process. To see you not being able to talk about certain parts due to NDA and closed source software is a turn off.
I don’t want some closed source module for this to run properly. I won’t be buying this if it’s using closed source OS software.
In addition if you add in software for packet inspection, nat, vpn tunneling, does the offloading still work as well? For a modern router there is more to it than just pure routing of packets.
Tomaž, sorry to sound like a noob, but would this router be a drop in replacement for our Slo fiber internet providers supplied routers like T-2 now that the "fiber everywhere" initiative is progressing nicely.
To me the CPU/RAM question boils down to how much extra load is anticipated for non routing activities, otherwise you just end up with hardware waiting as fast as it can. I do like the proposal made by others to have a "basic" and "pro" version if you can tolerate the separate SKU and BOM parts.
Do you happen to know if the HW offloading allows for port mirroring for analysis purposes?
Future marketing slogan for the device "Routing from Slo is Fast!" [OK, I'll go now. No need to push me into Soča :) ]
hello, any detail regarding DMZ, Port Forwarding, Multi WAN, OpenVPN, WineGuard and other VPN protocols?
Will test all of these! stay tuned!
I would say the per device unit costs are something to consider. When you are manufacturing 100s or 1000s of units that extra 70 eur per device adds up quickly. So once you have your final design then your challenge is to reduce the cost per unit as much as possible to keep manufacturing costs down and profits up.
I have a few thought questions to think about:
So I have to wonder at your current cpu/hardware design can you go faster than 10GbE? Is 25, 40, or 100GbE in this hardware’s future where you might need the CPU cores?
What impact will full stateful packet inspection have on CPU usage (this is the big question). You can move packet forwarding to asic but how well does the hardware handle layer 3 to layer 7 packet inspection? I would think there would need to be kernel intervention beyond just simple packet forwarding. Before deciding on removing hardware lets see a near final prototype.
With that development hardware can you underclock the hardware to see what the impact of 1.0 or 1.4Ghz means to the system’s operations? But from what you have shown there is no need for a 1.6Ghz 4 core cpu. Full packet inspection might change that opinion.
On the open source front, I could see two versions (depending on the hardware dev kit licensing). The advanced version of your board design could use the hardware acceleration kit with its licensing fees. And the “basic” version where it does all in kernel routing and packet inspection. The former could use a lesser CPU and ram. The latter one would need the 1.6Ghz and 8GB of ram. Or you can just use the same hardware and the only difference is if you use the close source with the acceleration kit or open source with kernel based packet routing. This isn’t a fully formed plan, only an idea.
I realize you have limited hardware for testing, but I wonder what the results would be if you add more senders with the one receiver. The idea is to see if you can flood the 10GbE link until you start to get retrans errors. This would see how the hardware would manage to much data to forward.
But all in all your hardware and your design team is doing a great job. You need to find the niche where your hardware has a market.
With regards to cost savings, my question would be are the CPU, RAM, and EMMC storage on sockets, so that we could easily upgrade them afterwards? If we can easily upgrade the device ourselves, then I don’t see a problem with shipping lower speed/capacity/cost components.
If you’re going to be like Apple and soldering them all onto the logic board, then I think you’re going to end up needing to provide multiple different versions with different capacities and speeds, much like Apple does. Because that would be our only way to upgrade.
It really depends on how much Physical SFP+ and actual 10G compatible ports you want to have in the final version of your product. If it’s going to be a solution up to 8 ports you can definitely go with a smaller requirement, if you go 16 and up you would need the additional power.
You should also consider a upgrade path for PoE++ input and PoE+ Output on Multiple Ports. Today that is a essential feature for a more expensive network appliances. Consider the price of you CPU alone I suspect you solution to end up above 250€ or even close to 400€ for that price point I expect a lot more of a device considering what you can get on the market right now.
Do note that hardware off-loading can get in the way of SQM which is important for many people.
Just sharing my journey, mate.
Definitely figure out how hard it is to break hardware offload, and what the performance is when you do.
But, even assuming the performance is great without hardware offload, I'd still like to see what you can do without the ASK. If you're stuck with the ASK, I would probably just stay within the unifi product line and not bother with fragmenting my network stack.
A bit more performance won't hurt for additional services like VPN, Proxy, system monitoring tools etc.
is the router with 1.4GHz CPU and 4GB still a networking beast?
if yes, use the cheaper components.
I believe so, but I wanted to check with my viewership whether ya'll agree.
I mean for comparison, I'm using an off the shelf router with what I believe is a 1.2 GHz CPU and 128 MB of RAM, serving as an edge router for 4 APs and 7 devices actively connected at any given time (give or take). Is the latency good? Absolutely not, but it's held up fine for several years. Pretty much the only issues I've had were some DNS outages (quad9) and some issues with the ISP modem.
@@tomazzamanmaybe hardware packet processing isn't as powerfull in slower CPU (marketing reasons) or it requires full RAM bandwidth. Please also check slower CPU with less RAM (it could mean less memory channels) for hardware accelerated performance.
@@graemewiebe2815128MB of RAM sounds like DDR3 (Samsung and SK Hynix no loger produce from May 2024)
How is performance altered with IP filtering?
One feature of pfsense I use is pfBlockerNG, which blocks IP ranges used by known bad actors. In my current config there are ~16000 ranges being blocked.
Is this also completely offloaded? Are there limits to the number of ranges we can block with offload?
I think the main purpose of the router is to perform flawlessly in the workloads you demonstrated in the video. I think therefore that the 1.6GHz or even the 1.4GHz cpu would not compromise the performance of the router. Similarly 8Gb vs 4Gb given one option or the other do not affect significantly the routing performance you should go with the option that makes the product as much competitive in the market as possible. Consideration should be given to what kind of UI and life comfort features you plan to make available in the product. But I don't think a UI, even with heavy graphical sophistication and maybe historical logging would be affected much by lower clock speed CPU or 4Gb of RAM. My 2 cents.
Nice video :) Personally I would strongly prefer to keep the current specs (1.6 GHz CPU and 8 GBs of RAM), but I would buy it even with the lovered specs.
I think it would make sense to keep the hardware modest. 1.5 to 1.6 Ghz would be fine, 2 or 4 GB of RAM would be fine. But I guess the big part of this is would be the shipping feature set.
Try doing iperf -c a.b.c.d -d (this should be the dual test which will make 2 connections one in each direction)
The faster CPU and more memory at the moment seems like a waste of resources. It's still going to be a beast either way.
I already see two version of this router emerging, depending on what traffic can really be offloaded and what software configurations would affect(and disable) hardware offloading, because that is important on a 10G router...
A cheaper version is preferred, yes, for the customers and for you guys... And that's probabably your main goal. But a higher end version might make sense too, maybe the faster CPU or something else added, maybe more connectivity options, or more RAM(example). That might be a different product, one that completes a product line. ;)
How's the bufferbloat score?
I would be fine with the lower clock version, but reducing RAM? Not something I think should be done. Maybe for simple routing not needed, but if you want to use the router with additional features on the software side, RAM could become more important. So I rather have the RAM available.
Well... speaking about performance vs price, it would be much easier to make a conclusion by seeing the practical tests and seeing approximate price.
Otherwise, pretty much interesting project, good luck with your routers!
Thank you!
This kind of hardware offloading is incredibly cool! One thing that would help with the decision is to see what the CPU utilization looks like at 10Gib without the hardware offloading. I would still prefer a higher-end chip and open-source core if we're talking about a difference between closed-source and open-source of 1 - 10%. But if the difference is more like 30 - 40%, I might lean more towards the closed-source side. Keep in mind, too, that reducing the CPU speed can also have more damaging effects on IPS/IDS.
Keep these videos coming; your level of engagement and openness is definitely giving me hope for the product!!
I believe V1 should be the beast version. Then if there is a markedt for it could a cheaper, less powerfull "lite" or V2 be made.
We were discussing the same thing internally. Thanks!
I'm a bit concerned that the packets handled by hardware offloading won't be available in the kernel - will the device support packet inspection or other forms of packet manipulation outside of the dedicated chip?
You can turn off hardware offloading to do the inspection. It'll be slower, granted, but only until you turn it back on again.
Many of these types of SoC support port-mirroring. So it would be a matter of providing an interface to these HW functions.
launch with the lower specs, which are overkill anyway. then come out with upgraded version later if necessary
i could see a market for undecided users that would purchase the router if it's inexpensive enough and would later purchase a top of the line one if it met expectations
How are you able to distribute a kernel binary with patches while adhering to the GPL license of the kernel?
The patches are likely GPLed, but load the closed source binary code.
Actually, that's a super valid question. I've put it on my list for when I talk to NXP.
@@tomazzaman
-
I love to see you being open to critical comments/questions like this!
@@tomazzaman You need to speak with an attorney in the US where NXP is located. Not their internal sales rep who's trying to sell it to you.
At the start you mentioned that FreeBSD is better than Linux for routing, but I've found the opposite to be true. I was testing opnsense vs OpenWrt on an old SFF PC with an Intel i5-9500 and X540-T2 NIC. OpenWrt was able to successfully reach 8Gbps over a single iperf connection over the internet. Opnsense struggled to even hit 3Gbps, with one CPU core pegged at 100% (which makes me think that maybe NAT or conntrack is single threaded... not sure)
My question is, what would 8GB of RAM even be used for if we kept it?
Do you reckon we could setup a home assistant server directly on the router? Would that benefit the extra specs that much?
Personally, I am just fine with the reduced specs. Specifically, 8GB RAM is overkill in my opinion, even for a top-of-the-line variant.
Use a CAMM module for ram :) is it possible to use something like IC Sockets for the CPU? Maybe you can make different hardware choices available.
Aim for €149 retail price, and go with 4GB and 1.4Ghz if you need. With offloading this good, no need to go up in specs, better to create a product first with mass apeal then later launch a more expensive option. If you want to make this more than a one product series, you need mass appeal, like the first EdgeRouter Lite, which really launched Ubiquiti.
Very cool project but for me if it can not be distribute like normal OpenWRT distribution is a deal breaker (not a fork crap staying years behind but true update one). Middle cpu variant is good option.
A 'lite' version first. Upgradeability and scalability offerings with pricings/discounts
are the CPUs not footprint compatible? for the memory, couldn't you just design for the full 8GB and either leave half of them unpopulated or use lower density ICs? It seems like you could pretty easily do variants; use the same boards and stencils and just different pick-and-place files. have the lower-end variant (4GB/1.4GHz) be the main product where you allocate all the inventory risk, and make the higher-end variant (8GB/1.6GHz or maybe 1.8GHz) in smaller "limited-edition" runs at whatever quantity you're confident you can sell through quickly. Due to the smaller purchasing quantities for the CPU and ram, the limited high-end variant's cost overheads will be higher, but you can just price it accordingly, and then if demand for the high-end variant is greater than expected, you can increase production, and pocket the reduced overhead as extra margin on your halo product, while still offering something reasonable for the rest of your customers.
As long as it can handle everything it is supposed to, why waste resources? Question is, what you want to ship and what people can do with it. From what I understand this is a router, not a server or NAS. So I’m not sure if we would run other software with it, that would require such extra headroom.
From a marketing standpoint and interesting opportunity: Segmentation - if you can show you can build high end first, then maybe you can then later release a less customisable version. You can see this in the 3D Printer space with Bambulab, they released their high end x1c model and then introduced more core feature focused versions that were cheaper.
My thoughts exactly, go with a more powerful version first, to both show we can and also test how the market responds, and then, if there's demand for it, release a "lite" version.
how much RAM does Suricata use??? I'm making some assumptions here that this thing will be able to run Suricata, though....
for ram: even 2GB should be more than enough. throw 32 gigs of emmc onto that (mostly just for logging. if it weren't for that, 8 gigs would be enough), and use the cheapest CPU you can get that still handles 10 gbps. would be nice to see the CPU be able to handle at least 500mbps of wireguard/openvpn throughput though.
It would be nice if this router support PPPoE Half-Bridge, we can connect to our own Firewall or IDS/IPS via DHCP
Are you still planning on this being capable of IDS/IPS inspection? I thought I remembered that was the original plan.
I am wondering if all this hardware acceleration and low power goes out the window as soon as you start doing that though? That was a problem on a lot of Ubiquiti earlier gen devices where their 10gb Unifi router went from 10gb throughput to 1~ gb throughput with IPS on because it had to move the traffic to general CPU cores. Newer gen Unifi devices changed that up by using different SoCs to get more IPS speed. This product doesnt have to be any kind of full decrypt/encrypt interception inspection with certificates on end devices sort of thing, but having a standard IPS type system (probably suricata running ET Open ruleset?) with pattern and packet signature matching to detect malware or hack attempts in encrypted traffic would be great to have. I know not everyone is of this opinion, but to me, all firewalls should have that type of traffic inspection as standard now days as a basic stateful firewall just isn't good enough anymore for evolving threats.
Id be fine with a 1.4ghz CPU as long as it has the same HW accel blocks in it since the CPU cores dont really matter.
Edit: Looking at the Layerscape CPUs, I see that the 1046A does not have a pattern matching HW engine, so I am guessing if you do enable IDS/IPS that the speed will drop a ton like I was thinking. I also see that the next gen LS2084A CPU steps things up a little bit (but for a lot more money) and increases the security engine from 10gb to 20gb, adds a 20gb data compression/decompression engine, and adds on a 10gb pattern matching engine. So this would be the CPU to use in order to get 10gbps IDS/IPS I would think. Sadly it costs $285~, though probably a bit less when buying by the thousand. At that point though IDK if it is even worth using the next gen CPU when you could step up to an even higher end on, the LX2082A that gets you 50gbps HW accel throughput for only an extra $15. lol. Though I dont see the LX2082A having a pattern matching engine like the LS2084A has... So maybe that's why it is not much more for so much more throughput, as it is missing the pattern match part the "lower end" part has.
1.4 GHz / 4GB RAM is probably good enough for a router with HW offloading in my opinion. You can release a more powerfull version at a later point in time. CPU is probably not an issue unless someone wants to run a firewall on it. The 8GB RAM could be useful for running containers to handle DNS, Caching, etc, but even with 4GB RAM it should be fine for basic stuff.
I understand the trade off with binary blobs and proprietary hardware. The biggest problem is security audits and ongoing support.
For security, we just have to trust NXP, but I expect there isn't any fully open source options out there with hardware offloading, without going full insanity mode and building a custom RISCV chip and drivers to do it.
The bigger issue IMO is ongoing support and maintenance, they are supplying not only kernel patches, but also patches to userspace programs as well. Those patches need to be kept in sync with subsequent releases of kernels / programs. This can be mitigated a bit by going with an LTS kernel but security updates are a certainty and again you are going to be relying on NXP to support that product.
I understand all hardware has a shelf life, but given this is a 10 gig router for home / small business users, outside of hardware failures I would expect these to last well past a decade.
It honestly highlights a design issue with the linux kernel if it requires patches to both the kernel and userspace programs in order to effectively make use of hardware features.
I'm kind of looking forward to a lower cost model, but if there was a higher cost option, I might pick that just because speed is important for some cases.
What about having two options:
1. base model : 1.4Ghz CPU, 4Gb RAM, eMMC
2. pro model : 1.8Ghz CPU, 16Gb RAM, M.2 SSD slot (SATA 3 or PCIe 3)
Do not go with cheaper only option. Use extra cpu/memory for extra features to justify price increase and this will give you more competitive product in the end. You are not in SOHO market, you're in enthusiast/lower pro segment. Having good IDS/IPS or, I do not know, maybe, graylog-style log ingesting server and viewer might be the reason for somebody to prefer your solution over putting another server or SBC for network management in remote location. For me that is a common problem with any small setup - you always need some box for things like pihole, syslog, ups monitoring or just to log into remotely and do speedtest.
@tomazzaman I've following your updates on this router for quite some time now. I personally believe if anyone if building new hardware in today era, it should be fast enough to handle most network security services/apps. I'm curious to know how many 10gbps streams can it handle simultaneously before it starts to throttle. Second, Will the cpu be able to handle IDS and perhaps IPS?
As for the number of streams, I can't answer that yet, as I've literally just gotten my hands on the ASK so it's something I plan to test (and report here). And as for IDS, from the documentation, it supports up to 32k rules with Snort, so it should still work full speed, but I need to test this too. Stay tuned!
If you guys only decided to go with MTK as I've said many videos ago....
Yea propietary BSPs dont age well, GPL violations etc... as always.
I'm not surprised anymore...
Man those chips are expensive! What they think they are? Broadcom or Nvidia?
Still thinking that MTK Filogic 880 would have been a lot cheaper and more functional for a FOSS router.
With Wireless, PPoE and NAPT accelleration available with FOSS drivers.
More accellerated protocols is supported by the SOC but those patches are available as FOSS.
No, this version is already locked, hardware-wise, 2x10Gb SFP+ ports and 3 GbE ports.
I figure it out right away.
That's why Windows has an advantage with well supported hardware, and Apple makes hardware and software, so there should be no issues there (as always, I have noticed on older hardware that even they control both hardware and software, they removed the very optimized drivers in favor to something which doesn't perform as good as it was, but it's so old people don't care about it). Drivers are what enable full use of the hardware, offloading all the operations to the ASICs (Application-Specific Integrated Circuits). And the software needs to call the driver functions to enable the hardware processing.
The same issue described here with Linux is present on "alternative firmware" for home routers / access points like the awful DD-WRT (and OpenWRT for that matter) and other variants which the developers have reverse engineered the hardware of those routers to provide support, but these reverse engineered drivers are basic most of the times and process everything on the CPU.
For those wondering what I'm talking about, think this way: you have a pretty decent GPU on your system, but no connection to the Internet or embedded drivers for that GPU. Windows enables it using the "STANDARD VGA ADAPTER" driver. It can enable a high resolution, SVGA, XVGA and the sorts, but window composition (moving windows, drawing) is bad. And it cannot draw video frames fast enough or accurately that way, simple tasks take up all your CPU cores. That's what's happens when you cannot offload processing to the many auxiliary processors, ASICs and utilize the hardware you own to the fullest.
Pretty interesting video.
If you keep asking public opinion on every step on the way, you are never going to complete anything. You have a product, you had a vision, you wanted something good for the world. Make it happen. Stop asking should I go this route, or that route, you need to choose the path for us. As a consumer yourself, you should know that a small fraction of the price is not going to keep us from buying. Make a great product that will cover everything, the sales will take care for themselves. It is okay if I pay for pricey hardware. I just do not want SaaS embedded into the software or do not lock it down, so we can freely load up any solution we desire. I strongly believe you will deliver something unique at the end goal.
You need to create the polls and do what most people want, because they are not doing it for themselves, but for sale.
I think you go with a cheaper version out of the gate, but know you could create a more performant version for those who want to pay for it. Not everyone needs all the bells and whistles.
One question on the 10 gig ports. Can they be used as a WAN or LAN interface? I have 5 gig fiber Internet.
10 gig ports can be used for pretty much anything you want - hardware offloading isn't port-specific (although it *can* be configured that way).
Target audience? Home? Go cheaper, but if you are able make a limited edition from the beast. Small office? Scalability is everything, keep it as is.
You should not save on RAM unless it's modular. RAM is never enough, and you should leave headroom for stuff to run services on top of the OS. Go for max RAM even if it ups the cost.
I'd prefer that we keep the 1.6Ghz and 8GB configuration. The more RAM, the better, in my opinion. Not to mention that with the larger amount of RAM, it's easier to keep the device going for much longer if you have the RAM to support it, not that it'd likely be needed given the impressive performance here, but still... I say keep it.
I know that this will be a significant overhead for logistics , but cant we have 2 versions?
For example i want to have the best of the best no mater the cost (in resonable ranges) , while others might not need a faster cpu or more ram
I liked your venture due to your open sharing and open source numbers. Been watching since the first videos and plan to buy a finished version. It has been fascinating to see your process. To see you not being able to talk about certain parts due to NDA and closed source software is a turn off.
I don’t want some closed source module for this to run properly. I won’t be buying this if it’s using closed source OS software.
I'm only slowly entering the world of custom networking gear and not using the ISP provided gear with a couple of the cheapest APs and switches possible, so apologies for the novice questions and random thoughts;
Would it make sense to choose a mix of the two, so the 1.4GHz with the 8GB of ram? To me it seems like the 0.2Ghz is much less of a downgrade, than the 50% less memory.
I don't know what sort of services would benefit from another 4GB of ram or, perhaps more interestingly, 0.2Ghz faster processor. Would something like WireGuard run noticeably quicker on the 1.8 vs the 1.4Ghz, would the 4GB ram be enough for it? What are some services that you would usually run on a router?
Nonetheless I do think the first version should probably be the fastest to get your name out and known as solid, if slightly over built, routers