I'd recommend everyone use logically seperated traefik instances for internal and external. Yes it does require a bit more configuration but it's going to be the most secure.
Excellent information! One addition thing I have done is restrict the entry of my NAT rules for ports 80 and 443 to the Cloudflare ASN in Opnsense. It doesn't even appear to be an open port any longer.
Did you have to find out all of the IP ranges used by the Cloudflare ASN and then apply these ranges to your firewall rules or does Opensense let you just specify the Cloudflare ASN ? I tried the former approach on pfSense (trying to find all of the IP ranges and then creating a rule for these IP addresses) but must have missed some because it didn't work for me. Would love to be able to just specify Cloudflare ASN.
@@luckbeforeleapi did this by creating an alias from a pair of urls (for v4 and v6) cloudflare provides that contain a list of all their current ips in use for dns proxy, then setting the source of my https port forward rule to that alias
I think the biggest flaws is in the volume mountpoint, mounting the Docker socket directly is not recommended, even the traefik documentation state you should not do that in prod. If somehow an external user get access to your traefik instance even through CloudFlare, they will have root access to your Docker, unless you are running traefik as nonroot
@@red_dautI used the same, then exposed it through TCP, also my Docker socket proxy is in another Docker network, with option internal=true and masquerade=false, and then my traefik are in Frontend network and socket network, container that need to be exposed are in the Frontend network
@@Jims-Garage By the way, what's the best way to expose bitwarden to the internet so we can use it thru the android APP ? Whitelistening isn't the best option as my ISP does not reserve an IP for mobile phones. Right now I have a VPN but it takes too steps to do when I need to open bitwarden...
I run the two instance setup. I have my docker compose set up so I use environment variables for configuration, then there is a merge feature where I have most of the config in a section that is then referenced and the internal/external portions only specify their unique elements (such as the IP whitelist). Additionally, on the external point, I have an additional docker constraint tag, so by default enabling items only registers them on the internal instance, but i can add an additional label to list them on the external provider as well. My setup is is all about doing as much of the work in the traefik configuration itself as possible, then the services themselves only have to specify a bare minimum, typically only a hostname and to enable (i have exposedByDefault set to false). I even have them sharing a cert store by mounting the same file, but only giving the external instance read-only access to it. Now THAT is something I wish I had a better way of managing, especially as most acme providers can run into conflict if you request multiple certs at the same time, and I don't want any internal subdomains explicitly listed on my publicly served cert.
@@Jims-Garage Thanks for the awesome videos I am learning so much! I wanted to setup the 2 instance way as well, but I wanted to use the CF tunnel, does it mean that for 1 domain you need to setup 2 different cloudflare tunnels? 1 per traefik instance?
Have you tried issuing a wildcard certificate? I've been using wildcard certificates for a long time now, and the greatest thing is that I don't have to request a new cert for every subdomain I add, because every single imaginable subdomain already points to my reverse proxy. The reverse proxy is then the decider of which subdomain is valid and which ones get redirected to the google homepage.
4:55 Yes there is! I know of a source where i get a list of all current DNS records. It's updated every 24 hours. And so it only takes a simple grep to get a list of your sub-domains - all of them.
Agreed, it's possible but not from legal methods (AFAIK). You often find this data on the dark web but it's unlikely anyone would be bothered for a homelab.
Godsent! Could not have come at a better moment in time for me. I just hope I can manage the complexity. I love the idea of docker-compose, but I've already let blood for a decent amount of time with the docker + docker compose documentation. And man if you already only half know what you're doing docker / compose / volumes /networking sure ain't gonna help that. The worst part is: it's difficult to diagnose. Is it fw rules? Is it vlan? Is it DNS? (it's always DNS) etc. I love the tinkering, but it to be a harsh mistress .. and hard on free time and money on top.
Haha yes, homelabbing isn't easy, and it takes a lot of time... There's tons of useful stuff to learn though that can be very useful outside of the lab. If you get stuck jump in the Discord, plenty of people to help you.
You're missing what I think is the best option. Just run 2 Traefiks. One on an internal-only IP, and one that's exposed to the outside world running your public projects. You can (and should) even put them on their own vlans.
The most secure option for home networks is to only use remote access vpn solutions such as wireguard to connect to your home resources. And with the wireguard connect on demand feature you don't have to manually enable your wireguard when away from home WiFi networks.
@@Jims-Garage Especially wireguard is very simple to setup and compared to autossh I found it much more reliable with that one crappy upstream connection that I have in my network spread across several locations. But you've cut into the right topic, most tutorials on docker and docker-compose are also not suitable for production as they don't bind service ports to certain ips, say making the db port to be accessible only on 127.0.0.1
Please keep in mind, that only applications which have 'Routers' in traefik are exposed... Like your SQL, Redis, Elastic, etc containers is probably not exposed to the public... I have two middlewares that I reference in my labels, public@file and private@file depending on if I want it exposed to the public or not, then in that middleware is the crowdsec bouncer and whitelist depending on which middleware
Why not Authelia/Authentik to add 2FA to your endpoints and use it as the middleware and make them have to go through multifactor authentication in order to get to the service?
I did two seperate traefik installs, but with the cloudflare proxy (Make sure to translate to the actual IP) and then I put up security software to monitor the traefik install behind it as well both internal and external. IE things like crowdstrike. i do have authelia but I'm lookint at oauth2 I'm not terrbily set on it though.
What about using Keycloak or something similar and having all applications OAuth2/OIDC enabled? All apps without a valid login session will redirect to the Keycloak login page...
Thank you, great video, Jim! There is one thing, I realized. When setting the sourcerange of ipwhitelist to my local network (eg. 192.168.0.0/24), it is for some reason not considered. Going on the Service in the Browser I get an "Forbidden". In trafik-log I saw that traefik "only sees" my WAN-IP, which is dynamic. Is there a workaround to constantly check the WAN-IP and alter the rule on the fly? Or is there another hint to make that work?
Very nice hints in this video. I'm improving a lot of my homelab based on your videos. It would be nice if you could do a video explaining with more detail how to create firewall rules in Sophos-XG and how to apply IPS and any other improved security mechanisms.
Could you please explain a bit more under what circumstances this security issue exists? And which local services are accessible? Do you mean services that are considered only for local usage and still have a config within traefik or how how does the infrastructure look when vulnerable?
Say for example you're running PiHole, Proxmox, vaultwarden, or anything else solely for internal use and it's routed through the proxy. If you also have something like Plex, or anything else designed for internet access that shares the same entrypoint, then it's accessible from the internet if you do not have some of the countermeasures I state in the video, provided I have your IP address. Test it out from your mobile or over a VPN. Create a hosts record as I showed in the video.
@@Jims-Garage Thanks for the clarification. So by "routed though the proxy" it means that if I for example just configure Plex in traefik and just use Bitwarden locally with it's own ip address and port there should be no way to connect to it from outside? Because it isn't routed through the proxy in that case in my understanding.
Another thing you can do is find out your cellular provider's IP address ranges (e.g. find one public IP address used by your phone and then use Hurricane Electric BGP Tools to find ASN and ranges), and apply these as Allow rules on your firewall. Then, when you are away from your house, only access your home network from your laptop when it is tethered to your phone's Internet connection. You will have limited your attack surface a lot with just that step. I also use a VPN with this so my firewall will only accept the VPN connection when it is coming from an IP address used by my cellular provider.
Love your videos. instead of opening up ports can you set up a cloudflare tunnel directly to traefik and let traefik manage which subdomain should go to which ip and port ? Something like NginxProxyManager behind a cloudflare tunnel but cooler
Its so great! Thinking about switching from nginx proxy to full on traefik like you.. One thing is holding me back is the thought of having managment interfaces/services accessable on the untrusted vlan. My internal services are on the same docker host as the traefik instance so puting those service on untrusted is a bit of a no go. for traefik to be able to access the gui's from different ip's would need to use a lot more "complicated" firewall rules. How did you do it or still doing it?
You can disable the dashboard, plus it's read only anyway, and there's nothing particularly sensitive. You can use separate entrypoints for internal and external services, and use whitelists to prevent access.
How about assigning an extra network interface to the docker vm, plug in an extra network cable (the port on the Switch of course assigned to a different vlan) and just let all the internal traffic go through that cable?
Tha ks very muchbfor the video, like someone commented have been trying to read ducumentation to understa d but is a task. Have two questions , so is tgos the way to protect the traefik dashboard? Also how can point traefik to another docker host for other applications? What i would like to do is have one docker with just traefik and crowdsec for enamplle and host the other apps on another docker host. What are youbthoughts on this eg which security apps beed to be on the same host as traefik and which ones dont. Thanks for any feedback
Thank you. You can restrict access to the dashboard this way. For none Docker service routing you need to create an external service reference, you do this in the traefik.yaml file. What you want to do with 2 hosts is possible but I'd do it all on the same host for simplicity and better Traefik integration. Multiple hosts is better suited for Docker Swarm or better yet, Kubernetes.
Great videos 👍 i just have a question about jellyfin. Will cloudflare not shut it down after time. Would love to stream my music through my jellyfin app on my mobile remotely. Is there so many mb you have?
hey great video! Qustion, out of curiosity, if you are not exposing your password manager (vaultwarden - I use it as well), how do you you it on your phone or when your are not connected to your local network?
Vaultwarden caches your passwords on the phone, you shouldn't need to connect to use it. If ever I did (creating a new password etc) I would just use my VPN to remote home (I have a few videos on VPNs).
hey again@@Jims-Garage just letting you know that i'm following your videos and creating a new homelab using them, so thanks! Also, I have 2 questions. 1. for jellyfin, when trying to connect if from within my network, it required me to add :444 at the end of the URL, otherwise it says "not found". 2) do I need to add a domain name fot it also in pihole a suggestion for a follow-up video, how do i use the domain I bought in cloudflare and exposing services?(for some reason, i'm struggling with it nad none of the videeos i wathc helped) tbnx again!
super tutorial. thanks Jim. i try to do. but it isn't work. have i do crowdsec (your tutorial video) first? than after traefik-secure tutorial video? thanks
I like to keep my admin interfaces on my admin VLAN/subnet. It's surprising how few tutorials cover this, and how buried in documentation it is. Even Debian is doing its best to make sure the ListenAddress option of sshd does not work:( Anyway, I wonder if i can use some similar approach to limit Docker stuff to particular subnets internally.
I love the ideas, and I'm trying to implement them all! I can't seem to get the cloudflare proxy service to play right with my hairpin NAT. It works great if my service is not behind cloudflare proxy, but if I put my service behind cloudflare proxy my NAT rules don't seem to catch it. Any tips?
Chapter title “demonstrating the problem”, … Opens a VM on screen share and talks 😄 Joking aside, great stuff on your channel! Thanks for sharing good info
Still learning along the way, mostly because of your videos! So, if you would like to expose only certain apps to certain users/ip-adresses, would it not be safer to just use something like Twiingate or OpenZiti? So you would not have to open any port whatsoever? Or would that be too strict and hamper usability?
That's always the trade-off. For apps you only want certain people to access that would be a valid approach, but for a public website or app that's clearly not a viable option.
I'm still trying to understand why using Traefik is superior to just setting port forwards with port translation and IP whitelists in the firewall (in my case pfSense) itself?
@@unmesh59 the proxy provides a layer in front of the application, it also provides things like SSL and DNS names, plus with Traefik you can add things like crowdsec for additional security.
My thought is, can I ipwhitelist services to private addresses for my internal services as a middleware on traefik, and for my external services allow any ip? and throw in Authlia in front of all my services anyway so it is blocked with auth to begin with for an additional layer. Would something like this work?
Does this vulnerability also apply to NGINX proxy manager? Eg port 443 coming from WAN routed to port 443 on nginx proxy manager which then redirects it to jellyfin for example?
A nice idea, another idea is to run two instances of reverse proxies (either Traffik or NPM) external services are hosted on the traffik-external, and internal services are hosted on the traffik-internal and just configure the firewall (pfSense etc) to forward ports 80,443 to just the traffik-external reverse proxy.
This is the more secure option. Separation of concerns/contexts. If an attack an take advantage of an external vulnerability in the reverse proxy, they can potentially escalate themselves or move latterly and gain access to internal services.
Hello Jim, any recommendations on improving the security of the Traefik docker container? Like it’s exposing the docker socket and no internal user this could lead to some security concerns, please share you’re thoughts, thanks
You're right, I'm due to cover in a video about not using the sock and root user. I will be following the official docs so feel free to check those out if you're interested.
@@Jims-Garage I’m looking into to it, just a hint I tried the read-only flag in the compose file and seems not to break anything , I’ll try to integrate crowdsec to test further, thanks for the awesome videos and content!
I use a VPN, but also, existing passwords are saved on the phone. The only issue you have is you cannot add new ones until you're back on the home network (hence the VPN).
the problem here is that you need a local DNS or modify the hosts file, both the external client and the internal client. Is not there a way so that the proxy redirect it?
@@Jims-Garage/videos Sorry for the bad English. I did set up everything except the option 3 and my problem is that i need to have a private DNS locally or modify my hosts file to be able to access the sites i have locally. And to be able to to access the sites i have externally i have to edit my hosts file and set my public ip to the site i want to access. I did discover the problem with the external sites is because of the cloudflare proxy option, if i disable it i can access the external sites with any problems, what can be causing this? I would like to enable the proxy option if it is a way to make it work. Also thanks for responding.
Thank for the tutorial though I don't know if I am missing something. I appreciate you hairpin NAT explanation but I'm unable to access my (https-external) service on my LAN. I have NAT reflection enabled on my OPNSense router but get a 404. My DNS overrides still list the internal IP's of all services. Is this correct?
My NAT rule was configured on the WAN interface only. I added the LAN interface to the NAT rule and bingo, LAN access to (http(s)-external) traefik services. Leaving this comment here for anybody else experiencing the same issue.
I'm having an issue where using the http-external tag works fine (sending traffic through port 81). As soon as I add the https-external label, the site is no longer reachable. Port 81 and 444 are routed correctly. Has anyone else run into this or have any advise?
Sure but why proxy your internal apps at all? Most guys suggest running a proxy docker network for traefik. Ideally you have separate networks for your dbs internal apps and one for the socket proxy. Then your internal apps only sit in the internal where there is no traefik endpoint. Meanwhile because these are all custom networks you can still call apps by their container name.
Most apps now want valid SSL certificates for web interfaces etc and not using them is difficult. Furthermore, some apps I want both internal and external access to (e.g. Jellyfin)
What about using cloudflare tunnel? Do i have the same issues when only defining some public hostnames? Is it also possible to access to internal apps, which have in my case slso another subdomain as my external apps?
Excellent video! I feel really dumb right now because I'm re-watching the video to find out why I couldn't get this setup to work on my end then I tried it a couple of days ago, and at the 5-minute mark I see why... I forgot to route ports 81 and 444 to Traefik in the docker-compose file. >_
@@Jims-Garage I got one step closer... But when I try to http to an external service, it redirects to :444 in the browser (fails), direct to https works. And services on the internal http and https just plain don't connect even though Traefik says everything is fine. Really frustrating, but I'll come back to it after a bit of rest.
You could, but it's still placing full trust into Cloudflare. These options enhance that by adding additional layers, especially with IPS/IDS on a firewall.
You don't need to guess a subdomain, It's really easy to get all subdomains just by knowing the main domain name. I would do this to yours, as a test to prove it to you but here in the UK without your explicit permission to do so it would be illegal under computer misuse laws.
If I proxy through Cloudflare, or use their Tunnels service, can they see the data that's being transmitted in plain text? Can the decrypt the data on their side?
When you’re using an SSL Cert from Cloudflare then theirs are technically capable of reading your traffic. This is the case when you use the Proxy or Tunnels, only when you’re the only one who is holding the Private key then only you can decrypt the content, but when using the cloudflare services they are holding the private key so it’s possible that they decrypt your traffic and they do it to protect you from DDoS etc.
While whitelisting seems to be the right thing, it doesn't seem to work. If I whitelist my LAN addresses and try to open the site, it says forbidden, and the log says that is not allowed to access. If I try to use the depth of ipstrategy, the log says that the IP is empty. Does anyone know a solution?
Great video. I use option 3 with the whitelists and restrict access for my internal apps to my local subnet and my vpn subnet ranges. Im currently trying to work out how i can have it give a 404 rather than a 403. I also want to create a custom error page. Would be great if you could explain and show something with ref to this
Thanks, yes. I've used custom error pages before, but I agree it would be much better if you could generate a 404. I'll consider it for a follow up video.
@@Jims-Garage I added domain a record in cloudflare. I also notice you have crowdsec I don’t have that installed so I’m working on that as well. This is new to me and I’m learning. Like when I go to install pi hole with unbound it uses port 80. Do I change it to a diffrent port. I’m using traefik with nginx proxy installed as well. I’m doing this with Ubuntu and raspberry pi 5
@@jordanpfeifer9587 You need to add a local DNS record that points to your traefik Docker IP. e.g., traefik-dashboard.yourdomain.com points to 192.168.1.100 - this needs to be done locally on your firewall, hosts, or PiHole etc, not in cloudflare (external).
@@Jims-Garage I added a a record in pi local dns settings. But the page don’t show. I have my pc setup to use my dns server that you have. I still don’t have the page. Same error. Do you have to use pi hole as your dhcp. Or can I still use my own router for that.
@@Jims-Garage but without pi-hole it is impossible to access local services? I precisely followed the tutorial btw and the external network is working except internal. For example i routed the traefik dash internally and cant access it.
Potentially, depends on the intended audience. If it's meant to be publicly accessible then that strategy doesn't work, you'd want additional layers of scanning and segmentation.
@@Jims-Garage For me, I have some apps exposed to the internet and some others like vaultwarden are only accessible via whitelist. For those, I just use WireGuard to access these apps. Seems easier
Considering method 3, is it possible to a hacker spoof its IP and pass as a whitelisted ip? My gut feeling is that option 2 is safer, but as I said, I’m not an expert, just curious hahaha
It's always possible but I find it very unlikely anyone would bother for a homelab. Technically it would require serious compromises of underlying infrastructure.
I just can't help it, but this is how this video reads to me: 'Are you using this proxy to expose services to the internet? Well, guess what, there is a significant security flaw, and you are probably not aware of it. You have exposed services to the internet.' And yes, I know there might be people who simply don't understand the concepts and just slap in a Traefik instance in the middle of their stack because they read one tutorial and want this specific thing accessible from the internet. But who are these people who 'have a stack' and then 'accidentally' expose it to the internet as a whole? I just cannot fathom the cross-section between these two groups of people who seemingly have knowledge of running a server with a stack but then don't realize what the Traefik instance they just added to the stack does. But again, I might be biased as I have an IT background.
No, not quite right. The issue is that by routing everything through Traefik, by default all applications are accessible if you know the IP address. That includes things you only want available locally (e.g., vaultwarden).
Why would you want to poke holes in your firewall? Just run a cloudflare tunnel and get rid of that proxy and firewall openings. Those are serious attack vectors.
I don't want Cloudflare to see all of my traffic, it's a privacy nightmare. Plus, you're putting all of your trust into a single vendor... Better to run everything though a firewall, even if you're using a tunnel IMO.
I don’t understand the port forwarding part, I don’t have a firewall like sophos are you saying I need to do a port forward from port 444 to 443 & 80 to 81 I found this video for UFW will this apply ruclips.net/video/F9TuhNmzz3g/видео.htmlsi=0wt8V8T0eItAIDsf
@@Jims-Garage I like the idea of only using one traefik instance, I just wasn’t sure about the port redirection part. I think i understand now but we’ll see if I get working. Thanks for your reply anyway :)
I'd recommend everyone use logically seperated traefik instances for internal and external. Yes it does require a bit more configuration but it's going to be the most secure.
Thanks for the video. I went with the dual traefik instances approach. Separation of concerns and less chance of adding mistakes to config.
@@nemac23 yes, likely the most bulletproof
Excellent information! One addition thing I have done is restrict the entry of my NAT rules for ports 80 and 443 to the Cloudflare ASN in Opnsense. It doesn't even appear to be an open port any longer.
Thanks, that's a good additional step.
Did you have to find out all of the IP ranges used by the Cloudflare ASN and then apply these ranges to your firewall rules or does Opensense let you just specify the Cloudflare ASN ? I tried the former approach on pfSense (trying to find all of the IP ranges and then creating a rule for these IP addresses) but must have missed some because it didn't work for me. Would love to be able to just specify Cloudflare ASN.
@@luckbeforeleapi did this by creating an alias from a pair of urls (for v4 and v6) cloudflare provides that contain a list of all their current ips in use for dns proxy, then setting the source of my https port forward rule to that alias
@@luckbeforeleap They have a page with a list of ranges
I think the biggest flaws is in the volume mountpoint, mounting the Docker socket directly is not recommended, even the traefik documentation state you should not do that in prod. If somehow an external user get access to your traefik instance even through CloudFlare, they will have root access to your Docker, unless you are running traefik as nonroot
That's an important point and something I will cover later. I mention it in my podman and kubernetes video.
@@red_dautI used the same, then exposed it through TCP, also my Docker socket proxy is in another Docker network, with option internal=true and masquerade=false, and then my traefik are in Frontend network and socket network, container that need to be exposed are in the Frontend network
That's why rootless docker install is advised Ithink.
Once again, another amazing video. Already waiting for the next one.
Thanks 👍
@@Jims-Garage By the way, what's the best way to expose bitwarden to the internet so we can use it thru the android APP ? Whitelistening isn't the best option as my ISP does not reserve an IP for mobile phones. Right now I have a VPN but it takes too steps to do when I need to open bitwarden...
@@joelfrojmowicz I use bitwarden on my mobile and I don't expose it. It should cache your passwords on the device (thus doesn't need to connect).
Jim is always there when I'm lost, thanks man for ideas!
Glad to help
I run the two instance setup. I have my docker compose set up so I use environment variables for configuration, then there is a merge feature where I have most of the config in a section that is then referenced and the internal/external portions only specify their unique elements (such as the IP whitelist).
Additionally, on the external point, I have an additional docker constraint tag, so by default enabling items only registers them on the internal instance, but i can add an additional label to list them on the external provider as well.
My setup is is all about doing as much of the work in the traefik configuration itself as possible, then the services themselves only have to specify a bare minimum, typically only a hostname and to enable (i have exposedByDefault set to false). I even have them sharing a cert store by mounting the same file, but only giving the external instance read-only access to it. Now THAT is something I wish I had a better way of managing, especially as most acme providers can run into conflict if you request multiple certs at the same time, and I don't want any internal subdomains explicitly listed on my publicly served cert.
That's a good setup and makes sense. I'm going to revisit Traefik soon for V3.
@@Jims-Garage Thanks for the awesome videos I am learning so much! I wanted to setup the 2 instance way as well, but I wanted to use the CF tunnel, does it mean that for 1 domain you need to setup 2 different cloudflare tunnels? 1 per traefik instance?
Have you tried issuing a wildcard certificate? I've been using wildcard certificates for a long time now, and the greatest thing is that I don't have to request a new cert for every subdomain I add, because every single imaginable subdomain already points to my reverse proxy. The reverse proxy is then the decider of which subdomain is valid and which ones get redirected to the google homepage.
Just don't do it. If you want to access internal apps externally, come into your home network with your own VPN. It's the only safe way.
I agree, but if you're not careful and you use a single Traefik for both then you can access internal apps via the web.
@@Jims-Garage 100% and the warning is much appreciated for those who may have otherwise overlooked this very dodgy situation.
4:55 Yes there is!
I know of a source where i get a list of all current DNS records. It's updated every 24 hours. And so it only takes a simple grep to get a list of your sub-domains - all of them.
Agreed, it's possible but not from legal methods (AFAIK). You often find this data on the dark web but it's unlikely anyone would be bothered for a homelab.
Godsent! Could not have come at a better moment in time for me. I just hope I can manage the complexity. I love the idea of docker-compose, but I've already let blood for a decent amount of time with the docker + docker compose documentation. And man if you already only half know what you're doing docker / compose / volumes /networking sure ain't gonna help that. The worst part is: it's difficult to diagnose. Is it fw rules? Is it vlan? Is it DNS? (it's always DNS) etc. I love the tinkering, but it to be a harsh mistress .. and hard on free time and money on top.
Haha yes, homelabbing isn't easy, and it takes a lot of time... There's tons of useful stuff to learn though that can be very useful outside of the lab. If you get stuck jump in the Discord, plenty of people to help you.
Thank you very much for this video ! Hardest part was to do the port forwarding/translation and not mess with docker iptables.
@@ethernaelis glad it was useful
You're missing what I think is the best option. Just run 2 Traefiks. One on an internal-only IP, and one that's exposed to the outside world running your public projects. You can (and should) even put them on their own vlans.
Thanks, I do mention running two Traefiks as an option.
The most secure option for home networks is to only use remote access vpn solutions such as wireguard to connect to your home resources. And with the wireguard connect on demand feature you don't have to manually enable your wireguard when away from home WiFi networks.
I agree, but I have some less tech savvy folk so I tend to IP whitelist and heavily segment my network.
@@Jims-Garage Especially wireguard is very simple to setup and compared to autossh I found it much more reliable with that one crappy upstream connection that I have in my network spread across several locations.
But you've cut into the right topic, most tutorials on docker and docker-compose are also not suitable for production as they don't bind service ports to certain ips, say making the db port to be accessible only on 127.0.0.1
Great video, though I would use different ports for the external, something like 8080 and 8443
Thanks! Yes, just choose whatever works for your setup (I already had those in use).
Nice video!
What should I do if my router doesn’t support port translation?
Thank you for your hard work!
Thanks for the demo and info, have a great day
Thanks, Chris. You too!
Please keep in mind, that only applications which have 'Routers' in traefik are exposed... Like your SQL, Redis, Elastic, etc containers is probably not exposed to the public... I have two middlewares that I reference in my labels, public@file and private@file depending on if I want it exposed to the public or not, then in that middleware is the crowdsec bouncer and whitelist depending on which middleware
Amazing video, your explanations are so clear and your talking flow is just perfect. I choose solution 2 for now. Thanks a lot for all your work.
Great to hear! Thanks for leaving a comment.
Why not Authelia/Authentik to add 2FA to your endpoints and use it as the middleware and make them have to go through multifactor authentication in order to get to the service?
That is an option, but a better one is not to expose it if you don't have to. Always reduce the attack surface.
Thanks for highlighting this point. It should be the most overlooked security flaw for most homelabbers. Updating my traefik config tonight!
Great, hop into Discord if you're not sure.
If you only use external access for yourself and maybe a few family members you should look at a Cloudflare tunnel
I did two seperate traefik installs, but with the cloudflare proxy (Make sure to translate to the actual IP) and then I put up security software to monitor the traefik install behind it as well both internal and external. IE things like crowdstrike. i do have authelia but I'm lookint at oauth2 I'm not terrbily set on it though.
What about using Keycloak or something similar and having all applications OAuth2/OIDC enabled? All apps without a valid login session will redirect to the Keycloak login page...
Thank you, great video, Jim!
There is one thing, I realized. When setting the sourcerange of ipwhitelist to my local network (eg. 192.168.0.0/24), it is for some reason not considered. Going on the Service in the Browser I get an "Forbidden". In trafik-log I saw that traefik "only sees" my WAN-IP, which is dynamic. Is there a workaround to constantly check the WAN-IP and alter the rule on the fly? Or is there another hint to make that work?
Very nice hints in this video. I'm improving a lot of my homelab based on your videos. It would be nice if you could do a video explaining with more detail how to create firewall rules in Sophos-XG and how to apply IPS and any other improved security mechanisms.
Thank you, do you mean this one? ruclips.net/video/tRBe1EbKyoU/видео.html
Could you please explain a bit more under what circumstances this security issue exists? And which local services are accessible? Do you mean services that are considered only for local usage and still have a config within traefik or how how does the infrastructure look when vulnerable?
Say for example you're running PiHole, Proxmox, vaultwarden, or anything else solely for internal use and it's routed through the proxy. If you also have something like Plex, or anything else designed for internet access that shares the same entrypoint, then it's accessible from the internet if you do not have some of the countermeasures I state in the video, provided I have your IP address.
Test it out from your mobile or over a VPN. Create a hosts record as I showed in the video.
@@Jims-Garage Thanks for the clarification. So by "routed though the proxy" it means that if I for example just configure Plex in traefik and just use Bitwarden locally with it's own ip address and port there should be no way to connect to it from outside? Because it isn't routed through the proxy in that case in my understanding.
@@jT-dj9sj correct, this is only for things using Traefik with a default config
Another thing you can do is find out your cellular provider's IP address ranges (e.g. find one public IP address used by your phone and then use Hurricane Electric BGP Tools to find ASN and ranges), and apply these as Allow rules on your firewall. Then, when you are away from your house, only access your home network from your laptop when it is tethered to your phone's Internet connection. You will have limited your attack surface a lot with just that step. I also use a VPN with this so my firewall will only accept the VPN connection when it is coming from an IP address used by my cellular provider.
Love your videos. instead of opening up ports can you set up a cloudflare tunnel directly to traefik and let traefik manage which subdomain should go to which ip and port ?
Something like NginxProxyManager behind a cloudflare tunnel but cooler
Its so great! Thinking about switching from nginx proxy to full on traefik like you..
One thing is holding me back is the thought of having managment interfaces/services accessable on the untrusted vlan.
My internal services are on the same docker host as the traefik instance so puting those service on untrusted is a bit of a no go.
for traefik to be able to access the gui's from different ip's would need to use a lot more "complicated" firewall rules.
How did you do it or still doing it?
You can disable the dashboard, plus it's read only anyway, and there's nothing particularly sensitive. You can use separate entrypoints for internal and external services, and use whitelists to prevent access.
@@Jims-Garage well sometimes i need to watch the video a bit further Xd
How about assigning an extra network interface to the docker vm, plug in an extra network cable (the port on the Switch of course assigned to a different vlan) and just let all the internal traffic go through that cable?
Tha ks very muchbfor the video, like someone commented have been trying to read ducumentation to understa d but is a task. Have two questions , so is tgos the way to protect the traefik dashboard? Also how can point traefik to another docker host for other applications? What i would like to do is have one docker with just traefik and crowdsec for enamplle and host the other apps on another docker host. What are youbthoughts on this eg which security apps beed to be on the same host as traefik and which ones dont. Thanks for any feedback
Thank you. You can restrict access to the dashboard this way. For none Docker service routing you need to create an external service reference, you do this in the traefik.yaml file. What you want to do with 2 hosts is possible but I'd do it all on the same host for simplicity and better Traefik integration. Multiple hosts is better suited for Docker Swarm or better yet, Kubernetes.
@Jims-Garage ok noted and thanks for the. Feedback, have to take a stab back at Crowdsec to see if I get that working
@@Jr-hv1ct check my recent config update on GitHub, the old one might have caused the issue.
@Jims-Garage ok thanks for that info.
Great videos 👍 i just have a question about jellyfin. Will cloudflare not shut it down after time. Would love to stream my music through my jellyfin app on my mobile remotely. Is there so many mb you have?
No quoted limit AFAIK. I'm streamed through it successfully with movies, suspect audio wouldn't be a problem.
hey great video!
Qustion, out of curiosity, if you are not exposing your password manager (vaultwarden - I use it as well), how do you you it on your phone or when your are not connected to your local network?
Vaultwarden caches your passwords on the phone, you shouldn't need to connect to use it. If ever I did (creating a new password etc) I would just use my VPN to remote home (I have a few videos on VPNs).
@@Jims-Garage tnx for the idea! never though about it!
hey again@@Jims-Garage just letting you know that i'm following your videos and creating a new homelab using them, so thanks!
Also, I have 2 questions. 1. for jellyfin, when trying to connect if from within my network, it required me to add :444 at the end of the URL, otherwise it says "not found".
2) do I need to add a domain name fot it also in pihole
a suggestion for a follow-up video, how do i use the domain I bought in cloudflare and exposing services?(for some reason, i'm struggling with it nad none of the videeos i wathc helped) tbnx again!
great video! So informative. Is there anything wrong with have all traffic (both internal and external) routed through the external entrypoint?
Nothing wrong per se, just that there are likely services that you don't want to be exposed.
super tutorial. thanks Jim.
i try to do. but it isn't work. have i do crowdsec (your tutorial video) first? than after traefik-secure tutorial video?
thanks
Yes, to add crowdsec you need to update Traefik. The config files are in the same folder.
I like to keep my admin interfaces on my admin VLAN/subnet. It's surprising how few tutorials cover this, and how buried in documentation it is. Even Debian is doing its best to make sure the ListenAddress option of sshd does not work:(
Anyway, I wonder if i can use some similar approach to limit Docker stuff to particular subnets internally.
You can use macvlan to put containers on different subnets. I do this for DMZ containers.
What I done, open remote port for once, get certbot do its things (filling my acme.Json) after that close the port, is I am doing wrong?
Use DNS challenge, you don't need any open ports 👍
I love the ideas, and I'm trying to implement them all! I can't seem to get the cloudflare proxy service to play right with my hairpin NAT. It works great if my service is not behind cloudflare proxy, but if I put my service behind cloudflare proxy my NAT rules don't seem to catch it. Any tips?
Chapter title “demonstrating the problem”,
…
Opens a VM on screen share and talks 😄
Joking aside, great stuff on your channel! Thanks for sharing good info
Still learning along the way, mostly because of your videos! So, if you would like to expose only certain apps to certain users/ip-adresses, would it not be safer to just use something like Twiingate or OpenZiti? So you would not have to open any port whatsoever? Or would that be too strict and hamper usability?
That's always the trade-off. For apps you only want certain people to access that would be a valid approach, but for a public website or app that's clearly not a viable option.
I'm still trying to understand why using Traefik is superior to just setting port forwards with port translation and IP whitelists in the firewall (in my case pfSense) itself?
@@unmesh59 the proxy provides a layer in front of the application, it also provides things like SSL and DNS names, plus with Traefik you can add things like crowdsec for additional security.
My thought is, can I ipwhitelist services to private addresses for my internal services as a middleware on traefik, and for my external services allow any ip? and throw in Authlia in front of all my services anyway so it is blocked with auth to begin with for an additional layer. Would something like this work?
Would there be a point to doing a setup like this if you keep everything internal and access it externally only via Tailscale?
Yes, but minimal gains. It helps you avoid annoying SSL warnings in the browser etc, and more and more services are becoming https only.
Would really love a video about caddy.
Thanks, it's on the list
Especially with Docker! :D @@Jims-Garage
I solve this issue with Authelia, only specified services are allowed either by bypass, one_factor or two_factor.
Yes, a default deny rule in Authelia is also a good option if you're using it. Authentik can also do a similar thing.
Does this vulnerability also apply to NGINX proxy manager? Eg port 443 coming from WAN routed to port 443 on nginx proxy manager which then redirects it to jellyfin for example?
I think you could also use MAC VLAN and have a separate ip for external, no ports needed.
Thanks, how do you mean? Two IPs for Traefik?
A nice idea, another idea is to run two instances of reverse proxies (either Traffik or NPM) external services are hosted on the traffik-external, and internal services are hosted on the traffik-internal and just configure the firewall (pfSense etc) to forward ports 80,443 to just the traffik-external reverse proxy.
This is the more secure option. Separation of concerns/contexts. If an attack an take advantage of an external vulnerability in the reverse proxy, they can potentially escalate themselves or move latterly and gain access to internal services.
Hello Jim, any recommendations on improving the security of the Traefik docker container? Like it’s exposing the docker socket and no internal user this could lead to some security concerns, please share you’re thoughts, thanks
You're right, I'm due to cover in a video about not using the sock and root user. I will be following the official docs so feel free to check those out if you're interested.
@@Jims-Garage I’m looking into to it, just a hint I tried the read-only flag in the compose file and seems not to break anything , I’ll try to integrate crowdsec to test further, thanks for the awesome videos and content!
@@jafandarcia you're welcome
Don’t you want your vaultwarden accessible to outside? What happens when you’re out and need to update a password? My partner won’t deal with VPNs
I use a VPN, but also, existing passwords are saved on the phone. The only issue you have is you cannot add new ones until you're back on the home network (hence the VPN).
the problem here is that you need a local DNS or modify the hosts file, both the external client and the internal client. Is not there a way so that the proxy redirect it?
Not sure I follow. You can add DNS records on your domain registrar for external sites.
@@Jims-Garage/videos Sorry for the bad English. I did set up everything except the option 3 and my problem is that i need to have a private DNS locally or modify my hosts file to be able to access the sites i have locally. And to be able to to access the sites i have externally i have to edit my hosts file and set my public ip to the site i want to access.
I did discover the problem with the external sites is because of the cloudflare proxy option, if i disable it i can access the external sites with any problems, what can be causing this? I would like to enable the proxy option if it is a way to make it work.
Also thanks for responding.
Thank for the tutorial though I don't know if I am missing something. I appreciate you hairpin NAT explanation but I'm unable to access my (https-external) service on my LAN. I have NAT reflection enabled on my OPNSense router but get a 404. My DNS overrides still list the internal IP's of all services. Is this correct?
My NAT rule was configured on the WAN interface only. I added the LAN interface to the NAT rule and bingo, LAN access to (http(s)-external) traefik services. Leaving this comment here for anybody else experiencing the same issue.
You need a nat rule that translates port 80 and 443 to your external port numbers. E.g., internal network -> translate port 80 to 81
Why not add a whitelist to the cloudflare tunnel as well?
I'm having an issue where using the http-external tag works fine (sending traffic through port 81). As soon as I add the https-external label, the site is no longer reachable.
Port 81 and 444 are routed correctly.
Has anyone else run into this or have any advise?
Sure but why proxy your internal apps at all? Most guys suggest running a proxy docker network for traefik. Ideally you have separate networks for your dbs internal apps and one for the socket proxy. Then your internal apps only sit in the internal where there is no traefik endpoint. Meanwhile because these are all custom networks you can still call apps by their container name.
Most apps now want valid SSL certificates for web interfaces etc and not using them is difficult. Furthermore, some apps I want both internal and external access to (e.g. Jellyfin)
What about using cloudflare tunnel? Do i have the same issues when only defining some public hostnames? Is it also possible to access to internal apps, which have in my case slso another subdomain as my external apps?
No, Cloudflare Tunnels shouldn't create the same issue. They create a whole other nightmare with zero privacy though...
@@Jims-Garage What are the other issues, except the case, that you have to trust cloudflare?
I´m curious about this statement too.
ive foolllowed your steps but after adding the ports ans the external parts i cant long in to the traefik dashboard i get 404 page not found error
If I may:
Does JC21 Reverse Proxy have the same issue?
I'm not familiar with that one.
Excellent video! I feel really dumb right now because I'm re-watching the video to find out why I couldn't get this setup to work on my end then I tried it a couple of days ago, and at the 5-minute mark I see why... I forgot to route ports 81 and 444 to Traefik in the docker-compose file. >_
Glad you managed to sort it
@@Jims-Garage I got one step closer... But when I try to http to an external service, it redirects to :444 in the browser (fails), direct to https works. And services on the internal http and https just plain don't connect even though Traefik says everything is fine. Really frustrating, but I'll come back to it after a bit of rest.
@@SnorreSelmer read up on hairpin NAT, that's what you need.
@@Jims-Garage Supposedly my UDM Pro has that automatically, it just doesn’t seem to work. But I’ll keep digging!
I wonder how this setup would look like with NGINX.
I will be covering nginx in the future, used it for a number of years prior to Traefik.
@@Jims-Garage I will wait patiently :)
Can't you make IP rules lists within Cloudflare (free)? Am I missing something?
You could, but it's still placing full trust into Cloudflare. These options enhance that by adding additional layers, especially with IPS/IDS on a firewall.
Does this problem exist when running traefic on K3S?
@@rcmnet yes. But you can fix much easier with network policies
traefik and crowdsec how do i reserve proxy outside docker
You don't need to guess a subdomain, It's really easy to get all subdomains just by knowing the main domain name. I would do this to yours, as a test to prove it to you but here in the UK without your explicit permission to do so it would be illegal under computer misuse laws.
If I proxy through Cloudflare, or use their Tunnels service, can they see the data that's being transmitted in plain text? Can the decrypt the data on their side?
Proxy, no. Tunnel, yes (if it's just HTTPS).
@@Jims-Garage So if I use 'davs', 'ftps' or any other protocol, they can't? Also, is there a way to prevent this from happening (when using Tunnel)?
@@kshitijkadlag anything that is end to end encrypted, no.
@@Jims-Garage Okay cool, thanks! :)
When you’re using an SSL Cert from Cloudflare then theirs are technically capable of reading your traffic. This is the case when you use the Proxy or Tunnels, only when you’re the only one who is holding the Private key then only you can decrypt the content, but when using the cloudflare services they are holding the private key so it’s possible that they decrypt your traffic and they do it to protect you from DDoS etc.
thanks. pls make a video how to access your services only via vpn (h. how to config firewall and how to config headscale for it.
Please check my recent videos, I have two videos on Headscale. One if you are able to port forward, and one using a VPS if you cannot.
While whitelisting seems to be the right thing, it doesn't seem to work. If I whitelist my LAN addresses and try to open the site, it says forbidden, and the log says that is not allowed to access. If I try to use the depth of ipstrategy, the log says that the IP is empty. Does anyone know a solution?
same problem
Great video. I use option 3 with the whitelists and restrict access for my internal apps to my local subnet and my vpn subnet ranges.
Im currently trying to work out how i can have it give a 404 rather than a 403. I also want to create a custom error page. Would be great if you could explain and show something with ref to this
Thanks, yes. I've used custom error pages before, but I agree it would be much better if you could generate a 404. I'll consider it for a follow up video.
i have trouble getting my dashboard to show up. i got a error message of 404 not found
Did you add a domain record in your DNS resolver?
@@Jims-Garage I added domain a record in cloudflare. I also notice you have crowdsec I don’t have that installed so I’m working on that as well. This is new to me and I’m learning. Like when I go to install pi hole with unbound it uses port 80. Do I change it to a diffrent port. I’m using traefik with nginx proxy installed as well. I’m doing this with Ubuntu and raspberry pi 5
@@jordanpfeifer9587 You need to add a local DNS record that points to your traefik Docker IP. e.g., traefik-dashboard.yourdomain.com points to 192.168.1.100 - this needs to be done locally on your firewall, hosts, or PiHole etc, not in cloudflare (external).
Can you access it outside your network if you wanted. Just curious
@@Jims-Garage I added a a record in pi local dns settings. But the page don’t show. I have my pc setup to use my dns server that you have. I still don’t have the page. Same error. Do you have to use pi hole as your dhcp. Or can I still use my own router for that.
You should run two proxies on diffrent vlans.
I dont understand how to access internal services. Can someone explain?
I assume you mean using the proxy with a URL? Check out my Traefik and PiHole videos, that should give you everything you need.
@@Jims-Garage but without pi-hole it is impossible to access local services? I precisely followed the tutorial btw and the external network is working except internal. For example i routed the traefik dash internally and cant access it.
Where is your traefik full tutorial?
Look for the thumbnail that pops up. ruclips.net/video/XH9XgiVM_z4/видео.html
@@Jims-Garage thanks a lot 👍
Wouldn’t it be easier to setup ip whitelists?
Potentially, depends on the intended audience. If it's meant to be publicly accessible then that strategy doesn't work, you'd want additional layers of scanning and segmentation.
@@Jims-Garage For me, I have some apps exposed to the internet and some others like vaultwarden are only accessible via whitelist. For those, I just use WireGuard to access these apps. Seems easier
Very interesting, and meanwhile so complicated for noobs as me
Thanks for the good video. Option 4 would be not do a port forwarding at all and use eg. Cloudflare tunnels to expose selected services instead.
True, provided that you fix the issue with Cloudflare Tunnels ruclips.net/video/1n9lCYCLUYI/видео.htmlsi=ZLhG173lomp9C0SF
Good luck doing that for Jellyfin/Plex without getting banned.
@@dustojnikhummer I've done it, albeit I'm a very light user. Not sure if there are specific rules on it?
@@Jims-Garage Cloudflare doesn't allow video streaming through Tunnel or Proxy
@@dustojnikhummer thanks for confirming
Considering method 3, is it possible to a hacker spoof its IP and pass as a whitelisted ip? My gut feeling is that option 2 is safer, but as I said, I’m not an expert, just curious hahaha
It's always possible but I find it very unlikely anyone would bother for a homelab. Technically it would require serious compromises of underlying infrastructure.
I just can't help it, but this is how this video reads to me: 'Are you using this proxy to expose services to the internet? Well, guess what, there is a significant security flaw, and you are probably not aware of it. You have exposed services to the internet.'
And yes, I know there might be people who simply don't understand the concepts and just slap in a Traefik instance in the middle of their stack because they read one tutorial and want this specific thing accessible from the internet. But who are these people who 'have a stack' and then 'accidentally' expose it to the internet as a whole? I just cannot fathom the cross-section between these two groups of people who seemingly have knowledge of running a server with a stack but then don't realize what the Traefik instance they just added to the stack does. But again, I might be biased as I have an IT background.
No, not quite right. The issue is that by routing everything through Traefik, by default all applications are accessible if you know the IP address. That includes things you only want available locally (e.g., vaultwarden).
How is possible that people thinks is a good practice to expose services to the exterior
OPNsense, and you're good to go
How so, what specifically are you using?
Go ahead, restrict it, see what happens when YOU can't connect to the internet.
@@Lulu-Rainsong not sure what you mean, this is for inbound not outbound traffic. Won't change internet routing.
Why would you want to poke holes in your firewall? Just run a cloudflare tunnel and get rid of that proxy and firewall openings. Those are serious attack vectors.
I don't want Cloudflare to see all of my traffic, it's a privacy nightmare. Plus, you're putting all of your trust into a single vendor... Better to run everything though a firewall, even if you're using a tunnel IMO.
The NSA just has to be all up inside Cloudflare.
All companies need to abide by the law of local jurisdictions, even VPNs...
I don’t understand the port forwarding part, I don’t have a firewall like sophos are you saying I need to do a port forward from port 444 to 443 & 80 to 81 I found this video for UFW will this apply ruclips.net/video/F9TuhNmzz3g/видео.htmlsi=0wt8V8T0eItAIDsf
Essentially, yes. Or you could use 2 traefik if you wanted.
@@Jims-Garage I like the idea of only using one traefik instance, I just wasn’t sure about the port redirection part. I think i understand now but we’ll see if I get working. Thanks for your reply anyway :)
will you update this on opnsense? 👀
Hopefully 🤞
dont you think that is too overhead? why not just play with rules for eg
Host(`resource.lan`) && ClientIP(`192.168.88.0/24`)