jokes aside, he's not a bad looking dude at all, getting jacked wouldn't hurt tho, as long as he doesn't covert to a life coach and talk about it non-stop like that other coder youtuber John Sonmez.
@@TechdubberStudios It may pay less than ads, but it's many times better. I support websites that responsibly use cryptomining, and I block ads. Please, don't say that ads are better. They have never been any good to anybody's web browsing experience. Oh, and you can use cryptomining along with Arc, another earning method that does not involve ads. I'm done with Google's creepy trackers. Cryptocurrency mining is the future.
@@ezshroom I am genuinely 100% with you on the crypto movement. I hate ads. Always have hated them. But there are at least 2....3 big corporations that come to mind that were built on the ads business model, but with crypto mining... can't find one. And browser-crypto-mining is not exactly a new technology. I really want it to replace ads. I really do. Hate the pop-ups, spying, tracking, that's going on. And the first corpo that comes to mind would be Netflix, when considering whom should adopt the crypto model. Because the users stay on netflix and binge-watch hours and hours!
@@ezshroom also, do you happen to know any website/forum/subreddit focusing on browser-based mining? I would really like to join and dig in more into this subject.
Fantastic job explaining this! As always, the hilarious dry humor and "next level" metaphors help drive home points and keep things entertaining. Really helped clear up a bunch of stuff and get me pointed in the right direction. Many thanks!
I avoided serverside rendering a meta tag by registering a sub-domain, doing the serverside-rendering there and making my app only compatible with a set number of user-agents. Brilliant!
7:20 would it be possible to use the same url but check the header value on the for example nginx server? like is this user agent a bot (twitter, fb, etc.) => proxy to your slim API for only the meta data response and if its a real user (mac, windows, chrome, firefox user agent etc.) => proxy to your real page / default response / SSR page. maybe im forgetting something. i dont know if this could work.
Lol just killed the whole argument. Never heard of it before. Just goes to say that tech is exponential. Wonder if it will cause the cosmic crash eventually.
Also check out the create-exact-app npm (that's exact not react). Like NextJS but Express-forward design, full control at the server side level of what's going on.
@@jpsimons Just FYI, next.js also gives you full server side control. You can just run next as a library within an express server. In my experience, it's super ergonomic while preserving the state-of-the-art benefits of next (code splitting, automatic static optimization, incremental static generation, etc.). Having said that, I have not yet checked out create-exact-app, and am not sure how it differs from nextjs.
Had this issue while using a MeteorJS website running with ReactJs as the client side is that we created a crawler ( i think there is an npm project for this) that would go to every page, render it and save it on our DB. When a non-human (Google) would access the site it would served this rendered HTML, making it SEO friendly. Basically our server would use the User Agent to define what type of content the user would get served. Hope this help
can you please explain what do you mean by 'render it and save it on db' , do you mean like render the dom elements and attach to html and store it in some link and add that link in db , or what exactly are you storing in db, if this is the case wouldn't it be too much for pages which are dynamic like /rob/photos, /sam/photos and likewise to be stored in db or am i missing something
there is a workaround : just add conditional tag in the small server that builds your page. you can still use client side rendering except for meta tags
2:05 i do it this way: Server serves response for parsers (meta, og, schema, jsonld and plain html content) and then comes along js that structures it up and takes over routing from this point, so when you navigate you actually don't "refresh"
A solution to your problem could be to build a single page application, with each end point for the app being pre rendered. It's basically jamstack. Once a user loads one page, the others do not need to be loaded.
Why do you want another domain for that. The server that delivers the app index.html could also deliver the meta response instead, based on user agent. One problem in both situations you could get (separate service or combined), is that the bots make checks whether the content they see is different to a regular user. Whether any of them does it, I don't know, but I would check this possibility.
I had to do that once, I used a Lambda function since it was hosted on AWS, and the function intercepts the CloudFront distribution request and updates the HTML if the request comes from a robot, adding the OpenGraph tags.
If you manage the web server, you could use the web server's router to do the same exact hack you described without the need for a different subdomain, just a route that checks the user-agent of the client and returns different HTML based on it.
Nice post. However I am curious as to why these FB bots can't read a client rendered app? I am sure that these bots are ultimately rendering the app on a web engine/browser, because they need to generate the preview of the rendered content. The static html will also most likely load some resources like CSS and JS over the network (unless the bot expects an inlined css/js), so it's not going to be an instant preview. So maybe the limitation is not technical but rather functional, in that the bot is probably not willing to wait too long for the page to load (which is the case with most client rendered apps)?
Greate video! I use Laravel on the server side to serve up everything. Static html pages and React apps or a combo of both. It's easy to embed a react app within a .blade template file. Meanwhile Laravel takes care of everything else, like API services, user registration and authentication, etc. Best of both worlds.
The problem I see with the magic link is that when a user just copies the url instead of pressing a share button to get the special share.* link that there will not be meta tags and when you do use the special share.* then there are meta tags. And I think most users will probably just copy the url they are on now instead of looking for a share button
I done this in a similar way. Except the share link page didn't check any user agent headers. It just featured a JavaScript redirect to the normal URL. Bots don't run JavaScript, don't redirect, they get the meta tags they need. User does run JS, redirects to the main SPA. And, for good measure, a fallback clickable link on the share page shows in case redirecting is blocked.
I just put my meta tags with variables like %PAGE_NAME% %PAGE_IMAGE% and replace these later while serving the page with express. doesn't work while client-side routing but It works for link previews.
my solution for this is: 1. set a lambda as an index 2. on the index lambda, it should run a headless tool (like a headless chrome) to render the page if the request is a bot, else just serve the raw js
If the only thing that needs to change is the meta tags (not the rendered bits), you can also modify the html before returning it to the client, inserting the relevant meta tags. It will probably lead to performance problems, but you could also perform an if condition on the referrer of the request to determine if you should perform such modifications.
kinda curious what the increased load would be if you'd do this on the normal domain already -> check the user agent... Would eliminate cdn usage tho... so maybe not foolproof.
Hello Ben, Have You thought about differences between react-snap and NextJS/Gatsby from SEO perspective? I mean is there reason to use NextJS instead just react-snap to get better results in search engines? Does NextJS/Gatsby do something extra to perform better in SEO? Regards
For some wierd cases like mine where only /particularRoute need to work like SSR what i actually tried was hosted gatsby project in particular route of CRA project and it worked just need to handle few re-routing cases
Our course -- I followed and understood all the way to the end. This is because I'm an unemployed ex-Tech Lead [who has never worked at a FANG company], and a thousandaire.
Looking for exactly this solution. Was thinking of using aws api gateway to check the routes and then redirect based on headers. Social og links are super important for shareability..
Yo Ben, just saw this video so my comment may be irrelevant by now but for a simpler solution could you just wrap your existing create-react-app codebase in an express app then serve the CRA to actual users and generate html for FaceBook bots etc all from one codebase? I imagine your express app would be super small so wouldn't be adding much weight to your existing codebase. You could then build your API into that express app too and treat it as a monorepo. I do see that this would completely negate the benefits of serving static sites via gh pages or Netlify but if you've got an active API anyways is it a large extra load for that API to serve your static assets as well? If you're using Lambda functions then fine, you win. Interested to hear your thoughts. Cheers :)
Ben Awad yeah fair. Maybe worth considering if you want to make it more static/ssr’ed in future but by then you might consider the full next.js recode that you discussed in the vid.
I think your points on pre-rendering are slightly off. Not mentioning tools like Gatsby, Scully, or Gridsome is a miss as they can be used to render those dynamic routes you say cannot be pre-rendered. It's worth mentioning that JAMStack options are becoming really incredible for developers and give you the same benefits as server side rendering out of the box.
Could you use something like pug generate a html file like the normal one that just contains the with meta tag links that have and just send that instead of the normal blank html (instead of having to different by client) ?
This helped me alot! I am working on a project and my backend was almost finished. I was using create-react-app with router but switched over to next.js! Thanks alot
Nice solution. Only downside guess would be that Google wants you to show the same content to their bot as the user. But probably doesn't matter if you don't want to index those pages.
I think you could easily do this in net core. In the startup class, in the configure for routing, you could filter each route with the correct meta tags. You could make this an extension and bing bang bosh, neat tidy job done
I wonder how you'll feel about Blazor, ASP.NET CORE released Blazor in order to do everything in C# and replace Javascript entirely ( I just started learning about it but it sounds like what they're doing, could be wrong)
You can write normal HTML with some normal CSS just to make the preview template, imagine that this is the loading page, and only once the client-side rendering is ready, you can show the content of that on top of that base temporary HTML template. I've never done client-side rendering before, and I've never used React, but I think this should work. Why are you saying that it's not possible?
netlify has a free experimental feature called pre-rendering, for me, it works with Facebook, it parses the right meta tags automatically with pictures also. My content comes from a backend via graphql and apollo. meta is being set with react helmet, the page is handled by react-router, and it's a create react app project. Hope this helps. You can also do prerendering very very easily with react-snap package, but you need to rebuild when data changes. (PS. Thanks for your work, I really like your videos)
This video is 2 years old, I don't know if it existed when you made it but today there is a tool called puppeteer that uses a headless Chrome browser to actually pre-render your HTML that you can feed to bots and still feed the client side rendered to humans
I think one issue is that most of the link preview code nowadays is kind of stuck in a pre-1page application world. Apart from making a curl request to the website and getting the HTML, the preview feature should also render it and stop assuming everybody's using server-side rendering. This does sound like it gets more into the "create a google scraper" type of situation, but if in-turn we're talking about a client-side rendered website trying to create a preview for another client-side rendered website, all the rendering is happening on the client so the server doesn't take much load at all.
There are benefits to client side rendering? Btw rather than checking the user agent, just include a client side redirect location="whatever.com" Because the bots dont have javascript they wont be redirected, but users will
You can use the header trick as you mentioned, and then simply use something like puppeteer to load the page on the server itself and then send the rendered HTML page to the client. If the header says it's from a normal user, then don't go to puppeteer, just return your usual index.html file.
Going through a similar issue myself. My static site is hosted on S3 / CloudFront, orchestrated by terraform. My plan is to use CloudFront Origin Response triggers to trigger a lambda function to add the correct open graph tags to the response. I think this is the lightest weight option.
Would the twitter/fb crawler follow a redirect? Or would it just grab the meta data and come back? In the latter case the user agent check wouldn't be required. Also, your users are going to grab the url from the browser bar and probably not user the share link you made special for them
Hey Ben, I'm starting a new react web app for something like delivery.com and doordash.com where SEO is a major thing. Like rich structured snippets of stars etc in google results. Is SSR the only option?
In that case Gatsby or NextJS or any other you recommend? I don't know if Gatsby will do but their web.dev score hits 💯 so was thinking that but in a dilemma
@@bawad do you know an easy way to migrate from create react app to next.js? I've found problems with REDUX, svg images and react router active route (NavLink). Great video btw ;)))
That's what I was thinking. Maybe something like this to handle the routing internally to an SSR-generated page: www.nginx.com/resources/wiki/modules/user_agent/ or this: serverfault.com/questions/775463/nginx-redirect-based-on-user-agent I'm sure there are other reverse proxies that support this, but this doesn't seem to be the most portable solution. Edit: added an example of a similar implementation from ServerFault
Well a simpler but similar way would be to always server side generate the index.html but only to change the meta data while letting the index.js generate the actual content. This wouldn't hurt human users and is pretty light on the server (only entry point html meta tags need to be generated depending on the link then the rest of the rendering navigation is done client side)
I liked the idea however, I think it is still good to have SSR for all users or maybe SEO as well. Would you share your ideas on uFrontends, too? Are you preparing some sort of tutorial on that or what?
Woah! My self esteem skyrocketed because I managed to keep up with you until the end :D Aside from that, your content is top notch, keep it coming man.
The pre-built html approach could work. If someone tries to visit a url while something is still building, you could serve up a special "page is still being built" page... one benefit here could be that, in 20 years when a website is taken down, you have all this pre-built static html, and it's very cheap (probably free) to host that.
Or when there's another sea-change with technology, MAYBE it's easier to migrate to something new, but I'm not sure, would depend on the specific situation.
I'm not a fan of server side unless it's to generate a HTML based email. Just seems so cluttered when doing work. Even the emails I generate on the server side are dreadful to work with but barrable since there aren't many.
I use client side rendering, and am able to return different HTML per page from the server. This is only a problem if you have a single, static, index.html.
I wonder how this solution for link previews would affect SEO, would Google consider this cloaking since one version of the page is displayed to bots and another to humans?
@@bawad thanks Ben. I guess my concern was maybe Google would think the difference between redirecting to the original URL for a human and not for a bot may be flagged up, more than an issue with just dynamic rendering in general :)
Doesn't Netlify have the option to pre render single page applications for bots? (Originally for a better SEO) I'm not sure about this because I have never used this feature before, but shouldn't it also be able to solve this problem?
If you decided to leave yourself open to refactoring your current solution to a server-side solution, will the performance hit be noticeable to the end-user?
With SSR no. The data management and computations happens at the server level. The components responsibility at the client level is dynamic UI rendering not manage API calls. Also the client devices do not have to be burdened with excessive large loads of JavaScript that they have to allocate memory resources to render JavaScript; that adds latency to the devices especially mobile devices. Components from an SSR perspective that contain the fetched data are going to be present and readily available to the client devices without worrying about spending computational resources on executing large JS bundles.
I made my "yarn deploy" script change react's build/index.html to build/index.php, then it throws in a little php script that grabs some info from the good ole' mongoDB and 'echo's the metatags into the , only when that url is directly accessed. (You can get the URL params in PHP) It's only a few lines of code. The hardest part was figuring out how to configure Nginx's reverse proxy to render PHP. O_O
Isn't this what client side routing or search queries are for? On document load before the window. Try some JSON data from thread, clone or populate a document fragment with your html and finally render the fragment using the query parameter and its prefix/suffix..
It works! I do exactly that with my react web SPAs. I use firebase and cloud functions to detect user agents and serve SSR version on the fly to robots and CSR version to users. This is also important to SEO indexing, cause some robots won't run any JS and expect html-only responses. Really enjoy your videos.
@@leisiyox I thought about using it, but never tried it. Don't know how well it works. It would also cost more than my current firebase cloud function solution.
@@leisiyox I like that they integrate lots of services in a single solution. When you create a firebase project, you instantly have access to file storage, hosting, database, authentication, and some other stuff that makes it really easy. I also like that Firestore has real-time listeners for your data. Really good for a client side rendered app. Also really like their documentation and the fact that you can easily access other Google Cloud services and API. There are many videos online about it. Check it out.
I just used a node.js express server to host compiled create-react-app, this way you can modify the page and add meta tags if needed before serving the page. Sort of a mix of server and client side rendering as he said.
Detecting the user agent and serving different content is called "dynamic rendering": developers.google.com/search/docs/guides/dynamic-rendering I've actually been wondering about this lately... whether to use Next.js or set up dynamic rendering so that only bots get the SSR version. But I don't get why you're talking about adding the "share." subdomain to serve different content? Why not just use dynamic rendering on the regular URLs?
Now you can make static pages for you dynamic frequently updated pages with Nextjs, How it works's is that it looks at the requested page and if it is present from the build time, sends it back and if it is not built during build time, builds on the go(run time) and adds it to the built pages for the next request. pretty amazing and game changing !
Your solution is nice, but people copy the url from the browser, and paste it where they wanna share it, and that won't work (they link will not have the "share" subdomain.. Or am I missing something? Anyway I use angular, and it has server side rendering so I think i will just use that.. You can also just switch to angular.. I know you like it
I wonder if doing something like this create-react-app.dev/docs/title-and-meta-tags#generating-dynamic-meta-tags-on-the-server would work for this use case and maybe solve this issue of having to check if the request is coming from a human or a bot.
I'm sure someone has already pointed out, but if someone copies the url from the browser, they won't have the `share.` prefix. Probably need a global UA interceptor that results in pages being rendered server-side. Year 2030: Browsers introduce a feature to request server-rendered pages by pretending to be a facebook/twitter bot.
At the end you said "In an ideal world, you would have picked the ideal tool to use in the first place." What would you consider the ideal tool in your case?
Guys I am not a Web developer so excuse me if I say something stupid, but why can't you just use the javascript to render a meta tag based on what route is invoked? Do link previews only parse the html without running any of the scripts on the page?
Because of the way that those websites scrap your content. They are not fully rendering the page as your browser does it. Just reading what the server is responding, the plain HTML of a SPA.
hey, can you talk about heroku, hasura and the likes? all this cloud tooling seems like the way to go to become a one full stack pony show, but I would love your opinion on it, thanks!
Another option would be to have a CDN and when you get content updates just render that single page to the CDN. Then you aren't having to rerender all the time. It would get more complex than that of course as you would need to rerender everything if you make a style change to your website. And it would be public. Unless you pay for something to make it private and have the CDN only return a partial page for bots.
I had this problem once but my focus was towards crawlers. I ended up using some php to "render" the important bits like title, descriptions and links. Then the javascript would remove those elements and do the single page app business. It was back in carrot farmer code days but I'm sure happy coders can accomplish this just as well.
Your metaphors are next level.
here's your big mac.
Aaaah I see what you did there...
🎈 🏠 🎈
"you are not a karen"
Ben's got 99 problems, but a girlfriend ain't one.
Bruh! He says it with a straight face
@@phantomKE I know right? He just leveled up with these jokes!
🤣🤣
jokes aside, he's not a bad looking dude at all, getting jacked wouldn't hurt tho, as long as he doesn't covert to a life coach and talk about it non-stop like that other coder youtuber John Sonmez.
He has a girlfriend in Canada.
I avoid client-side rendering in order to save CPU cycles for cryptocurrency mining.
hahaha
hilarious comment! but crypto mining is an inefficient form of revenue on client's computer, see TPB case experiment.
@@TechdubberStudios It may pay less than ads, but it's many times better. I support websites that responsibly use cryptomining, and I block ads. Please, don't say that ads are better. They have never been any good to anybody's web browsing experience.
Oh, and you can use cryptomining along with Arc, another earning method that does not involve ads.
I'm done with Google's creepy trackers. Cryptocurrency mining is the future.
@@ezshroom I am genuinely 100% with you on the crypto movement. I hate ads. Always have hated them. But there are at least 2....3 big corporations that come to mind that were built on the ads business model, but with crypto mining... can't find one. And browser-crypto-mining is not exactly a new technology. I really want it to replace ads. I really do. Hate the pop-ups, spying, tracking, that's going on. And the first corpo that comes to mind would be Netflix, when considering whom should adopt the crypto model. Because the users stay on netflix and binge-watch hours and hours!
@@ezshroom also, do you happen to know any website/forum/subreddit focusing on browser-based mining? I would really like to join and dig in more into this subject.
Fantastic job explaining this! As always, the hilarious dry humor and "next level" metaphors help drive home points and keep things entertaining. Really helped clear up a bunch of stuff and get me pointed in the right direction. Many thanks!
solutions:
0) pre-rendering with parcel or webpack
1) server side rendering
your solutions are not client side rendering. he mentioned it.
I avoided serverside rendering a meta tag by registering a sub-domain, doing the serverside-rendering there and making my app only compatible with a set number of user-agents. Brilliant!
I love the tint on your glasses, it's serial killer-ish, where can i get a pair like those?
a package arrives at your door after the 3rd kill
@@bawad respect
They are the left-behinds after each kill. That's the way you get it.
@@bawad quick scope no scopes?
Those tints are wiped off blood from killing
Love the joke about girlfriend and client side rendering at the beginning
For your sake and ours, I hope you DON'T get a girlfriend too soon.
7:20 would it be possible to use the same url but check the header value on the for example nginx server?
like is this user agent a bot (twitter, fb, etc.) => proxy to your slim API for only the meta data response
and if its a real user (mac, windows, chrome, firefox user agent etc.) => proxy to your real page / default response / SSR page.
maybe im forgetting something. i dont know if this could work.
the girlfriend problem might be solved if you stop walking around wearing asexual flag shirts
hahaha
lmao, good catch, respect
He's just playing hard to get. Karen gets it.
But with this if he ever gets one, she will be the right one. Lol
He do check a lot of aesthetic boxes from the virgin meme... Though I probably do too 😆
I guess u never heard of prerender.io
been using it for years
Lol just killed the whole argument. Never heard of it before. Just goes to say that tech is exponential. Wonder if it will cause the cosmic crash eventually.
yep yep yep , you just commented before me
@@ayushkhanduri2384 same case, I just searched prerender before I add a comment about it to check if it's already mentioned and here it was.
🤯
This was the best explanation video I've seen on the matter... Kudos to you Mister...
I love how Ben roasts Angular devs. I thought of that carrot farmer line off and on all day and cracked up every time.
Solution: NextJS, Angular Universal, Nuxt, etc.
Also check out the create-exact-app npm (that's exact not react). Like NextJS but Express-forward design, full control at the server side level of what's going on.
@@jpsimons Just FYI, next.js also gives you full server side control. You can just run next as a library within an express server. In my experience, it's super ergonomic while preserving the state-of-the-art benefits of next (code splitting, automatic static optimization, incremental static generation, etc.). Having said that, I have not yet checked out create-exact-app, and am not sure how it differs from nextjs.
Why do I not like the sound of Angular Universal?
@@angshu7589 because you are not carrot farmer. Although color of your profile picture kinda resembles the carrot :D
@Adithya R Svelte Sapper is still in early development. I love Svelte, but Sapper is still far away from production-ready
Had this issue while using a MeteorJS website running with ReactJs as the client side is that we created a crawler ( i think there is an npm project for this) that would go to every page, render it and save it on our DB. When a non-human (Google) would access the site it would served this rendered HTML, making it SEO friendly. Basically our server would use the User Agent to define what type of content the user would get served. Hope this help
can you please explain what do you mean by 'render it and save it on db' , do you mean like render the dom elements and attach to html and store it in some link and add that link in db , or what exactly are you storing in db, if this is the case wouldn't it be too much for pages which are dynamic like /rob/photos, /sam/photos and likewise to be stored in db or am i missing something
there is a workaround : just add conditional tag in the small server that builds your page. you can still use client side rendering except for meta tags
2:05 i do it this way:
Server serves response for parsers (meta, og, schema, jsonld and plain html content) and then comes along js that structures it up and takes over routing from this point, so when you navigate you actually don't "refresh"
A solution to your problem could be to build a single page application, with each end point for the app being pre rendered.
It's basically jamstack. Once a user loads one page, the others do not need to be loaded.
Why wouldn’t you just serve this over the server on a different port? As in send the result as a when someone searches that route?
Why do you want another domain for that. The server that delivers the app index.html could also deliver the meta response instead, based on user agent. One problem in both situations you could get (separate service or combined), is that the bots make checks whether the content they see is different to a regular user. Whether any of them does it, I don't know, but I would check this possibility.
I had to do that once, I used a Lambda function since it was hosted on AWS, and the function intercepts the CloudFront distribution request and updates the HTML if the request comes from a robot, adding the OpenGraph tags.
If you manage the web server, you could use the web server's router to do the same exact hack you described without the need for a different subdomain, just a route that checks the user-agent of the client and returns different HTML based on it.
Nice post. However I am curious as to why these FB bots can't read a client rendered app? I am sure that these bots are ultimately rendering the app on a web engine/browser, because they need to generate the preview of the rendered content. The static html will also most likely load some resources like CSS and JS over the network (unless the bot expects an inlined css/js), so it's not going to be an instant preview. So maybe the limitation is not technical but rather functional, in that the bot is probably not willing to wait too long for the page to load (which is the case with most client rendered apps)?
Greate video! I use Laravel on the server side to serve up everything. Static html pages and React apps or a combo of both. It's easy to embed a react app within a .blade template file. Meanwhile Laravel takes care of everything else, like API services, user registration and authentication, etc. Best of both worlds.
Sapper + svelte gives you the best of both worlds
Great video. Trying to wrap my head around server side rendering and this video definitely helped
Sorry I'm new here, and I've noticed that Ben clearly hates Angular.
Can someone give a quick background please???
I´ve been watching your videos and yes, the quality of the content is always awesome, new suscriber
The problem I see with the magic link is that when a user just copies the url instead of pressing a share button to get the special share.* link that there will not be meta tags and when you do use the special share.* then there are meta tags. And I think most users will probably just copy the url they are on now instead of looking for a share button
You ever tried NextJS? It does SSR for the initial request (b/c it could be a bot), but CSR when you click links.
I done this in a similar way. Except the share link page didn't check any user agent headers. It just featured a JavaScript redirect to the normal URL.
Bots don't run JavaScript, don't redirect, they get the meta tags they need. User does run JS, redirects to the main SPA. And, for good measure, a fallback clickable link on the share page shows in case redirecting is blocked.
"It's like I spent a bunch of time building a house and now I want that house to fly." LMAO
I just put my meta tags with variables like %PAGE_NAME% %PAGE_IMAGE% and replace these later while serving the page with express. doesn't work while client-side routing but It works for link previews.
The preview still won't work when users copy paste the link directly from the browser url bar
my solution for this is:
1. set a lambda as an index
2. on the index lambda, it should run a headless tool (like a headless chrome) to render the page if the request is a bot, else just serve the raw js
If the only thing that needs to change is the meta tags (not the rendered bits), you can also modify the html before returning it to the client, inserting the relevant meta tags.
It will probably lead to performance problems, but you could also perform an if condition on the referrer of the request to determine if you should perform such modifications.
kinda curious what the increased load would be if you'd do this on the normal domain already -> check the user agent... Would eliminate cdn usage tho... so maybe not foolproof.
I had similar issue. good thing you found a better solution.
Hello Ben, Have You thought about differences between react-snap and NextJS/Gatsby from SEO perspective? I mean is there reason to use NextJS instead just react-snap to get better results in search engines? Does NextJS/Gatsby do something extra to perform better in SEO? Regards
You are hilarious and informative my dude haha, relatable. And damn dude the lenght of your link
For some wierd cases like mine where only /particularRoute need to work like SSR what i actually tried was hosted gatsby project in particular route of CRA project and it worked just need to handle few re-routing cases
Our course -- I followed and understood all the way to the end. This is because I'm an unemployed ex-Tech Lead [who has never worked at a FANG company], and a thousandaire.
Hey I saw Wes Bos in one of his videos, he used cloud functions to generate the preview and puppeteer i guess to take a screenshot of the url
This channel is slowly becoming one of my favorites on RUclips! 😄
Looking for exactly this solution. Was thinking of using aws api gateway to check the routes and then redirect based on headers. Social og links are super important for shareability..
Yo Ben, just saw this video so my comment may be irrelevant by now but for a simpler solution could you just wrap your existing create-react-app codebase in an express app then serve the CRA to actual users and generate html for FaceBook bots etc all from one codebase? I imagine your express app would be super small so wouldn't be adding much weight to your existing codebase. You could then build your API into that express app too and treat it as a monorepo.
I do see that this would completely negate the benefits of serving static sites via gh pages or Netlify but if you've got an active API anyways is it a large extra load for that API to serve your static assets as well? If you're using Lambda functions then fine, you win.
Interested to hear your thoughts. Cheers :)
I could do it that way, but I only needed it one specific thing, so I wanted to try keeping the benefits of using a CDN
Ben Awad yeah fair. Maybe worth considering if you want to make it more static/ssr’ed in future but by then you might consider the full next.js recode that you discussed in the vid.
I think your points on pre-rendering are slightly off. Not mentioning tools like Gatsby, Scully, or Gridsome is a miss as they can be used to render those dynamic routes you say cannot be pre-rendered. It's worth mentioning that JAMStack options are becoming really incredible for developers and give you the same benefits as server side rendering out of the box.
Could you use something like pug generate a html file like the normal one that just contains the with meta tag links that have and just send that instead of the normal blank html (instead of having to different by client) ?
This helped me alot! I am working on a project and my backend was almost finished. I was using create-react-app with router but switched over to next.js! Thanks alot
Nice solution. Only downside guess would be that Google wants you to show the same content to their bot as the user. But probably doesn't matter if you don't want to index those pages.
I think you could easily do this in net core. In the startup class, in the configure for routing, you could filter each route with the correct meta tags. You could make this an extension and bing bang bosh, neat tidy job done
I wonder how you'll feel about Blazor, ASP.NET CORE released Blazor in order to do everything in C# and replace Javascript entirely ( I just started learning about it but it sounds like what they're doing, could be wrong)
You can write normal HTML with some normal CSS just to make the preview template, imagine that this is the loading page, and only once the client-side rendering is ready, you can show the content of that on top of that base temporary HTML template.
I've never done client-side rendering before, and I've never used React, but I think this should work. Why are you saying that it's not possible?
netlify has a free experimental feature called pre-rendering, for me, it works with Facebook, it parses the right meta tags automatically with pictures also. My content comes from a backend via graphql and apollo. meta is being set with react helmet, the page is handled by react-router, and it's a create react app project. Hope this helps. You can also do prerendering very very easily with react-snap package, but you need to rebuild when data changes. (PS. Thanks for your work, I really like your videos)
this is actually the first time I understood the difference between client side and server side rendering
This video is 2 years old, I don't know if it existed when you made it but today there is a tool called puppeteer that uses a headless Chrome browser to actually pre-render your HTML that you can feed to bots and still feed the client side rendered to humans
Nice GatsbyJS colorway on that shirt 🤙
I think one issue is that most of the link preview code nowadays is kind of stuck in a pre-1page application world. Apart from making a curl request to the website and getting the HTML, the preview feature should also render it and stop assuming everybody's using server-side rendering. This does sound like it gets more into the "create a google scraper" type of situation, but if in-turn we're talking about a client-side rendered website trying to create a preview for another client-side rendered website, all the rendering is happening on the client so the server doesn't take much load at all.
I use EJS and it allows for variables to be passed before sending the HTML to the client, so that can allow you to change the values in the meta tags.
There are benefits to client side rendering?
Btw rather than checking the user agent, just include a client side redirect location="whatever.com"
Because the bots dont have javascript they wont be redirected, but users will
You can use the header trick as you mentioned, and then simply use something like puppeteer to load the page on the server itself and then send the rendered HTML page to the client. If the header says it's from a normal user, then don't go to puppeteer, just return your usual index.html file.
I got this idea from here: ruclips.net/video/lhZOFUY1weo/видео.html
Going through a similar issue myself. My static site is hosted on S3 / CloudFront, orchestrated by terraform. My plan is to use CloudFront Origin Response triggers to trigger a lambda function to add the correct open graph tags to the response. I think this is the lightest weight option.
Did you do it? Am exploring some of this ideas :)
Would the twitter/fb crawler follow a redirect? Or would it just grab the meta data and come back? In the latter case the user agent check wouldn't be required. Also, your users are going to grab the url from the browser bar and probably not user the share link you made special for them
You make my day better
Hey Ben,
I'm starting a new react web app for something like delivery.com and doordash.com where SEO is a major thing.
Like rich structured snippets of stars etc in google results.
Is SSR the only option?
If your content could be pre-generated (for example, blog), one of the options would be to use Gatsby.js
yeah
In that case Gatsby or NextJS or any other you recommend?
I don't know if Gatsby will do but their web.dev score hits 💯 so was thinking that but in a dilemma
Next.js
@@bawad do you know an easy way to migrate from create react app to next.js? I've found problems with REDUX, svg images and react router active route (NavLink). Great video btw ;)))
Cant this be solved by a reverse proxy like NGINX just to cache the SSR rendered page but only serving it to bots?
That's what I was thinking. Maybe something like this to handle the routing internally to an SSR-generated page: www.nginx.com/resources/wiki/modules/user_agent/
or this: serverfault.com/questions/775463/nginx-redirect-based-on-user-agent
I'm sure there are other reverse proxies that support this, but this doesn't seem to be the most portable solution.
Edit: added an example of a similar implementation from ServerFault
Well a simpler but similar way would be to always server side generate the index.html but only to change the meta data while letting the index.js generate the actual content.
This wouldn't hurt human users and is pretty light on the server (only entry point html meta tags need to be generated depending on the link then the rest of the rendering navigation is done client side)
I liked the idea however, I think it is still good to have SSR for all users or maybe SEO as well.
Would you share your ideas on uFrontends, too? Are you preparing some sort of tutorial on that or what?
Woah! My self esteem skyrocketed because I managed to keep up with you until the end :D Aside from that, your content is top notch, keep it coming man.
I had the same issue last week and also was thinking about moving to nextjs, but having a separate domain and server makes a lot more sense.
well Next js or any other SSR solutions doesn't mean you're gonna use one server for the backend and the front-end.
The pre-built html approach could work. If someone tries to visit a url while something is still building, you could serve up a special "page is still being built" page... one benefit here could be that, in 20 years when a website is taken down, you have all this pre-built static html, and it's very cheap (probably free) to host that.
Or when there's another sea-change with technology, MAYBE it's easier to migrate to something new, but I'm not sure, would depend on the specific situation.
I'm not a fan of server side unless it's to generate a HTML based email. Just seems so cluttered when doing work. Even the emails I generate on the server side are dreadful to work with but barrable since there aren't many.
o-o anything wrong with just using twig/blade and just use vue compiled version when you want shiz to update in real time?
I use client side rendering, and am able to return different HTML per page from the server. This is only a problem if you have a single, static, index.html.
I wonder how this solution for link previews would affect SEO, would Google consider this cloaking since one version of the page is displayed to bots and another to humans?
developers.google.com/search/docs/guides/dynamic-rendering
@@bawad thanks Ben. I guess my concern was maybe Google would think the difference between redirecting to the original URL for a human and not for a bot may be flagged up, more than an issue with just dynamic rendering in general :)
Doesn't Netlify have the option to pre render single page applications for bots? (Originally for a better SEO)
I'm not sure about this because I have never used this feature before, but shouldn't it also be able to solve this problem?
If you decided to leave yourself open to refactoring your current solution to a server-side solution, will the performance hit be noticeable to the end-user?
With SSR no. The data management and computations happens at the server level. The components responsibility at the client level is dynamic UI rendering not manage API calls. Also the client devices do not have to be burdened with excessive large loads of JavaScript that they have to allocate memory resources to render JavaScript; that adds latency to the devices especially mobile devices. Components from an SSR perspective that contain the fetched data are going to be present and readily available to the client devices without worrying about spending computational resources on executing large JS bundles.
how to go back and forward if we are doing client side rendering?
i think we can use helmet in CRA also by generating the data for meta in server side
I made my "yarn deploy" script change react's build/index.html to build/index.php, then it throws in a little php script that grabs some info from the good ole' mongoDB and 'echo's the metatags into the , only when that url is directly accessed. (You can get the URL params in PHP) It's only a few lines of code. The hardest part was figuring out how to configure Nginx's reverse proxy to render PHP. O_O
Isn't this what client side routing or search queries are for? On document load before the window. Try some JSON data from thread, clone or populate a document fragment with your html and finally render the fragment using the query parameter and its prefix/suffix..
It works! I do exactly that with my react web SPAs. I use firebase and cloud functions to detect user agents and serve SSR version on the fly to robots and CSR version to users. This is also important to SEO indexing, cause some robots won't run any JS and expect html-only responses. Really enjoy your videos.
What about some prerender.io ?
@@leisiyox I thought about using it, but never tried it. Don't know how well it works. It would also cost more than my current firebase cloud function solution.
@@cauebahia what are the conditions that you recommend using firebase?
I thought about using it but I seek guidence
@@leisiyox I like that they integrate lots of services in a single solution. When you create a firebase project, you instantly have access to file storage, hosting, database, authentication, and some other stuff that makes it really easy. I also like that Firestore has real-time listeners for your data. Really good for a client side rendered app. Also really like their documentation and the fact that you can easily access other Google Cloud services and API. There are many videos online about it. Check it out.
I like how you explain, well done. Thank you for the quality content.
I just used a node.js express server to host compiled create-react-app, this way you can modify the page and add meta tags if needed before serving the page. Sort of a mix of server and client side rendering as he said.
Detecting the user agent and serving different content is called "dynamic rendering": developers.google.com/search/docs/guides/dynamic-rendering
I've actually been wondering about this lately... whether to use Next.js or set up dynamic rendering so that only bots get the SSR version.
But I don't get why you're talking about adding the "share." subdomain to serve different content? Why not just use dynamic rendering on the regular URLs?
has there been a followup to this? video/repo?
Facebook's user agent is there for the facebook app browser as well.
Now you can make static pages for you dynamic frequently updated pages with Nextjs, How it works's is that it looks at the requested page and if it is present from the build time, sends it back and if it is not built during build time, builds on the go(run time) and adds it to the built pages for the next request. pretty amazing and game changing !
Your solution is nice, but people copy the url from the browser, and paste it where they wanna share it, and that won't work (they link will not have the "share" subdomain..
Or am I missing something?
Anyway I use angular, and it has server side rendering so I think i will just use that.. You can also just switch to angular.. I know you like it
Gatsby also solves the React single-page problem, since we can generate all the individual HTML, CSS, and JS pages.
I wonder if doing something like this create-react-app.dev/docs/title-and-meta-tags#generating-dynamic-meta-tags-on-the-server would work for this use case and maybe solve this issue of having to check if the request is coming from a human or a bot.
How about mixing django template system with one spa like vue . What do you think ?
I'm sure someone has already pointed out, but if someone copies the url from the browser, they won't have the `share.` prefix. Probably need a global UA interceptor that results in pages being rendered server-side.
Year 2030: Browsers introduce a feature to request server-rendered pages by pretending to be a facebook/twitter bot.
At the end you said "In an ideal world, you would have picked the ideal tool to use in the first place." What would you consider the ideal tool in your case?
Guys I am not a Web developer so excuse me if I say something stupid, but why can't you just use the javascript to render a meta tag based on what route is invoked? Do link previews only parse the html without running any of the scripts on the page?
Because of the way that those websites scrap your content. They are not fully rendering the page as your browser does it. Just reading what the server is responding, the plain HTML of a SPA.
3:17 thanks for explaning this here lol, i was like why wouldn't you be able to just stick the meta tags in the requested html
hey, can you talk about heroku, hasura and the likes? all this cloud tooling seems like the way to go to become a one full stack pony show, but I would love your opinion on it, thanks!
Another option would be to have a CDN and when you get content updates just render that single page to the CDN. Then you aren't having to rerender all the time. It would get more complex than that of course as you would need to rerender everything if you make a style change to your website. And it would be public. Unless you pay for something to make it private and have the CDN only return a partial page for bots.
I had this problem once but my focus was towards crawlers. I ended up using some php to "render" the important bits like title, descriptions and links. Then the javascript would remove those elements and do the single page app business. It was back in carrot farmer code days but I'm sure happy coders can accomplish this just as well.