Build A Full Stack Chatbot in Rust (feat. Leptos & Rustformers)
HTML-код
- Опубликовано: 8 сен 2024
- A walkthrough of building a ChatGPT clone using Rust on the frontend and the backend. Quick introduction to the Leptos framework, Rustformers/llm, and open source language models.
Code: github.com/Me1...
Leptos TailwindCSS example: github.com/lep...
Camera: Canon EOS R5 amzn.to/3CCrxzl
Monitor: Dell U4914DW 49in amzn.to/3MJV1jx
SSD for Video Editing: VectoTech Rapid 8TB amzn.to/3hXz9TM
Microphone 1: Rode NT1-A amzn.to/3vWM4gL
Microphone 2: Seinheiser 416 amzn.to/3Fkti60
Microphone Interface: Focusrite Clarett+ 2Pre amzn.to/3J5dy7S
Tripod: JOBY GorillaPod 5K amzn.to/3JaPxMA
Mouse: Razer DeathAdder amzn.to/3J9fYCf
Computer: 2021 Macbook Pro amzn.to/3J7FXtW
Lens 1: Canon RF50mm F 1.2L USM amzn.to/3qeJrX6
Lens 2: Canon RF24mm F1.8 Macro is STM Lens amzn.to/3UUs1bB
Caffeine: High Brew Cold Brew Coffee amzn.to/3hXyx0q
More Caffeine: Monster Energy Juice, Pipeline Punch amzn.to/3Czmfox
Building A Second Brain book: amzn.to/3cIShWf
Keyboards
Redragon K552 - amzn.to/3oNtpD7
Keychron Q1 - amzn.to/3YkJNrB
Keebio Iris - keeb.io/collec...
Purple Gradient Keycaps on Iris - amzn.to/3UZq93f
Corne v3 - shop.beekeeb.c...
Apollo themed keycaps on Corne - amzn.to/3IXKPUc
Chocofi - shop.beekeeb.c...
Piantor - shop.beekeeb.c...
Kinesis Advantage360 Professional - amzn.to/3Ce5zUf
Glove80 - www.moergo.com...
![ARGENTINA vs. CHILE [3-0] | RESUMEN | ELIMINATORIAS SUDAMERICANAS | FECHA 7](http://i.ytimg.com/vi/De3AWmXzhuA/mqdefault.jpg)
Join us on Discord discord.gg/2G35Sp6kpQ
Does it have a bridge to Matrix?
It's so cool you use Rust on backend and frontend the same time! Would be great to see more videos on using Rust to build full stack! And the bot itself is super fun!
nice, glad to hear this one worked for many folks - looks like I'll be making more!
@@codetothemoon Please do!
Rusty Bros... Get ready for another great ride
🚀
Super interesting video! I love watching someone go trough a project small-ish but complete project. Would love to see more videos like this in future!
I’m just starting to code again. Last time I wrote code was in 1997 and was promptly discouraged as I am quite dyslexic. Not anymore with ChatGPT, holy heck has it been helpful for me! Watching projects from start to finish, and the methods for doing so is what I need to see and learn. =]
Your videos are some of the most timely, spot-on code tutorials available. It is as if I get an idea to do a project, and out of nowhere, you post a video showing me just exactly how to approach it. Really amazing stuff you produce.
someone's got a crush... but is it maybe likely that folks are about building chatgpt clones now, perhaps? :)
@@adicide9070 bahaaha :D
Just too nice, two things I wanted to have a look at united:
a Leptos example and
how to access G. Gerganov's language models in Rust.
So, thanks a lot! 🦀👍
thanks! glad you got something out of it!
Great video! I typically use these videos as reference when I'm building in Rust. Adding timestamps would be a 10x improvement for me!
thanks, really happy you got value out of it! and thanks for the feedback!
Thanks for this! You always bring out quality content.
glad you got something out of it! 😎
this is the most disgustingly insane thing I've seen in a while. I will be trying trying both the llm integration and the all rust NoJS stack immediately
Great let us know how it goes! hoping to be able to build more stuff like this...
That was the rustiest video I've seen in a while! Thank you! Keep up the good work. Really like the content and presentation. Also a bit of offtopic. I've noticed that you've switched from neovim to doom emacs several months ago. Maybe you can share your experience? I'd be very interested.
thanks for the kind words! my primary reason for switching to doom emacs is really org mode and org-roam, so I can take notes and manage knowledge in the same place I'm writing code. To me that's very valuable. I did made a video primarily about org mode ("Why I Prefer Emacs over vim") but I would like to do a more in depth comparison between doom emacs and a Neovim configuration framework like NvChad (which I've also been very impressed with)
Trpc i think gives you the same functionality in JavaScript land for not having to write fetches manually and just calling a function name.
15:10 that pattern reminds me of how tRPC works while using Typescript
Thanks, yeah I hadn't tried tRPC but a few others mentioned this as well! Good to know the pattern is available in the JavaScript ecosystem as well!
Cool to see Doom Emacs! The Monokai themes are nice but I prefer the default Doom One theme.
I agree the default one is pretty good. it's fun to switch things up now and then, maybe I'll give it another shot
Can confirm I was able to get this working on my desktop. It's got a ryzen 5 so responses do take a bit longer, but just wanna say awesome guide and ty.
Still trying to get my head around rust, and I don't use js/ts at all, so I just copied the tailwind stuff from the repo and it was fine.
The rust error handler is such a joy. I just went step by step and fixed everything (I had lots of typos from typing by hand to get used to the syntax), and leptos built at the end just fine.
Would be cool, if the LLM itself is streaming, to use a streaming HTTP call, updating the bot's message every time the client receives a chunk of the message. This is also what ChatGPT does-it's not just fancy effects! That way you could even halt the response while it's still running mid sentence as well.
Funny you mention this - after the video I actually implemented what you’re describing using web sockets. It’s in the repository if you’d like to have a look 😎
@@codetothemoon Yeah, that's cool! Though I specifially meant with Server-Sent Events since it's only a one-way streaming connection that is needed, and so there would not be a need for change of protocol nor the overhead of the WebSocket connection on the client. Though, WebSockets works too!
yeah I can't remember why I went with WebSockets instead of Server-sent events. I think maybe to keep the conversation context handy for the next prompt from the user? which you could do without WebSockets but there wouldn't be a guarantee the next request will go to the same host. not sure.@@dealloc
This is awesome. Can you release some content on how you got to the level of rust you are today? What projects do you think could elevate a novice??
thanks! Everything I've learned about Rust has come from making videos about it and doing little projects like the one in this video. But most of my knowledge of software development comes from my ~13 years in the industry prior to learning Rust 😎
As for projects that can elevate your skills, the best choice is always the project that you're most excited about. I don't think one specific type of project lends itself more to learning than another - I think for any given type of project there are are ways of approaching it that are more suited for a beginner and other approaches that an expert might take.
Examples: make a game, make a webapp, create generative art, create a command line utility, etc. All of these are suitable for beginners and experts alike, though the approach might differ depending on the level of experience.
The pattern you mentioned at 15:10 or so is very similar to the way things are done in PHP land via a package for the Laravel framework called Livewire. Essentially, two parts: A Livewire PHP component class, and a Blade (html templating engine) file that references it. So you might have
class MyComponent extends Livewire {
public string $someValue = "Test";
public function save() {
// Do saving stuff here
}
}
and then your HTML can just do stuff like
and
and you never have to write any of the transport logic.
Great work as always! Love your teaching style. Any plans on paid courses?
thank you! I do have something along those lines in the works...
Loved the video, please more Leptos!
Super useful, thank you!
nice, really happy you got something out of it!
supabase and featherjs offer similar front/back integration for ease of dev…
thanks for making this vid!!
ohh nice thanks for letting me know, I haven't used either of those - maybe I should check them out. Thanks for watching, glad you liked it!
Great stuff!
But `unwrap_or_else(|err| panic!(...))` looks a bit silly to me, considering the behavior of unwrap is to panic with the error message (and expect(...) to panic with a custom error message).
server actions are what you're looking for at 14:00
While there are robust web frameworks like React or Angular available for frontend development, it's worth considering Rust only for backend tasks to ensure high performance.
this is awesome!! excited to learn from this vid
nice! hope you get some value out of it!
The Leptos Server Functions pattern is effectively TRPC
yeah I took a look quick look at TRPC - at first glance it appeared to be similar but not quite as concise as clean as what Leptos has. could be completely wrong about that - maybe I just need to do a deeper dive.
15:00 honestly sounds a lot like some of the benefits u get from graphql but for a little less verbosity (depending on which gql libraries/frameworks u use)
Hey there! This's awesome! I have been wanting to do try full-stack Rust but for some reason I cannot get Leptos to deploy properly online. Locally everything works but when deployed to any host (e.g Github pages or AWS), it always shows an empty page LOL
I do have my backend running in Rust though via Actix Web :D
Wow, great job!
thanks, glad you liked it!
What is your neovim plugin for the file manager? I've looked over to your github but I didn't find your dotfiles. Great video btw!
Ah nvm you are using emacs
yep doom emacs! of which I'm a big fan
Exceptional example showing a lot of interesting and useful things/techs! Instant subscriber! :D
The `[profile.dev.package.ggml-sys]` specifies to compile the listed dependency package (or all if you write `[profile.dev.package."*"]`) with optimization level 3, the same that's used when compiling with --release: that way the dependencies (that don't change often unlike our code under active development) can __run__ fast while our code will still __compile__ faster and have debug data
thank you, very happy to have you onboard!
and thanks for clarifying the meaning of `[profile.dev.package.ggml-sys]`!
F#/websharper has this function call inside of api call. i tought this is coolest thing I ever seen but, in time this abstraction is to much for mind. I prefer standard http api call with http client(fetch,axios etc). Also last few month using htmx with maud(server side html generator :P) make things easier and more hypermedia style.
I also had to install "npm install -g sass" to make "cargo leptos build" work
oh interesting thanks for pointing this out!
Super awesome! ❤
thanks, glad you liked it!
holy, spent like 3 hours trying to figure out why mine didnt compile. turns out future_utils is not futures_util. *eye roll* there should be a law about naming your libraries too close to another. lol
hey, man thanks for this awesome video, I wanted to ask, which hardware specifications does your laptop has
thanks for watching! It's an M1 MacbookPro with an M1 Max and 32GB RAM
i think the closest thing i can think of to this in the JS world is NextJS server actions which is still beta
nice, I may have to check those out!
This is awesome, thanks
nice, glad you got something out of it!
Would the wizard-vicuna-13B-GGML model work if I switched it out with the current model you are using? Amazing video by the way taught me all the stuff needed to bootstrap a fullstack Rust project, also thanks a ton for the bonus tailwindcss integration part haha
I would like a Tailwind video! :D
nice thanks for the feedback! will try to make one...
@@codetothemoon nice!! It would be cool to see it integrated with Leptos. Thanks for all your work and for taking the time to answer me :))))
Great Thanks for Sharing.
thanks for watching!
Qwik has a pattern like Leptos's server APIs!
correct me if im wrong, but i think you can call the backend functions in a similar way in the T3 stack can you not? Also really great video.
Trpc which is what t3 uses for that l, is about as close to serverfn as your gonna get as far as I know when doing client and server in typescript
never tried T3 myself, been hearing great things about it though. I may have to give it a spin!
Interesting tutorial! I will definitely give it a try. I have a question. Is it also possible to define your own applications, similar to OpenAi? Example: Can I link a specifications document (.pdf/.text file) and then make requests for it? And then the bot gives me an answer in the context of the document.
thanks! what you're describing is absolutely possible, and there are a couple approaches you can take
1. If you're using a language model with a large context window, you might be able to simply prefix your prompt with the entire contents of the relevant documents
2. Implement RAG (Retrieval augmented generaton) yourself - where semantic search is done on the documents in question and the pieces deemed most relevant to the prompt are extracted and included in the prompt to the language model. There are databases that provide semantic search capability like Pinecone and Nuclia that can help with this
Wow. Thanks.
glad you got something out of it!
Some checkpoints in the video would help.
tRPC does the typesafe server side api thing you want.
thanks - I took a look at tRPC and it looked close but not quite as clean as what Leptos gives you. I didn't spend too much time investigating though - maybe I just need to dig a little deeper
Awesome tutorial!
Thanks!
thank you thank you so much for the support!
I've got most of the leptos components working great
but for some reason I can't seem to get Rustformers LLM tools to work with the LLM; I've snagged the same Vicuna model you linked.
I've git cloned the entire repository, and I've tried targeting a single node from it, and I've tried targeting the directory that all the nodes live in
Should I be zipping up the repo into a single file or what is the best way to connect the Rustformers/LLM tool to my actual LLM?
what do you mean by "targeting" in this context? Make sure to specify the path to the language model in the `.env` file in the root of the repository (by default it's set to the path on my system). More broadly, make sure you've followed all of the steps in the readme (I just updated it a bit yesterday) github.com/Me163/rusty_llama/blob/main/README.md
15:13 you might be looking for trpc? It kind of accomplishes the same thing.
Thank you for this, what is the plugin you're using to find your imports?
Yes, actix!
😎
Great video!
thank you!
Thanks!!
thanks for watching!
A link to your dotfiles please :) specifically vim, that's some awesome status line!
it's actually doom emacs! theme is doom-monokai-pro. other than that, everything else is pretty much what you get out of the box.
@@codetothemoon I haven't noticed the window title :facepalm: Thanks :) Great content (regardless of the inferior editor preference).
Why not liveview? And yes rust and elixir have liveview that can do SPA without JS(only for websocket) render everything at the the backend
I'd always been curious about this approach, and maybe it would be a better fit. it seems like it's well supported in the Elixir world - what do you think the best approach in Rust land would be? This crate? github.com/lunatic-solutions/submillisecond-live-view
you seem to be a lot into rust and so am i, however i've never used lisp much and i presume you did since you are a emacs user.
what comparison of the two would you make in term of workflow ?
N00b question: If I wanted to integrate such a chatbot into a website, what would be the best approach?😮
It really depends on the nature of the website that you're looking to integrate it into. You could use this project as a starting point and built the rest of the application around it, or you could use this project as an example for building something similar into whatever tech stack your website currently uses.
Great job man! What font are you using in your editor?
Is leptos or dioxus a single file component? I come from svelte and blazor and single file components are a lot cleaner from my experience
by "single file component" do you mean SPA? if so then the answer is yes. If you really mean single file component, it's completely up to you as the developer whether you want files to be 1:1 with components
Quick question, if I wanted to enable CUDA support where would I do that (just kind of a general direction thing). I went to the docs for Rustformers and *attempted* to try their implementation, however that didn't seem to do anything. This may just be a matter of "git-gud" for doc reading.
I actually had the same question, but in my case pertaining to metal support. I found instructions on how to enable it for the Rustformers command line tool, so my next step was going to be looking at the code for that to see what it's doing under the hood. Wasn't able to get there prior to making this video unfortunately :/
Good timing! I've just added CUDA/OpenCL support yesterday. Yeah, you're right, our docs on how to use GPU acceleration could use some work. I'll admit, documentation isn't really my strong suit.
@@lukaskreussel669 nice, thanks Lukas! Really appreciate your efforts!
Danke!
thank you so much for the support!!! 🙏
Now create_action needs to take a function which returns a future
Idk if im doing something massively wrong but inference doesn't work for me. I think cpu bottleneck as the request doesn't complete at all. i have an older ryzen 5 but can i not use gpu for inference? i have a 3060ti.
I think there is a way to enable CUDA for inference (or Metal) but I hadn't looked into that yet prior to creating this video. Maybe hop on the Rustformers/llm Discord, I'm pretty sure CUDA is supported, and I'd think the 3060ti should be plenty to run Vicuna 7B
can you do a video that, which areas rust is helpfull in using in react app like that.
you're asking what the advantages of using Rust on the frontend are? Type safety guarantees (because TypeScript is a superset of JavaScript, it can't really do such a thing), no garbage collection which yields better performance for certain use cases. Language isomorphism for projects where the backend is also in Rust (such as this one). Basically any reason you'd prefer using Rust over some compile-to-js language on the backend would likely apply to the frontend as well.
Great
thank you, glad you got something out of it!
It was really a great learning@@codetothemoon
How about dioxus?
it's extremely high on my list of stuff to check out!
Nice! what theme you using in Emacs?
thanks! doom-monokai-pro
I think the project is a bit high level for me for now, will need to learn th rust syntax and come back.
Great video! What window manager do you use?
thank you! I use Amethyst - I think it's fantastic, but it's also the only window manager I've really used for more than a trivial amount of time 😎
@@codetothemoon what is the code editor? thanks
@@guisoliven8022 doom emacs 😎
you got a sub
nice! very happy to have you onboard!
I just do a insta like when I see a video about rust with more than 10min.
How to get a Tailwind CSS, for some reason it doesn't work on my end. And how to swap over Axum from Actix? Thank you
check out this example if you haven't already: github.com/leptos-rs/leptos/tree/main/examples/tailwind
re: axum, I'm actually not sure as I've never actually done this. maybe check the Leptos docs and if it's not there, ask on their Discord server - Greg is very active on there answering questions!
trpc implment a similar pattern for api requests
Are you using leptos with stable or nightly rust ?
just took a quick look and it actually looks like I used nightly when I was making this video - that was a mistake, I had originally intended to use stable. Up until very recently, I believe Leptos required you to use nightly.
However, the code in the GitHub repository does use stable, so take a look at that if you want to avoid nightly.
@@codetothemoon If you read the leptos readme, it is explained that nightly permit to use a simpler syntax. But nightly should not be used in a long term project.
Thank you for your very instructive video. By curiosity, I have tried to generate it on Windows, but Visual studio 17 compiler fail with the C code of Rust former llm. I will try on my Linux computer or with WSL.
Very easy to understand! I still didn't understand the whole node_ref thing. Why is that necessary?
Noderefs allow you to pass a reference to a specific dom node around to say a function or closure, that way the closure can make changes and do things to the dom node and you don’t have to try to inline the statements as part of the dom
It's possible to use this lib to load personal documents and query them?
this is achievable but not without a bit of effort. the biggest obstacle there is the limited input size of the models, so in most cases you can't just prefix your prompts with the entirety of the documents. Many work around this by doing a semantic search on the document base to prune out the parts most relevant to the prompt, then prepend those excerpts to the prompt that is given to the language model. The problem with this approach is that there isn't a guarantee that the semantic search will surface all of the context necessary to accurately respond to the prompt.
I cannot compile the code on Windows 11 - both repository codes from the "video" branch and my hand-written variant gives the following error: error: linking with `link.exe` failed: exit code: 1120
Doom Emacs
much
Guys, why am I getting error: linking with cc failed exit status 1 when I run cargo leptos watch...
I hate my NixOS right now. I can't figure out how to get cargo-leptos to install. I'd probably already have it done with ubuntu or similar
I posted this. Right when I figured out the build. Hallelujah!
nice glad you got it sorted out!
@@codetothemoon gotta love a 45 minutes video turns into a 4 hour setup 😅. It's all good though. At least I learned how to get past it for next time.
@@codetothemoon Update. I never got it to work. I'm having a can't find cc error and it's a bit of a pain to figure out on NixOS to the point I needed to stop.
I was getting the model loaded and everything on the build I did. Unfortunately I couldn't get my version or your version. Working.
Yours less than mine.
Thanks for the video anyway. I'll have to come back later and try again sometime.
@@Script_Alchemist can you paste the exact error you're getting? or hop on the Discord and I can try to help there! discord.gg/2G35Sp6kpQ
why did you use the loop inj 37:11?
Is it possible to train this LLM to use some own database/documents?
Nice content! thks for share
this is kind of something I'm actively researching. Fine tuning using your own documents is an option, but I don't believe Rustformers supports this so you'd have to use a different library and I'm not sure how effective it is. The other option is the "few-shot" approach, where you provide your documents inline with your prompts - this is an easier and more effective approach as long as your documents fit in the context length of the model, but that can very quickly become limiting. To overcome the context length limitation, another option is to put your documents in a vector database and use semantic search to find the parts most relevant for a particular prompt, then feed just those pieces into the prompt that goes to the LLM. I think this is currently the most common pattern for this use case - check out something like NucliaDB (open source), Pinecone (not open source), or Weaviate (open source)
i'm a big fan of svelte, so poor to hear it slower than vue and leptos???
I'm a huge fan of Svelte as well! Incredible framework. I took a cursory glance at krausest.github.io/js-framework-benchmark/2023/table_chrome_114.0.5735.90.html and I'd personally describe the performance difference between it, Leptos and Vue as negligible at the moment. I definitely wouldn't switch from Svelte to Leptos for the performance - it'd be for the ability to use Rust on the frontend.
i think trpc is similar but not completely
yeah that was my impression as well - it seems like it aims to automatically manage the contract between the client and server, but just a quick glance at the examples led me to believe that it wasn't quite as clean as what Leptos offers. I may need to do a deeper dive to see whether that is actually true...
смотрю через перевод нейросетью, классно
I don't think you're going to have any FE perf issues for what your building haha "
An input field, button and some message box's
The bottleneck is the language model inference, which uses a non trivial amount of compute power. By “relatively fast” on the MBP, I’m talking about 10 full seconds or so
@@codetothemoon The LLM for sure! I just meant when you where talking about different FE framework pros / cons 🚀
Yeah this is a great video, but I also don't see the relevance of showing that FE performance chart here, That said, the video does go on to explain the other advantages to using a rust-based FE framework, including sharing type definitions across FE/BE, and that cool leptos feature where the FE can automatically (via macros, I think?) invoke a BE function to trigger an HTTP request.
@@mikejamesbelanger the main reason is that until recently, Rust frontend frameworks had performance profiles that were inferior to most of the popular JavaScript frameworks, so it was often cited as a good reason not to use Rust on the frontend. With these numbers it seems like that can't really be used as a reason anymore.
@@codetothemoon Oh I see. Fair enough.
Is rust good to learn vs go
Really depends heavily on both what you enjoy working on and your personal taste in programming languages
7:24 I think naming things is nobody’s forte 😅
hah thanks, glad I'm not the only one! 😎
Oooooooh what's this :)
A full stack rust chatbot!! 🎉
I prefer warp and Elm (elm-ui)
what code editor do you use. not like neovim, thanks
doom emacs!
@@codetothemoon thanks
use theme in video ?
doom-monokai-pro!
What font is that?
I think it was likely Fira Code, if not it would be Monaco. I made the switch from the latter to the former around the time of this video
error occurred: Failed to find tool. Is `gcc.exe` installed?
yikes - I don't have any experience building this on Windows, is that what you're on?
@@codetothemoon Yes
oo which font do you use!?
Monaco
@@780Chris thanks :D
nice video, but for me I'd like to keep the BE and FE apart, feels unnatural to write FE using rust but amazing video nonetheless and thank you!
thanks! hopefully it offered a new perspective even if it didn't turn out to be your thing!
Another opinion people (like myself) have is that it'll never be too early to see all of the JavaScript mess, language and tooling both, burn in hell 😂 so doing frontend using other languages (and Leptos especially thanks to its transparent function calling with client-server type guarantees) looks super-interesting 😀
Who designs a light mode app in 2023... Weird man... just weird.
lol, definitely one of my biggest regrets from this video. Feel free to submit a PR for dark mode - I'll approve 😎
@@codetothemoon hehe thx for the vid
Ain't nobody got 43 minutes to watch you type everything
Then don't watch.
You have the GitHub repo.
The most stereotypical RUclips comment. Pathetic.
@@elpolodiablo5486 i didn't
@@tylerweirtube i am sorry that i dont have 45 minutes to watch someone type every single command into the terminal. This video could be shortened into a 10 minutes.