tl;dr - one docker container per code submission + language specific optimizations to reduce container startup time + hooking into the various test frameworks to get the test results.
i dont mean to be so offtopic but does any of you know a trick to get back into an instagram account..? I somehow forgot the password. I appreciate any assistance you can give me
@Remy Wesson Thanks so much for your reply. I found the site through google and im in the hacking process now. I see it takes a while so I will reply here later when my account password hopefully is recovered.
Could you also maybe make another video talking about what software process do you use now that that you’re working full time for AlgoExpert? Like if you guys have sprints going on? Do you use JIRA or Asana for tracking tasks, do you have a bug tracking tool? Do you do or plan to do retrospectives at regular intervals? I’d love to know your working culture :)
I have been waiting for you to make a video with Antoine, another wonderful person to make algoExpert a huge success. Thanks for the wonderful content, Clement ! ☺
Very informative. Not a lot of people disclose behind the scenes and I’m loving all the videos where you reveal the magic happening in the background. Thank you and keep going.
Clement as an engineer who has used and setup containers using kubernetes both for work and personal projects. This was really interesting to hear how you guys implemented this. Hands off to y’all. This is what separates y’all from the competition. You guys will wreck this new industry. If I were starting a start up like y’all. Creating this remote code engine would be so cool
I really liked the idea of touching the real-world implementation and associated optimization challenges of your beloved Algoexpert app, well done! I hope it (discussing the architecture and implementation) will remain as a major part of your videos in the future.
Glad you found the content useful; we've certainly had a lot of hiccups with this feature / service, but we've also learnt a ton from it and are really excited to continue improving on it.
I am making a competitive programming website myself and for code execution, I was previously using docker too but, the problem was when I needed to deploy the entire thing as one big docker container and, running docker inside docker wasn't that straight forward so I ditched it and went for apparmor. I can define strict apparmor security profiles and it takes care of the sandboxing itself and because it runs natively and not inside docker, it's faster too. Since it's for competitive programming, output is read from stdout and then piped to a C++ program that reads it and compares it with output file line by line. I haven't implemented support for approximate problems or problems where multiple answers can exist and you need some other program to verify that answer but it wouldn't be hard to integrate in my current design.
Amazing. If ever possible can you release tutorials on building the complex system like RCE and all or just give a guideline on how to learn all these.
how do you spin up docker containers that quickly? awesome video btw, was mainly intrigued how you can deploy a docker environment in under 2 seconds for every code execution. super interesting and is definitely a topic im interested in!
For someone who isn't new to programming, how many months of learning development does it take to be able to make something like a code executer with both the front end and backend components of it?
Need more of such backend content.. Please backend is so underrated just cause people dont see, can you please bring Antwon again to explain top frameworks/technologies to be backend developer .. thanks
Thank you for your clear and open explanation. This is very informative and it gives me ideas. By the way, have you thought about selling your compiling engine as a SaaS Api or something?
I'm very keen in getting an overview of how you guys reduced the cold boot time for each docker container's spin up process. That part sort of left off in the middle unanswered.
Great video Clement! Gave a lot of good insights on how the decision making for a platform should be and how we can constantly improve on it. QUESTION/FEATURE REQUEST (If you get to this comment) Currently, all the stdout for the failed tests goes at the very end of the raw output section. At least to me, it makes it a bit hard to debug the code since that stdout is not separated by test cases (Although, I know you suggest not to look at raw output). Is there a way, we can get the stdout inside the failed test case raw output? I love your videos, and I am always motivated to do more after each one.
Glad you enjoyed the video, and thanks for the feedback! We're planning on making a pretty major overhaul of the workspace (specifically, the output / debugging flow), so we're taking note of all this feedback!
Awsm video....but I believe there has to be better way then running docker container for every single code request ! Maybe specific server setups for each lang with memory constraints for each process or thread... Or going as wild as compiling on frontend 😁🤯😈
Hi! Great video about an interesting topic! One thing I'm curious about is how do you stop people from uploading malicious code as "solutions" that actually takes over the server that they are being evaluated on?
they are all executed in a Docker container, isolated environment. The malicious code can do whatever it wants there and it does not affect the server, they kill the docker container afterwards and are only extracting information from it, not running code at that point.
@@ryancastner6200 Thanks for the reply. I'm not an expert in this but I've heard that Docker can actually be relatively easy to escape. I was wondering if they were taking extra security precautions. Although I wouldn't blame them if they wouldn't want to talk about that publicly.
Good to see these discussions for which people usually have to attend weekend meets in their cities. Good that we can listen to it right from our couch.
Interesting approach using Docker, did you use some kind of API to send the code to Docker? Or did you compile docker images on the fly? Anyways, awesome approach and thanks for being open! Much better than that TechPro canned video question stuff.
I know this is a one year old video, but still is awesome how you guys shared your thoughts. I got a question though, how do you manage to run potential threats, malicious code?
I would have designed the backend either on a lambda framework like fission which is something similar to AWS lambda but can support any language, Or the other option for a backend is to keep JVM as the centre of the universe and implement most host languages supported by JVM, like Python, Ruby, Kotlin, JAVA, Clojure etc .... And since there are decent mechanism for Sandboxing on JVM like java-sandbox you can have tight control on security. Ofcourse RCE is nothing new and people in the past have used BSD jailed shells and even file permissions to do this. Jailed shell of course required a deeper knowledge of linux kernels.
It is pretty neat as in if you use http handlers for API and inside of it you simply do cmd := exec.Command("node index.js") with cmd.Start() you can set Stdout for the command you can Wait for output and select on timeout to cmd.Process.Kill() if need be to avoid timeouts. I might make a similar remote code execution thingy. I like it. Main question is do you spawn docker containers for each "execution" or you simply spawn X docker containers or pods that handle the API for languages and you simply call them when needed. My idea was that you use something like kubernetes and in k8s you have simply services like: rce-nodejs-svc, rce-java-svc, rce-python-svc and each service is just a thin go http handler that runs in pod as container and waits for the code to run. Once it get code to run it will spawn process with timeout and respond with parsed or unparsed Stdout. Probably with Java to avoid startup time you can use some hot standup java thing that will keep spawned in this container or a sidecar container JVM process and you just send code to compile it it avoiding startup time and send code to run on it. Not a java expect but i think i used something like this years back. Back in the days i used "nailgun" now i remember so you could kill of JVM time like this. It isn't supported since 2017 which is a bummer but same time it kills compile and run times.
I really wonder, why you did not use open source solution. Currently there are some open source project for remote code execution? Did you evaluate them or they did not exist at that time when you built your platform? you did not like them? would you consider developing from scratch of use these solution if you would begin now? Great video!
@@t_kon Again, one could fake HTTP requests with code from Postman to unnecessary increase the cost of creating these docker environments. How do you prevent that?
So you guys use docker to run each instance of the test? Do you have any optimizations that you preferred in speeding up the process? Because spinning up a docker for each test might be time intensive. But I would love your point of view about this 😊
It seems like allowing remote code execution would have interesting security implications. How did you handle those? Was using docker containers enough to deal with those issues?
7th language is TypeScript or Pascal or Elixir or Erlang or Visual Basic or Assembly maybe? Do you optimize using Dynamic programming? Same input same result.
I dont understand the last part when you say that you instrumentize the testing frameworks to make sure that your backend receives a reliable output. Woul you care to give an example please ?
I don't think they'll share that part. It is like their IP stuff which Antoine was about to share and Clement being a good business guy switched the topic
I'll say that the content was very informative, which I greatly appreciate. Just a suggestion: I'll suggest you let your co-founder speak more during a conversation. Apparently, you dominated the conversation, which for me inhibited the other person's point of view. Nevertheless, I'm super proud of you guys. Also, I don't understand why people keep criricizing Techlead and Joma. I know things went wrong, but we all need to understand that things go wrong in life. But what matters most is being at peace, and being able to correct wrongs into right. Please, Clement, advocate for this. Thanks a lot 😊.
I believe both parties contributed to the growth of each other, which I believe should be cherished and be strong enough to bring both parties back to one 💪. Coz both parties as a team really made RUclips a huge family for me. Thank you, and looking forward to the full team reuniting together for greater impact.
So, I'm not sure whether there is still runtime latency or not. If so(even if it is not), try to show some useful language/coding tips and tricks to users, while the code is getting executed. Ex: "Use maps instead of arrays in a situation like blah blah......, because it has O(1) fetch time". "Create an array with dynamic memory space, as the input size could vary and it could save extra space". "Use a binary search when you have to find an element, within an array, because it has O(logN) complexity, whereas linear search has O(n) time complexity". Hope your users like those.
Great videos guys! Just one question, have you thought running the JavaScript code directly into the user's browser? And if yes, what stopped you? As far as I understood it also uses docker.
Liliyan Krumov propably fetching test framework plus all the test cases for each task is less efficient than posting short task answere and makes frontend slower and heavier
If the architecture is already unified on the backend there isn't a reason to have a special case for JS on the client side. Extra code and edge cases for one language, aka technical debt, and you can't guarantee performance like you can on your own hardware.
Clément : "How we built Algoexpert "
Techlead : Time to copy it!
*TechLead:* How we built AlgoExpert (as a millionaire)
Tech lead and Joma is Googling what is container
minor embellishment if you may: "how to copy AlgoeExpert's RCE (as a millionaire)" lmaooo :D
tl;dr - one docker container per code submission + language specific optimizations to reduce container startup time + hooking into the various test frameworks to get the test results.
thx :)
It's good to see you have a very good bond with Antoine. Its pretty rare.
Do I smell gay
i dont mean to be so offtopic but does any of you know a trick to get back into an instagram account..?
I somehow forgot the password. I appreciate any assistance you can give me
@Bowie Samir instablaster ;)
@Remy Wesson Thanks so much for your reply. I found the site through google and im in the hacking process now.
I see it takes a while so I will reply here later when my account password hopefully is recovered.
@Remy Wesson It worked and I finally got access to my account again. Im so happy!
Thank you so much, you saved my account !
Love this kind of videos. You can learn a lot from real life scenarios. Thanks!
Really appreciate how in-depth you guys go when doing these videos :)
Could you also maybe make another video talking about what software process do you use now that that you’re working full time for AlgoExpert? Like if you guys have sprints going on? Do you use JIRA or Asana for tracking tasks, do you have a bug tracking tool? Do you do or plan to do retrospectives at regular intervals? I’d love to know your working culture :)
Nice to meet you Antoine!! Hope Clement didn't trouble you during the project
😂
I have been waiting for you to make a video with Antoine, another wonderful person to make algoExpert a huge success.
Thanks for the wonderful content, Clement ! ☺
Neha Gandhi He has made a video with him in the past
It's just so inspiring if not smile inducing just seeing Clement so passionate about this!
Very informative. Not a lot of people disclose behind the scenes and I’m loving all the videos where you reveal the magic happening in the background. Thank you and keep going.
Clement as an engineer who has used and setup containers using kubernetes both for work and personal projects. This was really interesting to hear how you guys implemented this. Hands off to y’all. This is what separates y’all from the competition. You guys will wreck this new industry. If I were starting a start up like y’all. Creating this remote code engine would be so cool
I really liked the idea of touching the real-world implementation and associated optimization challenges of your beloved Algoexpert app, well done!
I hope it (discussing the architecture and implementation) will remain as a major part of your videos in the future.
Great content , I learn so much from hearing you two talk about how you built AlgoExpert ... it exposes me to things I haven’t heard about
I'm glad!
I think it's really cool that you guys are explaining the structure and working of your product. Very cool!
Very useful content, this is a doubt I had about your system. Very brave of you to clearly explain your difficulties like that.
#fucktechlead
#fucktechlead
Glad you found the content useful; we've certainly had a lot of hiccups with this feature / service, but we've also learnt a ton from it and are really excited to continue improving on it.
@@clem please oh please make a video about the tech lead and his dark side, I know you'll make it tasteful. PLEASE! I'm begging!
Comparison is the thief of joy man. Techlead is a ex-google, ex-facebook, ex husband millionaire. Don't say the "f" word to him.
Why would you have doubt about it, you didn't implement it
I am making a competitive programming website myself and for code execution, I was previously using docker too but, the problem was when I needed to deploy the entire thing as one big docker container and, running docker inside docker wasn't that straight forward so I ditched it and went for apparmor. I can define strict apparmor security profiles and it takes care of the sandboxing itself and because it runs natively and not inside docker, it's faster too. Since it's for competitive programming, output is read from stdout and then piped to a C++ program that reads it and compares it with output file line by line. I haven't implemented support for approximate problems or problems where multiple answers can exist and you need some other program to verify that answer but it wouldn't be hard to integrate in my current design.
Amazing. If ever possible can you release tutorials on building the complex system like RCE and all or just give a guideline on how to learn all these.
I've been curious about this for a while, thanks!
how do you spin up docker containers that quickly? awesome video btw, was mainly intrigued how you can deploy a docker environment in under 2 seconds for every code execution. super interesting and is definitely a topic im interested in!
We can do it in python using docker-py lib . I am doing some POC on it. I am stuck in few places. Can you pls help me out?
@@yogendramaarisetty i have some way. Wont exec command run the containers quick?
Did you guys ever trial using serverless? E.g. AWS Lambda? It comes with built in timeouts and you could provide a Lambda layer for most dependencies.
Really informative video, love everything you do Clem!
Video starts at 0:00
(As always ❤️)
tf?
Super interesting how that works. Thank you for sharing!
Really waiting for System Design to be added to your website. It'll be a complete package !!!
Great video! real-world problem and real-world solution/optimization. I'm a website builder, and I really like website building contents like this.
For someone who isn't new to programming, how many months of learning development does it take to be able to make something like a code executer with both the front end and backend components of it?
Need more of such backend content.. Please backend is so underrated just cause people dont see, can you please bring Antwon again to explain top frameworks/technologies to be backend developer .. thanks
It's so painfully obvious that algoexpert is the real the deal, on algo*. Nice work!
Informative thanks, any content like that appreciated !
Thanks a lot for the explanation of this feature.
Thank you for your clear and open explanation. This is very informative and it gives me ideas.
By the way, have you thought about selling your compiling engine as a SaaS Api or something?
Very informative, thank you very much for the video!
Thanks for the video. Really did learn something from video like this!
lol, Techlead & Joma are running ads on your videos 😂
I'm very keen in getting an overview of how you guys reduced the cold boot time for each docker container's spin up process. That part sort of left off in the middle unanswered.
Thanks a lot!
I had asked for this video
Clement, have you and your co-founder considered applying to an accelerator like YC? You have a great product and an incredible founding members.
Great video Clement! Gave a lot of good insights on how the decision making for a platform should be and how we can constantly improve on it.
QUESTION/FEATURE REQUEST (If you get to this comment)
Currently, all the stdout for the failed tests goes at the very end of the raw output section. At least to me, it makes it a bit hard to debug the code since that stdout is not separated by test cases (Although, I know you suggest not to look at raw output). Is there a way, we can get the stdout inside the failed test case raw output?
I love your videos, and I am always motivated to do more after each one.
Glad you enjoyed the video, and thanks for the feedback! We're planning on making a pretty major overhaul of the workspace (specifically, the output / debugging flow), so we're taking note of all this feedback!
Extremely informative!
Amazing stuff . The platform is also awesome.
Awsm video....but I believe there has to be better way then running docker container for every single code request !
Maybe specific server setups for each lang with memory constraints for each process or thread...
Or going as wild as compiling on frontend 😁🤯😈
Favorite Tech couple....
real real world problems.. so helpful
Backend and Frontend of AlgoExpert!
Hi! Great video about an interesting topic! One thing I'm curious about is how do you stop people from uploading malicious code as "solutions" that actually takes over the server that they are being evaluated on?
they are all executed in a Docker container, isolated environment. The malicious code can do whatever it wants there and it does not affect the server, they kill the docker container afterwards and are only extracting information from it, not running code at that point.
@@ryancastner6200 Thanks for the reply. I'm not an expert in this but I've heard that Docker can actually be relatively easy to escape. I was wondering if they were taking extra security precautions. Although I wouldn't blame them if they wouldn't want to talk about that publicly.
Hey clement, just make a video on how to build a remote code execution.
Good to see these discussions for which people usually have to attend weekend meets in their cities. Good that we can listen to it right from our couch.
Very informative!
Interesting approach using Docker, did you use some kind of API to send the code to Docker? Or did you compile docker images on the fly? Anyways, awesome approach and thanks for being open! Much better than that TechPro canned video question stuff.
I know this is a one year old video, but still is awesome how you guys shared your thoughts.
I got a question though, how do you manage to run potential threats, malicious code?
I would have designed the backend either on a lambda framework like fission which is something similar to AWS lambda but can support any language, Or the other option for a backend is to keep JVM as the centre of the universe and implement most host languages supported by JVM, like Python, Ruby, Kotlin, JAVA, Clojure etc .... And since there are decent mechanism for Sandboxing on JVM like java-sandbox you can have tight control on security. Ofcourse RCE is nothing new and people in the past have used BSD jailed shells and even file permissions to do this. Jailed shell of course required a deeper knowledge of linux kernels.
Great, actually very informative.
It is pretty neat as in if you use http handlers for API and inside of it you simply do
cmd := exec.Command("node index.js") with
cmd.Start()
you can set Stdout for the command
you can Wait for output
and select on timeout to
cmd.Process.Kill() if need be to avoid timeouts.
I might make a similar remote code execution thingy. I like it.
Main question is do you spawn docker containers for each "execution" or you simply spawn X docker containers or pods that handle the API for languages and you simply call them when needed.
My idea was that you use something like kubernetes and in k8s you have simply
services like: rce-nodejs-svc, rce-java-svc, rce-python-svc and each service is just a thin go http handler that runs in pod as container and waits for the code to run. Once it get code to run it will spawn process with timeout and respond with parsed or unparsed Stdout.
Probably with Java to avoid startup time you can use some hot standup java thing that will keep spawned in this container or a sidecar container JVM process and you just send code to compile it it avoiding startup time and send code to run on it.
Not a java expect but i think i used something like this years back.
Back in the days i used "nailgun" now i remember so you could kill of JVM time like this. It isn't supported since 2017 which is a bummer but same time it kills compile and run times.
I really wonder, why you did not use open source solution. Currently there are some open source project for remote code execution? Did you evaluate them or they did not exist at that time when you built your platform? you did not like them? would you consider developing from scratch of use these solution if you would begin now?
Great video!
Super interesting video - keep it up!
Im running a bulk code submission for testing docker conts using kubernates but every process take me 0.9 mins and it's too much.
Hi are there any drawbacks with local code execution on client machine itself using ace code editor api?
So in the end which "standard unit testing library for Python" are you using?
How do you ensure that someone doesn't write code to delete all files on the server in one of the test scripts?
Docker contains and isolate the code to prevent those happening.
@@t_kon Again, one could fake HTTP requests with code from Postman to unnecessary increase the cost of creating these docker environments. How do you prevent that?
@@my3m Well they must have that in mind too and possibly wont share it. I mean it's the core business feature for them.
when is system design coming?
So you guys use docker to run each instance of the test? Do you have any optimizations that you preferred in speeding up the process? Because spinning up a docker for each test might be time intensive. But I would love your point of view about this 😊
It seems like allowing remote code execution would have interesting security implications. How did you handle those? Was using docker containers enough to deal with those issues?
What does your backend infrastructure / architecture look like? You guys on AWS, Azure, etc? Great video though, love Docker!
very useful content. Can you tell me more about how you use docker to isolate your containers ?.
Are you using web sockets so your not making fetch calls constantly?
7th language is TypeScript or Pascal or Elixir or Erlang or Visual Basic or Assembly maybe? Do you optimize using Dynamic programming? Same input same result.
you guys could host programming competitions on algoexpert platform. target - schools or companies around the world
Have you looked into using protocol buffers and gRPC for sending data?
What about the security aspect of it? Use any seccomp profile with the containers?
When is data structures coming?!
Santa might come as early as the end of the week! (Oh crap...did I just publicly lock us into a deadline?)
@@clem lol, it's definatly a deadline now
@@clem I have an interview this Friday, I'd love to see it before then.
@@clem My google interview is in 3 weeks, so that could be ideal!
@@chrischristopher7102 Seems like he deleted the comment that said "It was coming this Saturday". Not sure why Clement deleted the comment :)
Awesome
I dont understand the last part when you say that you instrumentize the testing frameworks to make sure that your backend receives a reliable output. Woul you care to give an example please ?
just curious how did you meet with each other?
how did you optimized the new container creation and deletion on every request with the code execution inside the container?
It would be a good intro to Systems Design series )
how did you speed up rolling out new container for every code execution?
Hey can you briefly explain how you guys optimised the docker thing I am interested to know how used docker in your website
Really interesting!
Hi Clement make a video on how the Ds Like heaps, linked list Graphs ... Used in your product development.
But how can two different process effect each other if running in a same machine, considering our home computer which run multiple process perfectly?
Can you explain more of the docker optimizations, i feel like he was going to explain a little more on it, but then got sidetracked
I don't think they'll share that part. It is like their IP stuff which Antoine was about to share and Clement being a good business guy switched the topic
awesome video....please correct me if iam wrong, u are on high level using docker images of all the programming languages to run the code?
What is the backend written in?
I'll say that the content was very informative, which I greatly appreciate.
Just a suggestion: I'll suggest you let your co-founder speak more during a conversation. Apparently, you dominated the conversation, which for me inhibited the other person's point of view.
Nevertheless, I'm super proud of you guys.
Also, I don't understand why people keep criricizing Techlead and Joma. I know things went wrong, but we all need to understand that things go wrong in life. But what matters most is being at peace, and being able to correct wrongs into right. Please, Clement, advocate for this.
Thanks a lot 😊.
I believe both parties contributed to the growth of each other, which I believe should be cherished and be strong enough to bring both parties back to one 💪. Coz both parties as a team really made RUclips a huge family for me.
Thank you, and looking forward to the full team reuniting together for greater impact.
Did you guys ever look at other code execution engines like piston
So, I'm not sure whether there is still runtime latency or not. If so(even if it is not), try to show some useful language/coding tips and tricks to users, while the code is getting executed. Ex: "Use maps instead of arrays in a situation like blah blah......, because it has O(1) fetch time". "Create an array with dynamic memory space, as the input size could vary and it could save extra space". "Use a binary search when you have to find an element, within an array, because it has O(logN) complexity, whereas linear search has O(n) time complexity".
Hope your users like those.
Is spinning up the docket container is that much fast?
Is there anything you had to do in order to protect against malicious code that users might enter? Thanks.
Great videos guys! Just one question, have you thought running the JavaScript code directly into the user's browser? And if yes, what stopped you? As far as I understood it also uses docker.
Liliyan Krumov propably fetching test framework plus all the test cases for each task is less efficient than posting short task answere and makes frontend slower and heavier
Scala or Ruby?
still can't select and copy code from your site. is that a feature rather than a bug?
This shows that you can do magic even if you don't do card tricks. Great job both of you 😊
Will PHP be the next language on algoexpert?
Beautifully said 😛, and PHP is the next language after the next one!
@@clem That means I'm better at long term forecasting than I am at short term forecasting. Weird. 😁
Can you tell about how you use docker for your project?
What language and framework was used for the backend?
This is awesome content!
Javascript is available on almost all the client web browsers, will you even send that to the server?
If the architecture is already unified on the backend there isn't a reason to have a special case for JS on the client side. Extra code and edge cases for one language, aka technical debt, and you can't guarantee performance like you can on your own hardware.
What benchmark cases do you use to test alto expert?
When will data structures instruction on algoexpert come out?
Clement hope you make some tutorials on your channel specially front end dev and python
How does it compare with sphere engine