Interactive projection mapping done with Unity 3d, Leap Motion and Arduino / Uniduino / Adafruit
HTML-код
- Опубликовано: 5 авг 2024
- After my prerendered automotive #projection #mapping • Projection mapping tes... I wanted to do an #interactive one. So here is a prototype I built in my spare time to learn more about gesture recognition with the #leapmotion and #unity3d in general, programmed in C# using an #arduino / #uniduino board with an #adafruit motorshield to drive the cube / tesseract.
This personal project was an interesting diversion from my normal job as freelance interface and motion designer. But I have always liked to do things hands on anyways which is also why you can find my #DIY teleprompter here: pixelasm/pho...
Compositing and code by:
www.pixelasm.com
/ pixelasm
/ ralfbreninek
3D models from all over:
www.assetstore.unity3d.com
Thanks to:
Ilya Suzdalnitski for the portal shader which works like masks in Photoshop and After Effects
Timestamps:
00:00 Intro
01:52 Projection + interaction
03:03 Hardware setup
04:40 Unity 3D project
06:39 Conclusion 
Кино
This is truly inspiring. Working on a different but quite similar project right now. Your approach is really good. Thank you for sharing this information with the world.
Thanks for the feedback :-) Hope you can share what you are working on too.
Bis dahin viel Erfolg bei deinem Projekt (beruflich oder auch semi privat?)
This is awesome! You've just gained a subscriber. Your projects are very inspiring.
thanks for subbing :)
hi, you are the one man, your workflow is amazing ...
Thanks, looks like you are looking into video mapping as well. Nice Idea projecting onto a bride and a wedding cake, although I wonder how many couples got enough cash to contract a motion designer for 1-5 days.
Wow, thanks for share your project. Very nice!!! congrats!!
Thanks for the appreciation :)
You're my new role model.
@eliottheowa I am glad to inspire you and hope you will be able to use that inspiration to make your next steps closer to your goals :)
Very nicely done!
thanks :)
amazing! Will try to create something similar :)
Very nice work!
thanks :)
it is great work.
yo! some hypnotic voice you got. cool
LOL thanks I guess :D
Awesome, Ralf !!
Thanks +Emmanuel Henné :)
Amazing work! This is really great and inspirational!
Is the portal script available for public? I would love to know how that effect was achieved :)
AMAZING!
Thanks :)
why do i feel like im watching a asmr video
LOL, seriously? That was not intended. I am using a simple blue yeti to record. Are you wearing headphones?
lol it just kind of sounds like it
maybe too close to microphone and too much bass boost. sounds weird and artificial
pixelasm same here. thought asmr from first 2 seconds....also why so serious man. smile a little :) also change your tone and pitch a little up and down. your video and I appreciate your effort. so pls take this as constructive criticism
Thank you, nice project! I suppose young girls love it :)
Hi! It looks great, u did an awesome job! I was wondering how did u project the unity scene onto the cube? Did u maybe use Spout?
Thanks :) I used the game window of #Unity3D and moved it to the beamer, which I have setup as secondary monitor. So no dedicated mapping software, just a straight forward setup with the tools I have at hand.
Feel free to like and subscribe if you want further videos and backgroud infos ;-)
Can you exmplain a bit more.. how to managed to sync the movement of the cube with the mapped projection and Unity ?
That was the only reason I was watching the video. Sadly he did not mention it
thank you very much! I was looking for exactly that: a projection of the inside of the cube. Do you happen to know how this type of projection is called in professional terms?
I have not heard about a professional term for this kind of projection maybe also because I have not seen anything like it before :P Although this effect is seen in CGI quiet often due to a "bug" called backface culling but I have never before seen it used in conjunction with an interactive projection mapping.
Scientifically it is a tesseract or hyper cube as I mentioned but those terms you also can not find much about.
Wow nice work, may I know where you get the shader? Any link to the fie?
Kind sir, where can we find the shader? Is it on the asset store?
@pixelasm can you please share details with me how it was made source code,requirements etc i want to work and add something in it
I'm new to the whole visual scene recently as I started handling LED screen projection for events. I'm trying to learn more about projection mapping and live interaction to use as a personal art project in my hometown.
I saw some examples in Singapore where they used an Xbox Kinect to detect motion and have the visual interact with them, I wonder how they did it.
There is an asset for kinect in the asset store which gives you access to the data recorded by it. You can then for example attach things to the joints if the detected person. Be aware that you need C# knowledge to use this.
By the way I have used said Unity Asset for this test: ruclips.net/video/GYXTUf6UBUo/видео.html
what Unity are you using? I am having trouble using Uniduino with Unity 5. It connects and recognises the firmata protocol but does not display any IO ports nor does it recognise the correct COM ports.
If I do recall correctly I used 5.1 or 5.2 to build this. I only followed their tutorials to get everything connected and it worked seamless: www.uniduino.com/tutorials/
Can you provide me with the c# script you used here?
Awesome work! how did you go about syncing up the rotation of the motor to the rotation of the gameobject in unity?? I've been able to connect a servo motor to unity but with the uduino asset there is a delay
The trick is to use a stepper motor, an easy to miss detail I mentioned in the video. Steppermotors different to simple servos only move a predefined ammount of degrees per step. Therefore I used the same stepping for the digital model and had a really good match, although I had to tweak a bit. Ideally you would have another system checking the position independantly, for which some servo motors provide possibilities, but when setting everything up and buying the parts I wanted to keep everything a bit simpler and went for the approach shown here.
@@pixelasm Ah thank you! Was it hard to find documentation about controlling stepper motor from unity?
@@masayatodoroki339 I think it was ok.
It's wonderful challenge! I want to know how did you project the unity 3d model images on to each surface of the white cube? If I do this, I would map the surfaces of the unity model onto the surfaces of the white cube. But the white cube can revolve...It can't do this without a dynamic projection mapping method. But you didn't use any 3d scanner devices...?
Honestly finding answers is the most interesting thing to me. This is also what I find interesting about my professional and private work and the reason why I do it in the first place. Therefore I would like to "answer" your question with a hint: What is the difference between a DC motor and a STEPPER motor? The answer to this will give you the answer to how I "tracked" the cubes position.
Thank you for your consideration. After seeing your video I did a simple experiment goo.gl/photos/dS4zcwqp3yu949j5A. Your hint using a STEPPER motor indicates me everything I want to know.
Great to hear, looks like you are almost there too :)
Looking forward to your final result. Please keep me updated.
nice project man!! ,
i am still a noob and i need to know how do you send data from unity or leap motion to the arduino , i am trying to control my 6DOF Stewart platform using leap motion , do you have any tips from your personal experience
Thanks! Easiest way is to buy the uniduino asset, it also provides examples on how to comunicate with the pins.
TIP: You should also buy motorshields because turning the system on and off might have some backlash on the arduino board when used without a motorshield.
6DOF Stewart platform is a sick piece of hardware. Would be nice to hear / see what you are doing with it. Are you planing something different than a racing simulator?
i have no idea whats is uniduino asset,
is it a new chip like arduino ? or just extra libraries for arduino ?
i just want to let the stewart platform copy the movements of my hand that is being read by the leap motion sensor , so i just want to extract the XYZ coordinates from the leap motion and send it as serial data to the arduino
Oh! I suppose you would like to use the leap mption without Unity, right?
My data flow is leap motion -> unity -> arduino -> adafruit motorshield -> stepper motor and you would like to leave out unity, right? Thats a workflow I haven´t looked into because I wanted to use Unity for the visuals right from the start, sorry.
i am using mini servo motors i dont think i need the motor shields , i really dont know if i need the unity or not , but i read online there is a way to integrate the leap motion with arduino through unity ...
Now you playing 4d chess!!!!
cool
danke :)
Hast Du für das Projectionmapping eine Kamerakalibrierung wie Mapamok (openCV) in Unity genutzt, oder das "von Hand" angepasst ? Habe letzte Woche noch an einer Steppersteuerung über Raspberry Pi und Unity gearbeitet :-) Mit Projectionmappings arbeite ich auch super oft, bin aber bisher nie auf die Idee gekommen Unity dafür zu nutzen. Bisher waren das immer D3, Coolux oder wen es echt schnell gehen muss, mit Aftereffects :-)
Ja habs per Hand gemacht :P
Wollte mir zwar auch immer schon "echte" projectionmapping software anschauen hab mir dann aber gedacht: Warum schlägst du nicht drei Fliegen mit einer Klappe und lernst mehr über Projectionmapping, Unity und C# in einem.
Hey pixelasm,
für meine Thesis suche ich eine Möglichkeit UI-Elemente in Unity, per Projektor und LeapMotion auf meine Hand zu projizieren.
Die LeapMotion liefert die Tiefeninformationen und die Bilder, der Projektor projiziert als Ausgabegerät die Grafiken auf die Positionen der Hand.
Wissen Sie wie das funktionieren könnte?
Danke vorab :)
Wenn ich mich recht entsinne dann lädt LeapMotion ein virtuelles Hand Modell mit Bones in die Szene sobald eine Hand erkannt wird. An die Bones könnte man dann andere Objekte tracken und offsetten. Im schlimmsten fall könnte man in der update funktion permanent abfragen ob ein bestimmtes Null in der Szene existiert und dessen Koordination auf ein null mit den gefragten Inhalten übertragen.
ACHTUNG: Das ist so nicht performant.
Wenn ich mich recht entsinne liefert Leap Motion verschiedene Hand Modell prefabs mit die man auch editieren kann, ggf. könnte man die gewünschten Inhalte in das Prefab integrieren.
Könnte aber sein das ich das mit der API der Kinect verwechsle ist schon einpaar Jahre her ...
@@pixelasm Hallo pixelasm,
danke für die schnelle Antwort.
Meine Idee war mit OpenCV und der LeapMotion meine Handposition im 3d-Raum zu tracken und davon 2d-bilder zu machen. Die Bilder (die ein eigenes Koordinatensystem aus dem 3d-raum besitzen) transferiere ich in das Unity-Koordinatensystem und füge die UI-Elemente an die Koordinaten an, wo ich sie haben möchte. Anschließend spreche ich (leider weiß ich nicht wie das geht) den Projektor an und sage, projiziere dieses Bild.
Haben Sie das so ähnlich gemacht oder wissen Sie noch wie Sie den Projektor ansprechen konnten aus Unity raus?
@@eleonora7490 ich hatte den Projektor als zweiten Monitor an den Rechner angeschlossen und über Unity den Stepper Motor angesteuert der den echten Kubus dreht. Im gleichen Verhältnis habe ich dann auch den Kubus in der Unity Szene gedreht, so daß das projezierte Bild mit dem Kubus deckungsgleich bleibt.
Der Aufwändigere Teil war also eher das Ausrichten der virtuellen Kamera in der Szene, so daß die Projektion mit dem Kubus passte. Das Ausrichten alleine hat mich glaube ich einen Tag gekostet. Ähnlich müsste es aber auch mit der Projektion auf eine Hand klappen.
@@pixelasm danke für die schnelle Hilfe.
Wissen Sie als Experte wo ich gute Instruktionen zum Mapping finden kann? Meine Recherchen zum Thema projection mapping oder spatial mapping führen mich nur zu Projekten die keinerlei Tutorials beinhalten. Leider bin ich auch kein Mathegenie um solch komplexe Algorithmen zu entwerfen. Ihr Projekt ist das greifbarste Exemplar was ich finden konnte :/
Hi, Amazing job Sir,
I want know to Arduino model are use this project.
I used an arduino uno with an adafruit motorshield
Which projector is that one? could you please share model
Chop Labalagun it is an Epson TW 620 which is able to project 720p at max.
Can you recommend a projector for this type of test on a really low price? Or what should I look on a projector for projection mapping. I am using a LEAD pretty bad. Look at my latest video.
Chop Labalagun unfortunately I can not point out a specific one, just get the one with the highest lumen. Keep in mind though that those might also be the loudest. Aside from that I would prefere one with an optical system for trapez and other skewing instead of doing it digitally but that is just a minor thing.
Thanks for inputs.
bro chill with that low whisper ur making my boy parts tingle hehehehhe
LOL wonder why I get so many comments on my voice as I have not used my deep morning voice :D Aside from that I will do a more freestyle video next time, reading text sounds to stiff :-/
lol jokes but can u leave a lil noe of the software and hardware list so i may purchase and add to this project
Sure, but I need to look for all the receipts, because I bought some of the hardware over the course of several months. This might take some time ...
Kayak orwng indonesia anjir
set vid speed to 1.5 folks, this guys stuck in reverse!
gobble gobble gobble blah blah arduino so slow dude!!!
bob me