Arducam time of flight camera for 3d segmentation and distance estimation
HTML-код
- Опубликовано: 4 окт 2024
- This project combines an Arducam Time of flight depth camera with an esp32-cam to do 3-dimensional segmentation and distance estimation. I've used a raspberry pi 4B to capture and process the images from the depth and the esp32-cam.
UPDATE: If you get an error with the camera (bad cast or similar), it's because of the index. So, for cam.open(ac.TOFConnect.CSI,0), try cam.open(ac.TOFConnect.CSI,2). Basically to get the index, you can use find_webcam_index defined below. This usually returns 0 or 2. Try "v4l2-ctl-list-devices" to check if "unicam" is there first. You camera might be called something else.
camIDx = int(find_webcam_index("unicam"))
and then try cam.open(ac.TOFConnect.CSI, camIDx)
where:
import subprocess
def find_webcam_index(device_name):
command = "v4l2-ctl --list-devices"
output = subprocess.check_output(command, shell=True, text=True)
devices = output.split('
')
for device in devices:
if device_name in device:
lines = device.split('
')
for line in lines:
if "video" in line:
parts = line.split()
for part in parts:
if part.startswith('/dev/video'):
print(part)
return (part[10:])
The Arducam depth camera can be bought at the below site (also available on Amazon).
www.arducam.co...
To setup the arducam camera follow the instruction here:
github.com/Ard...
Files are on Github:
github.com/jon...
Solution for esp32-cam feed delay on raspberry pi is here:
stackoverflow....
Hi thanks for the vid, when trying to run the depth code I get a "bad file descriptor" error does not contain v4l subdev. Any ideas?
Hi Jonathan love your video. I am trying to do the same set up with the jetson nano with a B01 board ( Which has two csi ports) and Raspberry Pi Cam V2. However after i install dependencies for ArduCam Tof it changes the csi preferences so i can not see the RaspberryPi cam. Hence, i can only run the ToF cam. But if i change the csi configuration with jetson-io to one IMX219 and one ArduCam, then none of the cams work. If you have any idea how can i deal with this situation I would appreciate it.
Hi. Sorry, I haven’t tried with the jetson. Sounds like a struggle.
@@jonathanr4242 Thank you for your time. Please keep up the good work, you are the best :)
hi jonathan, i love your video, can you explain to me about the object segmentation algorithm? can i use only the ToF Arducam to measure the object segment distance? thanks
Thanks Daniel. All the segmentation is done with the tof. It happens in two parts. First, the tof image is grouped based on grey level only, and then adjacent pixels with the same grey level are clustered into connected regions, which represent blobs that are the same distance from the camera, in answer to your second question, yes, you only need the tof to do the distance segmentation. The image camera is just for colourisation, so the object can be fed into machine learning or something for higher level processing and identification.
There are two issues. 1. The number of grey level clusters for the k-means algorithm needs to be determined. I’ve just used five. 2. Some regions, such as the ground, will not be a constant distance from the camera. But these surface regions should have a constant gradient, which can be added as a feature for clustering.
@@jonathanr4242 thanks jonathan, i appreciate your response!
please help mine gives me probe failed in dmesg!
Are power leads connected to 5v/gnd? try connecting without any peripherals on the pi and run in headless node. It kept failing for me when I had a touch monitor connected.
Hey Jonathan, Thanks for the video. Not sure if you went over this but how can one save save the distance value calculated from the TOF camera? I'm assuming the depth_ptr would be the right function just not sure how to call or read that in another python code
I'm not sure what you mean? I just save the depth information as a Gray level image. My code is all on the github repo referenced in the description.
@@jonathanr4242 I'm trying to save the numerical data gotten from the TOF camera (the distance) instead of the grey level image and was wondering if there's a way to do that.
@@EstherEberechukwuJoshua you should be able to hack it in the library. I’ve only used python. The raw values of the TOF will be times for the light to be reflected. For the 2m, the camera sends 75 million pulses per second. So the times will be a fraction of 1/75 millionth of a second.
did you use any lens?
No. I just used the cameras out of the box.
did u have problems while installing the python depencies ?
I had to use a virtual environment to install the arducam library. Also some of the dependencies might be obsolete. I don’t think I got all of them installed.
@@jonathanr4242 thank you for your quick response. got the problem solved by installing an old legacy version :). still got the same errors ehile installing the depencies but it works now. oke of the biggest problems is that pip3 install does not work on any new distros. you have to use sudo apt-get python3-packagename
to install something ans some other problems.
Hi Jonathan Glad to hear from you and hope you are doing well !! I sent mail if possible kindly look Thank you for your cooperation and assistance.
Please do the needful !!
@@NIRMITKUMARSHAH thank you, for your message. I’m very flattered that you asked, but I don’t know if I can manage time wise, although I’m curious to know the best way to detect the tread quality of tires.
how to open arducam 16mp camera with using python. I can only use the libcamera options but can't access it using opencv in python
You have to install the arducam package with pip install.
@@jonathanr4242 i looked up on the internet but i did not see any documentations related about that
@@jonathanr4242 how? there is no documentation related about that
@@LeviAckerman-gt6ll
see docs.arducam.com/Raspberry-Pi-Camera/Tof-camera/Getting-Started/
I think you need to run "pip install ArduCamDepthCamera"
can i use this module on Raspberry pi 5?
Should be able to. I was using bullseye. It should work with buster. As long as you can install opencv which should just be one line in buster or bullseye. pip install opencv-python, and then you also need to install the arducam tof libraries.
@@jonathanr4242 thank you so much! Just one more question, as I understand the camera module can be used directly on opencv projects without need another camera module right? I mean is it possible to use it on object detection with yolo? I'm just new this area so I wonder.
I'll try to use it on object detection and distance estimation. I wonder that if it can detect objects on a bit far distance? Camera has low megapixels that's why I'm asking
@@Baris_FK the Arducam time of flight camera has a maximum distance of around 4m.
@@jonathanr4242 yes I know it thanks. I wonder that how this camera is accurate on object detection? It has 240x180 resolution and it seems that it's a pretty low resolution. Does it really effect the accurate when you work with a trained model?
@@Baris_FK It will depend on how big the object you want to detect is. It should still give you a fair bit of room to work with. Increasing the resolution will impact your processing time. So, I probably wouldn't try go too high on an rpi.
in this video, which one do you use? bullseye or buster?
Bullseye
@@jonathanr4242 thanks to you jonathan, finally the code works, i'm only use the ToF camera so i'll try to figure it out 😁
hi again jonathan, i'm using Raspberry Pi 3B 4GB, i can't install the OpenCV, it keep failing over and over again, once i successfully install the openCV, the program (python) can't find the library, could you please give me a suggestion to solve this problem? and which OpenCV version do you use?
are you using bullseye?
i'm not using bullseye, i installed the newest version of Raspberry OS using Raspberry Pi Imager@@jonathanr4242
Hi Jonathan , I have to use Xbox Kinect Camera for Distance estimation . How can i modify your code , so that it will work for me. Thanks!
First thing is to bring kinect image into python so you can experiment with it
@@jonathanr4242Yeah thanks for the Reply , I have done that. But am not really getting , how to get distance from it.
@@vipulpardeshi2868 I've never used the kinect camera, but I would think there is a 2d array that can be accessed somehow where the gray level is the distance.
What is the range of the ToF camera?
It has 2m and 4m mode. You can set this in the code. Realistically, the cutoff will be a bit below this because you will need to threshold pixels to avoid measuring 4.05m as 0.05m say.