Thanks for the demo and walkthrough of your flows. I was struggling with applying a simple flow to handle the csv log files and I am thankful that your video has helped me move forward.
Thank you for the tutorial, I had deployed and is working in my project. Just a few adds: folder permissions, share file and sent emails. 100 points to your video. TY again!
Super super super... Thank you. U r the best ... Really learn a lot from yr fantastic idea of sharing node flow and explain in details. U r doing more than a good job. Is there a possibility to teach the same thing apply to the video file or snapshot capture image file to a NAS backup and also able to watch the video or image snapshot under the column box, just like what u did to show a CSV graph. Can't wait to see that happen.
@@csongorvarga With refer to your another most fantastic ever video tutorial ruclips.net/video/ihZWrJmbGFY/видео.html In which, it does not play video clip on the Nodered UI but snapshot only with the previous image only. This new video tutorial given me an idea that it is good to have such file retriever for images and video, save as backup and image&video quick view. U r too brilliant...
How can I add more columns to the csv file? I am trying by adding more attributes to the "set data" function node, but nothing happens... Thank you very much Csongor, great great tutorial. Very useful idea.
Check the two CSV node as well. On both of them you have to list the columns you want to save. Those names have to match with the attribute names you add to the msg.payload in Set Data.
Thank for the excellent explanation for this tutorial. May I know any possible to create the file every 30minutes? Please enlighten me. Thank you in advance.
Could you kindly precise at which part/point of your video that I need let say instead of logging the data for every 10 secs, I'd like for it to log for every 10 minutes instead for example. Thank you!
Hi , i am using a windows machine. But getting en error : No filename specified" Anybody an idea ? (c:/datalogger/???) //Generate out file name pattern msg.fname = "battery_"+ yyyy + mm + dd + ".csv"; //Fullname filename with path for the file msg.filename = "c:/datalogger/" + msg.fname;
@@csongorvarga Yep but the way you use it is amazing! Im new with node red so im still figuring things out. Just found your video for "Browsing files in dashboard". Thanks bro love your work! 👍
@@csongorvarga Yeah but i can say its been really enjoyable learning it. Just a question if you have the time, im currently trying to use the date picker to select a range of dates (for example between 24th November to 28th), then merge the csv files of those dates and display it on a chart. If you have the time i would appreciate some tips or references that would help😁
Also have this problem, file upload block give out this error message: /home/pi/log_upload.sh: No such file or directory". Author, please help! Thank You!
first of all; Great Video !! I'm sure this will do a world of good for a bunch of users out there (me included :-) One question though; are all three variables being sent from the temp/humidity sensor at once or are they coming in sequentially? If my sensor is sending first temperature then humidity then pressure (three separate commands) do I need to change the code so that it sends the values all at once in order to have them fed into the csv file as one line?? thanks.
Yes, you are right. What I would do that when the temperature and the humidity comes in, store them in flow variables. And when the pressure comes, put the pressure into the data structure along with the temp, humidity which is stored in flow variables. Technically the readings are going to be still slightly off, but I assume these measurements come in close to each other.
Hi, thank you some much for this awesome video. I just have a few questions, would be awesome if you could help me: - Minute 2:00 "Set Data": Did you change the msg.msg433.TEMP names in the Node-Red mqtt instead of the standard msg.payload or is this the name you used in the arduino sketch for the string? - can i connect more the one mqtt in with one data set button? - do you know how i can change the date and time at my rapsi without internet? just like a command? Thank you some much in advance, Sarah
Thanks. My source was not MQTT and the data came this way. But if you are getting the data via MQTT, it can simply be just msg.payload. It really depends on what you get via MQTT. If your device sends multiple values in a single MQTT topic, for example in JSON format, you can use the JSON node to convert the JSON to a Javascript object, and you can reference to the various fields like msg.payload.temp, or msg.payload.humidity (depending on how your data looks like). But if it comes through in several MQTT topcis, it is different. In that case need to store the data in flow variable (e.g. send them to a change node where you set "Set" "flow." "something") to each of them, except one. And send one of the MQTT to Send Data, and in the code "temperature": msg.payload; so this mqtt will be stored in the temperature column, and on other lines "humidity": flow.get("something"); so that will fetch the last stored value in the "something" flow variable and store it in the humidity column. And so on as many as you need. Setting the time on the raspi, first make sure the timezone is set correctly (sudo raspi-config and it is under Internationalization options) and after you can use the date command like sudo date -s "2016-10-16 21:00". I never set time manually myself, but Google says this is the way to do it.
@@csongorvarga thank you so much for the quick answer! date changing worked perfect. but the other problem i can´t solve.. don´t know what i am doing wrong.
@@csongorvarga I changed the Set to and i fixed it with just separating everything. But now I want to have the real time and seconds in my csv. Do you have an idea how to change timestamp into it in the "set to" node? Thanks in regard!
This is very useful! I have been in doubt about how to store data from different devices in different .CSV files. Like in a scalable project, which is the smartest way to have the same flow for different devices?
@@csongorvarga I saw you have two devices on the turorial. They are publishing to two different topics right? each topic with it's subtopics (temp, Voltage). What it's still not clear it's how the flow knows in which CSV to write the data, maybe looking at the device ID? I will have 5 different Wheater Stations for a personal project and this flow is like perfect for me, except i still have doubt on how to handle the MQTT messages, like, is one single flow enough to handle different devices or i must have a flow for each device. Anyhow, i'm so happy to have found your Channel, it's probably the best Node-Red content in RUclips for sure.
@@exprymer If you have a lot of different topics, probably the best to do, that you save each payload to a different flow or global variable. That will always update and keep the latest value in those variables. And in the "Set data" function, you can recall the data from those flow variables. Like msg.payload = { "timestamp": now.getTime(), "temperature": flow.get("temp"), "humidity": flow.get("humidity") }. I did not have to do that because all the data I wanted to save was in a single object in one payload. So I could extract all of them at the same time.
Hi, really nice and useful video. If I wanted to generate dynamic names for my files, do you know how to only search for the date and not for the whole name to delete the older files? Many thanks
It is not part of the default installation. Click on the hamburger menu on the right, Manage Palette, on the Install tab, enter node-red-contrib-fs and install the new node. Once it is installed you can import the flow without error.
hello, have you ever tried to do a daily csv file using windows? I review your video and I am doing exactly the same configurarion, but I have an error in the list node, it says "node users list is not set". Any idea??
I am not sure about the username and password part. I don't know how it works under linux, in Windows you can maybe map it as a new driver with the user credentials, so anyone accessing the drive will not need to enter user and password again. Maybe something similar exists in Linux as well. The file name is getting generated in the function node. You can easily add a flow variable there. I show in the video where and how the file name is getting generated.
In one column? In Excel? Is it possible that your have a different field separator set in your windows settings as in the CSV file? How the CSV file looks like when you open it in notepad?
Hi, very good Tutorial! But I still have one question: My Sensor Data (from Electricity Meter) arrive every Minute, so the .csv file is written every minute, also. Is it possible to avoid this frequent writing in order to protect the SD card? Thank you!
Sure. You can save the readings in a flow variable. And have a separate inject that triggers let's hourly and get the latest reading from the flow variable.
@@csongorvarga well, I added a "join" node between "set data" and "filename generator". The join node is set to "Array" and generates an output every 60 inputs. This does the trick so far. Need to verify the data but looks good at the Moment...
Hello Csongor Varga! Do you have a video explaining how to access the RS232 serial ports of a Serial Device Server. For example, ZLAN5843A (8 ports RS232 / 485/422 To TCP / IP converter)?
I want to send my data to the following server: Host: 200.20.53.x/ external: 200.20.53.x Port: 21 Login: rsolutions Password: rsftpinea How could I configure this example?
Probably the easiest way to do is to log the data in your own server and use the exec node to start a shell script which FTPs the data onto your remote server.
Best videos on NR online. I have learned so much from You. Thank You.
Thank you. This video opened my eyes about what Node Red can do.
Thanks a lot for sharing your flow, it's always great to reflect on someone else's approach and it saved me a bunch of time. 🙌
Thanks to share this helpfull tutorial. I gonna use this to logging my weatherstation.
Thanks for the demo and walkthrough of your flows. I was struggling with applying a simple flow to handle the csv log files and I am thankful that your video has helped me move forward.
Thank you for the tutorial, I had deployed and is working in my project. Just a few adds: folder permissions, share file and sent emails. 100 points to your video. TY again!
Once again a great tutorial!
This was super helpful, thank you!
Keep up the good work. Subscribed!
Very useful. Thanks for all your videos. Grettings from Argentina!!
Great video, very well explained. Thanks.
Amazing tutorial! Worked great and now running in my project. Thanks
Super super super... Thank you. U r the best ... Really learn a lot from yr fantastic idea of sharing node flow and explain in details. U r doing more than a good job. Is there a possibility to teach the same thing apply to the video file or snapshot capture image file to a NAS backup and also able to watch the video or image snapshot under the column box, just like what u did to show a CSV graph. Can't wait to see that happen.
Thanks, what do you mean by snapshot a video file?
@@csongorvarga With refer to your another most fantastic ever video tutorial ruclips.net/video/ihZWrJmbGFY/видео.html
In which, it does not play video clip on the Nodered UI but snapshot only with the previous image only. This new video tutorial given me an idea that it is good to have such file retriever for images and video, save as backup and image&video quick view. U r too brilliant...
Appreciated . . .
Thanks very much for your great job
thank you thank you thank~~~ you very much
Can you please explain how CSV files gets downloaded when you click on the file name in UI? Which node / function you are using for this?
Great video and I have implemented the concept. can there be a timer function added? only store at the whole hr?
How can I add more columns to the csv file? I am trying by adding more attributes to the "set data" function node, but nothing happens... Thank you very much Csongor, great great tutorial. Very useful idea.
Check the two CSV node as well. On both of them you have to list the columns you want to save. Those names have to match with the attribute names you add to the msg.payload in Set Data.
Thanks for the demo, i've tried your flow by changing the path to store the file in windows desktop but it dsnt work for me, any suggestion ?
Thank for the excellent explanation for this tutorial. May I know any possible to create the file every 30minutes? Please enlighten me. Thank you in advance.
Could you kindly precise at which part/point of your video that I need let say instead of logging the data for every 10 secs, I'd like for it to log for every 10 minutes instead for example. Thank you!
Hi , i am using a windows machine.
But getting en error : No filename specified"
Anybody an idea ? (c:/datalogger/???)
//Generate out file name pattern
msg.fname = "battery_"+ yyyy + mm + dd + ".csv";
//Fullname filename with path for the file
msg.filename = "c:/datalogger/" + msg.fname;
Love the video! Very helpful with a very important function!! Can i ask, how did u do ur ui? If possible i would love to see that part of your flow😁
What do you mean. This is Node-Red and the Node-Red Dashboard. I just used the features available there.
@@csongorvarga Yep but the way you use it is amazing! Im new with node red so im still figuring things out. Just found your video for "Browsing files in dashboard". Thanks bro love your work! 👍
@@ahmadhafidzmuhdmawardi5743 Yes, it needs a lot of thinking, because you need to design the UI differently in order to accommodate the framework.
@@csongorvarga Yeah but i can say its been really enjoyable learning it. Just a question if you have the time, im currently trying to use the date picker to select a range of dates (for example between 24th November to 28th), then merge the csv files of those dates and display it on a chart. If you have the time i would appreciate some tips or references that would help😁
@Csongor Varga I dont manage to get the log_upload.sh to work and give exec error 127. What ftp server you install?
Also have this problem, file upload block give out this error message: /home/pi/log_upload.sh: No such file or directory". Author, please help! Thank You!
first of all; Great Video !! I'm sure this will do a world of good for a bunch of users out there (me included :-) One question though; are all three variables being sent from the temp/humidity sensor at once or are they coming in sequentially? If my sensor is sending first temperature then humidity then pressure (three separate commands) do I need to change the code so that it sends the values all at once in order to have them fed into the csv file as one line?? thanks.
Yes, you are right. What I would do that when the temperature and the humidity comes in, store them in flow variables. And when the pressure comes, put the pressure into the data structure along with the temp, humidity which is stored in flow variables. Technically the readings are going to be still slightly off, but I assume these measurements come in close to each other.
Hi, thank you some much for this awesome video. I just have a few questions, would be awesome if you could help me:
- Minute 2:00 "Set Data": Did you change the msg.msg433.TEMP names in the Node-Red mqtt instead of the standard msg.payload or is this the name you used in the arduino sketch for the string?
- can i connect more the one mqtt in with one data set button?
- do you know how i can change the date and time at my rapsi without internet? just like a command?
Thank you some much in advance, Sarah
Thanks. My source was not MQTT and the data came this way. But if you are getting the data via MQTT, it can simply be just msg.payload. It really depends on what you get via MQTT. If your device sends multiple values in a single MQTT topic, for example in JSON format, you can use the JSON node to convert the JSON to a Javascript object, and you can reference to the various fields like msg.payload.temp, or msg.payload.humidity (depending on how your data looks like). But if it comes through in several MQTT topcis, it is different. In that case need to store the data in flow variable (e.g. send them to a change node where you set "Set" "flow." "something") to each of them, except one. And send one of the MQTT to Send Data, and in the code "temperature": msg.payload; so this mqtt will be stored in the temperature column, and on other lines "humidity": flow.get("something"); so that will fetch the last stored value in the "something" flow variable and store it in the humidity column. And so on as many as you need.
Setting the time on the raspi, first make sure the timezone is set correctly (sudo raspi-config and it is under Internationalization options) and after you can use the date command like sudo date -s "2016-10-16 21:00". I never set time manually myself, but Google says this is the way to do it.
@@csongorvarga thank you so much for the quick answer! date changing worked perfect. but the other problem i can´t solve.. don´t know what i am doing wrong.
@@lovelyKINDofGIRL Which one is the other problem? What did you change so far?
@@csongorvarga I changed the Set to and i fixed it with just separating everything. But now I want to have the real time and seconds in my csv. Do you have an idea how to change timestamp into it in the "set to" node? Thanks in regard!
@@lovelyKINDofGIRL What do you mean real time?
beautiful guide, works perfectly. but how can I transform the "timestamps" into a readable timetable into excel? A thousand thanks
This is the formula to convert unix timestamp to Excel date: (A1/86400)+DATE(1970,1,1)
@@csongorvarga thank you very much for the very fast reply. I had found this formula too, but it doesn't work and I can't understand why
@@emanuelegiuliani2385 Oh yes, because it is =(A2/86400000)+DATE(1970,1,1). I just tried myself. Unix timestamp is milliseconds and not just seconds.
@@csongorvarga it work with =(A2/86400000) + DATA(1970;1;1) (excel in ita and ; ) thank you so much!
fs-file-lister where can I import that from?
From the Palette Manager, it is called node-red-contrib-fs. I talk about it somewhere in the video.
Can i used (File Listen) on nod-red-contrib.fs version 1.0.0, with this example?
This is very useful! I have been in doubt about how to store data from different devices in different .CSV files. Like in a scalable project, which is the smartest way to have the same flow for different devices?
You can just have copy and paste this flow multiple times and change the file pattern in the function node. I do the same.
@@csongorvarga I saw you have two devices on the turorial. They are publishing to two different topics right? each topic with it's subtopics (temp, Voltage). What it's still not clear it's how the flow knows in which CSV to write the data, maybe looking at the device ID?
I will have 5 different Wheater Stations for a personal project and this flow is like perfect for me, except i still have doubt on how to handle the MQTT messages, like, is one single flow enough to handle different devices or i must have a flow for each device.
Anyhow, i'm so happy to have found your Channel, it's probably the best Node-Red content in RUclips for sure.
@@exprymer If you have a lot of different topics, probably the best to do, that you save each payload to a different flow or global variable. That will always update and keep the latest value in those variables. And in the "Set data" function, you can recall the data from those flow variables. Like msg.payload = { "timestamp": now.getTime(), "temperature": flow.get("temp"), "humidity": flow.get("humidity") }. I did not have to do that because all the data I wanted to save was in a single object in one payload. So I could extract all of them at the same time.
Hi, really nice and useful video. If I wanted to generate dynamic names for my files, do you know how to only search for the date and not for the whole name to delete the older files? Many thanks
you could use regular expressions aka regex
I cannot find the file-listener node. for me it says unknown is that still a supported node or will another node do the same thing?
It is not part of the default installation. Click on the hamburger menu on the right, Manage Palette, on the Install tab, enter node-red-contrib-fs and install the new node. Once it is installed you can import the flow without error.
hello, have you ever tried to do a daily csv file using windows? I review your video and I am doing exactly the same configurarion, but I have an error in the list node, it says "node users list is not set". Any idea??
Which node you get this? On the "File Lister" node?
if daily it creating new csv files then what about storage
@@ramsai0268 what about it?
how can you save data in a network drive that requires username and password, and be able to change the file name based on a flow variable?
I am not sure about the username and password part. I don't know how it works under linux, in Windows you can maybe map it as a new driver with the user credentials, so anyone accessing the drive will not need to enter user and password again. Maybe something similar exists in Linux as well. The file name is getting generated in the function node. You can easily add a flow variable there. I show in the video where and how the file name is getting generated.
Great video man!
If I download the CSV all the data is in one column, how can this happen?
In one column? In Excel? Is it possible that your have a different field separator set in your windows settings as in the CSV file? How the CSV file looks like when you open it in notepad?
@@csongorvarga thank you very much for the fast reply. You are correct, in my country there are other settings in excel
Hi, very good Tutorial! But I still have one question: My Sensor Data (from Electricity Meter) arrive every Minute, so the .csv file is written every minute, also. Is it possible to avoid this frequent writing in order to protect the SD card? Thank you!
Sure. You can save the readings in a flow variable. And have a separate inject that triggers let's hourly and get the latest reading from the flow variable.
@@csongorvarga well, I added a "join" node between "set data" and "filename generator". The join node is set to "Array" and generates an output every 60 inputs. This does the trick so far. Need to verify the data but looks good at the Moment...
Hello Csongor Varga! Do you have a video explaining how to access the RS232 serial ports of a Serial Device Server. For example, ZLAN5843A (8 ports RS232 / 485/422 To TCP / IP converter)?
No, I don't. But I have used modbus TCP for other devices in the past, and the modbus node for Node-Red is pretty good.
@@csongorvarga Pode enviar o link da Aula Modbus TCP?
I want to send my data to the following server:
Host: 200.20.53.x/
external: 200.20.53.x
Port: 21
Login: rsolutions
Password: rsftpinea
How could I configure this example?
Probably the easiest way to do is to log the data in your own server and use the exec node to start a shell script which FTPs the data onto your remote server.
goodd
I Need your help. If you interessed per me know
What is your issue?