In case of Spiffs is not enough, library interface is the same as the one for SdCard. So, code for SPIFFS may be easily migrated to sd. Myself, I use SPIFFS to store html and is files for an embedded web server. You can find it as FSBrowser in my github account.
Two great things about this video - 1) as usual it is very clear and helpful 2) once again you pick a subject that is very relevant to my current projects! Your choice of subject is spot on every time. I will utilise some of your code in my projects. It would be helpful to understand a bit more about creating the JSON buffer in the sketch. I will look for other sources but maybe another video on this topic if other people have the same interest? Thanks again Andreas - brilliant material.
The steps you went through to read the config file into a JSON buffer. Around 8:44 - 9:15 minutes into the video. I understand this video was not to explain the use of the ArduinoJSON library but you do some interesting things as you read the config file ready for parsing. I would like to understand these steps. Maybe I can find the information somewhere - I only raised it in case other people are also interested and it might be another video for you to work on!!!! Thanks Robin
I also do not understand them. I just know how to use the commands. That is the charm of libraries... Baybe I will do a distinct video about json in the future.
Useful as usual! Just an FYI: JSON has very much to do with JavaScript is how you serialize objects in that language, which is the reason why most REST-endpoint now serve JSON instead of XML. Besides the nitpicking, awesome video again!!!!
after watching this video and found that it's the lesson number #121 and it was uploaded in 2017 and we are in 2021 ... i have a lot to learn i hope these tutorial don't end . let's start from beginning .. thanks alot
Fantastic, I don't know how you do it. You keep finding brilliant things that I haven't discovered. I have been using SPIFFS for some time to capture log information so that I can see how my devices are performing and to help with debugging. It involved about 200 lines of code (appending to the file data, displaying it, erasing it, listing the directory etc) but I have recently decided to use ThingSpeak and this works with about 10 lines - excellent. However, I didn't know about the ability to upload directly to SPIFFS for configuration information. I was thinking of using EPROM space but SPIFFS seems to be so much better. Can't wait to give it a go and I look forward to seeing your OTA video. Best wishes Arthur
+Arthur Yarnell In the IOTappstory I use UDP for debugging. This is also a great possibility as it is much faster than Thingspeak. I also have implemented that your device sends log infos in syslog format. This format easily can be monitored and logged even by a Raspby
Thank you very much!! I'm in the process of developing a senior design project in which I would have to play different sounds but I was going to use external storage to read the files from. This is just perfect timing :) I also watched your videos on sonars and i2c interface you made. You make very useful videos thanks again
For those working on mac, the ESP8266FS folder has to be placed in /Applications/Arduino.app/Content/Java/tools. This can be accessed with the Finder by right clicking on the Arduino application icon and selecting in the menu "Show Package Contents".
I didn't even know about SPIFFS !!! I Will buy ESP's from now on instead of Arduino Pro minis. Just won't use the WiFi for projects that don't need WiFi. Oh wait, I have 20 more pro minis left over :( Superb video. I know I learn something new every time I watch one of your videos. +1
wonderful work. Ive been away on work related projects and have come back to this. I am so glad I have sometime off to be truly absorbed and to follow your work. Thank you Andreas, your work flow is awesome, kind of hard to keep up with you pace, however do you manage to do it? You are indeed inspiring on so many levels
Andreas, could you please update this with a version based on JSON 6? I’m a little shaky with that library and your videos are great at explaining things👍 Thank you
The current JSON library is V6, so the example code from the Arduino example files does not compile correct, it will give an error, the example code is for JSON library V5, it needs to be rewritten and adapted for JSON V6 indeed.
I greatly enjoy your videos. Sadly when I attempted this on one of my boards, it became completely unusable. When I tried it on a second board, the software would not run even though the config file was apparently uploaded successfully. But like I said I really enjoy the videos. Please keep them coming.
Strang. I used the plain example and a Lolin board and had absolutely no problems. The same with a colleague of mine with a normal ESP12 module. What kind of configurations did you use?
Thanks ever so much for replying. I really did not expect that. I am using Windows 10 with Arduino IDE 1.8.1. The board is an XcSource D1 Mini which claims to be a node mcu lua device. It has a ch340g usb to uart on board. I selected the Wemos D1 R2 & mini board. Having thought about it a bit, the device was continually resetting which suggests either that the board might be faulty or that it could not get enough juice from the usb cable. I was able to load the blink sketch on both boards prior to this. I was able to load blink on the second board afterwards.
Really great tutorial and thanks for the heads-up about LittleFS, it's a great library! Have you ever tried to open and read or write to a file in the loop() method? I am getting Guru Meditation Error and I can't get around it. Thanks.
Hi Andreas, I know how to read a JSON from SPIFFS but not how to write it. In your code you seem to have a SaveConfig() function. Do you mind sharing that piece of code so we get an idea on how to do it. Thanks for your fantastic videos.
That was brilliant thank you! I was hoping for both of these topics. I didn't find it clear at the end how we would update individual variables within our config from within our sketch, can you elaborate a little? Thank you! :)
+excession I think, you find your answer in the saveConfig part of the example file. I will not update these variables from the sketch, because I would overwrite them by OTA. I still will use EEPROM for that.
For those interested about SPIFFS write cycles limit have a look here www.esp8266.com/viewtopic.php?f=32&t=6109 Could take a couple of years with the right usage scenario.
Hi Andreas! Thank you for your great contributions to the field, they helped me a lot! I was wondering if you can help me with a problem. My script is fully functioning as in your tutorial, but I have a problem passing the json object, or the assigned variables, to my setup and/or loop scope. I need this as I want to use them to start my WiFi.begin() and I want to be able to store config data from a web GUI to the JSON file. I am stuck on passing variables from scope to scope. Can you help me?
@@AndreasSpiess I am having the same issue: if I define const char* in a superior scope, I can't change their values; If I don't use const char* in a json object, they wont compile; If I declare const char* inside a function, it wont be seen in superior scope... i'm kinda stuck to use config.json for global variables... any idea? I am trying to export the mqtt server/port/user/password to use in reconnect functions for example.
Hi Andreas, Thanks for the good informative video. However I think I am missing something: In your recent commits to your IOTAppStory repository you changed to code for VirginSoil to also use the config.json to get the configuration parameters from. This is working fine, I updated my wireless tank level gauge program with this new code and it is getting the parameters from the config.json file, great! However, when I put the ESP into config mode, and change some paraters via the wifimanager page, they don't seem to be saved, they are always overwritten by whaterver I put in the original configuration from config.json... What am I doing wrong?
+Alex Goris You have to decide which parameters you want in json and which in Wifimanager and change your sketch accordingly. I kept both possibilities that you can choose and comment the unnecessary
Thanks very much for your videos. I have learned a great deal from them and I recently bought the Japanese Crimping tool you mentioned as I had the same cheap one from China and it was very disappointing! The Engineer PA-09 is brilliant! Anyway - to this video in particular. What impact will read/writes to this file system have on the lifespan of the device? I doubt very much that any kind of wear levelling is integrated? Thanks again
The chip is rated for 100'000 write cycles. If you just store from time-to time your config, even without any leveling (there is no levelling unless you program it), they all will survive me (I am 60)...
I am 52 so also would probably not outlive 100,000 write cycles ;-) I was considering using it as a local "buffer" to store some readings from attached sensors, then send the data less frequently back to my server; the thought being I can reduce the power consumption of the ESP by not using the WiFi so much. I am still not convinced this is a good use of SPIFFS though. Thank you for replying and please keep up your fascinating and very entertaining videos :-)
Hello, Andreas.I have 2 queries.. Do you have any tutorial for esp8266 to some database connectivity (fog and nosql/rdbms)? And 2nd one: Any tutorial for OTA spiffs loading pls?
1. No, I never worked with a database on the ESP. I just transfer the data to the cloud 2. If you watch the videos about IOTappstory.com you see, that we use also SPIFFS. There is one video which covers SPIFFS itself. Maybe it is worth watching it.
Last week it was Lorawan with TTN and this week esp8266 programming with Arduino IDE. Two of my favorite subjects. Any chance you will look at SDR with focus on GNU Radio in 2017? You are on a roll!
Really awesome video ... very interesting!, Thanks! Do you have some info if continuous writing to the flash to store f.e. the state of something every 5 seconds will be a problem over time ?
+Walter Steinchen I read, that the Winbond flash memory support up to 100'000 cycles. I still use Rtc memory for short-time info which has to survive deep-sleep, but does not need to survive power down.
Hmm .. with 5 sec intervall this would be less than a week .. so not really for high frequency storage ... better to blast stuff like that per UDP to a second Windows / Linux machine.
+Walter Steinchen I anyway do not think, connected devices should store data (other than for optimizing communication and battery live). As you write, they should transfer it to the cloud. That is, why they are "connected" ;-)
Hmmm, whenever possible I like an approach where my devices operate autonome and need a connection just for maintainance, offering a web Interface/API or stuff like that. Actual sample is a garage door opener that knows the state of the door. (1/2 , 3/4 open and so on) Easy to measure that with a IR-LED but then I have to store the state 'somewhere' to survive a power loss ...
Hi Andreas. I have just starting enjoying your videos. Very well done. I really enjoyed this video alot. Would be possible to make an add on video similar to this one using the platformio using spiffs. Anyway keep up the great work. thanks for your efforts.
Hi Andreas. I love your videos and learn a lot from them! Thanks! I have one question I don't know the best place to put it, but in #72 you were reverse engineering the display of a Load to hack your own micro and display in it. I'd like to do the same thing, but I can't find any follow-up videos on this topic. Have you continued working on it? Are you still planning to do a follow-up video on this or did I miss something? Best regards and keep up the great quality videos!
Thanks for the idea, i implemented it by myself and enhanced it. Now, if no WiFi is established within 60 seconds, a temporary WiFi Hotspot is created, and the User can connect to it and enter the new WiFi Credentials manually, which is then wrtten to SPIFFS again. Now, I don't even have to flash a new SPIFFS image when I change the location.
thanks... i tried to play with it a bit, it uploaded 'something' but i had not created a config file. i had some other sketches open in arduino ide. i am wondering, is there quick way to know what has been uploaded... at least file names. (i havent gone through detailed documentations of library... yet)
+Electromania There is the link to a very good documentation of the FS library. The whole data directory is converted into a bin file and this file is uploaded.
I really appreciate your videos Andreas. So, if I understand correctly SPIFFS performs an erase/write sequence according to sector or block size and not the size of the file. Thus SPIFFS is not going to be suitable for data logging where we must maintain a non-volatile circular buffer for fairly short files (timestamp and data sentence). Does anyone have any suggestions on how this can be done on the ESP32?
Thank you, this was just what I was looking for, just one suggestion - you might want to change the link for the ESP8266 filesystem uploader to point to the releases - if you said that I missed it, and that was really the only point I had a little bit of a problem - The releases have the jar file in the zip, the repository is the source code. Otherwise, very good and just what I was looking for to setup a config file for my current IoT project.
I still did not upgrade to version 6 because it is not compatible. If you want you can create a pull request when you changed the code to run with version 6
If I did more than the example files I usually post my files in GitHub. If I do not post them I usually assume they are close enough to the example files of the Arduino IDE
@@AndreasSpiess I had the same question and this answers it - thanks! So this means that the user could still update variables with WiFiManager via their cell phone AND the new variables get saved in the SPIFFS file AND those variables survive a power down. Now I'm seeing the beauty of the thing that I could not see before.
Great video! I am currently playing around with SPIFFS and have two questions: 1: Is there any more comfortable extension for accessing SPIFFS from within the Arduino IDE? I am thinking of a two way transfer (read and write) 2: Is there any extension doing the same for the ESP32?
I just got my hands and an 8266 about a week ago (six more on the way now) and your videos have been invaluable, thanks. Do you know is there any reason the config file size check is set to 1024 bytes?
Enjoy your new toy! Concerning 1024: This was just my decision to avoid problems. You can reduce it if you want. Keep in mind, that the ESPs have much more RAM than the Arduinos.
Thanks for the response. My interest was more if it could go bigger but I think in the best course of action would be to check my ram headroom for my particular chip and set the max size accordingly!
Hi Andreas, In your modified ConfigFile2 script, it looks like you've changed the generic : StaticJsonBuffer jsonBuffer; line to a : StaticJsonBuffer jsonBuffer; line. Considering that you handle a very small JSON configuration file (serverName and accessToken), is there any ***coding*** reason that we use a 1024-byte buffer instead of a smaller one (as long as the JsonBuffer size fits the size of the data to load or save) ? Thanks again for your super videos !
+Yann M Usually, I only use less than 50% of RAM and the script checks anyway for the 1024 byte length of the file. With 1024 buffer size, I can forget about that parameter. This library comes from Arduino, where we have to save every byte. Not anymore with the ESP...
Actually, I am a little bit confused with the differences between a **regular** sketch upload (with the SPIFFS data included in the saveConfig() routine) and the new "ESP8266 Sketch Data Upload" (with the SPIFFS data included in the /data/config.json file ). What is the exact procedure, what should come first : config.json or saveConfig() routine ? Is it better to write the data directly in the /data/config.json file or from the saveConfig() in the sketch ? Does the saveConfig() routine write the data **INTO** the /data/config.json file or **DIRECTLY** to the ESP disk ?
+Yann M There is no exact. You can either overwrite the SPIFFS disk with the tool of the IDE or change its files with the sketch. So, if you want remotely upload constants, the IDE tool is the right. If you want to store variables from the sketch, the second possibility is the right one.
Yeahhhhh ... I finally realized this in the mean time !!! The IDE option is quite slow to upload but is definitely the right one to use if one wants to keep a total confidentiality over the recorded data that will ***NOT*** show in the sketch. I suppose that you next-to-come OTA SPIFFS video will deal with this "confidentiality" point ! I also noticed that I had to switch OFF and ON the ESP after the 'Sketch Data Upload' otherwise it looks like the ESP cannot read back the SPIFFS content properly ... did you notice that ?
1. About speed: I use 1M SPIFFS and 921600 upload speed 2. I thust tried, and I had no need to reset the device. I uploaded the sketch first, then SPIFFS,. As a test, I read the conctants from spiffs, changed them by the sketch, and reloaded them. All went without reset
esp8266 does not forget credentials on restart or poweroff after you connect a network right? is this usefull for storing credentials if you are using wifi.disconnect which erases the wifi passwords?
I am not completely sure how this storing of credentials works. Normally, the ESP connects to the same WLAN without entering the credentials again. But I had situations, where it did not happen automatically. And I also do not know if you can erease the credentials completely.
Hi Andreas, Please can you show us a way to change SSID and Password programatically from a display or from HTTP requests, just like WiFiManager library just not by using the captive portal and web interface
after watching this video i am little confused now. i use eclipse for programming esp8266 mod 12 and it espressif has a library function to store and read data from its spi flash memory.(i have used it works fine). is this ssd memory different from that spi flash inside esp. i am using???
great one. I figured this out some time ago, but I don't know how to secure the data on the File System so nobody can read it. I mean, when someone hijacks the physical device (f.ex. garden lamp), he will have open access to our wifi network credentials. This in most cases will be a disaster of unknown outcome. I remember ATMEL chips have lockbits-fuses. Can you explore this topic a little bit more?
+aiklarung What do you mean with key-value parts? Spiffs is a file system, not an EEPROM. It only stores files and you have to take care of the file content.
Andreas Spiess I mean, is SPIFFS just a hash table ("dictionary" kind of structure) or does it have filesystem-like features, such as file dates, "directory" abstractions and such? (maybe my real question is, why didn't they just go with FAT, there's probably a FAT32 library somewhere asking to be used...)
FAT was probably "too fat" for these small devices. So, there is no directory structure. They solve this issue slightly different. You find a good explanation here: github.com/esp8266/Arduino/blob/master/doc/filesystem.md
can we give "ssid" and "password" to esp8266 using a webpage while esp is in AP mode. After rebooting the device it should automatically connect to that particular Access point with the credentials we have given before when esp8266 is on AP mode. i think this is possible with SPIFFS!
Thank you for the help..But while using spiffs I m facing one issue... I wrote the contents to a file in spiffs but when my code restarts due to some reason then all the configuration stored in my file is lost and re writes the configuration to spiffs again ?? Why is it happening?? What's the solution to it?? Plz help
could someone please tell me how do i know how much of flash memory is accessible to me...i expect it to be 4 M bytes....however we use if (size>1024).....why so...if i want to use whole 4 mb of flash memor, why would i care about size being greater than 1024...thanks
This may sound silly. But really how SPIFFS is beneficial here? Unless we have kilobytes of data stored on SPIFFS, i don’t see a useful case for smaller set of data. Do we also see any performance benefits?
For example if you are doing some kind of website for sensor data, and you need a picture for it, you can store small pictures on the SPIFFS. Also you can change the config files, Wifi ssid, password on the fly. If they were in the sketch you wont be able to update them other that re-uploading the whole sketch with arduino ide
Thanks for the video. I have some queries. I have placed my device(spiff uploaded) at a remote place. Now I need to change the SPIFF data. Is it possible to update a particular field of JSON array in the SPIFF? Example: {"Test1":[{"id":"1","Status":"On","StartHr":"0","StartMin":"5","EndHr":"0","EndMin":"10"}]} The data i got through Http get request. If I want to update the StartHr and StartMin, how to change it. Because I didn't find any command to change this. It's urgent. Thanks!!
No. You are great sir. I had learned many things from your videos;) Beside remote debugging, is it possible to change a particular field of the data stored in spiffs? coz I found command to get the file size, rename, create a new file, update the whole file, append file, and delete the file. but didn't found command for updating a data field. Thanks
Great video :) I like it and now I want to use SPIFFS to store multiple SSIDs and Passwords and connect with the one which have strong signals using WiFiMulti. I want ESP to store SSIDs and Passwords of each every place I move it and we I again go there it will automatically connect with that Wifi. My questions is that can I use SPIFFS to store them as JSON object and how many time I can read/write SPIFFS? Secondly is it better to use SPIFFS or EEPROM?
1. You can write files with any content to spiffs. Also text files containing a json string. 2. It is your choice. Both end up in flash memory. EEPROM uses (valuable) flash below 1M, Spiffs usually not. So, for big data, SPIFFS is probably better.
The ESP8266 isn't intended for professional and finished products, for instance, an end user being able to set a wireless login credentials trough a Web server. You are only able to set the WiFi login credentials when uploading the firmware something that voids the possibility of being general purpose!
I suggest you watch my videos about IOTappstory.com. There I update the credentials and other parameters through a Smartphone. The ESP8266 is used in many commercial products. Otherwise the manufacturer probably could not survive.
Shawn Woods just posted a video where he disassembled the Victor Smart Mouse Trap.... And soldered up the end of the board... An ESP module (probably ESP32). I think that says professional products are using them just fine.
Hello Andreas, very nice video. I've been following your adventures with the LoRa network, and I'm glad to report that I've learn a lot!. I was wondering if you could help me. I was wondering if you knew about any RFID modules that I could hookup to an ESP8266. Also, do you know of any sources of where I can find information of OTA flashing for this modules?. I'm a software developer getting my feet wet in electronics :). Anyhow, great video and keep up the good work!.
If you don't use upload tool, then any operation with SPIFFS will crash/fail because SPIFFS area is blank as default. Need to create sketch with a call to SPIFFS.format() (and run it once). Of course you can avoid this problem by using upload tool as shown in video.
hello thank you for this tutorial :) I have a problem, there is this message that appears: "Failed to open config file " could someone help me please. Thanks. (esp8266MOD)
@@AndreasSpiess Sir, which one is better EEPROM or SPIFFS ?? I want to store ssid and password in the flash memory of ESP8266 for my Home Automation Project.
This is something many esp8266 beginners didn't know about. Great tutorial.
+Germán Martín I also only discovered it recently when we were searching possibilities to transfer configurations from IOTappstory.com
In case of Spiffs is not enough, library interface is the same as the one for SdCard. So, code for SPIFFS may be easily migrated to sd.
Myself, I use SPIFFS to store html and is files for an embedded web server. You can find it as FSBrowser in my github account.
.ghfgh
Two great things about this video - 1) as usual it is very clear and helpful 2) once again you pick a subject that is very relevant to my current projects! Your choice of subject is spot on every time.
I will utilise some of your code in my projects. It would be helpful to understand a bit more about creating the JSON buffer in the sketch. I will look for other sources but maybe another video on this topic if other people have the same interest?
Thanks again Andreas - brilliant material.
What do you mean with "json buffer"?
The steps you went through to read the config file into a JSON buffer. Around 8:44 - 9:15 minutes into the video. I understand this video was not to explain the use of the ArduinoJSON library but you do some interesting things as you read the config file ready for parsing. I would like to understand these steps. Maybe I can find the information somewhere - I only raised it in case other people are also interested and it might be another video for you to work on!!!! Thanks Robin
I also do not understand them. I just know how to use the commands. That is the charm of libraries...
Baybe I will do a distinct video about json in the future.
Useful as usual! Just an FYI: JSON has very much to do with JavaScript is how you serialize objects in that language, which is the reason why most REST-endpoint now serve JSON instead of XML. Besides the nitpicking, awesome video again!!!!
+Andrea Buda Thanks for the clarification. What I meant was, that even if I have no idea about JS, I still am able to use JSON...
after watching this video and found that it's the lesson number #121 and it was uploaded in 2017 and we are in 2021 ... i have a lot to learn i hope these tutorial don't end . let's start from beginning .. thanks alot
Now I am around video #350...
wherever I am in my "ESP projects" stage, there is always a video on THIS channel to answer my questions ;-)
That is the intention of the creator, not only by chance ;-)
@@AndreasSpiess just: replace: SPIFFS with LittleFS and all is done ;-)
Fantastic, I don't know how you do it. You keep finding brilliant things that I haven't discovered. I have been using SPIFFS for some time to capture log information so that I can see how my devices are performing and to help with debugging. It involved about 200 lines of code (appending to the file data, displaying it, erasing it, listing the directory etc) but I have recently decided to use ThingSpeak and this works with about 10 lines - excellent. However, I didn't know about the ability to upload directly to SPIFFS for configuration information. I was thinking of using EPROM space but SPIFFS seems to be so much better. Can't wait to give it a go and I look forward to seeing your OTA video.
Best wishes
Arthur
+Arthur Yarnell In the IOTappstory I use UDP for debugging. This is also a great possibility as it is much faster than Thingspeak. I also have implemented that your device sends log infos in syslog format. This format easily can be monitored and logged even by a Raspby
Thank you very much!! I'm in the process of developing a senior design project in which I would have to play different sounds but I was going to use external storage to read the files from. This is just perfect timing :) I also watched your videos on sonars and i2c interface you made. You make very useful videos thanks again
You are welcome!
Perfect tutorial. Short, but contains everything required.
Thank you!
For those working on mac, the ESP8266FS folder has to be placed in /Applications/Arduino.app/Content/Java/tools. This can be accessed with the Finder by right clicking on the Arduino application icon and selecting in the menu "Show Package Contents".
Thanks!
Great example, thank you. DeltaJSON is really useful if you are working with JSON config files, it does compare, merge and graft.
This tool seems to be excellent for larger files!
Great video, you are very inspiring
Thanks!
Love your channel. I've been using C++ structs and EEPROM writes all this time!!!! This is very exciting. Thank you!
You are welcome!
I didn't even know about SPIFFS !!!
I Will buy ESP's from now on instead of Arduino Pro minis. Just won't use the WiFi for projects that don't need WiFi.
Oh wait, I have 20 more pro minis left over :(
Superb video. I know I learn something new every time I watch one of your videos. +1
lol, still using pro minis or they got over?
wonderful work. Ive been away on work related projects and have come back to this. I am so glad I have sometime off to be truly absorbed and to follow your work. Thank you Andreas, your work flow is awesome, kind of hard to keep up with you pace, however do you manage to do it? You are indeed inspiring on so many levels
+Dean Greenhough You are welcome. Last week it was not easy because I also was quite absorbed from "earning money"...
Awesome video. I was trying to understand the code through examples but it did not make any sense to me. Your video clarified everything perfectly!
:-)
Thanks for pointing me to this FS tool Andreas! Really helpful.
+Thomas Verschoof You are welcome!
Two main thing there are: sketch is no interference with spiffs, and the second: size of spiffs partition make influence on write speed (its amazing)
Thank you for your information.
Your videos are really helpful Andreas. thank you for the time and effort you put in to them. :)
Thank you!
You're a good damn bad ass! Thank you so much for the videos, best new channel find of late.
Thanks!
tHANK YOU SO much so you can store your ssid, temperature, humidity.. server info so no need an sd card ..
thank you !!!
you are the best!!! :)
Marc.
Tell this to my wife if I forget to do something "important" ;-) Thanks!
AHHAHAH :-) :d
This has helped me a lot to debug the ESP32 Thanks Andreas
You are welcome!
this is great esp32 info. 🙂 i am still new on this esp32 pcb. 🐕 thanks a lot. 🛠
Glad it was helpful!
As always very very inspiring video ! Great stuff please keep on rocking ;)
+Florian Hammers :-)
Great video Andreas!!! As usual.. Thanks!!
You are welcome!
Andreas, could you please update this with a version based on JSON 6? I’m a little shaky with that library and your videos are great at explaining things👍 Thank you
So far I did not upgrade to v6 of the JSON lib. It works completely different and I see no advantage for me.
The current JSON library is V6, so the example code from the Arduino example files does not compile correct, it will give an error, the example code is for JSON library V5, it needs to be rewritten and adapted for JSON V6 indeed.
I greatly enjoy your videos. Sadly when I attempted this on one of my boards, it became completely unusable. When I tried it on a second board, the software would not run even though the config file was apparently uploaded successfully. But like I said I really enjoy the videos. Please keep them coming.
Strang. I used the plain example and a Lolin board and had absolutely no problems. The same with a colleague of mine with a normal ESP12 module. What kind of configurations did you use?
Thanks ever so much for replying. I really did not expect that. I am using Windows 10 with Arduino IDE 1.8.1. The board is an XcSource D1 Mini which claims to be a node mcu lua device. It has a ch340g usb to uart on board. I selected the Wemos D1 R2 & mini board. Having thought about it a bit, the device was continually resetting which suggests either that the board might be faulty or that it could not get enough juice from the usb cable. I was able to load the blink sketch on both boards prior to this. I was able to load blink on the second board afterwards.
Really great tutorial and thanks for the heads-up about LittleFS, it's a great library!
Have you ever tried to open and read or write to a file in the loop() method? I am getting Guru Meditation Error and I can't get around it. Thanks.
I do not use LittleFS a lot :-(
@@AndreasSpiess SPIFFS and LittleFS are basically almost the same. But it's okay, I've figured it out. Thanks anyway!
can you save last relay status into esp8266 please tell me how to do it thank you
You can save it as a variable.
Hi Andreas. I love your videos and learn a lot from them! Have you tried to host an entire website with text and images on an esp32? 73's
No. I never tried that. But it should be possible.
@@AndreasSpiess Keep up the good work on your awesome channel best 73's
Great job -- your videos are very educational -- Thanks so much
You are welcome!
Hi Andreas,
I know how to read a JSON from SPIFFS but not how to write it. In your code you seem to have a SaveConfig() function. Do you mind sharing that piece of code so we get an idea on how to do it. Thanks for your fantastic videos.
Ooh, I think I've found the source code:
github.com/esp8266/Arduino/blob/master/libraries/esp8266/examples/ConfigFile/ConfigFile.ino
Google is your friend ;-)
I love JSON. Wasted so much time with XML in my first years.
I think, for our purpose, it is perfect.
That was brilliant thank you! I was hoping for both of these topics. I didn't find it clear at the end how we would update individual variables within our config from within our sketch, can you elaborate a little? Thank you! :)
+excession I think, you find your answer in the saveConfig part of the example file. I will not update these variables from the sketch, because I would overwrite them by OTA. I still will use EEPROM for that.
Pop
For those interested about SPIFFS write cycles limit have a look here www.esp8266.com/viewtopic.php?f=32&t=6109
Could take a couple of years with the right usage scenario.
Hi Andreas! Thank you for your great contributions to the field, they helped me a lot! I was wondering if you can help me with a problem. My script is fully functioning as in your tutorial, but I have a problem passing the json object, or the assigned variables, to my setup and/or loop scope. I need this as I want to use them to start my WiFi.begin() and I want to be able to store config data from a web GUI to the JSON file.
I am stuck on passing variables from scope to scope. Can you help me?
If I understand right, you just have to define the variables before setup(). Then, their scope covers the whole sketch.
@@AndreasSpiess I am having the same issue: if I define const char* in a superior scope, I can't change their values; If I don't use const char* in a json object, they wont compile; If I declare const char* inside a function, it wont be seen in superior scope... i'm kinda stuck to use config.json for global variables... any idea? I am trying to export the mqtt server/port/user/password to use in reconnect functions for example.
Hi Andreas, Thanks for the good informative video. However I think I am missing something: In your recent commits to your IOTAppStory repository you changed to code for VirginSoil to also use the config.json to get the configuration parameters from. This is working fine, I updated my wireless tank level gauge program with this new code and it is getting the parameters from the config.json file, great! However, when I put the ESP into config mode, and change some paraters via the wifimanager page, they don't seem to be saved, they are always overwritten by whaterver I put in the original configuration from config.json... What am I doing wrong?
+Alex Goris You have to decide which parameters you want in json and which in Wifimanager and change your sketch accordingly. I kept both possibilities that you can choose and comment the unnecessary
Thank you, very well made video, and easy to understand
You are welcome!
Thanks very much for your videos. I have learned a great deal from them and I recently bought the Japanese Crimping tool you mentioned as I had the same cheap one from China and it was very disappointing! The Engineer PA-09 is brilliant!
Anyway - to this video in particular. What impact will read/writes to this file system have on the lifespan of the device? I doubt very much that any kind of wear levelling is integrated?
Thanks again
The chip is rated for 100'000 write cycles. If you just store from time-to time your config, even without any leveling (there is no levelling unless you program it), they all will survive me (I am 60)...
I am 52 so also would probably not outlive 100,000 write cycles ;-)
I was considering using it as a local "buffer" to store some readings from attached sensors, then send the data less frequently back to my server; the thought being I can reduce the power consumption of the ESP by not using the WiFi so much. I am still not convinced this is a good use of SPIFFS though.
Thank you for replying and please keep up your fascinating and very entertaining videos :-)
I have a video where I use RTC memory for that, which fits better (no wear). Video #60 is about that.
Oh wow! Thanks - I will check it out later :-)
Thank you for this well explained video. So you can now save to your esp from arduino idea data, but is it possible to load it like this back too?
Yes.
Informative as always. Thank you.
:-)
Very nicely presented. Respect !
Thank you!
Hello, Andreas.I have 2 queries.. Do you have any tutorial for esp8266 to some database connectivity (fog and nosql/rdbms)? And 2nd one: Any tutorial for OTA spiffs loading pls?
1. No, I never worked with a database on the ESP. I just transfer the data to the cloud
2. If you watch the videos about IOTappstory.com you see, that we use also SPIFFS. There is one video which covers SPIFFS itself. Maybe it is worth watching it.
Thank you, hope to see more video specially on say beacons for retails, wearables for healthcare industry etc.
So far, I am not yet in Bluetooth. I still wait for the ESP32 and the Ardoino IDE covering this.
Last week it was Lorawan with TTN and this week esp8266 programming with Arduino IDE. Two of my favorite subjects. Any chance you will look at SDR with focus on GNU Radio in 2017? You are on a roll!
+Mark Troyer Maybe. At least it would be in the area of my interests... Why are you interested in GNU radio?
Did not understand anything. Thank you so much for making me see complex
Things in digital world.
Really awesome video ... very interesting!, Thanks!
Do you have some info if continuous writing to the flash to store f.e. the state of something every 5 seconds will be a problem over time ?
+Walter Steinchen I read, that the Winbond flash memory support up to 100'000 cycles. I still use Rtc memory for short-time info which has to survive deep-sleep, but does not need to survive power down.
Hmm .. with 5 sec intervall this would be less than a week .. so not really for high frequency storage ... better to blast stuff like that per UDP to a second Windows / Linux machine.
+Walter Steinchen I anyway do not think, connected devices should store data (other than for optimizing communication and battery live). As you write, they should transfer it to the cloud. That is, why they are "connected" ;-)
Hmmm, whenever possible I like an approach where my devices operate autonome and need a connection just for maintainance, offering a web Interface/API or stuff like that.
Actual sample is a garage door opener that knows the state of the door. (1/2 , 3/4 open and so on)
Easy to measure that with a IR-LED but then I have to store the state 'somewhere' to survive a power loss ...
+Walter Steinchen In a few months , autonomy will be defined as your cloud. And in a few years the internet;-)
Thank you Andreas, do you know whether the newer arduino library 'Preferences' works with esp8266 boards? Thank you.
This one should work: github.com/vshymanskyy/Preferences
Hi Andreas. I have just starting enjoying your videos. Very well done. I really enjoyed this video alot. Would be possible to make an add on video similar to this one using the platformio using spiffs. Anyway keep up the great work. thanks for your efforts.
Thank you! I do not use SPIFFS often. So the chance for such a video is small :-(
in my example config file has 3 libraries ... ArduinoJson.h, fs.h, LittleFS.h what is littleFS.h for ?
Little FS is similar to SPIFFS.
Hi Andreas. I love your videos and learn a lot from them! Thanks!
I have one question I don't know the best place to put it, but in #72 you were reverse engineering the display of a Load to hack your own micro and display in it. I'd like to do the same thing, but I can't find any follow-up videos on this topic. Have you continued working on it? Are you still planning to do a follow-up video on this or did I miss something?
Best regards and keep up the great quality videos!
9i
EXCELLENT. REALLY INFORMATIVE.
Thank you!
Very useful, as always ! Thanks !
You are welcome!
Thanks for the idea, i implemented it by myself and enhanced it. Now, if no WiFi is established within 60 seconds, a temporary WiFi Hotspot is created, and the User can connect to it and enter the new WiFi Credentials manually, which is then wrtten to SPIFFS again. Now, I don't even have to flash a new SPIFFS image when I change the location.
Cool!
Can you share that code ?
great video. truly remarkable effort.
Thanks!
Thank you!
Mr Spiess, is there any example using the IOTAPPSTORY with SD card?
Not that I know of
great tuto I have a question is it possible to modify the jsonfile uploaded by overwrite in the scetch ?
I think so. techtutorialsx.com/2019/05/28/esp8266-spiffs-writing-a-file/
Thanks for the video. Something like WiFi credentials, should we store it in flash or eeprom?
Because they are short it is easier in EEPROM
thanks...
i tried to play with it a bit, it uploaded 'something' but i had not created a config file. i had some other sketches open in arduino ide.
i am wondering, is there quick way to know what has been uploaded... at least file names.
(i havent gone through detailed documentations of library... yet)
+Electromania There is the link to a very good documentation of the FS library. The whole data directory is converted into a bin file and this file is uploaded.
I really appreciate your videos Andreas. So, if I understand correctly SPIFFS performs an erase/write sequence according to sector or block size and not the size of the file. Thus SPIFFS is not going to be suitable for data logging where we must maintain a non-volatile circular buffer for fairly short files (timestamp and data sentence). Does anyone have any suggestions on how this can be done on the ESP32?
I would do the buffering in RAM and transfer the data I want to keep in junks via Wi-Fi to a secure place.
Thanks for such a helpful video!
You are welcome!
Very informative, thank you sir!
+Kamran Aghlami You are welcome!
Thank you, this was just what I was looking for, just one suggestion - you might want to change the link for the ESP8266 filesystem uploader to point to the releases - if you said that I missed it, and that was really the only point I had a little bit of a problem - The releases have the jar file in the zip, the repository is the source code. Otherwise, very good and just what I was looking for to setup a config file for my current IoT project.
Be aware that this is quite an old video and things always change a little.
@@AndreasSpiess the only constant is change. Still really good information and very useful thank you
The Arduino Code in the example code of the Arduino IDE will not compile, its for JSON 5 while JSON 6 is now the later library.
I still did not upgrade to version 6 because it is not compatible. If you want you can create a pull request when you changed the code to run with version 6
Greetings Andreas! And how to do otherwise, that is, completely clean the files in SPIFFS to free memory on esp8266? Thank you
If I remember right there is a program around which empties the Flash. But I do not know where.
Hi Andreas, where can I get the files that you used for this tutorial?
If I did more than the example files I usually post my files in GitHub. If I do not post them I usually assume they are close enough to the example files of the Arduino IDE
In your last comment you mention saving will update flash memory, but not the spiff. Can you clarify that a bit please?
+DaveMody and No. It will update the SPIFFS, but not the file on your PC. I think, I was not clear enough.
@@AndreasSpiess I had the same question and this answers it - thanks! So this means that the user could still update variables with WiFiManager via their cell phone AND the new variables get saved in the SPIFFS file AND those variables survive a power down. Now I'm seeing the beauty of the thing that I could not see before.
Great video! I am currently playing around with SPIFFS and have two questions:
1: Is there any more comfortable extension for accessing SPIFFS from within the Arduino IDE? I am thinking of a two way transfer (read and write)
2: Is there any extension doing the same for the ESP32?
I do not use SPIFF often. And AFAIK it is not yet implemented on ESP32
You can put tools in a tools folder in your sketcbook root, instead of program files.
+tablatronix Good to know. I just followed the manual on GitHub.
I just got my hands and an 8266 about a week ago (six more on the way now) and your videos have been invaluable, thanks. Do you know is there any reason the config file size check is set to 1024 bytes?
Enjoy your new toy! Concerning 1024: This was just my decision to avoid problems. You can reduce it if you want. Keep in mind, that the ESPs have much more RAM than the Arduinos.
Thanks for the response. My interest was more if it could go bigger but I think in the best course of action would be to check my ram headroom for my particular chip and set the max size accordingly!
You can try. I newer had json files bigger than that.
where can I download the ESP Core library?
Maybe you search for a tutorial? You have to install a new board definition in the Arduino IDE
Hi Andreas,
In your modified ConfigFile2 script, it looks like you've changed the generic :
StaticJsonBuffer jsonBuffer;
line to a :
StaticJsonBuffer jsonBuffer;
line.
Considering that you handle a very small JSON configuration file (serverName and accessToken), is there any ***coding*** reason that we use a 1024-byte buffer instead of a smaller one (as long as the JsonBuffer size fits the size of the data to load or save) ?
Thanks again for your super videos !
+Yann M Usually, I only use less than 50% of RAM and the script checks anyway for the 1024 byte length of the file. With 1024 buffer size, I can forget about that parameter.
This library comes from Arduino, where we have to save every byte. Not anymore with the ESP...
Actually, I am a little bit confused with the differences between a **regular** sketch upload (with the SPIFFS data included in the saveConfig() routine) and the new "ESP8266 Sketch Data Upload" (with the SPIFFS data included in the /data/config.json file ).
What is the exact procedure, what should come first : config.json or saveConfig() routine ?
Is it better to write the data directly in the /data/config.json file or from the saveConfig() in the sketch ?
Does the saveConfig() routine write the data **INTO** the /data/config.json file or **DIRECTLY** to the ESP disk ?
+Yann M There is no exact. You can either overwrite the SPIFFS disk with the tool of the IDE or change its files with the sketch. So, if you want remotely upload constants, the IDE tool is the right. If you want to store variables from the sketch, the second possibility is the right one.
Yeahhhhh ... I finally realized this in the mean time !!!
The IDE option is quite slow to upload but is definitely the right one to use if one wants to keep a total confidentiality over the recorded data that will ***NOT*** show in the sketch.
I suppose that you next-to-come OTA SPIFFS video will deal with this "confidentiality" point !
I also noticed that I had to switch OFF and ON the ESP after the 'Sketch Data Upload' otherwise it looks like the ESP cannot read back the SPIFFS content properly ... did you notice that ?
1. About speed: I use 1M SPIFFS and 921600 upload speed
2. I thust tried, and I had no need to reset the device. I uploaded the sketch first, then SPIFFS,. As a test, I read the conctants from spiffs, changed them by the sketch, and reloaded them. All went without reset
esp8266 does not forget credentials on restart or poweroff after you connect a network right? is this usefull for storing credentials if you are using wifi.disconnect which erases the wifi passwords?
I am not completely sure how this storing of credentials works. Normally, the ESP connects to the same WLAN without entering the credentials again. But I had situations, where it did not happen automatically. And I also do not know if you can erease the credentials completely.
Hi Andreas,
Please can you show us a way to change SSID and Password programatically from a display or from HTTP requests, just like WiFiManager library just not by using the captive portal and web interface
If you do not want to use your smartphone you can use a display.
Hello. I have esp32 wrover 16mb. It works only with fat fs. Is there some way to upload my data on its FS? Please help...
I do not know. But google knows for sure.
Hello sir, how if i format the spiffs partitions frequently? let say once a day. what's the effect?
I think this should be no problem. Flash can be rewritten a few 10'000 times.
after watching this video i am little confused now. i use eclipse for programming esp8266 mod 12 and it espressif has a library function to store and read data from its spi flash memory.(i have used it works fine).
is this ssd memory different from that spi flash inside esp. i am using???
+Amit Singh I call the part of the ESP flash "SSD" to make it simple for beginners (and SSDs also use Flash memory). So, it is the same.
ok thanks alot
great one. I figured this out some time ago, but I don't know how to secure the data on the File System so nobody can read it. I mean, when someone hijacks the physical device (f.ex. garden lamp), he will have open access to our wifi network credentials. This in most cases will be a disaster of unknown outcome. I remember ATMEL chips have lockbits-fuses. Can you explore this topic a little bit more?
I do not think that it is possible to block this content.
great video and again a lot of inspirationthanks!
+Urs Marti You are welcome! You also helped me a lot!
What is the difference between SPIFFS and simple key-value pairs? Does it store metadata (e.g. dates)?
+aiklarung What do you mean with key-value parts? Spiffs is a file system, not an EEPROM. It only stores files and you have to take care of the file content.
Andreas Spiess I mean, is SPIFFS just a hash table ("dictionary" kind of structure) or does it have filesystem-like features, such as file dates, "directory" abstractions and such? (maybe my real question is, why didn't they just go with FAT, there's probably a FAT32 library somewhere asking to be used...)
FAT was probably "too fat" for these small devices. So, there is no directory structure. They solve this issue slightly different. You find a good explanation here: github.com/esp8266/Arduino/blob/master/doc/filesystem.md
Great video!
:-)
yeah but I still don't know how to save the data in the file with a program, does anyone know how to save the data?
How would you transfer a file on the esp storage to the windows disk? Let's say an image?
Via Wi-Fi? I do not know. Probably with calling a web service. With the Arduino IDE: As shown in the video.
Can we store the relay state in this ? So after reboot we can read the state from it and restore the state. !!??
Yes
can we give "ssid" and "password" to esp8266 using a webpage while esp is in AP mode. After rebooting the device it should automatically connect to that particular Access point with the credentials we have given before when esp8266 is on AP mode. i think this is possible with SPIFFS!
Yes. But it has nothing to do with SPIFFS. There are a few videos in my ESP8266 and IOTappstory playlist covering this topic.
Thank you sir! u r the best!
Thank you for the help..But while using spiffs I m facing one issue... I wrote the contents to a file in spiffs but when my code restarts due to some reason then all the configuration stored in my file is lost and re writes the configuration to spiffs again ?? Why is it happening?? What's the solution to it?? Plz help
I do not know. This should not happen. So either you do not commit() it to EEPROM or you erase the EEPROM during programming.
@@AndreasSpiess EEPROM??
I mixed something up. Forget my message, Sorry.
could someone please tell me how do i know how much of flash memory is accessible to me...i expect it to be 4 M bytes....however we use if (size>1024).....why so...if i want to use whole 4 mb of flash memor, why would i care about size being greater than 1024...thanks
You can partition the ESP memories. Not easy. You have to google for the ESP8266. I made a video about it for the ESP32
thanks @@AndreasSpiess for your reply.....i will see it
"Separate the cowards from the heroes" 😂😂😂😂
:-)
Usefull info, thanks a lot.
You are welcome!
This may sound silly. But really how SPIFFS is beneficial here? Unless we have kilobytes of data stored on SPIFFS, i don’t see a useful case for smaller set of data. Do we also see any performance benefits?
For example if you are doing some kind of website for sensor data, and you need a picture for it, you can store small pictures on the SPIFFS. Also you can change the config files, Wifi ssid, password on the fly. If they were in the sketch you wont be able to update them other that re-uploading the whole sketch with arduino ide
Thanks for a great video.
+Mehrdad Givehchi :-)
Thanks for the video.
I have some queries. I have placed my device(spiff uploaded) at a remote place.
Now I need to change the SPIFF data. Is it possible to update a particular field of JSON array in the SPIFF?
Example:
{"Test1":[{"id":"1","Status":"On","StartHr":"0","StartMin":"5","EndHr":"0","EndMin":"10"}]}
The data i got through Http get request.
If I want to update the StartHr and StartMin, how to change it. Because I didn't find any command to change this.
It's urgent.
Thanks!!
Unfortunately I cannot do remote debugging :-(
No. You are great sir. I had learned many things from your videos;)
Beside remote debugging,
is it possible to change a particular field of the data stored in spiffs?
coz I found command to get the file size, rename, create a new file, update the whole file, append file, and delete the file. but didn't found command for updating a data field.
Thanks
Great video :) I like it and now I want to use SPIFFS to store multiple SSIDs and Passwords and connect with the one which have strong signals using WiFiMulti. I want ESP to store SSIDs and Passwords of each every place I move it and we I again go there it will automatically connect with that Wifi.
My questions is that can I use SPIFFS to store them as JSON object and how many time I can read/write SPIFFS? Secondly is it better to use SPIFFS or EEPROM?
1. You can write files with any content to spiffs. Also text files containing a json string.
2. It is your choice. Both end up in flash memory. EEPROM uses (valuable) flash below 1M, Spiffs usually not. So, for big data, SPIFFS is probably better.
Thanks Andreas
The ESP8266 isn't intended for professional and finished products, for instance, an end user being able to set a wireless login credentials trough a Web server. You are only able to set the WiFi login credentials when uploading the firmware something that voids the possibility of being general purpose!
I suggest you watch my videos about IOTappstory.com. There I update the credentials and other parameters through a Smartphone.
The ESP8266 is used in many commercial products. Otherwise the manufacturer probably could not survive.
Shawn Woods just posted a video where he disassembled the Victor Smart Mouse Trap.... And soldered up the end of the board... An ESP module (probably ESP32). I think that says professional products are using them just fine.
Hello Andreas,
very nice video. I've been following your adventures with the LoRa network, and I'm glad to report that I've learn a lot!.
I was wondering if you could help me. I was wondering if you knew about any RFID modules that I could hookup to an ESP8266. Also, do you know of any sources of where I can find information of OTA flashing for this modules?. I'm a software developer getting my feet wet in electronics :).
Anyhow, great video and keep up the good work!.
I never worked with RFID modules. So, I have no knowledge about them.
Ah bummer. Well anyhow, thanks for the time :)
If you don't use upload tool, then any operation with SPIFFS will crash/fail because SPIFFS area is blank as default. Need to create sketch with a call to SPIFFS.format() (and run it once). Of course you can avoid this problem by using upload tool as shown in video.
Thanks for your input. I changed the ESP_loader in IOTappstory.com accordingly.
Great job, thank you
You are welcome!
On ESP8266 I just use MicroPython and there's already a file system.
+TheKetsa :-)
nice video!
:-)
hello thank you for this tutorial :)
I have a problem, there is this message that appears: "Failed to open config file
" could someone help me please. Thanks.
(esp8266MOD)
Unfortunately, I cannot do remote debugging.
replace "LittleFS.open" with "SPIFFS.open", also "LittleFS.begin" with "SPIFFS.begin"
Can you provide the sketch?
As you see in the video I used the example file in the Arduino IDE
Amazing tutorial. Sir, I'm a beginner to this field. Can you please tell me how to create .jar file???
This is not a simple question. You have to google for tutorials
@@AndreasSpiess okay. Thank you Sir.
@@AndreasSpiess Sir, which one is better EEPROM or SPIFFS ?? I want to store ssid and password in the flash memory of ESP8266 for my Home Automation Project.
for SSID/PW EEPROM is probably easier
@@AndreasSpiess Thank you Sir
Excellent
Thank you!
Thank you!