Please, please, make more videos. Change nothing, no intros, no fancy graphics. I'm not into electronics, but your voice is just spectacular, and keeps me attentive.
I will try. Not promising much. It's not like I've done a lot of videos, so I don't have particular style. To the extent that I think about it, I have mostly wanted the graphics to serve some kind of purpose. Most just pointing/indicating in some way. Or giving something hopefully okay to look at when it's just dialogue. The voice comment is nice. The audio track is heavily edited for ums and ahs and misspeaking, but I haven't done any fancy audio filtering or anything. It's about all I can handle now anyway.
So you've ("only") edited out umms and ahs, but "haven't done any fancy audio filtering".... Well that's admirable priorities. I can hear what you say well enough, and WHAT YOU SAY is easy to follow. Some would add a distracting music track to make it purdyer, and not have time to hightlight things in graphics. Or go on that tangent about the 4'th allocation. You keep it tight, and clear, and actually audioVISUAL. You have a style even if you're not aware of it. More please! (Maybe do another about functions etc that you prudently didn't cram into this video?)
This is very good advice, and we follow it in our big "Marlin" sketch to help ensure it can run on smaller AVR targets. We do need to build strings for some purposes, and for that I made my own string class that uses the stack, pre-defines the storage size, and uses sprintf or snprintf as preferred to deal with safe and unsafe copy situations. We also made our own serial string template functions for convenience which take multiple arguments and decompose them into separate print calls, replacing those plus signs with commas, basically. The cool thing there is we can wrap up float in a structure along with the precision to get a float-with-precision print, wrap a char in a struct to distinguish it from an int, or extend that concept to deal with any other printable type that we need. Templates can be a hassle to learn and make, but they can definitely help with reducing the number of lines you eventually have to type in a large project.
You bring up many truths, and indeed state these problems aren't so much of a problem on a bigger system such as an esp32. I feel this is a throwback from the original Arduino String/heap mismanagement bugs, which are now fixed. Seems to me your only leg-up is smaller code size, meaningful on the old Arduino. Heap and stack management is as important as all memory management. If as you said smashing out high speed serial to another device, of course you would consider string concatenation a slow down on an 8 or 16mhz device. Personally, I don't worry about it on ESP32. As for Arduino, I follow what, as you say use String() as little as possible.
Yeah, that is pretty close to how I think about it. And it is _mostly_ the usage on AVR-based Arduinos that motivated the video. More specifically, me helping very frustrated people and this ultimately being the source of their frustration. And you're right, it wouldn't have hurt to mention that on a more powerful device you're aren't going to have trouble feeding the UART. I scrapped the entire original video (twice I think) because I gave so many qualifications/caveats to what I was saying that I had a hard time imagining anyone putting up with it. It's been through at least three titles and had two thumbnails that I eventually removed because I thought they were too misleading for that same reason. I had to mention larger targets so it didn't sound quite so hyperbolic after cutting a lot of material. As it is is some probably still think its too long. But, I hope I'm at least putting enough nuance into it to be raising the average. On an ESP8266, I have seen people cobble together a large enough String object to cause a problem. Though they do it with a String object, a loop, and += with the end goal of passing their giant String to a single .print() call. Often this is for something like constructing an HTML formatted payload of logged data in an http server callback. So it doesn't quite manifest as the one-liner that's on screen. But the underlying thing of "you should just call .print() multiple times instead of concatenating in memory" is still in play there. From reading what you wrote it doesn't sound like you'd do that, but then I would guess you probably already knew what there was to learn in the video anyway. I don't remember yet having that problem on an ESP32, but all it would really take is a data log that was about three times larger. I'm sure someone will do it. =)
I've been using Arduinos for well over 10 years but mostly kept to higher level concerns and haven't really dug into compiler optimization quirks like this, so I really appreciate easily implementable tips like this!
People are so hyper focused on just doing something and getting results, they often failed to learn about the underlying side effects of their choices. Thanks to the cheap cost and high availability of technology in our modern world, most worry not about the underlying processes. This gives us a world of poorly written and often never optimized code waiting to cause a bottleneck of failure. You sir are bringing back the art of knowledge this generation needs. Please continue.
On real hardware, building the string first will delay output of the first character until the entire string is built, multiple prints to buffered serial won't. This applies to any platform implementation reasonably likely. Wasted cycles matter if your code does something else with those cycles, like capturing a waveform or bit banging the 1-wire bus to a temperature sensor.
Thanks for pointing out this better way, and the reasons to do it. I usually have done the concatenation. Now, to be a smart (or dumb) a??, the code is really efficient at getting to the point where the CPU blocks due to being I/O bound! Sorry, I couldn't resist. I like your points for all the right reasons.
I guess I have two comments. First, the memory and hardware considerations of using the dynamic memory in the string concatenations, and the optimization aspect of not using the string concatenantion. First, while most virtualized languages like Java have some gains from using memory like this, hardware based languages like C have larger detriments. I think the confusion that leads to the reason for this video is that greener programmers fail to understand the fundamental purposes of a given language are and how those differ between different languages. C is a lower level language that is designed so that software can be optimized for leaner execution, just like how JavaScript is intended to be used on client side devices or FORTRAN is intended to be used in maths heavy tools. Second, you give a great example of why optimization matters when designing software for embedded hardware. You cite that your example is very IO bound, which is true but also applies to similar operations such as recording data into a database. In this case the extra execution time could get in the way of recording timely data reliably, I've had cases where the overhead of concatenating on the fly like this has caused measurements to be missed due to the program taking just that much longer to put the log message together.
I also want to say that I love finding more creators talking about the more fundamental aspects of design and engineering. Especially in the areas of software! 🤓
C is NOT a "lower level" language. It's a "memory-oriented" and "undefined behavior-oriented (yeah)" language. Every object, be it code or variable, have size and address, and you can do whatever you want with that. But there is certain things, like overflowing signed integers or touching already dead object like variable or malloced memory, that by definition is undefined, thus allowing compiler to optimize the hell out of your code if you know what are you doing and not treating types and variable lifetime like its something that C-coders using to punish themselves.
@@rogo7330 When I said "lower level" language, I was referring to the fact that the C language is intended to be written for and compiled against hardware targets that are typically not a "full computer", i.e. some type of embedded device. I mean, you won't go bit-banging in a language such as Java or Perl or Rust, but you do in C. Also, do you know what "malloc" is? Because how you used it is backwards from what it does, "memory allocation". And, issues like overflowing values or interacting with a "dead" object are considered bad design choices and reflect on the skill of the programmer over any failings of the language. One of the specific reasons that things like "best practices" and "proper design" are a thing. Ya know, people go to school for that. A "C-coder", as you call it is what I grew up calling a "script-kiddy" - someone that takes a bit of code and copy-paste uses it without taking the time to understand what it does and how it does it. That is the difference between a Software Engineer and a novice. Take the time to learn and understand what the code is doing and why, then you can optimize and head for the O(n). That's a lesson I had to learn thru failure and time, and I think that anyone can follow that (in part) and become a great Software Engineer.
C++ on micrpcontrollers should have never been a thing. If we are limited to C then issues like this would mostly solve themselves because people would need to be more aware of the memory they are using/alocating
The fmt library uses a (fixed size) buffer on the stack while the string is small enough, should probably compile fine for the arduino and you can adjust the buffer size so you don't run into a stack overflow.
I have never used String in arduino code, and never had an aesthetic issue with using multiple Serial.print() calls, but if I had to, I'd probably go the rout of making the Serial.print() function return the Serial object, so that calls could be chained with a dot, like Serial.print().print().println(); or make a class wrapper that produces something similar to the cout
Optimizations are limited to cases where the code above and below behaves exactly the same which isn't the case here. The "inefficient string" calls Serial.println() with the entire serial string and as a result the entire string will be dumped onto the serial all at once, the code below doesn't guarentee this, it may dump the latter parts of the strings slightly later (as in milliseconds later). In cases like these, the compiler has to behave like the programming language spec requires, and what the code is saying according to this spec is "build this string, then call println."
Uhh, It's complicated. It's a somewhat big monolithic beast so far the compiler optimizer is concerned in large part because of the runtime parsing of its format string. For most Arduino boards you can complete the stdout implementation to allow normal printf. And you can make a buffer and snprintf into it and then Serial.println that. People are often surprised at the large jump in code size that happens on the first use of printf. Once you've used it somewhere you've kind of already payed the cost in code size. If you _really_ make use it, it may be worth it that way. Some of the smaller ATtiny probably don't have the flash space for the function at all. And printf is also really easy to misuse. So, it's a mixed bag. Normal (f)printf and snprintf don't do dynamic allocation though. So that much is helpful. Another thing to note is that avr-libc printf family as its compiled in the AVR board support doesn't have floating point. Not that you should really be using float broadly, but if you are and have decided to printf, you still can't use the floating point format specifiers on an UNO for example.
danke, can you discuss this: when i have a pre-concatenated string, and use serial1.println to transfer data between two mega2560, will there be the same effect you have described in your video, danke
Kann ich dir erklären: The memory allocation occurs when concating any string anywhere. String and number literals ("Literal") are part of the binary file that is completely loaded in memory once when the program starts. When you add 2 literals together there needs to be room for the 2 together in 1 contiguous region of memory. Since the 2 literals aren't likely next to each other a block of memory large enough for both together is allocated on the heap (dynamic memory) and the 2 literals are copied to the new memory. Meaning that yes the memory is wasted once you pre-concatenate the strings unnecessarily. Overall dynamic strings of text are the number 1 memory consumer in all of programming. Which is the reason that so many optimization tactics exist for them.
5:35 that's a bold assumption. I'd guess that the bottleneck of transmitting the data overshadows the performance loss of allocating and deallocating the additional memory in many cases
I'm not really that much into programming on arduinos, but what if I would concatenate the strings together into another string variable and then print out the resulting string variable , would that still be a memory hog ?
Interesting as I am guilty of this as well. I assumed that putting it all in one line would be more efficient but I never bothered to validate my assumption. Apparently, a shameful display 😁 Thanks for clearing that up.
You're totally right! So many websie try to teach us to use string in Arduino. I try to read 5 sensors data and translate to sring to send them by ENC28J60. When I try to combine the 5 sensor data to single one String, I met many strange error in Arduino IDE. I google for the reason, even ask for GPT, and found the String is a joke in Arduino. Now I have a question, can UDP.write can send the float data dierctly without convert to String?
I'd need to look at the specific library. A library that handles UDP will often expect just a block of characters though. You're right that it needn't be a String object or null terminated string or text. If both sides of the connection use ieee float in same size and same byte order you can .send((const float *)&my_float_var, sizeof my_float_var); Odds are you sensor doesn't produce floating point data but since fixed point integer format and it can make a lot more sense to just send that in the and manner as above. If you're on AVR and need to produce a temp c-string for a buffer you can try dtostrf(). You have a lot of options. Even if you stringize your data in a small array before putting them into the buffer you may still want to avoid float.
Thanks for reply. All of my 5 sensors data are float type. And I just use dtostrf to convert each to String. After that user String = String1 + String2 + String3...to combine them to one String. Finally sent this single String UDP data, which include all 5 sensors, to Server by LAN.@@TM-zs7ko
BMP180、SHT41、UV、Light sensor and ENC28J60@@TM-zs7ko . Now I've found the solution for this question. 1. use dtostrf to translate float into char array; 2 use strcpy and strcat to combine all sensors date into single char array; 3 send this new char array to LAN by ENC28J60. This way is more stable and easy than combine String for Arduino.
I wish we could fix this with macro, but afaik there is no way to loop through variadic arguments in C macro. We could use another custom pre-processor though.
Why couldn’t the compiler just split up the concatenation and do the prints individually? Is there situations where the concatenation would be necessary for the program to function?
It would be too specific of an "optimization" and it would change the behavior of the concatenation operator in certain circumstances. Additionally there are cases where concatenation has to be used in such ways.
@@Decimation001 being too specific isn't a reason to not do it, it should be possible to change it only in the case of print statements. That's what I was asking, in what cases would it not work?
@@conorstewart2214but it is too specific, it fundamentally changes what string is, once you roll out changes of such magnitude, you are creating a new language
@@avarise5607 but doing something like this isn't unheard of. For instance Java creates a stringBuilder object when the + operator is used more than once even taking loops into account. Especially with control over the compiler making a variadic print macro that is implemented by the compiler wouldn't be to far fetched for serious string optimization.
@@redcrafterlppa303 yea, but you dont use java to drive hardware directly and here you would. Especially on IC you want as little hidden mechanism as possible, especially since it convinience change with a lot of hidden costs
in C# you would do a template string or a string builder to make it more efficient... had not really thanked of this for Arduino programming though... i actually did something like this recently well now i know to do it better. duly noted.
In c# A template used directly in the print function with a params parameter is depending on the implementation similar to what multiple print statements do. A builder on the other hand creates the string as a whole in memory meaning that it suffers the same problem as the plus operator or string interpolation. A stringBuilder is simply a List under the hood. It's only better at concatenation than the plus operator because it grows with a factor instead of just adding room for the concatenated second string. But on devices that run c# code this level of performance is irrelevant most of the time.
I create a datatype that when I use serial print for debugging so I can just turn it off when I'm done or if I add features down the line. The compiler just skips it. I can't believe people use serial print like that. Doing it in python drives me nuts. I don't use python for uC projects haha
what if we concatenate it first? Suppose we already have a single String variable that holds a line of text. and then we link it next with the other text or variables. afterwards, we put it into the println. String str_data = "Result A : "; str_data += String(result_a); str_data += " | Result B : "; str_data += String(result_b); Serial.println(str_data);
avr-libc as it's compiled for the AVR doesn't support %f in printf. So, unfortunately for the people that this most relevant for they can't do that even if they use the fdev_setup stuff and supply their own callback. But yes, that's technically possible. Even if you may or may not need to recompile avr-libc and patch it into your Arduino support. My guess is anyone that knows how to do that won't learn much from this video anyway.
I rarely do embedded programming but hearing how bad the string situation is apparently it might be worth it to roll my own string implementation. On the other hand I assume that the rust community fixed those problems when they created the rust support for those embedded systems. So long live rust I guess 😂😅
Well, it's not _so_ bad if you have an easy to characterize usage pattern. Like if you do all your allocation dynamic allocation once early on, in main() or setup() (say according to configuration parameters) it's more reasonable, even on these controllers with just a few KB. But yeah, it's better if you can avoid it altogether. And with what I'm highlighting here there's just no need at all. I suppose if people are coding an MCU with only 2 KB of SRAM and they can only take away "heap is good" or "heap is evil" without more nuance, well, they're probably better served by "heap is evil."
@@TM-zs7ko I have to agree that the heap is not necessarily evil, but depending on the application unnecessary usage of the heap can introduce major debugging headaches. This was a good example of code that created hidden complexity just to make it pretty. On the Arduino there are resource constraints to consider. If using the heap can not be avoided I always match object creation and destruction then place the usage code between. This assists in memory management, particularly on larger systems which is where the majority of my experience lies.
I have never used Arduino because of its gobshite terminology and (in the early days) poor compiler. The world did not need to redefine perfectly well understood terms just to massage some developers ego. I write programs, I compile then, I flash them, I can choose to verify that the flash works. I do not write sketches, verify when I mean flash and run, have shields when I mean boards, all using a language that looks like unhappy offspring of a non consensual union between turbo pascal and an unwilling c++ compiler.
My reply started to become a novel, so this is really cut down: We're probably pretty similar that way. It took me a long time to get used to saying "sketch" to people. I still feel often the need to apply "arduino" to "library" to make clear when it's not a more conventional .a/.lib/.so/.dll/whatever or at least header-only library. "Sketch" is a holdover from Processing, where it was unnecessary, but at least made some sense thematically. After being brought through Wiring into Arduino it is still unnecessary. But at least it fairly unambiguously maps onto "Arduino program". I guess it has some value in that it sort of implies something more specific that just "program." or because it's shorter, I guess. But regardless of any rationale someone can give for the choice, there usage of "dynamic memory" memory seems to do people a disservice in a way that most of their other terms don't. Anyway, I try to steer people toward better tools/terms/thinking to the extent that they're will and interested and capable. Sometimes people are just trying to get their 3d printer to work and don't care about any of this. I try not to get worked up about these things if I can help it.
@@TM-zs7ko I get your point on "dynamic memory"... maybe they wanted to distinguish between read only memory and RAM in its many types, AKA 1970s speak, or maybe it is just careless wording. I have the same issue with "device"... when I design a system I always ending using "device" for multiple contexts, then realise that I am going to have change at least one context to a new word on confusion will rain. My dislike of Arduino may be partly I am not a fan of IDEs and partly I was working in embedded long before it came along. Now I have to compete in the commercial world with people who can chain a few library calls and make an LED flash. Arduino did for embedded much the same as Visual Basic did for coding - I am not fan, maybe it is just snobbery and my age ;-)
Adding strings to each other is uggly. It means you don't know wtf are you doing and "just want this two completly different to each other things became another completly different thing and I don't care how bad it is".
I really wanted to watch this, but listening to a list of what you won't do for 2 minutes of monotonic rambling made me click away. Don't tell me what you are not going to do. Simply go ahead and do the things you want to say/do/show.
probably someone already pointed it out, but the code you probably wanted people not to use was this: Serial.println("The temperature is " + String(t) + " degrees C") Since the String typecast should've been used in the float variable. And what about using printf?, do you like using it?
I'm not quite sure I get it. But maybe you're going for something subtle. You can write either of those and will end up with the save output and I believe nearly identical machinations at runtime.
@@TM-zs7ko When the complier deals with the code *String("The temperature is ") + t + " degrees C"* it will have to do an implicit cast of the numeric variable t to a string in order to do the concatenation. That produces a string, so the String() function isn't really doing anything at that point because the Serial.print() would implicitly convert its argument to a string anyway. The code that @pacsmile suggested does an explicit conversion of the numeric t to a string before concatenating it. Either way produces the same output - which is a good thing.
@@TM-zs7ko Yeah i mean, is there a reason why we should or shouldn't use: Serial.printf("The temperature is %f degrees ", t); instead of: Serial.println("The temperature is " + String(t) + " degrees C")
The voice adds a ludicrous amount of gravitas to this, it’s like being warned not to invade Poland 🫣 Useful stuff though, I’d have never thought about memory allocation for strings on a microcontroller but I guess it’s pretty important
This is cut down from what it was originally going to be, which was something like a half our long. In its original form it was less emphatic and a lot longer. Went into the alternatives more. It's also a recurring thing that I've had to help people with. I've kind of come to the conclusion that if I actually say everything I want to, the way I actually want to, nobody will watch anything. And if I take stuff out people ask why I didn't do it the other way and explain more.
@@TM-zs7ko I believe on YT everyone has an audience somewhere. You could always shoot the 2 hour version and then edit a 5 minute TL;DR and release both on the same day, people can choose what they want to watch and you'd get to feel like you've said everything you wanted to say (and being happy at what you are doing is just as important) 238K people watched 3 hours of Robert Feranec building a ESP32 board 😬 Probably a subset of those people watched him talk about RF reflection within a VIA stub, which is as obscure as it gets.
Please, please, make more videos. Change nothing, no intros, no fancy graphics. I'm not into electronics, but your voice is just spectacular, and keeps me attentive.
I will try. Not promising much. It's not like I've done a lot of videos, so I don't have particular style. To the extent that I think about it, I have mostly wanted the graphics to serve some kind of purpose. Most just pointing/indicating in some way. Or giving something hopefully okay to look at when it's just dialogue. The voice comment is nice. The audio track is heavily edited for ums and ahs and misspeaking, but I haven't done any fancy audio filtering or anything. It's about all I can handle now anyway.
Eh, I think the author could use a spoon of cough syrup.
@@sharpfangjealous of his wizened timbre now, aren't we
Yep
So you've ("only") edited out umms and ahs, but "haven't done any fancy audio filtering".... Well that's admirable priorities. I can hear what you say well enough, and WHAT YOU SAY is easy to follow.
Some would add a distracting music track to make it purdyer, and not have time to hightlight things in graphics. Or go on that tangent about the 4'th allocation.
You keep it tight, and clear, and actually audioVISUAL. You have a style even if you're not aware of it.
More please! (Maybe do another about functions etc that you prudently didn't cram into this video?)
This is very good advice, and we follow it in our big "Marlin" sketch to help ensure it can run on smaller AVR targets. We do need to build strings for some purposes, and for that I made my own string class that uses the stack, pre-defines the storage size, and uses sprintf or snprintf as preferred to deal with safe and unsafe copy situations. We also made our own serial string template functions for convenience which take multiple arguments and decompose them into separate print calls, replacing those plus signs with commas, basically. The cool thing there is we can wrap up float in a structure along with the precision to get a float-with-precision print, wrap a char in a struct to distinguish it from an int, or extend that concept to deal with any other printable type that we need. Templates can be a hassle to learn and make, but they can definitely help with reducing the number of lines you eventually have to type in a large project.
Didn't expect to find a Marlin dev in the comments, much less thinkyhead himself :D thank you for your work!
I would listen to a podcast of just you rambling about random coding things.
I love this kinda stuff. Keep it coming.
You bring up many truths, and indeed state these problems aren't so much of a problem on a bigger system such as an esp32. I feel this is a throwback from the original Arduino String/heap mismanagement bugs, which are now fixed. Seems to me your only leg-up is smaller code size, meaningful on the old Arduino. Heap and stack management is as important as all memory management. If as you said smashing out high speed serial to another device, of course you would consider string concatenation a slow down on an 8 or 16mhz device. Personally, I don't worry about it on ESP32. As for Arduino, I follow what, as you say use String() as little as possible.
Yeah, that is pretty close to how I think about it. And it is _mostly_ the usage on AVR-based Arduinos that motivated the video. More specifically, me helping very frustrated people and this ultimately being the source of their frustration. And you're right, it wouldn't have hurt to mention that on a more powerful device you're aren't going to have trouble feeding the UART.
I scrapped the entire original video (twice I think) because I gave so many qualifications/caveats to what I was saying that I had a hard time imagining anyone putting up with it. It's been through at least three titles and had two thumbnails that I eventually removed because I thought they were too misleading for that same reason. I had to mention larger targets so it didn't sound quite so hyperbolic after cutting a lot of material. As it is is some probably still think its too long. But, I hope I'm at least putting enough nuance into it to be raising the average.
On an ESP8266, I have seen people cobble together a large enough String object to cause a problem. Though they do it with a String object, a loop, and += with the end goal of passing their giant String to a single .print() call. Often this is for something like constructing an HTML formatted payload of logged data in an http server callback. So it doesn't quite manifest as the one-liner that's on screen. But the underlying thing of "you should just call .print() multiple times instead of concatenating in memory" is still in play there. From reading what you wrote it doesn't sound like you'd do that, but then I would guess you probably already knew what there was to learn in the video anyway. I don't remember yet having that problem on an ESP32, but all it would really take is a data log that was about three times larger. I'm sure someone will do it. =)
I've been using Arduinos for well over 10 years but mostly kept to higher level concerns and haven't really dug into compiler optimization quirks like this, so I really appreciate easily implementable tips like this!
People are so hyper focused on just doing something and getting results, they often failed to learn about the underlying side effects of their choices. Thanks to the cheap cost and high availability of technology in our modern world, most worry not about the underlying processes. This gives us a world of poorly written and often never optimized code waiting to cause a bottleneck of failure. You sir are bringing back the art of knowledge this generation needs. Please continue.
Subbed for presentation alone. All those RUclipsrs starting videos with “hey guys, how’s it going!?” should be sent to TM School 💁♂️
On real hardware, building the string first will delay output of the first character until the entire string is built, multiple prints to buffered serial won't. This applies to any platform implementation reasonably likely. Wasted cycles matter if your code does something else with those cycles, like capturing a waveform or bit banging the 1-wire bus to a temperature sensor.
Thanks for pointing out this better way, and the reasons to do it. I usually have done the concatenation. Now, to be a smart (or dumb) a??, the code is really efficient at getting to the point where the CPU blocks due to being I/O bound! Sorry, I couldn't resist. I like your points for all the right reasons.
I guess I have two comments. First, the memory and hardware considerations of using the dynamic memory in the string concatenations, and the optimization aspect of not using the string concatenantion.
First, while most virtualized languages like Java have some gains from using memory like this, hardware based languages like C have larger detriments. I think the confusion that leads to the reason for this video is that greener programmers fail to understand the fundamental purposes of a given language are and how those differ between different languages. C is a lower level language that is designed so that software can be optimized for leaner execution, just like how JavaScript is intended to be used on client side devices or FORTRAN is intended to be used in maths heavy tools.
Second, you give a great example of why optimization matters when designing software for embedded hardware. You cite that your example is very IO bound, which is true but also applies to similar operations such as recording data into a database. In this case the extra execution time could get in the way of recording timely data reliably, I've had cases where the overhead of concatenating on the fly like this has caused measurements to be missed due to the program taking just that much longer to put the log message together.
I also want to say that I love finding more creators talking about the more fundamental aspects of design and engineering. Especially in the areas of software! 🤓
C is NOT a "lower level" language. It's a "memory-oriented" and "undefined behavior-oriented (yeah)" language. Every object, be it code or variable, have size and address, and you can do whatever you want with that. But there is certain things, like overflowing signed integers or touching already dead object like variable or malloced memory, that by definition is undefined, thus allowing compiler to optimize the hell out of your code if you know what are you doing and not treating types and variable lifetime like its something that C-coders using to punish themselves.
@@rogo7330 When I said "lower level" language, I was referring to the fact that the C language is intended to be written for and compiled against hardware targets that are typically not a "full computer", i.e. some type of embedded device. I mean, you won't go bit-banging in a language such as Java or Perl or Rust, but you do in C.
Also, do you know what "malloc" is? Because how you used it is backwards from what it does, "memory allocation". And, issues like overflowing values or interacting with a "dead" object are considered bad design choices and reflect on the skill of the programmer over any failings of the language. One of the specific reasons that things like "best practices" and "proper design" are a thing. Ya know, people go to school for that. A "C-coder", as you call it is what I grew up calling a "script-kiddy" - someone that takes a bit of code and copy-paste uses it without taking the time to understand what it does and how it does it. That is the difference between a Software Engineer and a novice. Take the time to learn and understand what the code is doing and why, then you can optimize and head for the O(n). That's a lesson I had to learn thru failure and time, and I think that anyone can follow that (in part) and become a great Software Engineer.
C++ on micrpcontrollers should have never been a thing. If we are limited to C then issues like this would mostly solve themselves because people would need to be more aware of the memory they are using/alocating
Very interesting, wouldn't have thought something like this matters that much.
Thank you.
The fmt library uses a (fixed size) buffer on the stack while the string is small enough, should probably compile fine for the arduino and you can adjust the buffer size so you don't run into a stack overflow.
What is this simulator
Please make a video about it
Huh, a great tip on a topic that I’d never thought about before. Thanks!
I can't believe I have to worry about malloc in arduino, I never thought about this. Oh boy, I'm in for a ride.
I'll take your advice, thanks!!!
I wonder what the instructions look like in comparison (block by block) just to see step by step how that pans out.
Wait, does avr-gcc support templates ?
I have never used String in arduino code, and never had an aesthetic issue with using multiple Serial.print() calls, but if I had to, I'd probably go the rout of making the Serial.print() function return the Serial object, so that calls could be chained with a dot, like Serial.print().print().println();
or make a class wrapper that produces something similar to the cout
why doesn't the compiler optimize this out of the code?
Optimizations are limited to cases where the code above and below behaves exactly the same which isn't the case here. The "inefficient string" calls Serial.println() with the entire serial string and as a result the entire string will be dumped onto the serial all at once, the code below doesn't guarentee this, it may dump the latter parts of the strings slightly later (as in milliseconds later).
In cases like these, the compiler has to behave like the programming language spec requires, and what the code is saying according to this spec is "build this string, then call println."
How about using printf?
Uhh, It's complicated. It's a somewhat big monolithic beast so far the compiler optimizer is concerned in large part because of the runtime parsing of its format string. For most Arduino boards you can complete the stdout implementation to allow normal printf. And you can make a buffer and snprintf into it and then Serial.println that. People are often surprised at the large jump in code size that happens on the first use of printf. Once you've used it somewhere you've kind of already payed the cost in code size. If you _really_ make use it, it may be worth it that way. Some of the smaller ATtiny probably don't have the flash space for the function at all. And printf is also really easy to misuse. So, it's a mixed bag. Normal (f)printf and snprintf don't do dynamic allocation though. So that much is helpful. Another thing to note is that avr-libc printf family as its compiled in the AVR board support doesn't have floating point. Not that you should really be using float broadly, but if you are and have decided to printf, you still can't use the floating point format specifiers on an UNO for example.
LMAO “don’t do this,” then immediately shows code that looks suspiciously similar to mine ;,((
danke, can you discuss this: when i have a pre-concatenated string, and use serial1.println to transfer data between two mega2560, will there be the same effect you have described in your video, danke
Kann ich dir erklären:
The memory allocation occurs when concating any string anywhere. String and number literals ("Literal") are part of the binary file that is completely loaded in memory once when the program starts.
When you add 2 literals together there needs to be room for the 2 together in 1 contiguous region of memory. Since the 2 literals aren't likely next to each other a block of memory large enough for both together is allocated on the heap (dynamic memory) and the 2 literals are copied to the new memory.
Meaning that yes the memory is wasted once you pre-concatenate the strings unnecessarily.
Overall dynamic strings of text are the number 1 memory consumer in all of programming. Which is the reason that so many optimization tactics exist for them.
5:35 that's a bold assumption. I'd guess that the bottleneck of transmitting the data overshadows the performance loss of allocating and deallocating the additional memory in many cases
You seem to agree. So I think you may have missed something.
I'm not really that much into programming on arduinos, but what if I would concatenate the strings together into another string variable and then print out the resulting string variable , would that still be a memory hog ?
The issue is with the concatenation requiring temporary intermediate string variables, so the answer should be yes.
Interesting as I am guilty of this as well. I assumed that putting it all in one line would be more efficient but I never bothered to validate my assumption. Apparently, a shameful display 😁
Thanks for clearing that up.
You're totally right! So many websie try to teach us to use string in Arduino. I try to read 5 sensors data and translate to sring to send them by ENC28J60. When I try to combine the 5 sensor data to single one String, I met many strange error in Arduino IDE. I google for the reason, even ask for GPT, and found the String is a joke in Arduino. Now I have a question, can UDP.write can send the float data dierctly without convert to String?
I'd need to look at the specific library. A library that handles UDP will often expect just a block of characters though. You're right that it needn't be a String object or null terminated string or text. If both sides of the connection use ieee float in same size and same byte order you can .send((const float *)&my_float_var, sizeof my_float_var); Odds are you sensor doesn't produce floating point data but since fixed point integer format and it can make a lot more sense to just send that in the and manner as above. If you're on AVR and need to produce a temp c-string for a buffer you can try dtostrf(). You have a lot of options. Even if you stringize your data in a small array before putting them into the buffer you may still want to avoid float.
Thanks for reply. All of my 5 sensors data are float type. And I just use dtostrf to convert each to String. After that user String = String1 + String2 + String3...to combine them to one String. Finally sent this single String UDP data, which include all 5 sensors, to Server by LAN.@@TM-zs7ko
@@atlaslin1602 Your reply got held for some reason. Out of curiosity, what sensor is this?
BMP180、SHT41、UV、Light sensor and ENC28J60@@TM-zs7ko . Now I've found the solution for this question. 1. use dtostrf to translate float into char array; 2 use strcpy and strcat to combine all sensors date into single char array; 3 send this new char array to LAN by ENC28J60. This way is more stable and easy than combine String for Arduino.
I wish we could fix this with macro, but afaik there is no way to loop through variadic arguments in C macro. We could use another custom pre-processor though.
That's what templates were made for.
I think rust macros can do that. Just saying xd
And yes rust support for embedded is there and it's only getting better.
Why couldn’t the compiler just split up the concatenation and do the prints individually? Is there situations where the concatenation would be necessary for the program to function?
It would be too specific of an "optimization" and it would change the behavior of the concatenation operator in certain circumstances. Additionally there are cases where concatenation has to be used in such ways.
@@Decimation001 being too specific isn't a reason to not do it, it should be possible to change it only in the case of print statements.
That's what I was asking, in what cases would it not work?
@@conorstewart2214but it is too specific, it fundamentally changes what string is, once you roll out changes of such magnitude, you are creating a new language
@@avarise5607 but doing something like this isn't unheard of. For instance Java creates a stringBuilder object when the + operator is used more than once even taking loops into account.
Especially with control over the compiler making a variadic print macro that is implemented by the compiler wouldn't be to far fetched for serious string optimization.
@@redcrafterlppa303 yea, but you dont use java to drive hardware directly and here you would. Especially on IC you want as little hidden mechanism as possible, especially since it convinience change with a lot of hidden costs
This is just symantics.
But ...I don't see how saying
"Print Lin" is any easier than saying
"Print Line" ...which is what "println"
stands for ......
Just use `printf` instead?
Me using static buffers and sprintf on an ESP with oodles of ram…. Memory safe and beautiful!
Basically doing manual manual heap management lmao
in C# you would do a template string or a string builder to make it more efficient... had not really thanked of this for Arduino programming though... i actually did something like this recently well now i know to do it better. duly noted.
In c# A template used directly in the print function with a params parameter is depending on the implementation similar to what multiple print statements do. A builder on the other hand creates the string as a whole in memory meaning that it suffers the same problem as the plus operator or string interpolation.
A stringBuilder is simply a List under the hood. It's only better at concatenation than the plus operator because it grows with a factor instead of just adding room for the concatenated second string.
But on devices that run c# code this level of performance is irrelevant most of the time.
I create a datatype that when I use serial print for debugging so I can just turn it off when I'm done or if I add features down the line. The compiler just skips it.
I can't believe people use serial print like that. Doing it in python drives me nuts. I don't use python for uC projects haha
Please make more videos! Please!!!!
I hope produce something new at least, probably related to this video. We'll see.
what if we concatenate it first? Suppose we already have a single String variable that holds a line of text. and then we link it next with the other text or variables. afterwards, we put it into the println.
String str_data = "Result A : ";
str_data += String(result_a);
str_data += " | Result B : ";
str_data += String(result_b);
Serial.println(str_data);
That would not help because the extra code size and cycles are caused by concatenation and growing of one string
@@nickfarley2268 I see. Thanks for the answer!
string_view?
lol i used STRING BUFFER to print bulk string output in terminal like temperature , humidity is %d , %d lol {sprintf}
9:12 ln missing in the define ,)
That is true. Maybe I screwed up something during editing while cutting down the full file. Either way, you're right.
tldr, I avoid Arduino's String class like the plague.
printf("The temperature is %.2f degrees C.
",read_temperature());
avr-libc as it's compiled for the AVR doesn't support %f in printf. So, unfortunately for the people that this most relevant for they can't do that even if they use the fdev_setup stuff and supply their own callback. But yes, that's technically possible. Even if you may or may not need to recompile avr-libc and patch it into your Arduino support. My guess is anyone that knows how to do that won't learn much from this video anyway.
I find that strings in general cause instability.
I rarely do embedded programming but hearing how bad the string situation is apparently it might be worth it to roll my own string implementation.
On the other hand I assume that the rust community fixed those problems when they created the rust support for those embedded systems. So long live rust I guess 😂😅
heap is evil
Well, it's not _so_ bad if you have an easy to characterize usage pattern. Like if you do all your allocation dynamic allocation once early on, in main() or setup() (say according to configuration parameters) it's more reasonable, even on these controllers with just a few KB. But yeah, it's better if you can avoid it altogether. And with what I'm highlighting here there's just no need at all. I suppose if people are coding an MCU with only 2 KB of SRAM and they can only take away "heap is good" or "heap is evil" without more nuance, well, they're probably better served by "heap is evil."
@@TM-zs7ko I have to agree that the heap is not necessarily evil, but depending on the application unnecessary usage of the heap can introduce major debugging headaches.
This was a good example of code that created hidden complexity just to make it pretty. On the Arduino there are resource constraints to consider.
If using the heap can not be avoided I always match object creation and destruction then place the usage code between. This assists in memory management, particularly on larger systems which is where the majority of my experience lies.
I have never used Arduino because of its gobshite terminology and (in the early days) poor compiler. The world did not need to redefine perfectly well understood terms just to massage some developers ego. I write programs, I compile then, I flash them, I can choose to verify that the flash works. I do not write sketches, verify when I mean flash and run, have shields when I mean boards, all using a language that looks like unhappy offspring of a non consensual union between turbo pascal and an unwilling c++ compiler.
My reply started to become a novel, so this is really cut down:
We're probably pretty similar that way. It took me a long time to get used to saying "sketch" to people. I still feel often the need to apply "arduino" to "library" to make clear when it's not a more conventional .a/.lib/.so/.dll/whatever or at least header-only library.
"Sketch" is a holdover from Processing, where it was unnecessary, but at least made some sense thematically. After being brought through Wiring into Arduino it is still unnecessary. But at least it fairly unambiguously maps onto "Arduino program". I guess it has some value in that it sort of implies something more specific that just "program." or because it's shorter, I guess. But regardless of any rationale someone can give for the choice, there usage of "dynamic memory" memory seems to do people a disservice in a way that most of their other terms don't.
Anyway, I try to steer people toward better tools/terms/thinking to the extent that they're will and interested and capable. Sometimes people are just trying to get their 3d printer to work and don't care about any of this. I try not to get worked up about these things if I can help it.
@@TM-zs7ko I get your point on "dynamic memory"... maybe they wanted to distinguish between read only memory and RAM in its many types, AKA 1970s speak, or maybe it is just careless wording. I have the same issue with "device"... when I design a system I always ending using "device" for multiple contexts, then realise that I am going to have change at least one context to a new word on confusion will rain. My dislike of Arduino may be partly I am not a fan of IDEs and partly I was working in embedded long before it came along. Now I have to compete in the commercial world with people who can chain a few library calls and make an LED flash. Arduino did for embedded much the same as Visual Basic did for coding - I am not fan, maybe it is just snobbery and my age ;-)
Adding strings to each other is uggly. It means you don't know wtf are you doing and "just want this two completly different to each other things became another completly different thing and I don't care how bad it is".
I really wanted to watch this, but listening to a list of what you won't do for 2 minutes of monotonic rambling made me click away. Don't tell me what you are not going to do. Simply go ahead and do the things you want to say/do/show.
The person who seeks the why, without understanding the why not, is a terrible, and ignorant developer.
probably someone already pointed it out, but the code you probably wanted people not to use was this:
Serial.println("The temperature is " + String(t) + " degrees C")
Since the String typecast should've been used in the float variable.
And what about using printf?, do you like using it?
You mean vs Serial.println(String("The temperature is ") + t + " degrees C"); ?
@@TM-zs7ko yes
I'm not quite sure I get it. But maybe you're going for something subtle. You can write either of those and will end up with the save output and I believe nearly identical machinations at runtime.
@@TM-zs7ko When the complier deals with the code *String("The temperature is ") + t + " degrees C"* it will have to do an implicit cast of the numeric variable t to a string in order to do the concatenation. That produces a string, so the String() function isn't really doing anything at that point because the Serial.print() would implicitly convert its argument to a string anyway. The code that @pacsmile suggested does an explicit conversion of the numeric t to a string before concatenating it. Either way produces the same output - which is a good thing.
@@TM-zs7ko
Yeah i mean, is there a reason why we should or shouldn't use:
Serial.printf("The temperature is %f degrees
", t);
instead of:
Serial.println("The temperature is " + String(t) + " degrees C")
The voice adds a ludicrous amount of gravitas to this, it’s like being warned not to invade Poland 🫣 Useful stuff though, I’d have never thought about memory allocation for strings on a microcontroller but I guess it’s pretty important
This is cut down from what it was originally going to be, which was something like a half our long. In its original form it was less emphatic and a lot longer. Went into the alternatives more. It's also a recurring thing that I've had to help people with. I've kind of come to the conclusion that if I actually say everything I want to, the way I actually want to, nobody will watch anything. And if I take stuff out people ask why I didn't do it the other way and explain more.
@@TM-zs7ko I believe on YT everyone has an audience somewhere. You could always shoot the 2 hour version and then edit a 5 minute TL;DR and release both on the same day, people can choose what they want to watch and you'd get to feel like you've said everything you wanted to say (and being happy at what you are doing is just as important)
238K people watched 3 hours of Robert Feranec building a ESP32 board 😬 Probably a subset of those people watched him talk about RF reflection within a VIA stub, which is as obscure as it gets.