Code vs Data (Metaprogramming) - Computerphile
HTML-код
- Опубликовано: 15 сен 2024
- Why are code and data so separate? Robert Smith of Rigetti Quantum Computing explains how he uses Lisp code to generate Lisp data which he can use as Lisp code...
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottsco...
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com
Linear algebra has this same kind of duality between data and function. A matrix is just a collection of data in most cases but when you multiply a matrix by some other matrix or a vector the matrix acts as a transform or a "function/code" on the data that passes through it. A change of basis is a transform matrix that acts as a function while at the same time could be further transformed by rotations and scaling matrices and act again as just data.
Think of meta programming as inventing something new like multiplication or even pluss in math prior to those things being invented.
Or any other new concept.
For instance, there was a time when no one had invented zero.
Math at the time had to be done with numbers with a value since there was no zero.
In a programming language with metaprogramming you can invent such a new concept as zero, define it and use it instead of defining it every single time you need to use the concept.
Well, almost. A matrix is a *representation* of an operator relative to the given basis.
Love this comment
Please learn about geometry algebra
@@OverThinkKS
Linear algebra taught me to like math
Dude, please start a programmer's channel with a LOT more code. These videos are a tease.
Agreed. Show me some code.
If you want more code, check out "The Coding Train" channel ;-)
Its... Javascript... but there you go...
aah, that channel is so fun
Yes Daniel Shiffman is very cool. But hate JS are there any other channels which are not JS.
Yes and JS is very easy to understand. The problem arises when JS is used interchangeably with node and starts doing weird things.
@2:52: *Reversed reverse Polish Notation* or as sane people would call it... polish notation ;-)
@science no it's at 2:50
@Sourav Goswami - Thanks. Fixed it.
Or as people who prefer descriptive rather than arbitrary names for things, "prefix notation".
"So could you give us an example?"
"Oh sure, definitely. So lets say you wanted to draw an owl. First you just draw some circles. Right? Then you draw the rest of the owl, which I wont do here because its obvious."
as i understand it:
first, a function is a set of instructions.
second, we can create data during runtime.
third, we can interpret data as instructions.
fourth, we can run interpreted data as a function.
you created code at run time.
the program wrote parts of itself.
Andrew McAuliffe you can base code on input from the user if you desire so
The way he described it, metaprogramming is the same as creating a library, which doesn't sound right.
ElagabalusRex you can create libraries of syntactic abstractions. That’s part of the point. :)
Kris2510 great, written out example!
@5:20 - I've been there, too. Writing the loop in assembly and forgetting to INC the counter. Those days the only cure was rebooting your pc.
I love listening to this guy talk, I don't know why
This duality between data and code is much more common than most people think. One very simple example: most of us view a .c file as code. But to the compiler, it's data.
Exactly. The very existence of compilers and interpreters relies on that duality.
But can a C program evaluate, change, compile, and assemble it's own data at runtime?
I don't know, all this meta & functional programming seem so powerful because you can allegedly write compact code.
The problem I see is the same as with Perl: Can a third party actually decipher that code, or do we have to add 10 lines of comments for every line of code?
Well, you don't really have to decipher it. At least not any average Joe has to. This technique lets you develop your own DSL (domain specific language) which will be really intuitive to its users.
You can leave the writing of the actual interpreter to people that have the necessary skills to understand it.
@Arkhitekton Idk. So if I let users extend my program by letting them write plugins in a DSL is this already seen as meta-programming? What's the boundary - compilation vs. interpretation? Or has it more to do with Harvard vs. von Neumann architecture?
Can you give me a practical example?
Ever used CSS or JavaScript? You had the opportunity to use a DSL there. When you write "nav > ul li.top:hover" you are abstracted away from all of the details of how the interpretation of that line works. It is interpreted for you by some interpreter in the browser, and yet you know exactly what that line does.
This kind of support is quite hard to do in languages that do not support metaprogramming. It is not impossible in any way, but it does require you to spend more time on building up the scaffolding to make it work.
@@LordShigi yesss when I see ppl say "you can't do this in c/c++" I'm always like uhhhh yeah u can XD
Really nice broad pass over metaprogramming 👍
I remember back in the early 1980s, when programmable calculators starting blurring the lines between computers and calculators, I read a quote (which I'm sorry I don't recall the author of) which said that the difference between calculators and computers is that the latter can operate on their own programs as if they were data and vice-versa, whereas the former cannot.
I want more from Robert Smith. This guy is good.
"reverse RPN" is not Reverse Reverse PN, it is just PN. Polish notation is like + 1 2 or add 1 and 2, RPN is 1 2 +. Polish notation was specifically prefix notation, which was considered more natural than 1+2 because you can write +1 * 2 3 to mean 1+(2*3), and indeed, this is a common notation used in compiler intermediate codes. However, RPN is considered even more natural than this because 2 3* 1+ exactly expresses operands and the operators that use them in proper order. Thus get 2, get 3, multiply, then get 1, and then add to the result of 2*3. It also is stack machine friendly.
If there never were such a thing as polish notation, I'd agree that reverse RPN makes sense, but polish notation was a thing before RPN.
Oh, dear. The comment-formatting system has gotten tripped up by your example and parsed the asterisks for multiplication as instructions to make the text bold.
Thanks! Fixed it.
I really wish they'd just use a subset of HTML or pseudo-HTML for the formatting. These errors would be so much less likely.
Fun fact: ".. prefix notation is similar to postfix notation such as RPN." Beginning with the HP 48 calculator instead of using RPN the manual refers to RPL which is not defined. A search on the web gives the unofficial acronym of "Reverse Polish LISP" which it is. If i enter "{1 2 3 * +}" and press ENTER that program is placed on the stack. Pressing EVAL then evaluates the expression leaving "7" on the stack. Notice how the parens dissapear in RPN/RPL. There is no PROGRAM/RUN mode switch on this machine; everything is eval mode and is placed on the stack!
To me that for-loop example seems very similar to defining a function that takes other functions as its arguments, though perhaps with a nicer syntax. Would love some more in-depth examples.
the for loop takes a variable name, the starting value, the ending value, the increment and the body of code to be executed with each iteration. Can you make a function that you give as input a body of code? Can you bend the syntax of the language you use in any way you see fit?
Is the standard C++ loop...
The right example in C++ for metaprogramming are Virtual classes / methods
Forth is another wonderful language that still stands the test of time. This vid mentions Postscript, which is very much a variant of Forth.
Am I correct to say that Lisp as he describes it is better off for math problems while C is better off for dealing with memory... cause that makes scenes why you would rather use Lisp over C for something like Quantum problems as stated in last video
All abstract problems, while C is better for memory as you said
Would be interesting to see a clarification of how this topic compares to compilers in general -- because, of course, compilers are programs that write programs. I was waiting for the comparison and distinction to be made in this video, but it never happened.
I'd love to see someone debug a metaprogram that contains thousands of lines of business logic :)
During my job in a company i find a procedure that was modified costantly in 4 years ( a T-SQL store procedure) that has around 3000 rows with some part of code that write code that was executed...
It takes about 1 week to under stand it...and about 1 time over 2 month it need to be modified...The source of this madness is called SAP
A lot better than the Java mess.
Code as Data is more than just meta-programming, is about symbolic programming, and metaprogramming is one way of doing SP, there is also term rewrite and other stuff.
I will remain indebted for sparking my interest in Lisp.
I'm getting all warm and fuzzy when I see the _LISP_ syntax. Brings back good memories from when I used it with AutoCAD.
4:23 That’s what a compiler for a high-level language does. The original name for this concept, from the 1950s, was “automatic programming”. At that time, the high-level language source was not considered the “program”: that name was reserved for the actual machine code that the computer could interpret.
Duality: Tom Scott or John Green?
RE now I can’t unsee it 😂
Trick question, Griffin McElroy
Arguably better than both
Tom Green.
Or is is John Scott?
I did not really understand this, he seems to be skipping some important details in his explanation?
I mean, it's obvious what he's saying. I don't understand what you're finding difficult about it.
How different is that from ANY other interpreter? Or writing instructions directly to memory updating code on fly?
It's not. Most languages have some kind of 'eval' feature.
yeah i also don't understand what is so special about it.
It is a more abstracted way to modify code. You are modifying the code at an Abstract Syntax Tree level. (And with abstraction can come more power.)
TakanashiYuuji 0ⁿ
The special part is, that the program doesn't have to be recompiled by the programmer to introduce new features, formally this is called "at runtime". Simply speaking your program is conjuring itself while it's running, and doesn't come with all the necessary code (as defnined in the video) but instead it has the data necessary to create the code.
A popular counterexample would be C, which requires you to compile it for the given architecture before executing it. C cannot add (for example) a function that was passed from the keyboard as C-code, despite being valid C code, and it cannot be executed at runtime.
If you're looking for an "interpreted" language, you might find Java to be a suitable example (I know it's technically compiled, but it's also interpreted at the same time, so there's that). Although Java may modify the JVM runtime however it likes, changing the functionality of a method/function is prohibited, which severely limits its at-runtime capabilities (in practice it's possible but produces very patternized code and complex structures)
this idea that has been championed by lisp was how the original Assembly programmers did the code, they wrote Assembly to write Assembly because RAM was very expensive in the past. However when computers started to get cheaper, then people started to get lazy and try to achieve the dream of a programming language that looks like English.
More Robert! :D
I want to look at the video teased at the end, "fighting obsolescence", but can't find it anywhere. Anyone got a link?
Great video! I enjoyed the explanations and written examples. Thanks Mr. Robert Smith. 😊
Could have introduced defmacro since that is often what is used at pre-compile time to turn the new syntax into something that the compiler will understand.
This is a rather confusing example of the usage of Metaprogramming
Unusual Activities you didn’t have for loops before, now you do. Is it confusing because everybody already has them? Maybe a better example would be: Imagine you wanted to specify logical constraints a la prolog. You can use metaprogramming to add such a facility to your program. But that means you’d have to explain what prolog is...
that's what you get when you assign that "confusing" guy to present this video..
I mean, I get it, actually I would to turn data into code for a project I'm currently working on but the language im working in (C++) won't let me :p
you could implement a scripting language then you could execute instruction that were created
5:37 Assuming the increment of i lies at the bottom of that squiggle somewhere ... otherwise you’ve just written an endless loop.
Isn't RRPN just Polish Notation?
Hmmm... can i think about this similar to Lambda-Expression in C# or Java?
Which, like... IS program code at compile time and BECOMES data at runtime?
Or, is that too far fetched? ^^
A lambda expression is a functional concept so yes, that's somewhat similar.
The comparison breaks down when you try to pick apart the lambda-expression - which you can't do in C# or Java. You can create a lambda, but you can't pull it apart and pick bits and pieces out of it.
With metaprogramming, you would be able to compose functions together, and then later on, pick them apart and, say, recompose the functions in a different way based on some conditions for optimization.
C# and Java have no way to do this.
Currying using lambdas is an example of how to achieve some of what the video talks about.
You can then get data that becomes code at runtime too...
How is the for loop example (and how it is an example of metaprogramming if a for loop didnt originally exist in the syntax) different from inline functions or #define
2:53 normally called Forward Polish Notation
*_...but tight code is used repeatedly-on scads of data; (Amer. scads, Brit. jillions)..._*
*_...on the other hand, the similarity shows in loop parameters, i++, list descriptors,..._*
*_...((in the control/addressing variables-but not, in the data itself/results per se))..._*
It's strange as a comparability theorist, I consider data and code to be the same thing. Though we explicitly use numbers for both.
Von Neumann understood this perfectly well. The only ones who have forgotten tend to use languages like Java or C#. Oh, and PHP.
Don't be a hater. Use the best tool to solve problem at hand.
@@GreyDeathVaccine couldn't have said better
It doesn't seem like Robert Smith understands what he is saying. There wasn't much of an explanation here.
Where's the rest?
I would just like to suggest to anyone that is interested in this, check out terra lang :) It's basically a scriptable compiler and deserves more love
How is that different from other languages and interpreters?
I'm pretty sure you could do the same thing in C++ or Java or Haskell without any "duality".
Also I thought computer scientists had largely abandoned meta programming.
That "assembly" for the for-loop is not actually that correct. The evaluation of the repeat condition is actually made before you enter the loop in a standard for-loop. What he wrote down is more akin to a do-while loop, which runs trough the loop once unconditionally, and then repeats if the condition evaluates to true.
Actually, it's a bit of a hybrid, because he set the iteration-variable at the top, and is still using an iteration variable.
wouldnt reversed reverse polish notation just be polish notation?
Yes.
I'm not sure where he is going with this. So, you're not happy with the tools a language give you such as for loops. So, do you want to be able to make your code flow in whatever fashion you want? Use labels on C. Yes, it will be considered spaghetti code, because it is.
Also a c source file IS data, although stored less efficiently. So is Java code, and so is Java bytecode and assembly. A smart enough program could write programs in any of these languages. The difference is that some are readable, and some aren't. Judge for yourself which are.
It's not about not being happy about for loopes.
It's about being able to *implement* for loops on the fly in a language where it doesn't exist already.
Or any other feature for that matter.
If you can come up with a new consept you can implement it and use user input as a part of the implementation if desired.
You can change the code itself based on user input.
And sure, you can write a program that can write code for its own code.
However what about writing a program that can write the code used to run itself during the runtime?
Regarding the spagetti code argument.
Yes, partially.
But it doesn't have to be.
Love this man!
Interesting to learn a bit about lisp
Could you do a video on how to group logic gates to acomplish complex actions, such as multiplication, division, and converting binary to different two exponent bases
I think they already did, using domino bricks.
I would recommend you check the video game "Turing Complete" on steam, it goes there and beyond, allowing you to learn how to implement a full 8-bit Harvard Style computer.
Are macros the same thing as Metaprogramming?
Haritha Wickremasinghe They are a form of meta-programming.
Is it possible to get similar meta programming capabilities in languages other than lisp? IE are there other programming languages that can do similar meta programming?
Rust lang is another language with a somewhat powerful macro system. But it seems Homoiconicity is one of the qualities that make meta-programming easier to implement.
Nim, im not sure about "rust" but people have built whole DSL's with it in nim.
What a joy it must be to debug a Lisp program.
You bet.
Just don't write any bugs. :p
"write a programm that writes programms", many of us do that already. for example a few days ago i wrote a simple python script to dynamically to my needs created LaTeX source code. also,
automatically generating config files (config files may be considered as code sometimes) follows this scheme. more interesting: if the program and the generated program both are in the same language.
Please try to get the interviewees to explain their subjects more concretely (i.e. with code samples)
Hmm, since code is written exactly the same as data, I wonder if I could write a program that reads a different program and then tells me if that program would eventually stop?
I really like this guy
i feel bad for the computer scientists who worked on programs writing programs like this. because their way was the much more elegant, much more beautiful and pragmatic approach to programs that write programs. but instead we just went with this huge mess we call GPT-3 and language models. now nobody cares about this stuff even though it’s some of the coolest programming science out there.
is all of this related to code? like Metaprogramming and/or Software defined Networking? Where can i learn these things? Are they being taught in Computer Science?
5:37 this is what you had to write to iterate ten times, yes but in assembly you could:
times 10 instr
instr (some instruction you want to do 10
times)
At the end credit there's a snippet on the 'current state of quantum computing'. Where's that vid?
I can't find it.
Hmmm still not getting it, I hope there are more of these.
It's just a method of abstracting useful functions as an evaluable data
so how can we do this in ruby?
metaprogramming in ruby was a selling point but the only thing we hear today is rails. the language bent to the framework
how can we do metaprogramming in ruby?
Quite easy actually, due to the fact that everything is an object, you can construct many different things (classes, methods for specific objects, ...) with ease. Ruby's syntax also allows to create quite clean DSL's!
can you show me an example (external url)? im not a ruby dev so id like to see that working
people use metaprogramming in rails too?
Well, rails is already a full blown framework, I'm not sure if you ever really need to do metaprogramming there. However, a simple example would be the attr_accessor function, which creates getters and setters for a field of a class.
class Person
attr_accessor :name
end
Is a functions which basically gets converted to
class Person
def name
@name
end
def name=(value)
@name = value
end
end
This isn't exactly like the metaprogramming shown in the video, but you're still creating new "keywords" to make life easier. You can't do the same thing in Java for example (at least not when I used to write Java)
I think one of the best examples for metaprogramming are how ORMs (Object-Relational Mappers) create the Ruby classes for database tables. In Ruby you do not have to define methods to get and set the value for every single column. Most of the time, all you need to write down is your class name and point the ORM at the right table. The ORM will then for example use MySQL queries to get the column definitions of the table and define the methods for you, at run time. This defining of code on the fly and adding it to classes and objects is the metaprogramming. All this happens in memory. In contrast, Java for example would have the ORM literally write source code files for you, where it defines all those methods before compiling. You should then never touch those, because the next time you generate them, your changes are lost.
With Ruby then, you have the advantage that you don't have to change anything in case of a change to a part of a table, that your code does not touch.
Some keywords you can look up in ruby: "include", "extend" (both when called in a class definition and when called on an instance object), "send", "method_missing", "define_method" and of course the 'dreaded' "eval".
To iterate is imperative and to recuse is functional?
I had a guy demo lisp for me yesterday for the first time, now it is everywhere.
Is this the latest "Programmers are also human" video?
hey phile can you explain exactly about symbolic data from the book structure and interpration of computer program
>Is there a simple example of that?
he literally gave an example
Haskell doesn't have for-loops though :P
It uses recursion, it being functional and all.
Very well explained.
I'm assuming that in Lispland eval isn't evil?
You don't use eval directly in Lispland, you define macros instead.
nim-lang.org can do that and its procedural.
capoiosct If that is the case, then nim wouldn't just be procedural, it would be multi-paradigm, because then you could just write macros to add OOP, functional or any other paradigm functionality to the language. :-)
@Computerphile can we have a video about the Julia programming language?
RUclips just showed me a 24-minute ad--that’s over *three times* longer than the video I wanted to watch.
Never mind--I just let it play with the tab muted, while I did something else. That way you still get the ad revenue, right?
Please make a video about how code generation works. E.g. LLVM
I want to see a non-contrived example
The trouble with metaprogramming is that all it does is give the user the same power as a programming language designer. What that means is if your programming language doesn’t have something, you can add it. The problem for explaining metaprogramming is the non-contrived examples have all either been implemented in other languages by language designers (so they seem silly nowadays), are really trivial (but nice) conveniences, or take a lot more time than they had here (like adding a new programming paradigm as a library).
Elijah Malaby this is a really great explanation!
quadricode being hyper sarcastic :)
Meta programming is just another layer of abstraction. You have assembly code to make processors do what you want, you have languages to make assembly code do what you want. This is a language to make a language to do what people want. A really powerful tool especially in academia and research. Simple examples will make not convey the power of the concept and always feel contrived. Non simple examples are non contrived but hard to explain
@@badsyntax173 The power in 'code as data' is thanks to macros. there's no real equivalent of a macro in languages like Java or Python. While Java's added a 'reflect' library that keeps a table of functions created in namespace, it still doesn't treat them equally 'first class citizens'.
There's no way for example to define '+' (plus) as '-' (minus)...or multiply or any function you want really. There's no constraint or workaround in having predictable code adjusting itself.
Whats the difference between lisp and a langue where i can run the eval() from javascript?
Code is just data to a lower-level processor.
What is the lowest-level processor? Is it the laws of physics?
He forgot to increment i in the pseudocode
al Khwarismi And to have the initial check of the loop, which usually means an unconditional jump to its end, right after initializing i.
The pseudo code is definitely inaccurate!
Aren't compilers in general an implementation of metaprogramming?
So Macros are also a type of MetaProgramming?? right!
code = evaluated data
so its like the eval function in javascript and php which have been around for ages... and are generally considered as "dangerous", especially if you use it with user input.
Anytime you handle data from an unknown source it's dangerous... you have to predict what might come and handle it.
@@malusmundus-9605 and therein lies the hubris... unless you restrict the data, you can't possibly predict everything and handle it - especially with user input. A hacker can use these weak points to execute code without your knowledge and gain access to systems or data.
Isn't it true most programming languages have an eval function where you could pass data (string, var, number, etc) to be evaluated and executed?
Only some scripting languages have this, and compiled languages generally can't. Eval also usually operates on program text, rather than directly on a generated AST.
What do you think about using Marvin Minsky frames as the base for metaprogramming language?
I took his idea, only one element type: Frame object can hold scalar value, associative array (names slots points to other frames) and array holds pointers to other frames _in order_.
In results, I can construct networks of directed interconnected objects, looks very very close to the attribute grammar structure.
The key is some subset of data must have active nature: they must be executable, or more precise evaluatable in some execution context.
I played with this concept a lot, using some sort of FORTH script language to construct and manipulate such a structure type. Results look attractable. To represent say program-like structures I can inherit base Frame class to set of classes close to any generic script language -- functions, loops.
But what stops me going further this way: complexity rises too fast when I try to rewrite system in itself (metacircular implementation). Maybe I should not try to generate Python code too earlier, and I first should describe the system only at a very high level, I don't know.
data is to code what information is to knowledge
I've yet to hear a convincing argument for LISP including this vid.
A very powerful tool which I lost control over on day 2🤣🤣🤣now i kinda see where the elixir quote comes from...
Actually, everything is data (even machine code is made of "data").
reverse reverse polish notation = polish notation
Can't you also create assembler code in a C program and then execute it? Isn't that the same?
You can't take the generated code and make it into data again within the language. You'd have to disassemble the generated code, recreate the C from it (which is hard due to optimizations) and somehow make it recompilable.
The other way is for your C program to create a miniature VM to interpret byte code your program creates, but that, again, takes you out of the language.
So wrong what he shows as data is for all I know already code , data refers to the values of the datatypes(he refers to them as data)
"For loops exist in every language"...
ever used Haskell? No loops.
Skylar he probably meant "the concept of iteration/repetition"
Tail recursion or a fold/reduction give more or less the same thing as a for-loop.
Skylar, there is the forM function which is like a for loop for sequencing nomadic actions. Eg
forM [1..10] $ \i -> print i
Shout out Julia
this guy is great
Eval is a common misspelling of evil
Is this different than using eval in JavaScript?
so... in other words..... HUH ?!?!? where is the example again !?? i must have missed it.
as i understand it:
first, a function is a set of instructions.
second, we can create data during runtime.
third, we can interpret data as instructions.
fourth, we can run interpreted data as a function.
you created code at run time.
the program wrote parts of itself.
QWERTY man but writing a program to write parts of itself is just adding another layer of complexity. how is it any better than just writing the end program yourself? we are not using assembly anymore
Joel Koh an artificial intelligence might want to write an algorithm which solves a problem.
QWERTY man that kind of makes sense but i dont think AI is at that stage yet. if an AI does write its own code, it would be its initial program telling it to write that code if it encounters that specific case, which is no different from writing that end code yourself. i dont think the AI is smart enough to write code that noone programmed it to write in the first place. its definitely not how neural networks work. maybe long in the future, AI will be smart enough to write completely new code, although by that time, completely new programming languages would have been invented to suit that need
Joel Koh give it a year.
You can also directly obfuscate your code for exemple, turning it into base64 data and create "eval" function that decode and read it. So powerful !
So... functional programming then?
Not quite.
It's a programming paradigm.
You *can* do this with a functional programming language (well some of them).
But this and functional programming isn't the same exact thing.
Any actually useful real world examples of these features??
To me it seems the core idea of Lisp is now obsolete.