Cool ) Really cool. One question to Speaked if it is possible ) In the custom parser, how does the lexer backtrack in case of a complex item failure? From the video, it seems that the lexer only consumes what it accepts and doesn’t consume anything if it fails. But what if it consumes in the first step, fails in the next step, and needs to return to the state before the first step?
That is an excellent question, indeed :) The challenge I was solving (parsing fully qualified identifiers) was simple enough that backtracking was not a problem. Generally, if any matching method fails, the lexer's position should be the same as it was when the method was called. The lexer would have to support two more operations: getCurrentPosition () and resetCurrentPosition(). At the beginning of the matcher function, you'd save the lexer position and then reset it on a match failure. E.g.: /** sequence := identifier identifier */ private sequence() { const save = this.lexer.getCurrentPosition(); const first = this.identifier(); if(first) { const second = this.identifier(); if(second) { return [first,second]; // success } } this.lexer.resetPosition(save); return undefined; // failure }
@@Krzysztof.Rzymkowski-Snowflake thank you for your answer ) By the way, the parsing in this case is sync, so position management could be simplified by implementing the stack inside of the lexer. Or even better do something like: return lexer.run(() => [ this.identifier(), this.idenitfier(), ]);
Cool )
Really cool.
One question to Speaked if it is possible )
In the custom parser, how does the lexer backtrack in case of a complex item failure?
From the video, it seems that the lexer only consumes what it accepts and doesn’t consume anything if it fails.
But what if it consumes in the first step, fails in the next step, and needs to return to the state before the first step?
That's a great question, thanks! I forwarded it to Krzysztof, he should respond soon
That is an excellent question, indeed :)
The challenge I was solving (parsing fully qualified identifiers) was simple enough that backtracking was not a problem.
Generally, if any matching method fails, the lexer's position should be the same as it was when the method was called. The lexer would have to support two more operations: getCurrentPosition () and resetCurrentPosition().
At the beginning of the matcher function, you'd save the lexer position and then reset it on a match failure. E.g.:
/** sequence := identifier identifier */
private sequence() {
const save = this.lexer.getCurrentPosition();
const first = this.identifier();
if(first) {
const second = this.identifier();
if(second) {
return [first,second]; // success
}
}
this.lexer.resetPosition(save);
return undefined; // failure
}
@@Krzysztof.Rzymkowski-Snowflake thank you for your answer )
By the way, the parsing in this case is sync, so position management could be simplified by implementing the stack inside of the lexer.
Or even better do something like:
return lexer.run(() => [
this.identifier(),
this.idenitfier(),
]);