antlr - Reorder token stream in XText lexer

Question

I am trying to lex/parse an existing language by using XText. I have already decided I will use a custom ANTLRv3 lexer, as the language cannot be lexed in a context-free way. Fortunately, I do not need parser information; just the previously encountered tokens is enough to decide the lexing mode.

The target language has an InputSection that can be described as follows: InputSection: INPUT_SECTION A=ID B=ID;. However, it can be specified in two different ways.

; The canonical way
$InputSection Foo Bar
$SomeOtherSection Fonzie

; The haphazard way
$InputSection Foo
$SomeOtherSection Fonzie
$InputSection Bar

Could I use TokenStreamRewriter to reorder all tokens in the canonical way, before passing this on to the parser? Or will this generate issues in XText later?

标签： antlrxtext

Answer 1

经过大量调查，我得出的结论是编辑工具本身确实不适合这类问题。

如果您开始输入一条规则，则必须考虑后续部分的 AST 上下文才能知道如何自动完成。同时，这会给用户带来很大的困惑。

最后，我将因此根本不支持该语言的这种晦涩难懂的特性。相反，将构造 AST，以便（合理地）划分为两个部分的部分仍将正确解析。