When using templates in SDF3, the template option tokenize can be used to split certain characters to their own tokens. But this doesn’t work as I’d expect in combination with the template option keyword.

Example:

module cfs/Statement

imports
  cfs/Exp

context-free start-symbols
  Statement

template options
  tokenize: "()"
  keyword -/- ~[A-Za-z]

templates
  Statement.ForLoop  = [for([Exp];[Exp];[Exp]) do [Statement]]

results in:

module cfs/Statement
imports cfs/Exp

exports
  context-free start-symbols
Statement

context-free syntax
  "for" "(" Exp ";" Exp ";" Exp ")" Statement               -> Statement {cons("ForLoop")}

context-free syntax
  CONTENTCOMPLETE -> Statement {cons("COMPLETION-Statement")}

lexical restrictions
  "do" -/- ~[A-Za-z]

Whereas I was expecting "for" to be recognized as well.

Submitted by Jeff Smits on 28 September 2013 at 14:28

Perhaps it doesn’t matter as much as I thought at first, because in this case "if" and "while" can only be followed the ( anyway according to the grammar rules.

I edited the issue. It now uses an example which does not include parts of a MiniJava implementation.

I was also puzzled by the described behaviour, but I think it is not an error. When layout is mandatory, this layout needs to be part of the template as well and no tokenize option is needed.