I’ve run into an issue when trying to use normalized syntax productions to disallow layout in context-free productions. It appears to be due to the treatment of variables for lexical sorts in SDF3. I’m using Spoofax 1.4.

My SDF3 grammar for a language CL originally included the following productions:

context-free syntax
  CL-char-literal.CLCharLiteral = 
  	"`" CL-character "`"

  CL-character.CLRegularChar = CL-regular-char
  CL-character.CLEscapedChar = "\\" CL-escaped-char
  CL-character.CLEscapedCharcode = "\\" CL-escaped-charcode
  
lexical syntax
  CL-regular-char = ~[\\\`\n\r\t]
  CL-escaped-char = [\\\`ntbr]
  CL-escaped-charcode = [0-9] [0-9] [0-9]

The above syntax is transformed to a language FCT by the following transformation rules:

to-funcons:
  |[ char[: ` ~CH ` :] ]| ->
  |[ capture[: ~CH :] ]|
to-funcons:
  |[ capture[: ~RCH :] ]| ->
  |[ ascii-character("~RCH") ]|
to-funcons:
  |[ capture[: \\ :] ]| ->
  |[ '\' ]|
...

The syntax involved in the transformation rules includes:

variables
  CL-character = "~CH" [1-9]? {prefer}
  CL-regular-char = "~RCH" [1-9]? {prefer}
context-free syntax
  FCT.FCTDoubleQuoted = <"<FCT-Quoted>"> {prefer}
  FCT-Quoted.FCTregularchar = CL-regular-char  {avoid}
  FCT.Xchar4charliteral = "char" "[:" CL-char-literal ":]"
  FCT.Xcapture4character = "capture" "[:" CL-character ":]"

I managed to suppress layout in CL-char-literal by using the following normalized rule:

syntax
  CL-char-literal-CF.CLCharLiteral = "`" CL-character-CF "`"

(Based on the description of normalization in Eelco’s thesis, the addition of -CF above appears to be necessary in SDF2, and presumably also in SDF3.)

After rebuilding, closing Eclipse and rebuilding a couple of times, the transformation rules above all still work.

I then tried to suppress layout in CL-character as follows:

syntax
  CL-character-CF.CLRegularChar = CL-regular-char-LEX
  CL-character-CF.CLEscapedChar = "\\" CL-escaped-char-LEX
  CL-character-CF.CLEscapedCharcode = "\\" CL-escaped-charcode-LEX

That syntax works as expected when parsing CL. But when parsing the transformation rules (after (rebuild;quit)^N), ~RCH is no longer accepted as a meta-variable in capture[: ~RCH :].

I’ve tried changing the variables section to lexical variables, or by using

  CL-regular-char-LEX = "~RCH" [1-9]? {prefer}

but neither appears to work. They also provoke the following error report, which makes me suspect that support for lexical variables in the Spoofax version of SDF2 is incomplete:

sdf2rtg:
        [echo] ========================================================================================
        [echo] include/CL.def -> include/CL.rtg
        [echo] ========================================================================================
        [java] [ org.strategoxt.tools.main-sdf2rtg | info ] Calling external tool /Users/pdm/Applications/spoofax-1.4/plugins/org.strategoxt.imp.nativebundle_1.4.0/native/macosx///sdf2table -i /Volumes/HDD/pdm/SVN/CL-Editor/include/CL.def -o /var/folders/l6/zbrrl3zn41l9bf3n4z_l7nhw0000gn/T/StrategoXT9034639934386655114.tmp -n -m CL
        [java] [ org.strategoxt.tools.main-sdf2rtg | info ] Calling external tool /Users/pdm/Applications/spoofax-1.4/plugins/org.strategoxt.imp.nativebundle_1.4.0/native/macosx///implodePT -i /var/folders/l6/zbrrl3zn41l9bf3n4z_l7nhw0000gn/T/StrategoXT9034639934386655114.tmp -o /var/folders/l6/zbrrl3zn41l9bf3n4z_l7nhw0000gn/T/StrategoXT6390393975346510769.tmp
        [java] [ org.strategoxt.tools.main-sdf2rtg | warning ] Cannot generate a nice name for symbol
        [java] [ org.strategoxt.tools.main-sdf2rtg | warning ]   varsym(lex(sort("CL-regular-char")))
        [java] [ org.strategoxt.tools.main-sdf2rtg | warning ]   Please report this bug at 
        [java] [ org.strategoxt.tools.main-sdf2rtg | warning ]     - https://bugs.cs.uu.nl/browse/STR
        [java] [ org.strategoxt.tools.main-sdf2rtg | warning ]     - or martin.bravenboer@gmail.com
        [java] [ org.strategoxt.tools.main-sdf2rtg | warning ]   Resolution: falling back to the ugly name 
        [java]           "l_21198"

Lexical variables were crucial in ASF+SDF for transforming inside the structured lexical tokens of SDF2. It looks as if Spoofax is trying to avoid the need to indicate whether variables are lexical or not, but it doesn’t work with raw syntax productions.

Any hints about how to address this issue would be most welcome!

Submitted by Peter Mosses on 10 August 2015 at 14:53

P.S. Inserting an extra level of context-free syntax in the CL grammar circumvents the reported issue:

syntax
  CL-character-CF.CLRegularChar = CL-regular-char-CF
  CL-character-CF.CLEscapedChar = "\\" CL-escaped-char-CF
  CL-character-CF.CLEscapedCharcode = "\\" CL-escaped-charcode-CF

context-free syntax
  CL-regular-char = CL-regular-char-lex
  CL-escaped-char = CL-escaped-char-lex
  CL-escaped-charcode = CL-escaped-charcode-lex
  
lexical syntax
  CL-regular-char-lex = ~[\\\`\n\r\t]
  CL-escaped-char-lex = [\\\`ntbr]
  CL-escaped-charcode-lex = [0-9] [0-9] [0-9]