There are reports that the error markers and error highlights (underlines) are out of sync. Behaviour explained below:

1. Make a one-character change that causes a semantic-error
2. An error marker is shown, but no highlighting appears
3. Make a one-character change in the editor (without invalidating the previously introduced error)
4. The error marker stays and the highlighting appears as well
5. Fix the one-character semantic-error
6. The error marker disappears, the highlighting remains
7. Make a one-character change in the file
8. The error marker remains gone, the highlighting also disappears

I attached an Eclipse configuration which uses WebDSL and Spoofax stable (1.0.2.0-r24315). But there are reports this happens in other languages as well. Need to investigate.

UPDATE (4/2/13):

This issue is caused by feature interaction between Spoofax/IMP and XML validators included in the Web Tools Platform (WTP) Eclipse packages. When a language is defined which has file extensions that are also handled by the XML editors, because Spoofax language do not have a content-type contribution, Eclipse infers a content-type based on the file extension. This inferred content type is org.eclipse.core.runtime.xml. When such a file is opened both the IMP editor and the XML validators are initialised. The two marker providers appear to cause this issue.

The solution appears to be to have Spoofax plugins declare a content type which is based on org.eclipse.core.runtime.text. Simply adding an extension such as:

	<extension point="org.eclipse.core.contenttype.contentTypes">
	      <content-type base-type="org.eclipse.core.runtime.properties"
	        file-extensions="ent"
	        id="entlang.entfile"
	        name="Entlang file"
	        priority="normal">
	      </content-type>
	</extension>

resolves this issue for deployed plugins as reported by Chris, but does nothing for dynamically loaded languages. Therefore a solution for dynamic loading is required, probably at Spoofax-runtime or IMP level. The other direction is to find out why the markers get mangled if both Spoofax and XML editors provide them. Suggestion may be that it has something to do with Spoofax incrementally adding/removing markers between two analyses.

Submitted by Vlad Vergu on 18 January 2013 at 15:29

I have also encountered this behaviour in the default Entity language in the latest Spoofax (r25838).

The title suggests that error markers are also delayed, but I have not seen this behaviour at all. Only the error underlines (or squigglies) are delayed. They also seem to be delayed by exactly one analysis.

Also see this in the Entity editor of the unstable branch. How come this is suddenly appearing in old editors??

It seems the delayed squigglies and reference resolution showing no underline and hand icon are related.

The issue seems to occur not just with an Entity editor, but any editor that uses extension .ent. However, the issue does not occur on some Eclipse version/editions.

• Clean Eclipse 3.7.2 for Java Developers: Issue occurs
• Clean Eclipse 3.7.2 classic: No issue
• Clean Eclipse 3.8.1: No issue

For WebDSL however, the issue also occurs on a Clean Eclipse 3.8.1..

Results of investigation in Eclipse 3.7.2 for Java Developers, with latest Spoofax nightly (hence without WebDSL):

• Occurs for .ent extension
• Occurs for .mod extension as well
• Occurs for .dtd extension as well
• Does not occur for .app extension

The .ent, .mod and .dtd extensions are also associated with the DTD editor provided by plugins org.eclipse.wst.dtd*.*.

At least in 3.7.2 for Java Devs this happens due to a interaction with the org.eclipse.wst.xml.core plugin (version 1.1.602 over here). If this plugin is disabled the issue no longer occurs. Disabling the org.eclipse.wst.dtd*.* does not resolve the issue. Suspect that these extensions are hardcoded in org.eclipse.wst.xml.*; semi-logical since DTD uses XML syntax.

The org.eclipse.wst.xml.core plugin is shipped with Web Tools Platform (WTP) which is included in Eclipse for Java Developers but not in Eclipse Classic. This partially explains why the issue occurs in WebDSL since it has some dependencies on WTP which pull in the named plugin.

To clarify, Web Standard Tools (WST) appears to have been renamed to Web Tools Platform (WTP) sometime between 3.7 and 3.8.

Ah, so it IS the DTD editor that is interfering with .ent. That explains why it occurs for .ent, but not for .app in the WebDSL plugin.

Is this interference with other editor plugins a Spoofax issue? Can we somehow avoid such interference? Or is it a pure WTP/WebDSL installation issue?

This is a Spoofax usability issue. It is probably caused by interference between the IMP Universal Editor and the core of the XML editors shipped with WTP.

This is an issue which affects pure installations of Spoofax or WebDSL. For Spoofax we can avoid by saying we only support Eclipse Classic, which is a bit nasty. For WebDSL no clue since it has a dependency on WTP.

Could somebody please investigate whether this issue occurs in Spoofax nightly on Eclipse 3.8/3.8.1 for Java Developers? If it does not maybe we could have a quick diff look at the sources of org.eclipse.wst.xml.core. If we find out what they fixed we can patch the same fix in IMP maybe.

Also happens with .xmi extension. Support for this is provided by org.eclipse.wst.xml.core.

I didn’t find an Eclipse 3.8.1 for Java Developers, but after installing http://eclipse.org/webtools/releases/3.4.0/ the issue also occurs in Eclipse 3.8.1.

Does not look like an IMP issue. Error markers (both markers and highlights are provided through a single class IMarker) are provided by Spoofax itself, IMP does not seem to be involved.

Right Gabriël, so it’s definitely a WTP related issue.

Also happens with .xsd extension. Support for this is provided by org.eclipse.wst.xsd.core.

I would not rule out an IMP issue unless IMP also does not use markers (not just creation). But IMP definitely uses markers to provide squigglies in the editor?
There’s also the reference resolution issue where no underlining and no hand cursor is shown.

It is also not entirely WST/WTP related. Using .app in Eclipse with Spoofax + WST/WTP installed does NOT cause the issue. However in Eclipse with WebDSL (+ Spoofax and WST/WTP as dependency) using .app DOES cause the issue. Maybe there is another plugin interfering with .app?

Right, of course IMP provides functionality for marking, but Spoofax does not seem to use it. Indeed, may be premature to rule out IMP.

Yes, it would be worthy to investigate which plugin interferes with .app.

I’ve tried commenting out everything related to Markers in IMP runtime and error markers in Spoofax are still functional. Leads me to think again that IMP may not be involved here.

I’d like to add that this happens for both syntactic and semantic errors/warnings.

The errors appear and disappear in the Problems view on time, slightly after the gutter marker (probably the expected behaviour).

The issue seems to be purely a presentation issue.

the strange behaviour of .app with and without webdsl, might have to do with this line: plugin.xml

@Chris. That may indeed be a thing. Did you try removing the base-type attribute and rebuilding the editor

@Danny: what is the reason for having that base-type on .app files?

Sorry I was not at a place to test this, hopefully I will get the change to try this tomorrow.

I guess it was even my code try to get the Tasks stuff working there was some free integration with XML editors, which at that point didn’t seem dangerous and didn’t seem to break anything. I guess that even the task scanner isn’t even using it anymore so could be removed or replaced.

That sounds very good Chris. If you can find a changeset in WebDSL that introduces this issue there, we can definitely find a workaround for now. If you find such a changeset say WebDSL revision R, such that R-1 still works but R breaks, it would be great to check whether the Spoofax that’s included in R still causes the issue to occur in R-1; i.e. backport the Spoofax used in R to R-1. This would help in figuring out whether it’s a change in Spoofax that broke or this issue has always existed but was never noticed.

This issue may have gone unnoticed for a long time (in Spoofax) due to the recommendation at http://strategoxt.org/Spoofax/Download to use Eclipse classic.

Vlad indeed the problem was that one line I changed it like this

- content-type base-type=“org.eclipse.core.runtime.xml”

+ content-type base-type=“org.eclipse.core.runtime.properties”

which solved it for the .app

There was no change in spoofax itself, it is just that indeed this WST/WTP implements a universal editor for structured files(so xml, dtd and more like the extensions that you already have shown), which I guess is conflicting with IMP.

The reason I didn’t notice this while implementing was that I tested it in the running eclipse, but those changes in plugin.xml are not used there, it uses the plugin.xml from the installed plugin.xml. Use running as eclipse plugin, did work(partly) so to be sure, I will wait for the nightly to finish and check if still works.

I got some feeling that something similar could work for the other extensions.
Short answer yes this works !

I added this to the entlang plugin.xml

   <extension point="org.eclipse.core.contenttype.contentTypes">
      <content-type base-type="org.eclipse.core.runtime.properties"
        file-extensions="ent"
        id="entlang.entfile"
        name="Entlang file"
        priority="normal">
      </content-type>
</extension> 

This solves it partly, only works on deployed/installed plugin not when building inside eclipse.

So making it totally work would mean to make the inside build less like a hack
or another way would be to have the universal editor that we have for imp have an extension point like this which we can then extend for every other plugin.

This is good news Chris. Making that change in WebDSL gives a workaround for the next release. The workaround for Spoofax would be to use Eclipse classic.

We should still look for a fix for this issue, but it’s good that we now have a workaround.

@Chris: Changing the way Spoofax dynamically loads languages is definitely not doable for the upcoming release, unfortunately. I’m curious why you chose the base-type="org.eclipse.core.runtime.properties" instead of simply base-type="org.eclipse.core.runtime.text"?

Update: the culprit plugin seems to be org.eclipse.wst.sse.ui. Classes such as StructuredMarkerAnnotation and XMLStructuredDocumentRegion are being instantiated when an IMP editor for a file with one of the mentioned extension is opened.

Here’s a stack triggered when opening a Spoofax editor handling the .xml extension, maybe somebody has an idea:

Daemon Thread [Thread-1] (Suspended (breakpoint at line 23 in XMLStructuredDocumentRegion))	
	XMLStructuredDocumentRegion.<init>() line: 23	
	XMLStructuredRegionFactory.createRegion(int) line: 31	
	XMLSourceParser.createStructuredDocumentRegion(String) line: 112	
	XMLSourceParser.parseNodes() line: 367	
	XMLSourceParser.getDocumentRegions() line: 173	
	XMLStructuredDocumentReParser(StructuredDocumentReParser)._core_reparse_text(int, int) line: 355	
	XMLStructuredDocumentReParser(StructuredDocumentReParser).core_reparse(int, int, CoreNodeList, boolean) line: 752	
	XMLStructuredDocumentReParser(StructuredDocumentReParser).reparse(IStructuredDocumentRegion, IStructuredDocumentRegion) line: 1401	
	XMLStructuredDocumentReParser(StructuredDocumentReParser).reparse() line: 1333	
	JobSafeStructuredDocument(BasicStructuredDocument).updateModel(Object, int, int, String) line: 2713	
	JobSafeStructuredDocument(BasicStructuredDocument).internalReplaceText(Object, int, int, String, long, boolean) line: 1923	
	JobSafeStructuredDocument(BasicStructuredDocument).replaceText(Object, int, int, String, long, boolean) line: 2423	
	JobSafeStructuredDocument(BasicStructuredDocument).set(String, long) line: 2935	
	ResourceTextFileBuffer.setDocumentContent(IDocument, IFile, String) line: 575	
	ResourceTextFileBuffer.initializeFileBufferContent(IProgressMonitor) line: 286	
	ResourceTextFileBuffer(ResourceFileBuffer).create(IPath, IProgressMonitor) line: 245	
	ResourceTextFileBufferManager(TextFileBufferManager).connect(IPath, LocationKind, IProgressMonitor) line: 112	
	TextFileDocumentProvider.createFileInfo(Object) line: 559	
	TextFileDocumentProvider.connect(Object) line: 478	
	UniversalEditor(AbstractTextEditor).doSetInput(IEditorInput) line: 4222	
	UniversalEditor(StatusTextEditor).doSetInput(IEditorInput) line: 237	
	UniversalEditor(AbstractDecoratedTextEditor).doSetInput(IEditorInput) line: 1480	
	UniversalEditor(TextEditor).doSetInput(IEditorInput) line: 169	
	UniversalEditor.doSetInput(IEditorInput) line: 1488	
	AbstractTextEditor$19.run(IProgressMonitor) line: 3209	
	ModalContext.runInCurrentThread(IRunnableWithProgress, IProgressMonitor) line: 464	
	ModalContext.run(IRunnableWithProgress, boolean, IProgressMonitor, Display) line: 372	
	ApplicationWindow$1.run() line: 759	
	BusyIndicator.showWhile(Display, Runnable) line: 70	
	WorkbenchWindow(ApplicationWindow).run(boolean, boolean, IRunnableWithProgress) line: 756	
	WorkbenchWindow.run(boolean, boolean, IRunnableWithProgress) line: 2649	
	UniversalEditor(AbstractTextEditor).internalInit(IWorkbenchWindow, IEditorSite, IEditorInput) line: 3227	
	UniversalEditor(AbstractTextEditor).init(IEditorSite, IEditorInput) line: 3254	
	EditorManager.createSite(IEditorReference, IEditorPart, EditorDescriptor, IEditorInput) line: 828	
	EditorReference.createPartHelper() line: 647	
	EditorReference.createPart() line: 465	
	EditorReference(WorkbenchPartReference).getPart(boolean) line: 595	
	EditorPane(PartPane).setVisible(boolean) line: 315	
	PresentablePart.setVisible(boolean) line: 180	
	PresentablePartFolder.select(IPresentablePart) line: 270	
	LeftToRightTabOrder.select(IPresentablePart) line: 65	
	TabbedStackPresentation.selectPart(IPresentablePart) line: 473	
	EditorStack(PartStack).refreshPresentationSelection() line: 1245	
	EditorStack(PartStack).setSelection(LayoutPart) line: 1198	
	EditorStack(PartStack).showPart(LayoutPart, Object) line: 1597	
	EditorStack(PartStack).add(LayoutPart, Object) line: 493	
	EditorStack.add(LayoutPart, Object) line: 103	
	EditorStack(PartStack).add(LayoutPart) line: 479	
	EditorStack.add(LayoutPart) line: 112	
	EditorSashContainer.addEditor(EditorPane, EditorStack) line: 63	
	EditorAreaHelper.addToLayout(EditorPane, EditorStack) line: 225	
	EditorAreaHelper.addEditor(EditorReference, String, boolean) line: 213	
	EditorManager.createEditorTab(EditorReference, String) line: 808	
	EditorManager.openEditorFromDescriptor(EditorDescriptor, IEditorInput, IMemento) line: 707	
	EditorManager.openEditor(String, IEditorInput, boolean, IMemento) line: 666	
	...

Ok, so here’s my take on this.

Eclipse uses two mechanism for determining who handles what file:

1. File associations: associate file extensions with editors <- this determines which editor part etc. to instantiate
2. Content type: associate content types (determined by encoding and file extension) with handlers <- this determines who handles the actual content of the file

Spoofax/IMP registers file associations for language but does not register content types for them. In fact, i believe that IMP does not obey the above structure for determining the content handler and does it solely based on file associations. When a file with extension say .xml is opened, the correct editor part (UniversalEditor) is created based on the file association. Also based on the file extension, the org.eclipse.core.runtime.xml content type is determined. This content-type mapping is used to obtain a document factory in the [ResourceExtensionRegistry](https://www.cct.lsu.edu/~rguidry/eclipse-doc36/org/eclipse/core/internal/filebuffers/ExtensionsRegistry.html#getDocumentFactory(org.eclipse.core.runtime.content.IContentType[]) class. Because Spoofax/IMP does not contribute a content type the XML handlers are instantiated.

I believe the resulting behaviour described in this issue is caused by having two document handlers interact. One is the IMP one which is associated solely by file extensions and the other is the XML one which is associated by content-type via file extension. This explains the behaviour reported by Chris if he sets the content-type for specific Spoofax languages.

I believe the solution is for Spoofax/IMP to contribute a content-type for each language/file extension. We need to figure out how to do that, and probably the first move would be to look whether upstream IMP has resolved this issue already.

Thus, we have a work around for 1.1 and a new, more general issue (providing content types for languages) for 1.2?

@Vlad: this is a revert of the change so maybe Danny can tell why that one. I was thinking about using the text one instead, but that could be just as wrong, since we actually need a base one from spoofax to extend for every languages. Any other choice could conflict with other plugins.

The content type based on org.eclipse.core.runtime.properties used in the WebDSL plugin.xml was part of an example used for getting custom file icons to work: https://svn.strategoxt.org/websvn/WebDSL?op=comp&compare[]=/webdsl-editor/trunk/webdsl.editor/plugin.xml@4977&compare[]=/webdsl-editor/trunk/webdsl.editor/plugin.xml@4978

It would be nice to provide better support for custom file icons in any Spoofax project, so probably something to look into as well when digging into content types.

Nice work! I think we should add a note about incompatibility with WST/WTP and Eclipse for Java Developers to the download site, and create a separate issue for solving the incompatibility as Vlad explained. Or is solving this issue easy and can it be done for the 1.1 release Vlad?

I do not know how much work this would be to fix. As @Chris pointed out, simply using the Eclipse extension points is easy but only covers deployed plugins, not for dynamically loaded ones. For dynamic loading i suspect we need a dynamic solution which sets the content type on the File, or maybe using a content describer.

Some more investigation & reading up on Eclipse literature is required before trying a fix. In particular it would be useful to test whether other IMP-based frameworks have similar issues. While we now understand what is happening at Eclipse level that causes the WTP plugins to be called, we still do not understand why the effect is that of partially messing up the markers. I suspect that it is a deeper interaction with Spoofax’s incremental style of adding/removing markers (AstMessageBatch*.java).

Let’s postpone this issue for v1.2 since (1) we’re not close to a fix and (2) there would be insufficient testing time. I think we should keep this issue, in an updated form, since it documents all the investigation and findings.

• Postponed for 1.2
• Updated issue description
• Opened https://yellowgrass.org/issue/Spoofax/628 for the documentation issue.

I think there are more general issues in the comments, which should become separate, but related issues:

• Supporting content types.
• Supporting custom file icons.

We have looked at the AstMessageHandler and AstMessageBatch classes to see how markers are added to Eclipse. We tried a simplified version without batching that simply clears markers before analysis and adds markers as they arrive. The issue still occurs with the simple approach as well.

It probably is related to how WTP handles markers.