When adding a lot of text together with undo (undo deleting a full line) the Editor gives an exception.

I cannot reproduce this always.
And I have only been able to reproduce this in my own language.

java.lang.IndexOutOfBoundsException: No token at offset 139
at org.spoofax.jsglr.client.imploder.Tokenizer.getTokenAtOffset(Tokenizer.java:121)
at org.strategoxt.imp.runtime.parser.tokens.SGLRTokenIterator.getStartIndex(SGLRTokenIterator.java:42)
at org.strategoxt.imp.runtime.parser.tokens.SGLRTokenIterator.<init>(SGLRTokenIterator.java:37)
at org.strategoxt.imp.runtime.parser.SGLRParseController.getTokenIterator(SGLRParseController.java:516)
at org.strategoxt.imp.runtime.editor.SelectionUtil.getSelectionAst(SelectionUtil.java:22)
at org.strategoxt.imp.runtime.services.views.properties.PropertiesService.getProperties(PropertiesService.java:54)
at org.strategoxt.imp.runtime.editor.StrategoTermSelection.getFirstElement(StrategoTermSelection.java:62)
at org.strategoxt.imp.runtime.editor.StrategoTermSelection.toList(StrategoTermSelection.java:86)
at org.eclipse.ui.internal.services.EvaluationService$2.compute(EvaluationService.java:97)
at org.eclipse.e4.core.commands.ExpressionContext.getDefaultVariable(ExpressionContext.java:94)
at org.eclipse.core.internal.expressions.IterateExpression.evaluate(IterateExpression.java:138)
at org.eclipse.ui.internal.services.EvaluationReference.evaluate(EvaluationReference.java:94)
at org.eclipse.ui.internal.services.EvaluationReference.evaluate(EvaluationReference.java:131)
at org.eclipse.ui.internal.services.EvaluationReference.changed(EvaluationReference.java:125)
at org.eclipse.e4.core.internal.contexts.TrackableComputationExt.update(TrackableComputationExt.java:110)
at org.eclipse.e4.core.internal.contexts.EclipseContext.processScheduled(EclipseContext.java:334)
at org.eclipse.e4.core.internal.contexts.EclipseContext.set(EclipseContext.java:348)
at org.eclipse.ui.internal.services.EvaluationService$1.changed(EvaluationService.java:73)
at org.eclipse.e4.core.internal.contexts.TrackableComputationExt.update(TrackableComputationExt.java:110)
at org.eclipse.e4.core.internal.contexts.EclipseContext.processScheduled(EclipseContext.java:334)
at org.eclipse.e4.core.internal.contexts.EclipseContext.set(EclipseContext.java:348)
at org.eclipse.e4.ui.internal.workbench.swt.E4Application$4.changed(E4Application.java:789)
at org.eclipse.e4.core.internal.contexts.TrackableComputationExt.update(TrackableComputationExt.java:110)
at org.eclipse.e4.core.internal.contexts.EclipseContext.processScheduled(EclipseContext.java:334)
at org.eclipse.e4.core.internal.contexts.EclipseContext.set(EclipseContext.java:348)
at org.eclipse.e4.ui.internal.workbench.SelectionAggregator$7.changed(SelectionAggregator.java:213)
at org.eclipse.e4.core.internal.contexts.TrackableComputationExt.update(TrackableComputationExt.java:110)
at org.eclipse.e4.core.internal.contexts.EclipseContext.processScheduled(EclipseContext.java:334)
at org.eclipse.e4.core.internal.contexts.EclipseContext.set(EclipseContext.java:348)
at org.eclipse.e4.ui.internal.workbench.SelectionServiceImpl.setSelection(SelectionServiceImpl.java:30)
at org.eclipse.ui.internal.e4.compatibility.CompatibilityPart.selectionChanged(CompatibilityPart.java:437)
at org.strategoxt.imp.runtime.editor.SelectionProvider.setSelection(SelectionProvider.java:35)
at org.strategoxt.imp.runtime.editor.SpoofaxEditor$2.caretMoved(SpoofaxEditor.java:78)
at org.eclipse.swt.custom.StyledTextListener.handleEvent(StyledTextListener.java:96)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4166)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1489)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1474)
at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1279)
at org.eclipse.swt.custom.StyledText.setCaretOffset(StyledText.java:8512)
at org.eclipse.swt.custom.StyledText.setSelection(StyledText.java:9542)
at org.eclipse.swt.custom.StyledText.setSelectionRange(StyledText.java:9584)
at org.eclipse.jface.text.TextViewer.setSelectedRange(TextViewer.java:2401)
at org.eclipse.jface.text.TextViewerUndoManager.selectAndReveal(TextViewerUndoManager.java:407)
at org.eclipse.jface.text.TextViewerUndoManager.access$5(TextViewerUndoManager.java:400)
at org.eclipse.jface.text.TextViewerUndoManager$DocumentUndoListener.documentUndoNotification(TextViewerUndoManager.java:181)
at org.eclipse.text.undo.DocumentUndoManager.fireDocumentUndo(DocumentUndoManager.java:929)
at org.eclipse.text.undo.DocumentUndoManager$UndoableTextChange.undo(DocumentUndoManager.java:266)
at org.eclipse.core.commands.operations.DefaultOperationHistory.doUndo(DefaultOperationHistory.java:417)
at org.eclipse.core.commands.operations.DefaultOperationHistory.undo(DefaultOperationHistory.java:1282)
at org.eclipse.ui.operations.UndoActionHandler.runCommand(UndoActionHandler.java:80)
at org.eclipse.ui.operations.OperationHistoryActionHandler$1.run(OperationHistoryActionHandler.java:306)
at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:464)
at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:372)
at org.eclipse.jface.dialogs.ProgressMonitorDialog.run(ProgressMonitorDialog.java:507)
at org.eclipse.ui.internal.operations.TimeTriggeredProgressMonitorDialog.access$6(TimeTriggeredProgressMonitorDialog.java:1)
at org.eclipse.ui.internal.operations.TimeTriggeredProgressMonitorDialog$2.run(TimeTriggeredProgressMonitorDialog.java:203)
at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:70)
at org.eclipse.ui.internal.operations.TimeTriggeredProgressMonitorDialog.run(TimeTriggeredProgressMonitorDialog.java:216)
at org.eclipse.ui.operations.OperationHistoryActionHandler.run(OperationHistoryActionHandler.java:321)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:499)
at org.eclipse.jface.commands.ActionHandler.execute(ActionHandler.java:119)
at org.eclipse.ui.internal.handlers.E4HandlerProxy.execute(E4HandlerProxy.java:90)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.e4.core.internal.di.MethodRequestor.execute(MethodRequestor.java:56)
at org.eclipse.e4.core.internal.di.InjectorImpl.invokeUsingClass(InjectorImpl.java:243)
at org.eclipse.e4.core.internal.di.InjectorImpl.invoke(InjectorImpl.java:224)
at org.eclipse.e4.core.contexts.ContextInjectionFactory.invoke(ContextInjectionFactory.java:132)
at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.execute(HandlerServiceHandler.java:167)
at org.eclipse.core.commands.Command.executeWithChecks(Command.java:499)
at org.eclipse.core.commands.ParameterizedCommand.executeWithChecks(ParameterizedCommand.java:508)
at org.eclipse.e4.core.commands.internal.HandlerServiceImpl.executeHandler(HandlerServiceImpl.java:213)
at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher.executeCommand(KeyBindingDispatcher.java:285)
at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher.press(KeyBindingDispatcher.java:504)
at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher.processKeyEvent(KeyBindingDispatcher.java:555)
at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher.filterKeySequenceBindings(KeyBindingDispatcher.java:376)
at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher.access$0(KeyBindingDispatcher.java:322)
at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher$KeyDownFilter.handleEvent(KeyBindingDispatcher.java:84)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Display.filterEvent(Display.java:1080)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4165)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1489)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1474)
at org.eclipse.swt.widgets.Widget.sendKeyEvent(Widget.java:1503)
at org.eclipse.swt.widgets.Widget.sendKeyEvent(Widget.java:1499)
at org.eclipse.swt.widgets.Canvas.sendKeyEvent(Canvas.java:496)
at org.eclipse.swt.widgets.Control.doCommandBySelector(Control.java:1060)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:5648)
at org.eclipse.swt.internal.cocoa.OS.objc_msgSend(Native Method)
at org.eclipse.swt.internal.cocoa.NSResponder.interpretKeyEvents(NSResponder.java:68)
at org.eclipse.swt.widgets.Composite.keyDown(Composite.java:594)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:5558)
at org.eclipse.swt.internal.cocoa.OS.objc_msgSendSuper(Native Method)
at org.eclipse.swt.widgets.Widget.callSuper(Widget.java:221)
at org.eclipse.swt.widgets.Widget.windowSendEvent(Widget.java:2104)
at org.eclipse.swt.widgets.Shell.windowSendEvent(Shell.java:2318)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:5620)
at org.eclipse.swt.internal.cocoa.OS.objc_msgSendSuper(Native Method)
at org.eclipse.swt.widgets.Display.applicationSendEvent(Display.java:5057)
at org.eclipse.swt.widgets.Display.applicationProc(Display.java:5206)
at org.eclipse.swt.internal.cocoa.OS.objc_msgSend(Native Method)
at org.eclipse.swt.internal.cocoa.NSApplication.sendEvent(NSApplication.java:128)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3648)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1113)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:997)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:140)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:611)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:567)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:354)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:181)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:636)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:591)
at org.eclipse.equinox.launcher.Main.run(Main.java:1450)

Submitted by Daco Harkes on 21 March 2014 at 17:56

Further undos are not possible after this exception.

The undo that throws the exception does happen.

It can happen in a very small example

module a
model

Delete model and undo: No token at offset 13

There are exactly 14 tokens in this file without undo: 6 chars + 1 whitespace + 1 chars + 1 newline + 5 chars, so after undo the character at offset 13 should exist.

Also in stratego the identical example does not throw an error -> so the problem seems language specific.

module a
rules

Can it happen because of a certain combination of syntax definitions?

Apparently the problem is not Undo specific.
Commenting a line of code gives the same error in the Error Log.
(Moreover the whole editor returns grey (rendering no text). The line is really commented out, if you save the file close editor and open the editor.)

I think this is a race condition. Given a new offset, it tries to find the selection in the old AST because the text has not yet been re-parsed and a new AST is not yet available. The old AST has no node corresponding to the new offset.

Optimally we should wait until the new AST becomes available and then try to find the selection using the new offset, but it probably makes more sense to just catch the exception and return null as the selection AST. Then, no properties will be visible in the properties view until the user moves the cursor, but that’s not a big deal. I’ll fix this.

Ah, that explains why it is only in my own language, the other languages do not have the properties view yet.

(Note that while typing at the end of the file it constantly throws errors in the Error Log with this error. Leave one character before the cursor and its fine.)

I have a seen a similar issue in PCF when trying to comment out the entire program. In PCF specifically the empty file is not valid syntax. Perhaps this has something to do with error recovery failing to produce an AST?

I don’t think that’s the case here though because the stack trace really shows that it fails when it tries to obtain the selection AST when it creates the model for the properties view:

at org.strategoxt.imp.runtime.editor.SelectionUtil.getSelectionAst(SelectionUtil.java:22)
at org.strategoxt.imp.runtime.services.views.properties.PropertiesService.getProperties(PropertiesService.java:54)

I have added properties view to PCF.

Should be fixed now. If not, please reopen.
https://github.com/metaborg/spoofax/commit/f08f6f9518546137b88905c372c5f4ecb6c68841