When typing something in a SpoofaxLang editor, it seems background jobs get queued really fast. Sometimes a lot faster than the speed at which they are processed.

The effect is that sometimes after doing a bunch of editing, you have to wait a long time for all pending background jobs to get processed, before you can do a compile. (Or you can restart Eclipse as a workaround :-))

Might be nice if there is some limit on the queue size, though maybe this bug will be obsoleted by other index work.

Submitted by Tobi Vollebregt on 7 March 2012 at 15:54

This is likely interaction with the new file system notifications: Spoofax/224. Since SpoofaxLang currently has its own file system notification, maybe FileNotificationServer should ignore .spx files?

No, I watched it a bit closer, and the job spam happens in particular when SpoofaxLang is generating files. So .spx need not be ignored, but everything in .shadowdir needs to be ignored. (Or whichever dir is configured in spx.config as target for code generation, or maybe in all hidden directories?)

But .shadowdir should already be ignored in the FileNotificationServer class. Plus it would only trigger if Eclipse actually manually or automatically gets a refresh event.

Hmm, yeah, I just noticed that.

Nevertheless I doubt the number of threads rising rapidly while generating code is accidental :-)
Also there happen to be ~750 threads created when generating ~750 files in .shadowdir. (even needed to lower stack size to get this far ;-))

When putting a breakpoint in FileNotificationServer, I see that isIgnoredChange is invoked, and returns true for the files in .shadowdir.

Plus it would only trigger if Eclipse actually manually or automatically gets a refresh event.

Hmm, so that may happen for me cause I always set ‘Refresh using native hooks or polling’ and ‘Refresh on access’ on in the Eclipse workspace settings. I’ll experiment a little with those settings later on.

---

Also, I think I may be talking about a different issue now, than the one I originally posted.
In particular the issue is now that n threads seem to be spawned when n files are generated, which tends to cause OOM errors etc.

The background job / worker thread spam is solved, now that Adil changed the code generator to not invoke SSL_EXT_refreshresource for every generated file.

Unfortunately I think this was a separate issue than what I originally posted, so I’ll leave this open for now.

Great that the background job/worker thread spam is now fixed! Now, the .shadowdir is refreshed just once in every build. Hence, SSL_EXT_refreshresource is invoked once in every build.

Regarding the open issue: the trunk version of SpoofaxLang does not have an additional file event listener. Therefore, it seems to be same issue related to editor-analyze of spoofax-based language editor : every key stroke is initiating a new analysis.

In case of SpoofaxLang, it is performing just one additional step. It is setting error/warning markers for the packages that are dependent on the current package. However, it seems like SSL_EXT_set_markers is not spawning a new background job/thread for that.

I will investigate further to see why it is creating many background jobs.

Indeed, in the open issue, there are many analysis jobs queued, but the worker threads are managed well: there is no thread spam. The jobs are really only queued, and it takes a while for the single (?) worker to go through them all.

This is particularly frustrating as Eclipse locks up (not the UI thread, but nearly everything else) from the moment you save the file you are working on, until the moment all queued jobs are finished.

At the moment I suspect this is a runtime issue, and not really specific to SpoofaxLang (except that SpoofaxLang analysis jobs take relatively much time). That is:

• Analysis jobs may be queued a bit too often
• Maybe pending analysis jobs should be cancelled as soon as a new analysis job for the same file is added to the queue. That way, there can be at most two jobs for the same file in the queue: one currently running analysis, and one queued analysis.

This hypothesis is supported by the fact that if I put a breakpoint in StrategoAnalysisQueue and inspect the queue after I’ve typed a bit in some file, then the queue easily contains 100s of StrategoAnalysisQueue$UpdateJobs for the same .spx file.

IIRC Nathan said that the queue avoids duplicate jobs like that. But it might depend on what kind of job is queued.

Yeah, I recall from reading the code some months back that only manually queued jobs (using queue-analysis) are ignored if a job for the same file is already present. But automatically queued analysis jobs can have duplicates.

Background analyses avoid duplicate jobs, only one analysis can be active per path.

Indeed.

Apparently, editor-analyze invokes StrategoAnalysisQueue.queue(StrategoObserver observer, IParseController parseController, long delay). Unlike StrategoAnalysisQueue.queueAnalysis, it does not verify if an update job is already pending with the path.