Buddy class loading in Eclipse is a mechanism to load a plugin into another (third-party) plugin that you do not control, to inverse the dependencies. Oskar added buddy class loading to Spoofax and its languages in 2012 (by commit: https://github.com/metaborg/spoofax/commit/673d44b9d575064ef342b4a65b74d7b3ca652026) for some use case with modelware.

Say we have the language Dynsem, that does buddy class loading into the Spoofax runtime. When the Spoofax runtime is loaded, Dynsem will also be loaded and its classes will be included into the classpath of the Spoofax runtime. Dynsem’s Stratego files are compiled to Java classes, so that means that the generated Stratego Java classes are included with Dynsem, and those classes get loaded into the Spoofax runtime by buddy class loading.

When analyzing a Spoofax language, say Stratego, a HybridInterpreter is created and the JARs of Stratego are loaded into that HybridInterpreter, to load the generated Stratego Java classes. Java uses class loaders to manage class loading, and typically have a parent class loader that loads classes that cannot be found in the current class loader. When loading classes for the HybridInterpreter, the Spoofax runtime class loader is used as the parent class loader to load common classes, such as the Stratego standard library, into languages, without duplicating them in memory.
However, this parent class loader also has the Dynsem classes on the classpath, because of buddy class loading! So when loading classes into the HybridInterpreter, it may actually use classes from the Dynsem language, causing the analysis to be done by Dynsem instead of Stratego.

This problem will occur with any language that buddy class loads into the Spoofax runtime, that compiles Stratego to Java classes. This issue is most likely the cause of https://yellowgrass.org/issue/Spoofax/784 as well.

A quick fix/workaround for this issue is to disable buddy class loading for all languages, but we are not sure if Oskar still needs this feature, so we should ask him. The proper fix would not use the Spoofax runtime class loader as the parent class loader, but rather a separate class loader that includes everything that a language needs. Another problem that triggers this problem, is that the generated Stratego Java class files are included on the classpath of the language plugins, which should not be the case. Those classes should be compiled to include/[langname].jar, and not included on the classpath.

Submitted by Gabriël Konat on 1 October 2014 at 15:05

Good find, that explains a lot.
Spoofax Modelware doesn’t rely on it anymore. I’m using it for something else, but can easily work around.

Besides the proper fix that you propose, I see three workarounds that can be used in the meantime.
Note to others: what we try to achieve is to have Spoofax projects depend on other projects in the workspace. These projects are not packed as Jar. For Jars, we already have the provider solution.

1) Let Spoofax runtime still use buddies for class loading. However, remove buddy registration from every Spoofax project. Then, have users register a buddy for every non-Spoofax project that they depend on. Note: dependencies between two Spoofax projects can be handled via foreign-call(|lang,strategy), so no need to register a buddy for a Spoofax project.

2) Remove buddy class loading functionality altogether. Instead, compile dependent projects when building include/entity-java.jar.
This can be achieved by replacing <property name="src-dirs" location="editor/java"/> in your build.main.xml to <property name="src-dirs" location="editor/java../HelloWorld/src"/>, assuming that HelloWorld is a project that we want to depend on. In contrast to solution 1, this even allows you to load and use functionality from HelloWorld dynamically, which means that as a language developer, you don’t need to work with two Eclipse instances in the case HelloWorld doesn’t contribute to the user interface. However, class files for HelloWorld end up in include/entity-java.jar and when deploying the projects, they also end up in HelloWorld_someVersionNumber.jar. This duplication is not very neat.

3) Remove buddy class loading functionality altogether. Instead, have users add a task to their build.main.xml that creates a include/HelloWorld.jar and have them add include/HelloWorld.jar as a provider. When deploying the projects, they will end up with a HelloWorld_someVersionNumber.jar and a include/HelloWorld.jar, but they could delete the latter and make the former a provider using some scripting. This gets a bit complicated though.

I propose to remove buddy class loading functionality altogether and use solution 2 or 3 for the time being. Removing buddy class loading functionality from Spoofax runtime, means that legacy projects don’t need to be updated. They would still register themselves as a buddy, but nothing is done with this information. Also, solution 2 and 3 allow things to be dynamically loaded, which is nice.

I’ll remove buddy class loading functionality for now if nobody disapproves.

I’m happy we finally figured out what had been causing that weird behavior for so long now :). If i understand correctly the buddy class loading mechanism was added to have allow a Spoofax project invoke strategies of another Spoofax project. The buddy class loading ensured that the two plugins (languages, projects) were loaded by the same class loader and could see each other’s classes thus.

I think the solutions we discussed above address the issue in Spoofax. But they do not solve the issues of fragility and that of the HybridInterpreter’s leaky loading. At any time that the build of a Spoofax project’s Java classes will include the trans.strategies package in the plugin’s jar this issue will re-appear. If say we have a conflicting strategy between plugin A and B, and one editor of A was opened first (thus language A loaded first) then B will leak into strategies from A. The buddy classloading made sure that this issue did occur because it ensured that everybody was loaded by the same class loader.

I think we need a more robust solution that eliminates the possibility that leaks across HybridInterpreters can occur. One idea is to not root the HybridInterpreter loading in the parent class loader (that of Equinox/Eclipse) and instead load all classes ourselves. But this comes at the price of not being able to make calls across languages and that of duplication of classes in memory. A third alternative is to not redesign the loading mechanism but just add serious defenses against leaky loading. Defenses such that leaky loading is detected and logged. What’s your take on this?

I have already removed the buddy class loading from our meta-languages and from the generator, the problem does not occur any more, but could occur again if buddy class loading is added again. I think the buddy class loading is not required at all, let me explain why.

There are 3 ‘class spaces’ where classes from Spoofax languages reside in, all originating from the editor/java folder.

1. Eclipse plugin space, classes from all packages except [language].strategies and trans. Compiled to an Eclipse plugin JAR.
2. Stratego generated Java space, classes only from package trans. Compiled to include/[language].jar.
3. Java strategies (primitives), classes only from package [language].strategies. Compiled to include/[language]-java.jar.

The HybridInterpreter will load the JARs from class spaces 2 and 3, so that the Stratego generated Java classes and primitive classes are available in the HybridInterpreter.

Eclipse loads the classes from class space 1, those are only available to Eclipse plugins. Buddy class loading will only load classes from class space 1 into another plugin’s class space 1, so buddy class loading can normally never influence the HybridInterpreter, so we don’t need buddy class loading!

The only reason why buddy class loading was influencing the HybridInterpreter, is because the trans package is being included in the plugin, which is a bug I am going to look at.

The reason I added buddy class loading was to allow for having a Spoofax project that invokes Java code from a non-Spoofax plug-in project. The feature wasn’t meant to influence the HybridInterpreter.

The way I see it is that Eclipse doesn’t support dynamically loading plug-ins, yet we treat a Spoofax project as an Eclipse plug-in and try to load it dynamically. As a result, we run into all kind of user-interface limitations and are required to re-implement some of Eclipse’ class loading functionality, etc.

If I understand things correctly, Spoofax Eclipse has two modes of loading a language: dynamically (e.g. when developing and testing a language in the same Eclipse instance) and non-dynamically (e.g. when using a deployed language). However, both modes share most of the functionality such that loading non-dynamically basically means loading 95% of the features dynamically and 5% non-dynamically. For example, a Spoofax project has a MANIFEST.MF. This MANIFEST.MF defines the dependencies between projects, necessary for class loading purposes. However, in both modes, we try to do things dynamic and completely ignore the MANIFEST.MF. Instead, we have a single class loader and a single MANIFEST.MF (the one from org.strategoxt.imp.runtime) that is shared among all languages. That’s really not the way it should be.

IMO, we should either make things completely dynamic, which means that Spoofax projects should be regular Eclipse projects and not Eclipse plug-in projects (no plugin.xml or MANIFEST.MF), or they should be completely non-dynamic and be regular Eclipse plug-ins. I would actually argue for the later. Even though developing and testing in a single Eclipse instance is a great feature of Spoofax, it prevents us from properly integrating with Eclipse. Adding new user-interface features (think menus, code coloring schemes, formatters, etc) is either not possible or requires very ugly hacking. To me, Eclipse is simple not meant for this and we should either drop the dynamic feature or look for a more suitable IDE.

We shouldn’t force a dynamic-only, static-only, or eclipse-only solution. A Spoofax language project should be a regular Java project, a separate plugin project can then lift the language into a plugin so that we can use it in Eclipse. That needs to be done anyway if we want to use Spoofax languages on different platforms such as NetBeans.

Executing Java code from Stratego code is a separate issue. It is possible but it is tricky to set up, especially if you have third-party dependencies. The third-party dependencies have to be available during compilation of Stratego generated Java files and primitive Java strategies, on the classpath when running the Stratego code, and shipped with the Eclipse plugin so that Eclipse loads them when deploying the plugin.
I think the real issue here is the configuration, which is spread over several files of different systems (ESV, Eclipse META-INF, Ant scripts), making it complex to set up and maintain. A more unified configuration, and more modular code that handles the loading, will solve the problems you talk about.

Not being able to add (or having to hack-in) new user-interface features is another separate problem, which is caused by Spoofax’s crappy architecture where everything is lazily instantiated and destroyed, without any control. The only requirement that dynamic loading enforces on editor services, is that editor services need to be reloadable, because the language can change dynamically. This is solvable by a more reactive/event-driven architecture.

Anyway, this is going quite off topic, we should probably discuss this somewhere else :)