tl;dr: Altran’s Xtext / Sirius Integration provides the full Xtext experience for Sirius direct editors (a.k.a. pressing F2 in diagram) and properties editors. It’s highly configurable, well-documented, and released under EPL 2. I presented it at EclipseCon 2018. We intend to contribute this to the Sirius project.

Example of Xtext’s error checking and auto-completion support within Sirius diagram figure

Both textual and graphical modeling have their well-known strengths and weaknesses – obviously, we want the best of both worlds!

Xtext and Sirius are among the most-used Eclipse Modeling Framework technologies, covering textual and graphical modeling, respectively. Previous   EclipseCon talks discussed the different usage scenarios and difficulties of integrating these technologies. We focused on the “Xtext editors within Sirius diagrams and properties view” scenario and could solve most of the integration difficulties. The result is called Xtext / Sirius Integration.

Our main goals were to be a good Sirius citizen, i.e. configuring all details in the Odesign model, and to support as much of Xtext’s features as possible without requiring changes to the Xtext language.

We would like to contribute the code to the Sirius project. Initial discussions already started, so stay tuned.

Main Features

Use Cases

  • Use Xtext as direct editor for Sirius diagram figures and connections
  • Use Xtext in Sirius property editors
  • Edit parts of the Sirius editor’s model with Xtext
  • Edit text stored in the model with Xtext

Configuration Options

  • Single-line or multi-line Xtext editor
  • Limit the editable features
  • Configure pre-selected features
  • Prepend or append text to model contents in order to provide a valid Xtext document
  • Ignore selected nested features
  • Use a different Xtext grammar for persisting and editing a model

Examples

The source code of all examples is available at github.

The github readme provides more details on the examples.

Smart UML Class Attributes

Assume a model and diagram akin to UML class diagrams.

Each attribute has a visibility, a name, a type, a multiplicity with lower and upper bounds, and a (possibly long) description text. We can edit these features in one line with full Xtext support.

Xtext reports invalid visibility

Xtext provides auto-completion for referenced constants

Xtext / Sirius Integration allows to edit all the visible features of the attribute just as we see them, without fiddling around in property editors. It automatically checks for valid input (i.e. does not accept ~ as visibility) and provides auto-completion for references to declared constants.

Pre-selecting the most edited part

We hide the description feature from the Xtext editor, as it makes no sense to edit it in-line. Also, we pre-select only the attribute’s name. This way, we changed only the name without touching the other features, if the user pressed F2, typed something, and pressed enter.

We can still edit all of the features separately in the properties view.

Markup Language for Descriptions

Assume a model where every element can have a description. Formally, the description is just a string feature of the element. However, the user may use HTML-like tags in the description. A build server task collects the descriptions of all elements of one model and combines them into one HTML page. We’d have an Xtext grammar for HTML.

HTML-like documentation language with Xtext support in Properties View

As HTML goes, we need to start with a <html> tag, followed by a <head> section, and only in the <body> the user may add their description text. Also, we need to finish the text by closing both <body> and <html>. We don’t want the user to add this boilerplate to every description, as it’s cumbersome, error-prone, and we’d need to remove it in the build step. Still, we’d like to use our Xtext grammar.

With Xtext / Sirius Integration, we define a prefix (<html><head><title>dummy</title></head><body>) and suffix (</body></html>). This way, the model contains only the actual description, and the user still benefits from all the goodies of our Xtext HTML language, like using only valid tags, or closing them in the correct order.

We can provide such an editor to the user within the diagram and/or in the properties view.

Smart UML Class Associations

Assume a model and diagram akin to UML class diagrams. Each class is represented by a rectangle, associations between classes are shown as connections.

The model is persisted with Xtext. Example:

constant MAX_ROOMS = 23
constant MAX_PEOPLE = 42
constant MORE_CONSTANT = MAX_ROOMS

class House {
  public inhabitants: string[1..MAX_PEOPLE]
}
class Room {
  public size: integer[2..2]
}

association rooms House --> Room[1..MAX_ROOMS]

The Xtext grammar might be:

grammar com.example.classes with org.eclipse.xtext.common.Terminals

generate classes "http://example.com/Classes"

ClassModel:
	content+=Content*
;

Content:
	Constant
	| Class
	| Association
;

Constant:
	'constant'
	name=ID
	'=' initial=Value

// omitting Class, Value, ...

Association:
	'association'
	name=ID
	source=[Class] '-->' target=[Class]
	'[' lowerBound=Value '..' upperBound=Value ']'

We want to display and edit the name and multiplicity of the association as connection label. We want to get full Xtext support (e.g. auto-completion for referenced constants in multiplicity).

However, for technical reasons, we can only hide features at the beginning and/or end of the element’s text. To solve this issue, we create another Xtext language and use this one in our diagrams:

grammar com.example.classes.edit with com.example.classes

import "http://example.com/Classes"

EditClassModel returns ClassModel:
	ClassModel
;

@Override
Association:
	'association'
	name=ID
	'[' lowerBound=Value '..' upperBound=Value ']' 
	source=[Class] '-->' target=[Class]
;

We extend the original grammar and override the Association grammar rule. All other grammar rules remain untouched. We move the source and target features to the end of the element’s text. This way, we can hide them from the user: They should edit the source and target by dragging the connection.

Model in original grammar. Note the association’s source and target are in-between its name and multiplicity.

Xtext editor using the alternative grammar, hiding the association’s source and target features.

Complications

We spent more than six months on this development, and had to overcome quite a few hurdles. Please read about some of them below.

Deep Technology Stack

Deep technology stack

Both Xtext and Sirius are very mature, flexible, and complex frameworks. It’s not surprising they both stand on the shoulder of giants, i.e. re-use lots of other technologies. Xtext / Sirius Integration balances on top of these behemoths standing on other giants. In this height, the air is thin and we feel the Law of Leaky Abstractions.

To achieve our goal of best possible integration, we have to weave together both technology stacks on various levels, leading to lots of gory gut work.

Sirius follows a very rigorous “separation of concerns” approach (partially because it seems to be required by underlying technologies). As a result, for each “thing” in Sirius (e.g. a node, connection, or tool), there is one class in each concern. As soon as we required even the slightest adjustment on any aspect, we had to duplicate this whole thread of classes throughout all concerns.

Graphical / Event Integration

I was most scared about the graphical and event integration, as my previous experience in this field suggested. Luckily, there is another Open Source project called Yakindu Statecharts. It provides, among lots of other features, graphical state machine modeling. It’s not based on Sirius, but shares the same base (Graphical Editing Framework GEF, to be precise). Yakindu Statecharts also provides Xtext editors within the diagram – jackpot! We could re-use most of the drawing, sizing, and event integration.

As a sidenote, I think this is a prime example of Open Source’s power: Xtext is Open Source, Sirius is Open Source, and so is Yakindu Statecharts. This whole integration would never have been possible without access to the source code! Statecharts happily accepted a pull request to improve re-usability of their code. Thanks again!

Model Integration

Both Xtext and Sirius edit a model, but in very different ways. Integrating them was the biggest technical challenge – as we knew from previous EclipseCon talks.

Note: The following discussion touches quite intricate issues; explaining them with text only is tough. If you’re lost, view my talk at EclipseCon on YouTube.

Xtext / Sirius Integration can work on two different bases:

  1. Storing the Xtext-edited model as text in the Sirius model
  2. Editing the same model with Xtext and Sirius

Base 1) is not that hard: When the user presses F2, we open an Xtext editor, read the text attribute from the Sirius model, Xtext parses it and creates its own, independent model. The user edits the Xtext model, and when the editor is closed, we ask Xtext for the changed text and write it back to the text attribute in the Sirius model.

Base 2) is much more challenging. To understand why, we have to look at some details.

(We assume the Sirius model is persisted via Xtext. At Altran, this is the most common use-case, as this allows sensible diff/merge on models without taming EMFCompare. Anyways, most of the discussion applies equally to other persistence formats.)

Let’s have a look at the lifecycle of a Sirius diagram persisted via Xtext:

Lifecycle of a Sirius diagram

  1. The user opens a Sirius diagram.
  2. Sirius figures out the diagram is based on a model that’s persisted via Xtext. Sirius asks Xtext to read the model.
  3. Xtext reads the model text file.
  4. Xtext parses the file contents and creates an read-only Abstract Syntax Tree (AST).
  5. Xtext converts the AST into a model instance.
  6. The model instance is ready to use. Sirius uses the model instance to display the diagram.
  7. The user changes the diagram, e.g. they add some connections or remove some elements.
  8. Sirius changes the underlying model without saving it. At this point, the contents of the file and the model instance differ.
  9. The user saves the diagram.
  10. Sirius asks Xtext to persist the current state of the model instance.
  11. Xtext serializes the model into text. Xtext does not change the AST.
  12. Xtext writes the text into the text file.

So what changes if the user didn’t use Sirius to change the model, but the Xtext / Sirius Integration?

Lifecycle of Xtext edit within Sirius diagram

We should only show to the user the interesting part of the underlying model. If they pressed F2 on one UML Class Attribute (as an example), they should only edit this Attribute, not the whole model.
We cannot use the existing AST, as it might be outdated. Also, for fundamental technical reasons, Xtext cannot parse/serialize parts of a model file.
Thus, we ask Xtext to serialize the complete model instance into a memory buffer. We create a dummy Xtext file and copy the text there. Xtext does its usual work of parsing and creating the dummy model instance (which, at this point, is identical to Sirius’ model instance).
We hide all text from the user they should not edit. Again for technical reasons, we can only hide text before or after the visible text, not somewhere in between. This allows also to hide some features of the edited attribute: If the UML Class Attribute’s description was the first feature in the model text, we can hide it – and the user cannot edit it any more.

The user changed the text to their wishes. As Xtext flags any issues directly, we assume the user did not introduce any errors. But all these changes happened in the dummy Xtext file; we have to get them back into the Sirius model instance.

We could try to merge the changed text from Xtext into the model’s text. But we all know text merge is not reliable on big changes, so we prefer to merge on a model basis. Given the known restrictions, model merge works mostly ok. It gets complicated if the user changed names or similar attributes that are part of an element’s URI.

Even more complicated are cross-references. Assume our model instance contains Constants and Classes with Attributes. The Attributes’ initial values can refer to the Constants. If the user pressed F2 on the Attribute, we create a complete copy into a dummy model instance. The user asks auto-complete for available Constants, and auto-complete proposes the Constants from the dummy model instance.
Once the user is finished, we merge their changes back into the Sirius model instance. But we only merge the edited Attribute and throw away the rest of the model – including the referenced Constant!

Luckily, both Xtext and Sirius are based on EMF, and EMF knows a concept called proxies. If EMF loads a model and finds a reference to an element outside of this model, it only creates a placeholder, a.k.a. proxy, for the external element. This placeholder stores the URI of the external element. Once required, EMF loads the external resource and replaces the proxy with the real thing.
We abuse this mechanism a bit to solve our dangling reference problem: Before merging the Attribute back into the Sirius model instance, we check the Attribute for any references to the dummy model. If we find one, we convert the target into a proxy. After finishing the merge we ask EMF to resolve all proxies in the Sirius model instance. As the target was there in the copied dummy model instance, and we didn’t change anything outside the merged part, we can be sure these proxies will be resolved.

We can follow a similar approach for references from the Sirius model instance to element merged back from the dummy model instance.

odesign Integration

The developer defines all aspects of a Sirius diagram in a odesign file. We wanted to keep this approach. Thankfully, Sirius provides all the necessary hooks – but they are hard to find. Interestingly, the hooks work very different for seemingly similar aspects: Adding context menu entries to the Properties part of the odesign file requires a completely different approach than adding context menu entries to the Diagram part of the odesign file.

I’m quite satisfied with the result: Only the most advanced use cases require coding outside the odesign file.

Further Reading

Please refer to the extensive user guide (also available as PDF) for more details and known issues.

The github readme provides more details on the examples and information about required dependencies.

Xtext / Sirius integration is released under EPL 2 at github.

We provide an update site.