Design Log

Most of current language extensibility work focuses on macros. This precludes alternative tools for the languages. Sometimes tools are interested in the deep semantics of the source. Such tools would need to expand constructs to more primitive ones. Others like text editors are only interested in surface syntax. Expansion into deeper constructs only would confuse them. However without expansion it is often impossible to check for correctness of the syntax.

The language consciously limits recursion forms to only expressions and blocks. It would have been possible to introduce other recursive constructs too. For example def construct might have invocation semantics instead of replace semantics.

The current intuition about other constructs is that it would have made the language so rich that it would be possible to define new constructs in non-uniform way. Therefore we would get back in the hell of languages that looks completely unlike each other.

At some point of language design alternative phrase syntaxes were considered. The sections below list rejected alternatives.

The parser logically works as function that takes input as argument and returns AST.

Tokens are mapped to simple values in the AST like string, integers, or even enumeration. There are also objects that have properties that hold these simple values. Also objects might have properties that hold other objects.

When it is straightforwardly mapped to object models of programming languages, there is a problem. Position information might be attached only to objects. It cannot be attached to primitive values.

This led to design of token wrappers which is shorter form of the following construct. For example there is the following construct:

integer wrapper xj:IntegerLiteral.value

That is shorthand for the following:

^ xj:IntegerLiteral {
  @ value = integer;
}

However possibly it is a misplaced workaround. The position information is reported by term parser for primitive values too. It is specific object model what cannot represent them. So possibly we should just make tokens a primitive in the model. And request that if the tokens should be wrapped to object to represent position in object model, than it should be done by mapping level rather than by parser, because parser returns all required information.

Also it looks like in some cases we might not care about whether token wrapped or not, and usage of unwrapped token would be faster.

Somewhat similar problem is with includes. It is unreasonable to expect that Class statement will implement abstract method statement subclass. So currently a wrapper that wraps class statement might be specified in include from method content context to classifier context in EJ grammar.

However it is a fix to limitation of the object model. It is not clear whether this fix is well placed.

At one time context include construct allowed the following form:

grammar A {
	import B = "other.g.etl";
	context c {
		include e from B;
	};
};

It looked like a nice construct at first. The problem were that it were hard to define a clean include semantics for it. Particularly how it will interact with grammar and context includes.

This construct has been eliminated in favor of the future suppress/rename constructs that allow achieving the same goals in more clean way. However it might be possible to return to this issue in the future when include semantics is well defined.

Attributes and documentation statements insert values at some attribute inside normal statement. However these objects act as prefix operators that act on statements rather than on expressions. So more logical way would be generating objects like AttributedStatement and DocumentedStatement .

Primary problem with such approach is usability. The user needs to retrieve attributes and documentation associated with statement. Not a statement associated with attributes or documentation.

As one of hacks currently implemented are fallback statements. These statements are considered to become active if there error in attributes or documentation comments. However the hack is quite dirty from point of view of implementation. It will be yet another special case from point of view of LL1 grammar.

If there is a statement with empty look-ahead, it is chosen as fallback. Otherwise arbitrary statement is chosen.