PS The cpp changes were needed because hxcpp uses the same template format as haxe interpolation for its xml directives, but Haxe is not supposed to process those, so I made the double-quoted strings required for @:buildXml and other c meta.

Ok, I'm all for the part that fixes things. But I would request at that it be EFormat(original:String, parts:Ref<Array<FormatFragment>>);.

From my experience, when you're parsing JSX-like syntax, the node syntax comes first and then the interpolated expressions. Just consider what you get from <foo bar=${123}>${321}</foo>. Even the opening tag is split apart. I'm sure it's well-intended, but in reality it's not that useful.

Take a look at the haxe-react parser, you will see that it first uses Xml.parse to get the raw structure, and then picks up interpolation within that. Similarly, tink_hxx parses the tag structure primarily and identifies code blocks within it. In both cases the raw string is needed. Also, for compatibility with JSX, both libraries support <tag>{expr}</tag>, so having some parts preparsed and others not only adds to the complexity.

With these changes merged, we'd have to reconstitute the string. I assume that the parser for the "mini-AST" is necessary for string interpolation to work properly. But pushing down the information onto macros is pretty wasteful, if it's not needed. First the AST has to be allocated as interpreter value. And if the original string is not available, you will need to pass it to haxe.macro.Printer to get back the string which then still requires unescaping, for which I guess one would use haxe.Json.parse to not have to do it by hand. Only then do you have the String you wanted, actually start parsing it, and call back to Context.parseInlineString to parse again the expressions that you've just converted to a String \o/

I would request at that it be EFormat(original:String, parts:Ref<Array>);.

That sounds fair. I'm not sure about Ref because IIRC we don't really use it for untyped AST structures, but I see no particular reason for that. And yeah having original strings might be useful (also it'll help avoiding the c metadata issue - we can just use the original string).

That said, I don't see much complexity in parsing <foo bar=${123}>${321}</foo>, if anything one can do something like:

var xmlParts = [];
var interpolatedExprs = new Map();
var nextId = 0;
for (part in parts) {
    switch part.kind {
        case FRaw(s):
            xmlParts.push(s);
        case FIdent(i):
            var id = nextId  ;
            xmlParts.push('{$id}');
            interpolatedExprs[id] = macro $i{i};
        case FExpr(e):
            var id = nextId  ;
            xmlParts.push('{$id}');
            interpolatedExprs[id] = e;
    }
}
var xml = xmlParts.join("");

// ...parse and process xml, taking exprs from interpolatedExprs map

I get that this might not be enough for haxe-react because it wants to support that <tag>{expr}</tag> syntax which Haxe has no idea about. But I can easily imagine valid JSX-like (and not only) DSLs based on actual haxe String interpolation syntax, if we expose it in macros.

Meh, I don't know, having both original string and format segments makes it easy to construct invalid ast nodes (that is with original string and segments that don't correspond to each other), and having segments as a Ref prevents modifying it though macros. Gotta think about it some more.

The thing is I want to have accurate position tracking down to the character and tink_hxx accomplishes that by parsing the XML itself.

So then I could try to pad the placeholder with whitespace to the length of the expression the place is held for (which I assume I can get from the fragment's position). All this works only if the original expression was not shorter than the placeholder, e.g. can't replace $a by {$42} and somehow get the offsets to remain fine.

That leaves you with some pretty pesky gymnastics to get the offsets right. Yes, it's doable, but I would very much prefer not to have to do it ;)

As for your concerns: I think the typer should just ignore the raw string.

A different approach would be to have CString(s:String, doubleQuote:Bool) and MacroStringTools.getStringFragments. Both map and iter could do:

case CString(s, false): f(MacroStringTools.formatString(s, e.pos));

If you can somehow avoid to decompose the string twice into fragments (or if parsing cost is negligible compared to sending the result down to the interpreter), this would probably be the simplest and fastest approach.

Alright, I'm closing this for now so it's not accidentally merged, will reopen when ready :)

What's the agenda here? I would really like to resolve the string quoting mess...

We gotta figure out what API we want for this, so I know what exactly to implement :) It seems that we want to keep strings at the untyped AST level and parse it later when typing, right? But I think it could be useful to provide access to the mini-ast in macro somehow, so people can easily use interpolated strings for their DSLs.

Could we do this in two steps?

1. Clean up our internal mess.
2. Figure out how to expose it to macros.

I have to look into this again...