Hyperlinks in source code being ignored

[opoudjis]

These are a requirement in ISO-10303: both xref and eref inside of sourcecode need to be supported. They are not supported currently at all, even in the grammar, and are stripped from rendering.

I know why I did not want to support them (for years), and processing both them and Rouge code highlighting is going to be nasty; but the processing model for Rouge tells us how to deal with them: store their locations in the source code, and then reinsert them one by one at the right spot.

This is a project I would love to delegate. It involves generalising the code in lib/isodoc/presentation_function/sourcecode.rb , from stripping and then restoring callouts and annotations, to also stripping and then restoring xref and eref opening and closing tags. The catch is, callout and annotations only record the line location, and that is enough to restore the markup: callout only appears at the end of a line, annotations are dumped in order after the source code.

For eref and xref, the obvious thing that can be done is record (a) the line number, (b) the string being hyperlinked; (c) if there are multiple instances of that string, the iteration of the string to hyperlink.

Markup is removed and then added back in to allow rouge to do syntax highlighting (which means adding <span> tags to the lines); so tracking byte locations is not going to help us. erefs and xrefs will possibly conflict with spans, and that will mean (a) recursing to add hyperlinks within each span separately, and (b) doing something clever if the hyperlinked text spans across spans.

So, we get:

<sourcecode>
<xref target="abc">begin
  print</xref> "hello"
end
</sourcecode>

We strip the markup so we can do syntax highlighting:

<sourcecode>
begin
  print "hello"
end
</sourcecode>

We do syntax highlighting using Rouge:

<sourcecode>
<span class="a">begin</span>
  <span class="b">print</span> "hello"
<span class="a">end</span>
</sourcecode>

And then, SOMEHOW, we work out the right place to reinsert the xref and eref tags, dodging the spans hierarchy:

<sourcecode>
<span class="a"><xref target="abc">begin</xref></span>
  <span class="b"><xref target="abc">print</xref></span> "hello"
<span class="a">end</span>
</sourcecode>

We can't guarantee that the hyperlinked text are complete space-delimited words; but if that would help, we could require that. ISO-10303 should be ok with that. We could then still use regex matches for text, one word at a time.

[opoudjis]

https://github.com/metanorma/annotated-express/issues/124

[opoudjis]

It is true that (a) ISO-10303 is unlikely to hyperlink anything more than single words, and (b) ISO-10303 may not use code highlighting at all for Express. But a solution that does these both would be highly desirable as future-proofing this: the ability to hyperlink arbitrary code to code definitions is going to be a selling point of Metanorma strategically.

[ronaldtse]

This is a separate project by itself and requires a lot more specification.

This is actually a separate program to structure and annotate source code. Just like how VSCode handles source code highlighting. Technically, it's a document model for source code.

[ronaldtse]

But a solution that does these both would be highly desirable as future-proofing this: the ability to hyperlink arbitrary code to code definitions is going to be a selling point of Metanorma strategically.

It looks like we should integrate internal Rouge code directly instead of parsing it ourselves?

i.e. a Metanorma class that binds to a Rouge class?

[opoudjis]

This may not be urgent specifically for ISO 10303, because there doesn't seem to be any code highlighting going on currently with Express (because there is no code highlighter currently defined for it, presumably.) It does need to be done though, and I wouldn't rule out that a code highlighter for Express is coming.