Raku, Go, Chroma, Germanium, Hugo and my blog

When I created my Raku programming language website with Hugo static site generator, I had to manually create the syntax highlighted html files for code examples shown in the front page, because Hugo used Chroma(written in Go and uses a mostly similar API to Pygments) for syntax highlighting which didn't support Raku. So I opened an issue on Chroma, the discussion showed that you cannot just convert the Raku lexer from Pygments to Chroma; You would need to create some functions manually, and Chroma needed a new Emitter interface which would take the LexerState as an argument, because Raku has some complex syntax; the Raku lexer in Pygments had functions for finding closing brackets matching the opening brackets, and for regex nesting in tokens, regexes and rules. In the end I realized functions for token nesting were unnecessary.

But I was curious if I could do something with the broken lexer I generated with the pygments2chroma.py script that comes with Chroma. So I began working on the Raku lexer without knowing Go, Chroma, Python or Pygments.

Chroma

After playing with the lexer for a while, I was very close to giving up. But after looking at the Chroma code and the functions it provided and also looking at some other lexers, I began to come up with some ideas.

For finding closing brackets I used the function generated by pygments2chroma and modified it a lot and also as a workaround I placed empty capture groups inside the patterns that needed bracket finding; and I filled those placeholders with the bracketsFinder function(or mutator).

I also used a placeholder rule inside regex which would be replaced by bracketsFinder with the appropriate rule which Poped from the regex state when the delimiter matched.

I did the same thing with POD formatters(C<code>, B<bold>,...), although I didn't have to, and could create a state for each formatter that needed different tokens for highlighting.

I also created Emitters for Raku's colon pair syntax and POD config.

So I finally managed to create an imperfect Raku lexer for Chroma.

I added some things to Chroma itself as well; Added support for named capture groups(although there is something people need to watch for. regexp2, the regex package Chroma uses, implements .Nets' regex engine which doesn't maintain the order of captures, in other words, the number capturing groups come first then named capturing groups are appended, which I found surprising, and made a pull request to add an option for maintaining the capture order (I don't know if it'll get merged), added the ByGroupNames helper function, and made the emitter functions take *LexerState as argument, so they have access to more things.

I should have done these things before I made the Raku lexer, so I could use them while I was making it, but at the time, I didn't know I could do it! And also improved the Svelte, SCSS and Fish lexers. Additionally, added two new styles, doom-one and doom-one2 which are inspired by Atom One and Doom Emacs's Doom One themes. You can use them in version 0.9.2.

Lastly, the things I learned Chroma command line tool can do(which of course you can find by running chroma --help):

Printing syntax highlighted files to terminal:

chroma  -f terminal256 -s doom-one source.raku

Generating SVG file:

chroma -s doom-one source.raku --svg > output.svg

Generating styled HTML files:

chroma -s doom-one --html --html-all-styles --html-lines source.raku > output.html

You can install Chroma with:

go install github.com/alecthomas/chroma@latest

Or if you use Arch Linux, you can install it from AUR.

Generate images from source code with Germanium

Germanium(written in Go) is an alternative to Silicon, which is written in Rust and uses Syntect which uses Sublime Text syntax definitions for syntax highlighting, which does not support Raku. Syntax definitions are written in YAML files, I don't know if you can put logic in there. Also they say they don't accept new packages! Maybe Syntect accepts packages in their submodule? I don't know.

Anyway, I added support for options choosing a style and also listing styles. Also changed the Chroma version in the dependencies, so new lexers and styles can be used in Germanium v1.2.0.

Germanium: raku and go

Generating images:

germanium -s doom-one -o output.png source.raku

You can install Germanium with:

go install github.com/matsuyoshi30/germanium@latest

Or if you use Arch Linux, you can install it from AUR.

Golang

While writing the lexer, I often felt frustrated, because I wanted some functionality and would search for it and find three things:

  • A GitHub issue with a proposal for what I wanted, which would be a closed issue and not implemented
  • A Stack Overflow answer which said: you can do it with "this long and manually written code"
  • A Stack Overflow answer which said: you can emulate it like this

Which made me make a joke about it.

Go, to me is the complete opposite of Raku. And yet even though I didn't like it and the frustration that came with it, strangely I began to kinda like it! I contributed to different Go projects in a relatively short time, that says something nice about Go. So I may actually try to learn it. Even though I see this when I visit their website(because of Google App Engine and sanctions! Google usually goes to the extreme).

google-403-forbidden-iran

Lastly, there are things the go command line tool doesn't provide(or maybe I'm unaware of it). Such as updating all modules installed globally(especially useful for command line apps).

Hugo and my blog

My blog was generated by Jekyll until now. I don't blog often, so I didn't have much motivation to migrate to Hugo, even though Hugo has more features. But I said what the heck, I created a lexer that Hugo uses so I might as well use it. Therefore, I migrated my blog to Hugo and even used some of its features, for example I created categories and tags, added TOC and used the markdown render hooks feature for headings to linkify them and images to linkify them to their sources which is useful when the image is large.

Raku lexer is available since Hugo v0.84.0.

Raku

Finally some syntax highlighting for Raku inside Hugo with random code from here and there:

#!/usr/bin/env raku

=head1 Raku syntax highlighting

=begin code :lang<go>
package main

import "fmt"

func main() {
    fmt.Println(`Hello, Raku`)
}
=end code

put 'Hello, Go!';

#`(
multi
line
comment
)

say qqsome $scalar text」;
say qwwsome $variable 'some other text' text」;
say q:wwsome $variable 'some other text' text」;
say q:wsome $variable 'some other text' text」;
say qq:wsome $variable 'some other text' text」;
say Q:csome $variable 'some text' { 2 + 1 } text」;
say q:asome @array[1] 'some text' { 2 + 1 } text」;
say Q:a:csome @array 'some text' { 2 + 1 } text」;
say Q[some \qq[$variable.method()] testing];
say 'some \qq[$variable.method()] testing';
say "some @array[2] %hash{'test'} $variable.method() func() testing";

use Some::Module:auth<author>:ver(v1.*.0+);

say X::SomeException;

#| B<Fibonacci> with multiple dispatch
multi sub fib (0 --> 0) {}
multi sub fib (1 --> 1) {}
multi sub fib (\n where * > 1) {
    fib(n - 1) + fib(n - 2)
}
say fib 10;
# OUTPUT: 55

# Indirect invocant
notes $trip: "Almost there";

LABEL:
for <a b c> {
    .put;
}

# Roles and classes
role Shape {
    method area { ... }

    method print_area {
        say "Area of {self.^name} is {self.area}.";
    }
}

class Rectangle does Shape {
    has $.width is required;
    has $.height is required;

    method area {
        $!width * $!height
    }
}

Rectangle.new(width => 5, height => 7).print_area;
# OUTPUT: Area of Rectangle is 35.

#| U<INI> Parser
grammar INIParser {
    token TOP { <block> <section>* }
    token section { <header> <block> }
    token header { '[' ~ ']' \w+ \n+ }
    token block { [<pair> | <comment>]* }
    rule pair { <key> '=' <value> }
    token comment { ';' \N* \n+ }
    token key { \w+ }
    token value { <-[\n ;]>+ }
}

my $match = INIParser.parse: q:to/END/;
; Comment
key1=value1
key2 = value2

; Section 1
[section1]
key3=value3
END

say $match<block><pair>[0]<value>;
# OUTPUT: 「value1」

say $match<section>[0]<block><pair>[0]<value>;
# OUTPUT: 「value3」

grammar Calculator {
    token TOP { <calc-op> }

    proto rule calc-op          {*}
    rule calc-op:sym<add> { <num> '+' <num> }
    rule calc-op:sym<sub> { <num> '-' <num> }

    token num { \d+ }
}

class Calculations {
    method TOP              ($/) { make $<calc-op>.made; }
    method calc-op:sym<add> ($/) { make [+] $<num>; }
    method calc-op:sym<sub> ($/) { make [-] $<num>; }
}

say Calculator.parse('2 + 3', actions => Calculations).made;
# OUTPUT: «5␤»

grammar G {
    rule TOP { <function-define> }
    rule function-define {
        'sub' <identifier>
        {
            say "func " ~ $<identifier>.made;
            make $<identifier>.made;
        }
        '(' <parameter> ')' '{' '}'
        { say "end " ~ $/.made; }
    }
    token identifier { \w+ { make ~$/; } }
    token parameter { \w+ { say "param " ~ $/; } }
}

G.parse('sub f ( a ) { }');
# OUTPUT: «func f␤param a␤end f␤»

# Inifinite and lazy list
my @fib = 0, 1, * + * ... ;
say @fib[^11];
# OUTPUT: (0 1 1 2 3 5 8 13 21 34 55)

# Feed operator
@fib[^20] ==> grep(&is-prime) ==> say();
# OUTPUT: (2 3 5 13 89 233 1597)

# Function composition
my &reverse_primes = &reverse  &grep.assuming(&is-prime);
say reverse_primes ^20;
# OUTPUT: (19 17 13 11 7 5 3 2)

my @a = 1..4;
my @b = 'a'..'d';
# Zip two lists using Z meta operator
say @a Z @b;
# OUTPUT: ((1 a) (2 b) (3 c) (4 d))
say @a Z=> @b;
# OUTPUT: (1 => a 2 => b 3 => c 4 => d)

# Hyper Operators
say @b «~» @a;
# OUTPUT: [a1 b2 c3 d4]

# Junctions
say 'Find all the words starting with a lowercase vowel'.words.grep: *.starts-with: any <a e i o u>;
# OUTPUT: (all a)

ay '🦋'.chars;
# OUTPUT: 1
say '🦋'.codes;
# OUTPUT: 1
say '🦋'.encode.bytes;
# OUTPUT: 4

my $raku = 'راکو';
say $raku.chars;
# OUTPUT: 4
say $raku.uninames;
# OUTPUT: (ARABIC LETTER REH ARABIC LETTER ALEF ARABIC LETTER KEHEH ARABIC LETTER WAW)
say $raku.comb;
# OUTPUT: (ر ا ک و)
say +$raku.comb;
# OUTPUT: 4

subset  of Int where * > 0;

sub f ( $a,  $b --> Array of ) {
    Array[].new: $a², $b²;
}

say f 1,2;
# OUTPUT: [1 4]

# Native Types
my int @a = ^10_000_000;
say [+] @a;
# OUTPUT: 49999995000000

sub MAIN(
    Str   $file where *.IO.f = 'file.dat',  #= an existing file to frobnicate 
    Int  :size(:$length) = 24,              #= length/size needed for frobnication 
    Bool :$verbose,                         #= required verbosity 
) {
    say $length if $length.defined;
    say $file   if $file.defined;
    say 'Verbosity ', ($verbose ?? 'on' !! 'off');
}

# $ script-name
# Usage:
# script-name [--size|--length=<Int>] [--verbose] [<file>]

#    [<file>]                 an existing file to frobnicate
#    --size|--length=<Int>    length/size needed for frobnication
#    --verbose                required verbosity

# Using NativeCall to access libnotify and show a notification
use NativeCall;

sub notify_init (str $appname --> int32) is native('notify') { * }
sub notify_uninit is native('notify') { * }
class NotifyNotification is repr('CPointer') { * }
sub notify_notification_new (str $summary, str $body, str $icon --> NotifyNotification) is native('notify') { * }
sub notify_notification_show (NotifyNotification) is native('notify') { * }

if notify_init 'MyApp' {
    notify_notification_show notify_notification_new 'My Notification', 'Notification Body', Str;
    notify_uninit;
}

# Concurrency
react {
    my $current-proc;
    whenever $script.watch.unique(:as(*.path), :expires(1)) {
        .kill with $current-proc;
        $current-proc = Proc::Async.new($*EXECUTABLE, $script);
        my $done = $current-proc.start;
        whenever $done {
            $current-proc = Nil;
        }
    }
}

my $modules-load = start @files
    .grep(/ \.(yaml|yml) $/)
    .sort(-*.s)
    .race(batch => 1, degree => 6)
    .map(-> $file {
        my $yaml = self!load-yaml($file, $schema, $problems);
        with $yaml {
            Easii::Model::Module.new(parsed => $yaml, source => $file.basename)
        }
    })
    .eager;


# Supply
my $bread-supplier = Supplier.new;
my $vegetable-supplier = Supplier.new;

my $supply = supply {
    whenever $bread-supplier.Supply {
        emit("We've got bread: " ~ $_);
    };
    whenever $vegetable-supplier.Supply {
        emit("We've got a vegetable: " ~ $_);
    };
}
$supply.tap(-> $v { say "$v" });

$vegetable-supplier.emit("Radish");
# OUTPUT: «We've got a vegetable: Radish␤» 
$bread-supplier.emit("Thick sliced");
# OUTPUT: «We've got bread: Thick sliced␤» 
$vegetable-supplier.emit("Lettuce");
# OUTPUT: «We've got a vegetable: Lettuce␤»

=finish
Goodbye!