JVM weirdness

May 8, 2014
Compiling lib/ABC/BrokenRhythm.pm to jar
===SORRY!===
Function 'ABC::Note' needs parens to avoid taking the block
at lib/ABC/BrokenRhythm.pm:36
------>             }⏏<EOL>
Missing block (apparently taken by 'ABC::Note')
at lib/ABC/BrokenRhythm.pm:37
------>             ⏏when ABC::Stem { ABC::Stem.new($note.not

The code in question (from BrokenRhythm.pm) is

when ABC::Note { 
    ABC::Note.new($note.accidental,
                  $note.basenote,
                  $note.octave,
                  ABC::Duration.new(:$ticks), 
                  $note.is-tie); 
}

I believe it’s the first ABC::Note there that is causing the problem.

This appears to be legal to me, and works fine under Parrot and Moar.

Tune Reminder

May 5, 2014

We’re going to Newfoundland later this year, and as a result I have a list of tunes I’d really like to learn before I get there, and I need to brush up on the ones I already know. (The sad truth is there isn’t a lot of chance in Michigan to practice many of these tunes with other musicians.)

Years ago I wrote a Perl 6 script to help me track what I needed to practice. I meant to make it use spaced repetition, but the truth is I got so caught up using Niecza and GTK# to implement a simple GUI interface for the program I never got around to actually adding the spaced repetition part. I used it for a bit, and then it fell into disrepair.

Well, I’ve dusted off the repo it was in and added a new script which attempts to do an approximation of actual spaced repitition. And it was beautifully simple to write, barring a small bug in the Mix class that lizmat++ fixed before I could even break out the rakudo source.

The first slightly tricky bit is initializing the record of how you’re doing on each team. I wanted it to be able to handle the case where you’re running the code for the first time and have no records yet, and the case where there have been new tunes added. Each tune gets a level between 0 and 5.

my @status;
if $status-database.IO.e {
    my $file = from-json($status-database.IO.slurp);
    @status = @$file, 0 xx ($tunes.GetNumberTunes - @$file);
} else {
    @status = 0 xx $tunes.GetNumberTunes;
}

I used the JSON::Tiny module to handle I/O of the records, and initialize tunes not seen before with 0.

The main loop of the code is very straightforward:

loop {
    my $mix = @status.kv.map(-> $tune, $status { $tune => 1 / 2 ** $status }).Mix;
    my $tune = $mix.roll;
    say $tunes.GetTuneName($tune);
    given prompt "Know / Don't Know / Quit: " {
        when "q" | "Q" { last; }
        when "k" | "K" { @status[$tune] = (@status[$tune] + 1) min 5 }
        when "d" | "D" { @status[$tune] = (@status[$tune] - 1) max 0 }
    }
    say "";
}

First I create a Mix based on the each tune’s status. The idea is that the higher the status level for the tune, the less likely you are to be prodded about it. Mix.roll is used to choose a random tunes with those weights taken into account. Then we print the name of the tune and use a simple combination of given and prompt to record how you did. (Note that this version doesn’t have any error checking here!)

Once you’ve hit “q” to exit the loop, the final line just records changes to the tune status file:

spurt($status-database, to-json($@status));

All-in-all, quite easy to code, and I’m already using it to help direct my practice!

rakudobrew

May 5, 2014

Sorry for the long silence here.  My autumn was a mess, and my winter was buried in snow.  In all that time I have kept working with Perl 6, and things actually keep on getting better and better, which is quite exciting.

One really exciting thing for me is rakudobrew.  tadzik’s lovely little script makes it easy to build and rebuild multiple different versions of rakudo and switch between them.  In the past, keeping up-to-date with Rakudo was a matter of typing and executing four commands, the second of which was very fiddly to remember and type.  Now it’s just
rakudobrew build moar and everything moar-related is updated, including all your installed modules.  That may sound like a small change, but it works and it’s terrifically handy.

So, last night I tried to move the module smoke tester to using rakudobrew.  This allowed me to delete nearly half the lines in my smoke_test script, and added the potential to test all three of rakudo’s backends instead of just Parrot.  However, I ran into problems on all three backends, none of which are rakudobrew’s fault.

First and most vividly, rakudo under parrot is completely broken at the moment.  benabik++ tracked down the breaking patch two days ago.  It’s been broken since May 1st.  It really says something about the current state of rakudo use on parrot that almost nobody seems to have even noticed this.  (Hint: I’ve been using rakudo on moar for pretty much everything for a month or two now.)

rakudobrew does a beautiful job building and installing rakudo on moar.  However, the module smoke test (which tries to install every module) fails because installing File::Find breaks the module system, possibly because File::Find is used in that installation process.

==> Successfully installed File::Find
Invalid opcode executed (corrupt bytecode stream?) opcode 4352

This one has been the case for a while now.  Last night I was hoping I could get around it by just having emmentaler (the smoker) specifically skip testing File::Find, but it turns out a number of other modules depend on it.

rakudobrew built the JVM version of rakudo with no issues.  But the smoke test ran very slowly and died after making it through a few modules.  I haven’t really had time to look into this one, just wanted to mention it.

One other issue I’ve been bumping into, again not rakudobrew’s fault but making it less than perfectly awesome on OS X.  When you build rakudo on moar on the Mac, the makefile dies with the message make: write error after it has finished building and installing rakudo.  Attempts turn on make’s debugging tools cause a seg fault.  This is pretty innocuous when you’re building by hand, but rakudobrew actually (implicitly?) checks the results of the make, and stops before going on to rebuild the modules.  As nearly as I could tell, this seems to be an actual bug in GNU make on OS X, but I have no idea how to make further progress fixing it.

This may sound like a lot of complaints, so let me close by pointing out how amazingly awesome the progress has been in the last year.  I don’t know that anyone but jnthn believed in early May last year that we’d have rakudo fully working on JVM and yet another backend by this time.  Rakudo on moar has become clearly the best Perl 6 we have.  For ages I’ve needed to keep Niecza around to handle a few things that rakudo couldn’t do, but rakudo on moar is rapidly overtaking it.  And development continues at a furious pace.  This is really a great time to be a Perl 6 user.

Rakudo performance

August 2, 2013

Last week I had an amazing adventure in Perl 6. I had a specific goal I needed to accomplish for my day job: determining which face in a STEP model was disappearing when the model was tessellated.

I could determine a face I knew was connected to the face I was looking for, so I decided to try to find all the faces that shared an edge with that face. I have a set of Perl 6 tools for parsing STEP files. It was pretty easy to modify the code I already had to create a script to read & parse the file, walk the topology tree to each edge on the specified face, and then back from those edges to the faces that use them.

This worked well enough when I fed it a small sample file. It did not work so well when I fed it the actual 13M file I needed to work with. I left it running overnight, and it had a fatal error (presumably out of memory) sometime in mid-morning the next day. I think at that point I switched from trying it on Rakudo-Parrot to Rakudo-JVM, with essentially the same results. I added a bunch of say statements, and eventually determined that the line

my %reverse = $step-data.entities.keys.categorize({ $step-data.entities{$_}.map(*.entity_instances) });

was the guilty party. This line built the reverse lookup table, so you could find all the entities that reference a given entity — once built, this lets you quickly walk from an edge to each face that used it. Unfortunately, it seemed to eat up all the memory on my 32G Linux box.

At this point, I figured using categorize was a mistake. I wasn’t sure why it was, mind you. (Though my test program examining the question ended up leading to a separate impressive optimization from jnthn++.) But it occurred to me there was a different approach that avoided it entirely. Instead of walking back from the edges to the faces, just walk from all the faces to their edges, and then look for faces whose set of edges intersected the given face’s set of edges. Promisingly, this approach was about 33% faster on my small test file. And it didn’t use categorize.

However, when I tried this on the big model, this led to the same exact same ballooning memory problem as the first script. Now, though, I had something more concrete to work with, and a few minutes later, I established that the issue was $step-data.entities.kv. On a big hash (this one had 182500 elements), calling .kv or .keys would consume all available memory on my system. You could create the hash, no problem, and if you knew the keys you could access them quite quickly. But asking it to give you the keys killed Rakudo.

With this I had a nice simple golfed test case I could give to jnthn. And before the morning was out, he gave me a patch to fix it. With the patch, the script executed correctly in 11 minutes under Rakudo-JVM. (Note that I did have to change the Java -XmX maximum memory setting in the perl6 script to get it to work.) It also executed correctly in Rakudo-Parrot -- but in 7 hours, 52 minutes.

Let me emphasize that. For this real-world task of significant size, Rakudo-JVM was 40 times faster than Rakudo-Parrot.

Note that I don't know why. But I do know that jnthn++'s fix which was needed to make this work on both Rakudos was in no way JVM-specific. This isn't an example of optimized Rakudo-JVM versus non-optimized Rakudo-Parrot.

But the important thing here is that this represents a very significant improvement in Rakudo's performance by switching to JVM. The script is pretty basic core stuff, mostly file I/O, grammar parsing, and hashes. The improvement is much smaller on a small data-set -- on my small test file, Rakudo-JVM is not even twice as fast as Rakudo-Parrot. But throw a big task (well, this big task, anyway) at Rakudo, and Rakudo-JVM crushes Rakudo-Parrot.

Set Operations

June 21, 2013

A month or so ago, while working on ABC, I tried to use [∪] (reduce on set union) in my grammar actions. Unfortunately, it blew up when I tried it in Niecza. I assumed the problem was that I hadn’t implemented zero or one arguments forms of set union, and made a quick effort to add them. It didn’t work, so I worked around the problem by using the set constructor directly and added it to my to-do list.

I finally got around to tackling the problem again this week, and my earlier difficulties were quickly explained when I looked at the stack trace for the error message. turns out to be list associative. That means if you write $a ∪ $b ∪ $c, the actual call generated is infix:<∪> ($a, $b, $c), and the reduce meta-op generates that call internally. So I needed to change infix:<∪> from a binary sub to an N-ary sub.

Previously the code looked like this

proto sub infix:<∪>($, $ --> Set) is equiv(&infix:<|>) {*}
multi sub infix:<∪>(Any $a, Any $b --> Set) { $a.Set ∪ $b.Set }
multi sub infix:<∪>(Set $a, Set $b --> Set) { Set.new: $a.keys, $b.keys }
multi sub infix:<∪>(Baggy $a, Any $b --> Bag) { $a ∪ $b.Bag }
multi sub infix:<∪>(Any $a, Baggy $b --> Bag) { $a.Bag ∪ $b }
multi sub infix:<∪>(Baggy $a, Baggy $b --> Bag) {
    Bag.new-from-pairs(($a.Set ∪ $b.Set).map({ ; $_ => $a{$_} max $b{$_} }))
}

So the rules were, if you had a Baggy (ie Bag or KeyBag), you promoted both arguments to Bag. Otherwise both arguments were promoted to Set. Also note that, because of this, the proto signature was wrong — it could generate a Set or a Bag.

I replaced them with a single sub:

only sub infix:<∪>(\|$p) is equiv(&infix:<|>) {
    my $set = Set.new: $p.map(*.Set.keys);
    if $p.grep(Baggy) {
        my @bags = $p.map(*.Bag);
        Bag.new-from-pairs($set.map({ ; $_ => [max] @bags>>.{$_} }));
    } else {
        $set;
    }
}

Line 2 creates the Set version of the union. Then we check to see if any of the arguments are Baggy using $p.grep(Baggy). (That filters out all the non-Baggy from the argument list, and then (because it is used in a boolean context) returns true if the resulting list has any elements in it.) If there is a Baggy, then we convert all the arguments to Bag. Line 5 is a spiffy bit of code that creates a new Bag using the maximum count for each element found in $set. Finally, if there were no Baggyarguments, just return $set.

Once I had that working, I did the same for infix:<∩>. Then I started looking at infix:<(-)>. I’ve only ever seen set difference used as a binary operation, so my first step was to think about how to generalize it to N arguments. My thought was $a (-) $b (-) $c should be ($a (-) $b) (-) $c. If someone has a good reason this shouldn’t be the case, please let me know!

Then the next question was, what should Baggy objects do here? Previously set difference always converted its arguments to Set. But it seemed to me there was a pretty obvious way to do it. After consideration, I concluded it should only use the Bag form if the first argument (the “set” things are being subtracted from) was a Baggy. Here’s my current code for infix:<∖> (note the new name, which is the ISO symbol for set difference rather than the common backslash):

only sub infix:<∖>(\|$p) is equiv(&infix:<^>) {
    return ∅ unless $p;
    if $p[0] ~~ Baggy {
        my @bags = $p.map(*.Bag);
        my $base = @bags.shift;
        Bag.new-from-pairs($base.keys.map({ ; $_ => $base{$_} - [+] @bags>>.{$_} }));
    } else {
        my @sets = $p.map(*.Set);
        my $base = @sets.shift;
        Set.new: $base.keys.grep(* ∉ @sets.any );
    }
}

This works (I think — I haven’t really tested the Baggy functionality, I’ve only run the existing Set tests). But it breaks some spectests (which assume set difference always returns a Set). And it involves the unspecified details mentioned above, so I’m stepping out on a bit of a limb. So I’m posting here looking for feedback before I commit these changes.

(Edited to add: I’ve also added infix:<⊖> for symmetric difference; that’s one of the standard symbols used, but it’s not ISO standard.)

Philosophical Issues with Rakudo’s .parse

March 6, 2013

For a little bit now I’ve had the ABC module working on both Niecza and Rakudo. I got a new Linux box yesterday; this morning I was playing around with my ABC tools there, trying to see if I could get them to work. I installed them with panda, so I was using ABC with Niecza by default on my MBP and with Rakudo by default on my new machine. I tried it with a file of my own compositions. Apparently I hadn’t tried this in a while, because it had several issues. And the two systems handled them completely differently.

On Niecza, I got this message:

Unhandled exception: Did not match ABC grammar: last tune understood:
 A Hat for the Whales
d|:e~A3 BGEF|G2BG DG (3ABd|eBdB ~A3G|Bdef gfed|
e~A3 BGEF|G2BG DG (3ABd|eBdB AAGA|[1Beed e3d:|

On Rakudo, there were no error messages, but "A Hat for the Whales" was the last tune processed, even though there were a dozen more tunes in the ABC file after it.

Without thinking about it too much I corrected this batch of errors. Now on Niecza the entire file parsed, but I got the error

Unhandled exception: Illegal key signature B minor

Under Rakudo, the same ABC file just appeared to work.

What's going on here? Well, key signature processing looks something like this (simplified):

    token key-def { <basenote> <chord_accidental>? <mode>? }
    token mode { <minor> | <major> }
    token minor { "m" ["in" ["o" ["r"]?]?]? }
    token major { "maj" ["o" ["r"]?]? }

No space allowed between basenote and mode. So when you invoke parse with that rule on “B minor”, in Niecza it fails, because the entire string was not successfully parsed. In Rakudo it succeeds, because “B” is a valid key signature, and it doesn’t care that the entire string was not parsed. In practice, this means that three of the tunes in the collection got the wrong key signature: B major instead of the correct B minor.

To my mind, Niecza’s behavior is more intuitive. If you’re trying to parse a file, parsing half of it and then silently ignoring the rest is wildly unhelpful behavior. In every use I’ve had for parse so far, I’ve wanted it to parse everything in the string I sent to the function.

But hey, I understand that some people might have different needs. Can we at least add a flag that will give you a parsing failure if the entire string is not parsed? :all, perhaps?

STEP Actions

October 16, 2012

I’ve started adding actions to the STEP code, targeted (at least at first) at collecting up the entity instance names in a parameter list to make it easier to analyze what’s in a STEP file.

I need to do this more often, because I spent about an hour fumbling about making stupid mistakes. And even now that I’ve got it working, I’m haunted by the notion there should be a simpler approach. Anyway, here’s what I’ve got at the moment:

class ISO_10303_21::Actions {
    method entity_instance_name($/) { make [~$/] }
    method parameter($/) {
        for <typed_parameter untyped_parameter omitted_parameter> -> $s {
            return make $/{$s}.ast if $/{$s}.defined;
        }
    }
    method omitted_parameter($/) { make [] }
    method untyped_parameter($/) {
        for <entity_instance_name list_of_parameters> -> $s {
            return make $/{$s}.ast if $/{$s}.defined;
        }
        make [];
    }
    method typed_parameter($/) { make $<parameter>.ast }
    method list_of_parameters($/) { make merge-arrays(@($<parameter>)».ast); }
    method parameter_list($/) { make merge-arrays(@($<parameter>)».ast); }
}

What’s bugging me are the parameter and untyped_parameter definitions — those seem like they should be easier to express somehow. At least, that basic functionality is going to be pretty common.

Oh, and I just found (and fixed) a bug in the code looking at this blog post.

Feel like I had something more to say, but I’m tired and heading to bed.

Parsing STEP

October 10, 2012

A month ago I started perl6-ISO_10303-21. In plain English, it’s the beginnings of a toolset for working with CAD files in the STEP format. All it does is parse so far.

The STEP (that’s ISO-10303) Part 21 standard provides a BNF (I think) grammar for ASCII STEP files. It was downright trivial to translate it to a Perl 6 grammar — looking at the commits, it took me about two days to get it to read the first file. THen it sat there for a long time until I starting throwing files at it.

Since then I’ve made a couple of simple changes. First, I replace C-style comments in the file with spaces before parsing; the spec’s grammar doesn’t mention them at all.

Second, I discovered three of the basic sample files I was using were actually illegal, with newlines embedded in strings. I fretted about what to do for a while: should I expand the grammar to accept illegal but “standard” files, or keep the grammar strict so that I could use it to test files my own code exports for correctness?

Then I realized I could have my cake and eat it too:

grammar ISO_10303_21::LooseGrammar is ISO_10303_21::Grammar {
    token non_q_char { <special> | <digit> | <space> | <lower> | <upper> | \v }
}

I just defined a new subclass, changing the non_q_char token to also accept vertical whitespaces, too. Now ISO_10303_21::Grammar is strict, and ISO_10303_21::LooseGrammar is loose.

The next obstacle I’ve run into was speed. The 1.3 meg sample file took about five minutes to prase in Rakudo, and 10+ hours (!) in Niecza. Luckily jnthn++ was on the job, and sent me a patch about fifteen minutes after I sent him the profile of the parse today. (Rakudo’s got --profile built in.) That shaved off 1/6th of the running time.

He also pointed out that tokens like

token standard_keyword { <upper> [ <upper> | <digit> ]* }

were unnecessarily creating a Match object for each character in the keyword. Since all we want is the full text of the keyword, this could be changed to

token standard_keyword { <.upper> [ <.upper> | <.digit> ]* }

and it would suppress those Match objects. That reduced the running time by another 10%.

Another few days like this and it will be fast enough to do some useful work for me…

Sums of Fourth Powers

October 5, 2012

Another fun John Cook post has sent me off into playing with numbers and Perl 6.

He gives a footnote saying that Euler discovered 635318657 = 158^4 + 59^4 = 134^4 + 133^4 and that this was the smallest number known to be the sum of two fourth powers in two ways. It seems odd now to think of such questions being unresolved. Today we’d ask Hardy “What do you mean 635318657 is the smallest known example? Why didn’t you write a little program to find out whether it really is the smallest?”

Perl 6 has features that make that fun to explore, so let’s go to it! Here was my first attempt:

my @fourth-powers = (1..*).map(* ** 4) ...^ * > 635318657;
my %counts;
for @fourth-powers X+ @fourth-powers -> $sum {
    %counts{$sum}++;
}

for %counts.keys.grep({ %counts{$_} > 2 }) -> $sum {
    say "$sum has { %counts{$sum} / 2 } fourth power sums";
}

This works, and is quite fast in Niecza. (It’s much slower in Rakudo for some reason, though it still finishes in about ten seconds.) I love that first line, which constructs a lazy infinite list of fourth powers, then truncates it at 635318657. Then it uses X+ to build all the sums of two fourth powers less than (or equal to) 635318657. The rest of the code is pretty mundane, I fear.

I decided to try to use classify to get rid of the explicit loops, resulting in this code:

my @fourth-powers = (1..*).map(* ** 4) ...^ * > 635318657;
my @results = (@fourth-powers X+ @fourth-powers).classify({ $_ }).pairs.grep(*.value > 2);
say @results.map({ $_.key ~ " " ~ $_.value.Int / 2 ~ " times" });

Unfortunately, this doesn’t work in Niecza. So now I’ve got some bugs to fix…

Complaining about Mono

July 2, 2012

So, I’ve hit on something extremely annoying working with Niecza on my MacBook Pro. I upgraded to Mono 2.10.9 a few months back. Since then, Glib# has not worked. Even after I uninstalled Mono 2.10.9. It is incredibly frustrating.

Here’s the error I’m getting:

Unhandled exception: System.TypeInitializationException: An exception was thrown by the type initializer for GLib.GType ---> System.DllNotFoundException: glibsharpglue-2

Here’s the call:

constant $GLIB = "glib-sharp, Version=2.12.0.0, Culture=neutral, PublicKeyToken=35e10195dab3c99f";

constant $G_TYPE_STRING = CLR::("GLib.GType,$GLIB").String;

Here’s what

gacutil

says about glib-sharp:

glib-sharp, Version=2.12.0.0, Culture=neutral, PublicKeyToken=35e10195dab3c99f

If anyone has any suggestions on how to fix this, I’m all ears.


Follow

Get every new post delivered to your Inbox.