I've been a vivid fan of thesixtyone.com - until they changed the design. The old site can still be found here: http://old.thesixtyone.com/.

Many of my most favorite songs are from this site and I have only been able to listen to them through the site. However, since the aforementioned design change, the site has really died down in my opinion and I was fearful ever since that it would simply go down some day and vanish - taking my songs and playlists with it.

As counter-measure I've written a java console application to extract playlists and songs from thesixtyone. It uses Selenium to remote-control a FireFox instance that uses the normal user-interface to play songs and read in playlists.

I've uploaded the code to launchpad: https://launchpad.net/t61extractor/trunk
I've tested and used it myself and it runs alright.

This was a small weekend project (or rather two weekend project) that I did a few months ago but I only found time now to write about it.

The code should be mostly self-explanatory and it's not a lot of code either.

Cheers,
Andreas

I've been playing around with a Java grammar for ANTLR that was supposed to work straight-away but it did not, with very strange warnings and errors, that made it look like ANTLR only supports lexers with a lookahead of 1 character:

warning(209): ...: Multiple token rules can match input such as "'*'": STAR, STAREQ

while STAR matches only '*' and STAREQ only matches '*='. This is a huge w-t-f, especially if you have worked with ANTLR before and didn't have issues with this. This also contradicted all documentation you can find about ANTLR and its lexer rules.

I've spent a considerable amount of time with Google trying to find how to fix it. First I've found lots of posts on the ANTLR mailing list [antlr-interest] from people who had the same issue and no replies to them (really helpful, eh?). People had issues with replacing character ranges with unicode ranges (or rather a huge list of unicode characters), which probably caused the problem in my grammar, too. Others found that ANTLR suddenly behaved as if it only had a one character lookahead, but only if more than 300 lexer rules were used in the grammar.

After searching for a long time and almost giving up on the mini-project I've wanted to use ANTLR for, I've found this post: http://www.antlr.org/pipermail/antlr-interest/2009-September/035954.html (which matches my problem more or less but with additional insight)
and someone even replied (someone being the guy who maintains the C runtime of ANTLR):
http://www.antlr.org/pipermail/antlr-interest/2009-September/035955.html

If you are sure that the messages are not correct and the lexer rules
are not ambiguous, then you probably need to increase the conversion
timeout:

-Xconversiontimeout 30000

if that does not work, then there is a conflict in your rules.

Jim

And that turns out to be the right advice and the remedy to my problems and the problems of lots of other people probably.
However, no warning or error message I encountered mentioned that ANTLR's internal processing actually timed-out and there was no ambiguity in the grammar itself...

This comes to show that any good tool like ANTLR can quickly degrade to a piece of crap and a major source of annoyance, if error and warning messages aren't clear and helpful.

On further investigation, you can trigger warnings that the conversion times out:

internal error: org.antlr.tool.Grammar.createLookaheadDFA(Grammar.java:1279):
    could not even do k=1 for decision 121; reason: timed out (>1ms)

but not consistently. I guess this is a bug - either in ANTLR or in ANTLRWorks...

Because people complained to me about the formula feature in my PowerPointLaTeX add-in, which used a somewhat experimental approach to editing formula objects by adding an editing text shape that contained the formula code and that would be merged back into the formula as soon as you deselect it, I decided to rewrite it to use a standard modal dialog to edit formula objects:

Updated Ribbon (above) and Formula Editor Dialog (below)

The editor isn't perfect (yet), but it certainly shouldn't add any bugs to the add-in and solve some natural issues the old approach created.

Implementation Note

The idea was pretty straight-forward but the actual UI design was a PITA due me not knowning the panel/flow/table layout concepts very well and the code still has some annoying quirks with auto-scroll, so I need to fix that later.

I almost rewrote the whole cache system, because I'm using a background thread for updating the preview (if the text is changed, a 500 msec timer is started which triggers an update) and the update accesses the cache system, which in turn accesses PowerPoint to return some data, which in turn is busy because of the modal dialog -> dead-lock.

The solution to this is very simple but was not obvious to me at first (I actually began to rewrite the cache system with a feeling that there should be an easier solution):
The background thread needs an Invoke call to update the preview picture because the control has been created by a different thread (the main thread) and the code to get an updated picture can simply be moved into Invoke delegate function.

This solved all my problems and made 4 hours of previous work and thinking about a new cache system obsolete

Download the new build at: http://code.google.com/p/powerpointtools/downloads/list

Cheers,
Andreas

Today I want to write about something I've been working ages ago - specifically in March I wanted to see if I can extend a Java compiler to support LINQ´ expressions, too.

I probably spend more time on finding a good open-source compiler to experiment with than I later spent on trying things out, so let me share my preferred source with you: http://openjdk.java.net/ is a good address to start with.
More specifically http://openjdk.java.net/groups/compiler/ contains some valuable information about the way the compiler works.
A nice thing is that there is a branch that has added support for ANTLR which makes added language a tad bit easier since you get to change a grammar file instead of tweaking hand-written lexers and parsers. More info about it can be found at http://openjdk.java.net/projects/compiler-grammar/.
You can download the source code from http://hg.openjdk.java.net/ - don't follow the link to http://hg.openjdk.java.net/compiler-grammar/compiler-grammar, that one will only allow you to download part of the branch´.

I didn't come around to add support for LINQ in the end, but to get known to the compiler and the ANTLR grammer, I added support for the var keyword as known from C#, which allows for automatic type deduction and for anonymous objects (again using the C# syntax). Thus my changes allowed for the following to compile and execute correctly:

public class Test {
	public  static void main(String[] args) {
		// automatic type deduction
		var t = Math.atan(1);
		System.out.println( t );

		// anonymous type
		var i = new { Amount = 108, message = "hello" };
		System.out.println( i.Amount );
	}
}

Automatic Type Deduction

Let's take a look at how I added support for the var keyword, which requires an initializer at the variable declaration and automatically deduces the type used and uses that.

This itself is not a feature that I'd actually recommend using in general because it obfuscates a variable's type´ and makes it harder to read and understand the code but in conjunction with LINQ and anonymous types, it is very useful, because you don't want to know the type of the query´.

The nice thing about ANTLR is that it's.. nice.
I'm going post the diff from my repository to show how easy it was to add support for the var keyword in the grammar file:

It originally looked like this:

localVariableDeclaration returns [com.sun.tools.javac.util.List<JCStatement> list]
        @init {
            [...]
        }
        @after {
            [...]
        }
    :   variableModifiers type
            {
                mods = $variableModifiers.tree;
                type = $type.tree;
            }
        va1=variableDeclarator
            {
                JCExpression ntype = pu.makeTypeArray(type,$va1.i, $va1.dimPosition, $va1.endPosition);
                JCStatement ntype1 = T.at($va1.pos).VarDef(mods, $va1.name, ntype, $va1.tree);
                //pu.storeEnd(ntype1, $va1.stop);
                ptree = ntype1;
                listBuffer.append(ntype1);
            }
        (cm=',' va2=variableDeclarator
            {
                JCExpression ntype = pu.makeTypeArray(type,$va2.i, $va2.dimPosition, $va2.endPosition);
                JCStatement ntype1 = T.at($va2.pos).VarDef(mods, $va2.name, ntype, $va2.tree);
                pu.storeEnd(ptree, $cm);
                ptree = ntype1;
                listBuffer.append(ntype1);
            }
        )*
    ;

I changed it to:

localVariableDeclaration returns [com.sun.tools.javac.util.List<JCStatement> list]
        @init {
            [...]
        }
        @after {
            [...]
        }
    :   variableModifiers
            {
                mods = $variableModifiers.tree;
            }
        (VAR |
         type
        	{
        		type = $type.tree;
        	}
        )

        va1=variableDeclarator
            {
                JCExpression ntype = null;
                if( type != null )
	                ntype = pu.makeTypeArray(type,$va1.i, $va1.dimPosition, $va1.endPosition);
                JCStatement ntype1 = T.at($va1.pos).VarDef(mods, $va1.name, ntype, $va1.tree);
                //pu.storeEnd(ntype1, $va1.stop);
                ptree = ntype1;
                listBuffer.append(ntype1);
            }
        (cm=',' va2=variableDeclarator
            {
                JCExpression ntype = null;
                if( type != null )
	                ntype = pu.makeTypeArray(type,$va1.i, $va1.dimPosition, $va1.endPosition);
                JCStatement ntype1 = T.at($va2.pos).VarDef(mods, $va2.name, ntype, $va2.tree);
                pu.storeEnd(ptree, $cm);
                ptree = ntype1;
                listBuffer.append(ntype1);
            }
        )*
    ;

The code changes the way a local variable declaration works by using (VAR | type) instead of type and later in the grammar the VAR token is added

VAR	:	'var';

and a lookahead rule also needs to be adapted´:

localVariableHeader
    :   variableModifiers (type|VAR) IDENTIFIER ('['']')* ('='|','|';')
    ;

The code doesn't enforce that a var variable needs to have an initializer, this is later done in the actual code. It would be easy to add a flag to the variableDeclarator but it would require even more changes to the grammar file.
Now we only need to run ANTLR and regenerate the parser and lexer from the grammar and we're done with this part.

The main change is in visitVarDef in MemberEnter (which completes the Enter stage - see http://openjdk.java.net/groups/compiler/doc/compilation-overview/index.html for more info):

public void visitVarDef(JCVariableDecl tree) {
        Env<AttrContext> localEnv = env;
        if ((tree.mods.flags & STATIC) != 0 ||
            (env.info.scope.owner.flags() & INTERFACE) != 0) {
            localEnv = env.dup(tree, env.info.dup());
            localEnv.info.staticLevel++;
        }
        // old: attr.attribType(tree.vartype, localEnv);
        // BlackHC: deduce the type from the initializer if we have a variant
        if( tree.vartype == null ) {
            if( tree.init != null ) {
                tree.vartype = make.Type(attr.attribExpr(tree.init, localEnv));
                tree.vartype.type = tree.init.type;
            }
            else {
                log.error(tree.pos, &amp;quot;initializer.required.for.implicit.type&amp;quot;);
                return;
            }
        }
        else {
            attr.attribType(tree.vartype, localEnv);
        }

        Scope enclScope = enter.enterScope(env);
        VarSymbol v =
            new VarSymbol(0, tree.name, tree.vartype.type, enclScope.owner);
        v.flags_field = chk.checkFlags(tree.pos(), tree.mods.flags, v, tree);
        tree.sym = v;
        if (tree.init != null) {
            v.flags_field |= HASINIT;
            if ((v.flags_field & FINAL) != 0 && tree.init.getTag() != JCTree.NEWCLASS) {
                Env<AttrContext> initEnv = getInitEnv(tree, env);
                initEnv.info.enclVar = v;
                v.setLazyConstValue(initEnv(tree, initEnv), log, attr, tree.init);
            }
        }
        if (chk.checkUnique(tree.pos(), v, enclScope)) {
            chk.checkTransparentVar(tree.pos(), v, enclScope);
            enclScope.enter(v);
        }
        annotateLater(tree.mods.annotations, localEnv, v);
        v.pos = tree.pos;
    }

The code simply initializes the initializer expression's type early and sets the variable's type to it.
Because of this Attr's visitVarDef needs to be adapted to avoid recreating the type later´:

                    // BlackHC: this if condition is a hack to keep anonymous objects, etc. from breaking >_<
                    if( tree.init.type != tree.vartype.type )
                        attribExpr(tree.init, initEnv, v.type);

Now only an additional line needs to be added to res/compiler.properties to add the error message text that should appear if the initializer is missing and we're done:

compiler.err.initializer.required.for.implicit.type=
    initializer required for implicitly-typed variables

I also added a line to com.sun.tools.javac.main.Main's compile function to display a custom string to make sure that the correct compiler is run, but that's just cosmetic.

With this, a new keyword has been added to the Java compiler with a few lines being changed only. The compiler itself is not that straight-forward to understand if you're not used to its design, but it's still amazing that it's that easy.
It took me about 15 hours at most to implement this feature. 80% of the time was spent looking through the code and grammar and identifying how to best add the keyword and implement it.

If you want to test your compiler something like the following command-line is needed (on Windows)´:

java -cp MyJavaC\bin;antlrworks-1.2.3.jar com.sun.tools.javac.Main Test\Test.java

Anonymous Objects

This was an even simpler feature to implement that did not require any code changes at all.
The change only allows for local var variables but this is just because we only changed the localVariableDeclaration rule.
Adding anonymous objects (ie. new { fieldName = initializer [, ...] }) is straight-forward once you have automatic type deduction and if you think about it, it's obvious that it's nothing but a rewrite of new Object() { public var fieldName = initializer; [...] }.
ANTLR shows its strength here:

creator returns [JCExpression tree]
        @init {
                [...]
        }
    :   'new' nonWildcardTypeArguments cr1=classOrInterfaceType cl1=classCreatorRest
            {
                [...]
            }
    |   'new' cr2=classOrInterfaceType cl2=classCreatorRest
            {
                createdName = $cr2.tree;
                args = $cl2.list;
                body = $cl2.tree;
                $tree = T.at(pos).NewClass(null, typeArgs, createdName, args, body);
                pu.storeEnd($tree, $cl2.stop);
            }
    // BlacHC: add C# anonymous types
    |	'new' '{' typebody=anonymousTypeBody b2='}'
		    {
		    	createdName = T.at(pos).Ident(names.fromString("Object"));
		    	$tree = T.at(pos).NewClass(null, typeArgs, createdName, args, $typebody.tree);
		    	pu.storeEnd($tree, $b2);
		    }
    |   arrayCreator
            {
                $tree = $arrayCreator.tree;
            }
    ;

anonymousTypeBody returns [JCClassDecl tree]
	    @init {
            ListBuffer<JCTree> defs = new ListBuffer<JCTree>();
            JCTree ptree = null;
            String dc = ((AntlrJavacToken) $start).docComment;
        }
		@after {
 			JCModifiers mods = T.at(Position.NOPOS).Modifiers(0);
            $tree = T.at(((AntlrJavacToken) $start).getStartIndex()).AnonymousClassDef(mods, defs.toList());
            if (ptree != null) {
               	pu.storeEnd(ptree, $stop);
           	}
        }
    :	(va1=variableDeclarator
            {
                JCVariableDecl tree = T.at($va1.pos).VarDef(T.at(Position.NOPOS).Modifiers(Flags.PUBLIC | Flags.FINAL), $va1.name, null, $va1.tree);
                pu.attach(tree, dc);
                ptree = tree;
                defs.append(tree);
             }
        (cm=',' va2=variableDeclarator
            {
                JCVariableDecl tree = T.at(va2.pos).VarDef(T.at(Position.NOPOS).Modifiers(Flags.PUBLIC | Flags.FINAL), $va2.name, null, $va2.tree);
                pu.storeEnd(ptree, $cm);
                ptree = tree;
                pu.attach(tree, dc);
                defs.append(tree);
            }
        )*)?
	;

This is all that is needed. The code is mostly copy'n'pasted from other rules (classOrInterfaceType and classCreatorRest) and it wasn't really that difficult. The long compile times of ANTLR were the only obstacles when writing it.

Try it

I've uploaded my current sources (ready to compile and run) and you can download the zip here.
Just execute the compileAndRun.bat and my javac and the test should be compiled and run.

What Else?

This is it for today. But let me tell you about a few final thoughts:

• Hacking away on compiler code and grammar files is a lot fun´
• It's not feasible for real projects, because you don't want to start questioning the validity of the compiler you're using - I had that with QuakeC and it wasn't fun at all - and chasing compiler bugs is terrible in general when you want to spend your time working on project's actual code
• I have started working on a preprocessor that would read in Java code with the extended syntax and emit normal 1.6 Java code.
  If you want to do something like this, you can probably find a grammar of your language on ANTLR's homepage - for Java it is: http://openjdk.java.net/projects/compiler-grammar/antlrworks/Java.g.
• This approach is more difficult though because you suddenly lose the nice functionality that gives you an expression's type for free (which is a non-trivial thing to code on your own if you consider imports and local classes, etc.).
• It's best thing to do, if you want to extend the language and it's a good idea to give it a thought if you have a project whose code could greatly benefit from some additional language features that can be easily emulated using normal code, to´.
• ANTLR is nice for quick prototyping even though the hand-written Java parser and lexer are faster in general.

I've finally managed to upload a version of my PowerPoint LaTeX add-in for PowerPoint 2007.

I'm just going to post a short Vimeo video here that shows how inline formulas work (the main feature):

Click here to view the embedded video.

I've tried to work on the add-in during my spring vacation but somehow I have instead spent all my time watching four seasons of House M.D. (which was totally worth it though ).

There are still some features like support for MikeTeX, code clean-ups, small bugfixes and a preference window that I should work on, but I don't plan on selling it, so I don't really care if it's still somewhat work in progress.
I'm going to continue working on it when I have to use PowerPoint again.

You can check it out (and another add-in dubbed Language Painter that I wrote to fix some annoyances when writing presentations in languages different from your keyboard layout) at http://code.google.com/p/powerpointtools/.

"Inter-Mortem"

I'm not really done with the project but I think it's time nonetheless to talk about the project itself. Initially I looked into using TexPoint for a presentation I had to prepare, but it wasn't very user-friendly in my opinion and I also didn't really want to install MiKTeX again, so the idea was born to use a web service (from codecogs.com or wordpress.com) to render the equations and embed the formulas as pictures. A few tests later I knew that it was feasible, so I started to work on the project.

The bulk of the code was written in 4 days during my winter vacation (so that was 4 months ago) - I've only added minor things since then.

This warrants the question what took me so long to upload it and/or write about it. The main reason why it was delayed by such a long time is that I wanted to polish it some more and make sure it really has a well-rounded feature set. In retrospect I shouldn't have added support for non-inline equations because people will hardly use them and the way you edit them is kind of crazy (I create a text shape in the current slide to let the user change the equation) and I reverted lots of the later changes because I wasn't sure which direction to take UI-wise.

The user interface is mostly okay in my opinion but the ribbon might need a redesign to streamline it some more and also maybe come up with new things users would want to do. For that I'll probably wait until I have to use it again (probably in a few weeks' time) and then take notes about what to improve.

The design of the code is somewhat messy when you look at it, but that's because it grew out of a experimental project to see if it was possible to embed LaTeX code the way I wanted, and most of the functionality ended up in one big file/class (LaTeXTool.cs)  because everything is centered around inline formulas.