19 Oct 09 Extracting Information from StudiVZ

Some time ago somebody stole 1 million data records from StudiVZ, the German Facebook clone. I’m not exactly sure why people call the person a hacker who stole data, because it appears he simply wrote a tool that harvested the publicly available data from StudiVZ (which everyone with an account can view).

People on StudiVZ share all their data by default—contrary to Facebook which values a person’s private data a lot more. Thus by simply opening each profile from a dummy user and processing the HTML data from StudiVZ one can extract a lot and some more information from random people who probably don’t even know about it or don’t care.. so I’m not sure about the stealing part.

Apparently there are some captcha’s when you start browsing searches beyond a few pages. I guess that is where the hacking part comes in, because getting around a captcha probably constitutes hacking—maybe?

Anyway I think part of the media coverage is a bit ridiculous because anyone can write a simple harvester in an hour or two. It took me one and half hours, so I think I’m on the safe side with this estimate and I didn’t really have a clue about this stuff before either.

Since I don’t want to “hack”, I’ve only written a very tame harvester. It connects to your personal StudiVZ account, and retrieves the name and profile ID (and thus profile URL) of all your friends in the “Meine Freunde” pages.

It could do a lot more with that like retrieving everybody’s birthday or random pictures, but I’m too lazy to code that because you use the same pattern for extracting data over and over again and it stops being interesting quite fast.

You can download the project here. It is a one file C# project. I’m releasing it under GPL (whatever).

It’s really easy to explain how it works:

• It uses System.Net’s HttpWebRequest and HttpWebResponse to get (and post) web pages.
• StudiVZ (like every other portal) uses cookies, so I create a CookieContainer and use it in every http request.
• There are a few hidden values that StudiVZ expects during login. I’m retrieving them from the main page using custom built regular expressions. I’ve found a handy AJAX tester for .NET regular expressions. It was really useful for building the expressions and debugging them. (BTW you can find all URLs I used in the comments.)
• After login I use the same pattern: get page & parse using regex for everything.
• Visual Studio has an awesome “HTML Visualizer” for strings. It displays the content of a string as HTML page, which is really nifty if you’re doing anything related to HTML processing.

The code is quite ugly. Well, it’s not production code and this is only meant as a proof of concept.

Also note that I have at most violated the AGB of StudiVZ and not committed any criminal acts and I’m not planning to sell my friend’s profile IDs or data either

Maybe someone can extend the code and make it more useful. I guess it would be fun to automatically download all your pictures (including tags) and feed them into flickr or picasa… but someone else can do that.

Cheers,
Andreas

Tags: Profiles, Regex, StudiVZ

23 Aug 09 Sploidz Revisited (Unofficially)

I’ve already written about the semi-conductor project and how I’ve written some Flash animations/applications for it. Of course, I’m more interested in making fun stuff´, so I decided to put my knowledge to good use and write a small game to see how difficult/awkward Flash actually is.

To sum it up, it is somewhat awkward, at least if you use the IDE itself. FlashDevelop still is as nice as ever, but you can quickly develop games nonetheless. I prefer Torque Game Builder though in retrospect.

Before I continue talking about the development itself, let’s take a look at the actual game. Sploidz was the first game I wrote using Torque Game Builder for Joshua Dallman, and since I still had all the assets in my subversion repository, it was an easy decision to try and port this game. If you want to play it, you can download it for free here.
I haven’t ported everything: I’ve just rewritten the main characteristic features that make up Sploidz’s code in ActionScript.

Without further ado´ here is the game:

Click to open Sploidz in its own window

Because the art is still copyrighted and I haven’t heard back from Joshua yet ´, I decided to create a free version that only uses “coder art” – in this hand-drawn coder art

Some´ have said that this version looks cuter, decide for yourself:

Click to open SploidzCC in its own window

Below you’ll find a description of the development and at least one helpful trick and most importantly a link to the source code of the “copyright-free” version.

Because the orginal version is way too difficult to be really fun, I actually sat down one more time and added code to make the platform slower if you’re in danger of losing (up to 3 times slower):

Click to open SploidzMoreFun in its own window

(more…)

Tags: ActionScript, Flash, FlashDevelop, game, Photoshop, Sploidz

05 Aug 09 PowerPointLaTeX Update

Because people complained to me about the formula feature in my PowerPointLaTeX add-in, which used a somewhat experimental approach to editing formula objects by adding an editing text shape that contained the formula code and that would be merged back into the formula as soon as you deselect it, I decided to rewrite it to use a standard modal dialog to edit formula objects:

The editor isn’t perfect (yet), but it certainly shouldn’t add any bugs to the add-in and solve some natural issues the old approach created.

Implementation Note

The idea was pretty straight-forward but the actual UI design was a PITA due me not knowning the panel/flow/table layout concepts very well and the code still has some annoying quirks with auto-scroll, so I need to fix that later.

I almost rewrote the whole cache system, because I’m using a background thread for updating the preview (if the text is changed, a 500 msec timer is started which triggers an update) and the update accesses the cache system, which in turn accesses PowerPoint to return some data, which in turn is busy because of the modal dialog -> dead-lock.

The solution to this is very simple but was not obvious to me at first (I actually began to rewrite the cache system with a feeling that there should be an easier solution):
The background thread needs an Invoke call to update the preview picture because the control has been created by a different thread (the main thread) and the code to get an updated picture can simply be moved into Invoke delegate function.

This solved all my problems and made 4 hours of previous work and thinking about a new cache system obsolete

Download the new build at: http://code.google.com/p/powerpointtools/downloads/list

Cheers,
Andreas

Tags: Formula Object, Invoke, LaTeX, PowerPoint

02 Aug 09 Semi-Conductor Optimization (Uni Project)

I’ve written my last exam yesterday (except for two oral exams in September), so now I have got some spare time before I start working on my Bachelor Thesis tomorrow and I want to use it to wrap up a few things.

During this term I took part in a course that was both a (research) project/presentation/lecture thing, which was fun but also a lot of work.
I’ve already written about one mathematical aspect of it in my post about Analysis, Cauchy-Schwarz and Reciprocal Sums.

The project was about optimizing semi-conductor wiring placement. We wrote a small paper about our findings and the work it was based one – you can look download it here.

We also created a self-running presentation that doesn’t contain any Maths at all but makes heavy use of Flash animations (which were exported to .gif manually, which was a huge pain in the ass, which I will never do again if possible) to visualize all the concepts and algorithms.

You can download a PowerPoint 2007 (.pptx) version here or one that works with PowerPoint 2003 here.

For the Student’s I sat down and wrote a small Flash application to show the algorithms at work. It’s not obvious how it works, so let me explain the major points:

• On the right you have a number of panels that you can enlarge by clicking on the small button in the upper right of each panel.
• The upper three panels show different views of the same dataset. They will all be updated as you run the algorithm step by step.
• The fourth panel lets you change the number of wires and/or their activity.
• The last panel shows the electrical field that is created by one active wire in the center. It was created using MatLAB. I’ve also uploaded the script here.

Abitag

Last but not least I’ve also uploaded the current version of all my .fla and .as files. You can download it here.

ActionScript is a nice language and you can quickly learn it using the available resources from Adobe.
While ActionScript 2.0 is arguably weird, ActionScript 3.0 is quite logical and it’s syntax is straight-forward and consistent, too. You can’t say that about the IDE (Flash CS4), which is braindead, but if you’re only interested in writing ActionScript code, FlashDevelop is an excellent and free alternative.

This is it for now, maybe I’ll play around with Flash some more another time.
Cheers,
Andreas

Tags: ActionScript, Flash, MatLAB, PowerPoint, Semi-Conductor Optimization

18 Jul 09 From Vertex Positions to Packed Element Arrays

At my new workplace at university I’m currently porting an advanced terrain rendering engine from DirectX to OpenGL.
One of the performance optimizations the engine uses is that it draws the terrain tiles right from the index buffer without using a vertex buffer at all – that is it packs the vertex position into the 32-bit index and unpacks it in the vertex shader.

Why is this faster than rendering using a vertex buffer and no index buffer?

When using an index buffer the graphics card can make use of a cache of already transformed (vertex-shaded) vertices and when an index is reused, it can use the cached result instead of running the vertex shader again. Of course, this only works if there exists a certain temporal locality, but that is given.
If no index buffer is used, the vertex cache won’t be used, because the implicit index is different for each vertex.

Since I need to port the engine from DirectX to OpenGL, I did some research to see if it is possible to do the same in OpenGL.
It’s not really possible but you can achieve something quite similar in OpenGL 3.0.

Using the Input-Assembler Stage without Buffers (OpenGL)

This is meant as OpenGL analogon for Using the Input-Assembler Stage without Buffers (Direct3D 10).

I think the title is self-explanatory but for greater clarity let me rephrase it:
The aim is to render something without using vertex or index buffers, that is (in OpenGL speak) using neither vertex data nor an elements array to render something.
Instead the automatically supplied gl_VertexID attribute  (vertexId in DirectX) is used to determine the vertex the shader is currently processing.

The example in MSDN simply draws a triangle using vertexId:

VSOut VSmain(VSIn input)
{
VSOut output;

if (input.vertexId == 0)
output.pos = float4(0.0, 0.5, 0.5, 1.0);
else if (input.vertexId == 2)
output.pos = float4(0.5, -0.5, 0.5, 1.0);
else if (input.vertexId == 1)
output.pos = float4(-0.5, -0.5, 0.5, 1.0);

output.color = clamp(output.pos, 0, 1);

return output;
}

If you want to do the same thing in OpenGL, you have at least two problems:

• OpenGL uses glVertex* as end marker for the specification of a single vertex and the array rendering commands (glDrawArrays, glDrawElements, etc.) all require an enabled vertex array.
• gl_VertexID is only supplied if:
  • the vertex comes from a vertex array command that specifies a complete primitive (e.g. DrawArrays, DrawElements)
  • all enabled vertex arrays have non-zero buffer object bindings, and
  • the vertex does not come from a display list, even if the display list was compiled using DrawArrays / DrawElements with data sourced from buffer objects.

  (from GL_EXT_gpu_shader_4)

There is no way around these requirements, but what you can do is to create dummy vertex buffer with one element, bind it as vertex array and simply draw as many vertices as you want. If you don’t access gl_Vertex there is no way that uninitialized data can affect the shader and although behavior is generally undefined in OpenGL, if you render beyond the vertex buffer size, it has worked so far that I’ve test this on.

You can download the source code here.

Packing Vertex Positions into the Elements Array

The next step is to start packing and unpacking data in the gl_VertexID. For this an integer type and bit operations (shifting and masking at least) are required in the vertex shader, so it requires GLSL 1.30 at least.

The code is quite short from my proof of concept project, so I’m pasting the shader here:

#version 130
#extension GL_EXT_gpu_shader4 : enable

out vec4 color;

vec3 unpackVertex(int index) {
	return vec3( index & 0xFF, (index >> 8 ) & 0xFF, (index >> 16) & 0xFF ) * (2 / 255.0) - vec3(1.0);
}

void main()
{
	vec3 unpackedData = unpackVertex( gl_VertexID );
	gl_Position = vec4( unpackedData, 1.0 );
	color = vec4( (unpackedData + 1.0) / 2.0, 1.0 );
}

In main.cpp the equivalent can be found for setting up the elements array:

#define packFloat(v)	(int(((v) + 1.0) / 2 * 255) & 255)
#define packVertex(x,y,z) (packFloat( x ) + (packFloat( y ) << 8 ) + (packFloat( z ) << 16))

void display(void) {
[...]

unsigned indices[] = {
packVertex( 0.0, 0.0, -1.0 ), packVertex( 1.0, 0.0, -1.0 ), packVertex( 1.0, 1.0, -1.0 ),
packVertex( 0.0, 0.0, -1.0 ), packVertex( -1.0, 0.0, -1.0 ), packVertex( -1.0, -1.0, -1.0 )
};
glDrawElements( GL_TRIANGLES, sizeof( indices ) / sizeof( *indices ), GL_UNSIGNED_INT, indices );

[...]
}

For the code to actually make sense I should initialize an index buffer/elements array buffer in OpenGL and upload the indices into it but this was just for testing.

You can download the source code here.

PS: Wordpress is incredibly annoying – or rather this stupid SyntaxHighlighter Evolved plugin >_<

Tags: DirectX, no vertex buffer, OpenGL

21 Jun 09 Extending Java and Javac

Today I want to write about something I’ve been working ages ago – specifically in March I wanted to see if I can extend a Java compiler to support LINQ´ expressions, too.

I probably spend more time on finding a good open-source compiler to experiment with than I later spent on trying things out, so let me share my preferred source with you: http://openjdk.java.net/ is a good address to start with.
More specifically http://openjdk.java.net/groups/compiler/ contains some valuable information about the way the compiler works.
A nice thing is that there is a branch that has added support for ANTLR which makes added language a tad bit easier since you get to change a grammar file instead of tweaking hand-written lexers and parsers. More info about it can be found at http://openjdk.java.net/projects/compiler-grammar/.
You can download the source code from http://hg.openjdk.java.net/ – don’t follow the link to http://hg.openjdk.java.net/compiler-grammar/compiler-grammar, that one will only allow you to download part of the branch´.

I didn’t come around to add support for LINQ in the end, but to get known to the compiler and the ANTLR grammer, I added support for the var keyword as known from C#, which allows for automatic type deduction and for anonymous objects (again using the C# syntax). Thus my changes allowed for the following to compile and execute correctly:

public class Test {
	public  static void main(String[] args) {
		// automatic type deduction
		var t = Math.atan(1);
		System.out.println( t );

		// anonymous type
		var i = new { Amount = 108, message = "hello" };
		System.out.println( i.Amount );
	}
}

(more…)

Tags: ANTLR, C#, Compiler, Java, JavaC, OpenJDK

03 Apr 09 PowerPoint Tools

I’ve finally managed to upload a version of my PowerPoint LaTeX add-in for PowerPoint 2007.

I’m just going to post a short Vimeo video here that shows how inline formulas work (the main feature):

http://www.vimeo.com/4442353

I’ve tried to work on the add-in during my spring vacation but somehow I have instead spent all my time watching four seasons of House M.D. (which was totally worth it though ).

There are still some features like support for MikeTeX, code clean-ups, small bugfixes and a preference window that I should work on, but I don’t plan on selling it, so I don’t really care if it’s still somewhat work in progress.
I’m going to continue working on it when I have to use PowerPoint again.

You can check it out (and another add-in dubbed Language Painter that I wrote to fix some annoyances when writing presentations in languages different from your keyboard layout) at http://code.google.com/p/powerpointtools/.

(more…)

Tags: AddIn, C#, LaTeX, MiKTeX, PowerPoint, TexPoint

12 Mar 09 What I (don’t) like about C#

I’ve been using C# personally quite a bit lately and want to share with you a list of the things I like or don’t about it.

First the things I like about it (compared to C++ and Java sometimes):

• The out and ref keywords are awesome
  You actually need to use them both when declaring out and reference parameters and in the actual function calls.
  This really avoids confusion and makes the code clearer compared to the way references are transparently handled in C++.
• a class’s definition and declaration is kept in just one file
  Avid C++ coders will probably disagree with me here, but I really like this way of coding things. It allows me to code faster and also reduces one source of compiler errors. To be honest I would probably go insane when coding in C++ if it wasn’t for Visual Assist’s nifty “Move Implementation to Source File” and “Create Declaration/Implementation” refactorings.
  Likewise one could say that C#’s (or .NET’s) notion of assemblies is better and less messy than C/C++’s compilation units.
• Properties are simply awesome
  Properties look like normal member variables but you can write your own getter and setter functions and also give both different access levels, so e.g. you can have a property that is public for reading but protected for writing.
  Moreover, you can have C# create the code automatically, so you just end up with a way to specify separate access levels.
• Read-only variables can be set anywhere in the constructor
  If you know C++’s initialization lists, you will thank god for this feature, because this feature gives you a lot more freedom and again results in more readable code (at least in my opinion).
• The override and new keywords are really handy
  Again this is a feature to avoid unnecessary bugs because you get warnings as soon as you “override” a method that doesn’t exist.

What I don’t like:

• No typedefs
  This really bugs me, because the concept of typedefs is a good way to add abstraction to your code and the using keyword in C# only marginally improves the situation, because it only affects the current module.
• No default/optional parameters
  You have to overload methods instead, which is kind of annoying. I hope this gets fixed sooner or later.
• No inner classes like in Java
  Passing on references to constructors is annoying and inner classes in Java really help it quite often.
• No synchronized keyword like in Java
  Again this is a nice-to-have feature because it would  clean up some code.

31 Jan 09 How to make WordPress display Full Text in RSS Feeds

… and why WordPress Moderators are obviously idiots

Today I got an email asking me to enable full text RSS feeds – so far so good, only when I activated the “Full Text” option nothing changed – neither in FireFox, nor in Google Reader, nor anywhere else. Only in Feed Proxy I saw a <content:encoded> tag appear that contained the full text (including html tags), but the description still only contained the excerpt (without html tags).

Before I link you to the thread that made me conclude the statement in the post title, let me quote from the RSS 2.0 specs:

A channel may contain any number of <item>s. An item may represent a “story” — much like a story in a newspaper or magazine; if so its description is a synopsis of the story, and the link points to the full story. An item may also be complete in itself, if so, the description contains the text (entity-encoded HTML is allowed; see examples), and the link and title may be omitted. All elements of an item are optional, however at least one of title or description must be present.

That said take a look at http://wordpress.org/support/topic/190901 – having the problem described above and reading the replies, it just left me with one question:

If Otto42 is a moderator, where does WordPress get its trolls from?

The RSS Specs are pretty unspecific and blurry when it comes to the issue, one can at most point to http://www.feedvalidator.org/docs/warning/DuplicateDescriptionSemantics.html, but the main issue I have is that the rants of this “moderator” don’t help me fix my problem, because Google Reader and FireFox still don’t display the full entries.

A fix for it

If you want to fix it, you can do the following (in WP 2.7):

Open up wp-includes/feed-rss2.php and change

<?php if (get_option('rss_use_excerpt')) : ?>
<description><![CDATA[<?php the_excerpt_rss() ?>]]></description>
<?php else : ?>
<description><![CDATA[<?php the_excerpt_rss() ?>]]></description>
<?php if ( strlen( $post->post_content ) > 0 ) : ?>
<content:encoded><![CDATA[<?php the_content() ?>]]></content:encoded>
<?php else : ?>
<content:encoded><![CDATA[<?php the_excerpt_rss() ?>]]></content:encoded>
<?php endif; ?>
<?php endif; ?>

to

<?php if (get_option('rss_use_excerpt')) : ?>
<description><![CDATA[<?php the_excerpt_rss() ?>]]></description>
<?php else : ?>
<description><![CDATA[<?php the_content() ?>]]></description>
<?php endif; ?>

Similarly, if you want to fix your comments, too, open wp-includes/feed-rss2-comments.php and change

<?php if ( post_password_required($comment_post) ) : ?>
<description><?php echo ent2ncr(__('Protected Comments: Please enter your password to view comments.')); ?></description>
<content:encoded><![CDATA[<?php echo get_the_password_form() ?>]]></content:encoded>
<?php else : // post pass ?>
<description><?php comment_text_rss() ?></description>
<content:encoded><![CDATA[<?php comment_text() ?>]]></content:encoded>
<?php endif; // post pass

to

<?php if ( post_password_required($comment_post) ) : ?>
<description><![CDATA[<?php echo get_the_password_form() ?>]]></description>
<?php else : // post pass ?>
<description><![CDATA[<?php comment_text() ?>]]></description>
<?php endif; // post pass

Cheers,
Andreas

Tags: Full Text, PHP, RSS 2.0, WordPress

31 Jan 09 Seminar about Motion Retargeting

Two weeks ago I had to give a presentation about Motion Retargeting, which I want to share with you now.
I created it due to me attending a seminar about the latest developments in Computer Graphics at university and my presentation was about the Siggraph ‘08 paper “Real-time Motion Retargeting to Highly Varied User-Created Morphologies” from Chris Hecker et al.
You can check it out on Chris Hecker’s homepage – his website also contains a bunch of other really cool articles and presentations from various conferences, so it certainly is worth taking a look at it.

I’ve also sifted through quite a lot of IK papers and lectures for my presentation to understand the later part about the IK solver in Spore and I’ve found a few links that are a nice read:

• Chris Hecker’s talk about IK is a nice introduction (you also get an audiostream which is nice, too).
• Chris Welman’s Master Thesis is nice for reading about some of the common methods to solve IK problems.
• Steve Rotenberg’s Computer Animation course is awesome – the slides about IK are nice, but the ones about locomotion are just awesome´!
• Particle IK and particle physics in games are related a lot (the constraint solver is pretty much the same) and for a nice read I can recommend Thomas Jakobsen’s Advanced Character Physics.
• Pixar’s Physically Based Modelling course from 2001 is also nice to get going with physics again – it also explains everything you need to understand the idea behind ODE solvers in rigid body simulations  and also has a very nice explanation of stiffness in ODEs.
• Principles of Animal Locomotion is a very neat book that is about, well, the principles of animal locomotion, which I have read a bit thanks to Google books and it certainly is quite interesting.
• Numerical Robot Kinematics Based on Stochastic and Molecular Simulation Methods is interesting, too, just to see what’s possible with particle IK

Ive created a huge PowerPoint presentation for my seminar  It includes a few videos (thanks again to Chris Hecker for uploading them and replying to my emails incredibly fast ´) and two awesome IK solvers that I’ve implemented with VBA macros´ to show how CCD and Particle IK solvers work.

You can find a zip with all the videos and high resolution images here (includes both a PPT 2003 file and a 2007 file). I’ve also uploaded a small version without videos, macros as PPT 2003 file here, if you don’t feel like downloading the 23 MB .zip file.

Here’s a YouTube video of the two IK solvers:

http://www.vimeo.com/3021779

I’ve exported the code into an additional IK Playground presentation which contains just one slide and the two IK solvers with the setup you see in the video above. You can find the PPT 2003 version of it here and the 2007 one here.

I’ve zipped up the macros here if you want to use them in your own slides. I’ve also written a handy VBA form that allows one to edit everything more easily (the macros are hardly documented though, but if anyone really wants to use them and runs into problems – you can always drop me a line or two in a comment )

BTW I’m not sure you know about it´, but Blender contains an awesome video editor – the UI needs some time to get used to, but the online documentation has improved a lot and with it, it works like a charm. Blender also supports some pretty professional filters, so it’s going to be my video editing tool of choice from now on.

Cheers,
Andreas

Tags: CCD, Inverse Kinematics, Macros, Motion Retargeting, Particle IK, PowerPoint, Spore, VBA