I've been a vivid fan of thesixtyone.com - until they changed the design. The old site can still be found here: http://old.thesixtyone.com/.

Many of my most favorite songs are from this site and I have only been able to listen to them through the site. However, since the aforementioned design change, the site has really died down in my opinion and I was fearful ever since that it would simply go down some day and vanish - taking my songs and playlists with it.

As counter-measure I've written a java console application to extract playlists and songs from thesixtyone. It uses Selenium to remote-control a FireFox instance that uses the normal user-interface to play songs and read in playlists.

I've uploaded the code to launchpad: https://launchpad.net/t61extractor/trunk
I've tested and used it myself and it runs alright.

This was a small weekend project (or rather two weekend project) that I did a few months ago but I only found time now to write about it.

The code should be mostly self-explanatory and it's not a lot of code either.

Cheers,
Andreas

Some time ago somebody stole 1 million data records from StudiVZ, the German Facebook clone. I'm not exactly sure why people call the person a hacker who stole data, because it appears he simply wrote a tool that harvested the publicly available data from StudiVZ (which everyone with an account can view).

People on StudiVZ share all their data by default---contrary to Facebook which values a person's private data a lot more. Thus by simply opening each profile from a dummy user and processing the HTML data from StudiVZ one can extract a lot and some more information from random people who probably don't even know about it or don't care.. so I'm not sure about the stealing part.

Apparently there are some captcha's when you start browsing searches beyond a few pages. I guess that is where the hacking part comes in, because getting around a captcha probably constitutes hacking---maybe?

Anyway I think part of the media coverage is a bit ridiculous because anyone can write a simple harvester in an hour or two. It took me one and half hours, so I think I'm on the safe side with this estimate and I didn't really have a clue about this stuff before either.

Since I don't want to "hack", I've only written a very tame harvester. It connects to your personal StudiVZ account, and retrieves the name and profile ID (and thus profile URL) of all your friends in the "Meine Freunde" pages.

It could do a lot more with that like retrieving everybody's birthday or random pictures, but I'm too lazy to code that because you use the same pattern for extracting data over and over again and it stops being interesting quite fast.

You can download the project here. It is a one file C# project. I'm releasing it under GPL (whatever).

It's really easy to explain how it works:

• It uses System.Net's HttpWebRequest and HttpWebResponse to get (and post) web pages.
• StudiVZ (like every other portal) uses cookies, so I create a CookieContainer and use it in every http request.
• There are a few hidden values that StudiVZ expects during login. I'm retrieving them from the main page using custom built regular expressions. I've found a handy AJAX tester for .NET regular expressions. It was really useful for building the expressions and debugging them. (BTW you can find all URLs I used in the comments.)
• After login I use the same pattern: get page & parse using regex for everything.
• Visual Studio has an awesome "HTML Visualizer" for strings. It displays the content of a string as HTML page, which is really nifty if you're doing anything related to HTML processing.

The code is quite ugly. Well, it's not production code and this is only meant as a proof of concept.

Also note that I have at most violated the AGB of StudiVZ and not committed any criminal acts and I'm not planning to sell my friend's profile IDs or data either

Maybe someone can extend the code and make it more useful. I guess it would be fun to automatically download all your pictures (including tags) and feed them into flickr or picasa... but someone else can do that.

Cheers,
Andreas

... and why WordPress Moderators are obviously idiots

Today I got an email asking me to enable full text RSS feeds - so far so good, only when I activated the "Full Text" option nothing changed - neither in FireFox, nor in Google Reader, nor anywhere else. Only in Feed Proxy I saw a <content:encoded> tag appear that contained the full text (including html tags), but the description still only contained the excerpt (without html tags).

Before I link you to the thread that made me conclude the statement in the post title, let me quote from the RSS 2.0 specs:

A channel may contain any number of <item>s. An item may represent a "story" -- much like a story in a newspaper or magazine; if so its description is a synopsis of the story, and the link points to the full story. An item may also be complete in itself, if so, the description contains the text (entity-encoded HTML is allowed; see examples), and the link and title may be omitted. All elements of an item are optional, however at least one of title or description must be present.

That said take a look at http://wordpress.org/support/topic/190901 - having the problem described above and reading the replies, it just left me with one question:

If Otto42 is a moderator, where does WordPress get its trolls from?

The RSS Specs are pretty unspecific and blurry when it comes to the issue, one can at most point to http://www.feedvalidator.org/docs/warning/DuplicateDescriptionSemantics.html, but the main issue I have is that the rants of this "moderator" don't help me fix my problem, because Google Reader and FireFox still don't display the full entries.

A fix for it

If you want to fix it, you can do the following (in WP 2.7):

Open up wp-includes/feed-rss2.php and change

<?php if (get_option('rss_use_excerpt')) : ?>
<description><![CDATA[<?php the_excerpt_rss() ?>]]></description>
<?php else : ?>
<description><![CDATA[<?php the_excerpt_rss() ?>]]></description>
<?php if ( strlen( $post->post_content ) > 0 ) : ?>
<content:encoded><![CDATA[<?php the_content() ?>]]></content:encoded>
<?php else : ?>
<content:encoded><![CDATA[<?php the_excerpt_rss() ?>]]></content:encoded>
<?php endif; ?>
<?php endif; ?>

to

<?php if (get_option('rss_use_excerpt')) : ?>
<description><![CDATA[<?php the_excerpt_rss() ?>]]></description>
<?php else : ?>
<description><![CDATA[<?php the_content() ?>]]></description>
<?php endif; ?>

Similarly, if you want to fix your comments, too, open wp-includes/feed-rss2-comments.php and change

<?php if ( post_password_required($comment_post) ) : ?>
<description><?php echo ent2ncr(__('Protected Comments: Please enter your password to view comments.')); ?></description>
<content:encoded><![CDATA[<?php echo get_the_password_form() ?>]]></content:encoded>
<?php else : // post pass ?>
<description><?php comment_text_rss() ?></description>
<content:encoded><![CDATA[<?php comment_text() ?>]]></content:encoded>
<?php endif; // post pass

to

<?php if ( post_password_required($comment_post) ) : ?>
<description><![CDATA[<?php echo get_the_password_form() ?>]]></description>
<?php else : // post pass ?>
<description><![CDATA[<?php comment_text() ?>]]></description>
<?php endif; // post pass

Cheers,
Andreas

I have a few private pages that I use to store random stuff and ideas and private pages (for a reason I don't understand) don't show up in the pages widget.

It turns out that WordPress's get_pages function always filters them out, while get_posts doesn't (it actually has some logic to figure out whether to show private pages or not).

I've decided to fix this. A real fix would probably be merging get_pages and get_posts because both seem to do pretty much the same except that get_posts is a tad bit more advanced, but I'm all for quick fixes at the moment, because I don't have much time and I don't want to end up considering myself a PHP developer

To make get_pages return private pages, too, you have to open the file wp-includes/post.php and change the following lines near the bottom of the get_pages function:

	$query = "SELECT * FROM $wpdb->posts $join WHERE (post_type = 'page' AND post_status = 'publish') $where ";
	$query .= $author_query;
	$query .= " ORDER BY " . $sort_column . " " . $sort_order ;

to:

	$private_pages_inclusion_where = "";
	if ( is_user_logged_in() ) {
		$private_pages_inclusion_where  = current_user_can( "read_private_pages" ) ? " OR post_status = 'private'" : " OR post_author = $user_ID AND post_status = 'private'";
	}

	$query = "SELECT * FROM $wpdb->posts $join WHERE (post_type = 'page' AND (post_status = 'publish' $private_pages_inclusion_where)) $where ";
	$query .= $author_query;
	$query .= " ORDER BY " . $sort_column . " " . $sort_order ;

If you also want to make it easier to distinguish private pages from normal ones, you can also change the following bits in wp-includes/classes.php in the method Walker_Page::start_el:

$output .= $indent . '<li class="' . $css_class . '"><a href="' . get_page_link($page->ID) .
	'" title="' . attribute_escape(apply_filters('the_title', $page->post_title)) . '">' .
	$link_before . apply_filters('the_title', $page->post_title) . $link_after . '</a>';

to:

$output .= $indent . '<li class="' . $css_class . '"><a href="' . get_page_link($page->ID) .
	'" title="' . attribute_escape(apply_filters('the_title', $page->post_title)) . '">' .
	$link_before . apply_filters('the_title', $page->post_title) . ($page->post_status == 'private' ? " (Private)" : "") . $link_after . '</a>';

Cheers,
Andreas

Yahoo Pipes are an interesting concept, as are the other existing mashup tools (like Microsoft's Popfly or Ubiquity), and it is amazing what can be done with a few clicks with them.

Because I've wanted to learn how to work with Yahoo Pipes (before the Google Mashup Editor is released to the public), I've decided to take my best-of-explosm viewer to a different level and prototype it with Yahoo Pipes.

This just shows the first page (it's easier to view - and create - the feed in groups of 20 pictures than all at once). You can follow the link if you want to view more pages.

Stay tuned for more.
Cheers,
Andreas

PHP and CSS and all that funky web design stuff are funky, because they are so easy to use and you quickly get results. I like it.

I've decided to release my first plugin into the world (it's big hack, but it has turned out to be very useful on my other blog http://info1.blackhc.net).

It supports 3 text replacements:

Xkcd-style Tooltips as Replacement for Footnotes

Example:

[[:This is a default footnote. Hover over it to display it.]]

Result:

´

You can use a different marker, too:
Example:

[[:My Footnote#This is a default footnote. Hover over it to display it.]]

Result:
^{My Footnote}

Linking to Wikipedia

Example:

[[wikipedia:Pie]]

Result:

Pie

The link text can be changed, too, of course.
Example (you can use : as seperator, too):

[[wikipedia:pi#Fake Pie]]

Result:

Fake Pie

Expandable Text

Example:

[[expandable#Expandable Text...]]

Result:

The link text can be changed, too:
Example:

[[expandable#Click me! Click me! Pleeease!#Yay!]]

Result:

And we can nest it and it works with other tags.
Example:

[[expandable:#Expandable Text...

[[expandable:Click me! Click me! Pleeease!#Yay!]]
// some Java code

]]

Result:

// some Java code

We can also use a Heredoc-like mode:
Example:

[[expandable_ex:EOD#Heredoc-like Expandable#[[[[[[[]][[EOD]]

Result:

Escaping [[ and ]]

Example:

Escaping [[escape_left]] and [[escape_right]]

Source Code

<?php
/*
Plugin Name: Useful text replacements
Plugin URI: http://blog.blackhc.net
Description: Some useful text replacement for annotations and other nifty things.
Version: 1.01
Author: Andreas 'BlackHC' Kirsch
Author URI: http://blog.blackhc.net

Plugin Name: Latex for WordPress
Plugin URI: http://zhiqiang.org/blog/plugin/mimetex
Description: using WordPress.com or public MimeTex service to add latex formula in post and comment. You don't need to install your own latex service.
Version: 2.01
Author: zhiqiang
Author URI: http://zhiqiang.org
*/

/*
  1.3 and higher are maintained by Zhiqiang.

   Copyright 2006  Anders Dahnielson (email : anders@dahnielson.com)

  This program is free software; you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by
  the Free Software Foundation; either version 2 of the License, or
  (at your option) any later version.

  This program is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.

  You should have received a copy of the GNU General Public License
  along with this program; if not, write to the Free Software
  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
*/

/*
  Version-History:
  1.1.2:
  Tweaker: Martin Becker
  Email:   mbecker@physik.uni-wuerzburg.de
  Homepage: http://fstyle.de
  -   Edited discription of plugin, so correct tags will be shown
  -   Put inclusion of Snoopy-Class down into the class (mimetex),
    for in some cases WP will intialize the Snoopy-Class of its own
    after we do here, causing it to not to be able to reinitialize.
    For some reason I don't understand.
    Note: I think all this workaround using our own extra Snoopy-Class can be
    fixed by the wordpress-team by removing the if statement in their Snoopy-Class.
    That would be the case for ./wp-includes/rss-functions.php in ./wp-admin/index.php
  -   Cleaner code

  1.1.1:
  Tweaker: Baris Evrim Demiroz
  Email:   b.evrim /AT/ anlak /DOT/ com
  Homepage: http://www.anlak.com
  -   Switched to Snoopy
  -   Working in comments as well: Thanks to Robert Jones, he gave the code snippet for plugin to work with comments. You may wish to visit him: http://www.jonesieboy.co.uk/blog

  1.1.0:
  Anders Dahnielson, URI: http://dahnielson.com
  -   Anders Dahnielson's original version.
*/

class blackhc_replacements{
  var $uniqueID = 0;

  var $server = "http://l.wordpress.com/latex.php?bg=ffffff&amp;amp;amp;amp;fg=000000&amp;amp;amp;amp;latex=";
  var $img_format = "png";
  // $img_format is 'gif' when using mimetex service.
  // more server:
  // "http://l.wordpress.com/latex.php?latex=";
  // "http://www.bytea.net/cgi-bin/mimetex.cgi?formdata="; 

  function getUniqueID() {
    return $this->uniqueID++;
  }   

  // parsing the text to display tex by putting tex-images-tags into the code created by createTex
  function parse($toParse) {
    // tag specification (which tags are to be replaced)
    // change it to
    // $regex = '#\1 stype in your post
        //$regex = '#\$\$(.*?)\$\$#si';

      $text = $toParse;

      $oldText = "";
      while( $oldText != $text ) {
        $oldText = $text;
        $text = preg_replace_callback('#\(\)!expandable\:(?:(?P<link>[^ $]*?)\#)?(?P<content>(?:.|(?P<brackets>$!(?:(?>[^
]+)|(?P>brackets))* $))*?)$ $#si', array(&amp;amp;amp;amp;$this, 'createExpandableTextFix'), $text);

        $text = preg_replace_callback('#$!
 $$expandable\_ex\:(?P<eod>[^$$ 
]*?)\#(?P<link>[^ $]*?)\#(?P<content>.*?)(?P=eod)$ $#si', array(&amp;amp;amp;amp;$this, 'createExpandableTextExFix'), $text);
      }
    $text = preg_replace_callback('#$!
 $$\:(?:([^$$ 
]*?)\#)?(.*?) #si', array(&amp;amp;amp;amp;$this, 'createAnnotation'), $text);

    $text = preg_replace_callback('#\(\)!wikipedia\:(.*?)(?:(?:\#|\:)(.*?))? #si', array(&amp;amp;amp;amp;$this, 'createWikiLink'), $text);

    $text = preg_replace( array('#\(\)!escape_left #si', '#\(\)!escape_right #si'), array( '[' . '[', ']' . ']'), $text );

    return $text;
  }

  function createWikiLink($match)
  {
    $text = $match[2];
    $page = $match[1];
    if( !$text )
      $text = $page;
    return "<a href=\"http://en.wikipedia.org/wiki/".($page)."\" target=\"_blank\" >".($text)."</a>";
  }

  function createAnnotation($match)
  {
    $superscript = $match[1];
    $text = $match[2];
    if( $superscript == ""  )
      $marker = "<strong>&amp;amp;amp;amp;#180;</strong>";
    else
      $marker = "<sup><strong>" . ($superscript) . "</strong></sup>";

  return "<a class=\"annotation\" title=\"".($text)."\" href=\"javascript:;\">" . ($marker) . "</a>";
  }

  // Big Thanks to Salteh!
  function createExpandableText($match)
  {
    $linkText = $match["link"];
    $text = $match["content"];

  if ($linkText == "") {
    $linkText = "Click to expand";
  }

    $id = "blackhc_expandable_" . ($this->getUniqueID());
  return '<div class="expandable">' .
         '<a class="expandable_link" href="javascript:blackhc_replacements_togglediv(\''. $id . '\');">' . $linkText . ' &amp;amp;amp;amp;#187;</a>' .
         '<div class="expandable_text" name="' . $id . '" id="' . $id . '" style="display:none">' . $text . '</div><br>' .
         '</div>';
  }

  // fix for a php bug
  // the two functions can be removed as soon as preg_replace_callback passes named subpatterns correctly: http://bugs.php.net/bug.php?id=45748
  function createExpandableTextFix( $match ) {
    return $this->createExpandableText( array( "link" => $match[1], "content" => $match[2] ) );
  }

  function createExpandableTextExFix( $match ) {
    return $this->createExpandableText( array( "link" => $match[2], "content" => $match[3] ) );
  }

  // reading the tex-expression and create an image and a image-tag representing that expression
  function createTex($toTex) {
    $formula_text = $toTex[1];
    $imgtext=false;
    if (substr($formula_text, -1, 1) == "!")  return "$".substr($formula_text, 0, -1)."$";
    if (substr($formula_text, 0, 1) == "!") {  $imgtext=true;$formula_text=substr($formula_text, 1);}
    $formula_hash = md5($formula_text);
    $formula_filename = 'tex_'.$formula_hash.'.'.$this->img_format;

    $cache_path = ABSPATH . '/wp-content/cache/';
    $cache_formula_path = $cache_path . $formula_filename;
    $cache_url = get_bloginfo('wpurl') . '/wp-content/cache/';
    $cache_formula_url = $cache_url . $formula_filename;

    if ( !is_file($cache_formula_path)) {
       if (!class_exists('Snoopy')) require_once (ABSPATH.'wp-includes/class-snoopy.php');

      $snoopy = new Snoopy;

      $snoopy->fetch( $this->server.rawurlencode(html_entity_decode($formula_text)));
      // this will copy the created tex-image to your cache-folder
      if(strlen($snoopy->results)){
        $cache_file = fopen($cache_formula_path, 'w');
        fputs($cache_file, $snoopy->results);
        fclose($cache_file);
      }
    }

    // returning the image-tag, referring to the image in your cache folder
    if($imgtext) return "<center><img src=\"$cache_formula_url\" align=\"absmiddle\" class=\"tex\" alt=\"".($formula_text)."\" /></center>";
    return "<img src=\"$cache_formula_url\" align=\"absmiddle\" class=\"tex\" alt=\"".($formula_text)."\" />";
  }  

  function headerAdditions() {
    // additions for expandable text
    echo '
  <script language="JavaScript">
  function blackhc_replacements_togglediv(divname) {
    var div = document.getElementById(divname);
    div.style.display = (div.style.display=="none") ? "" : "none";
  }
  </script>
  ';
   }
}

$mimetex_object = new blackhc_replacements;
// this specifies where parsing should be done. one can look up further information on wordpress.org
add_action('wp_head', array($mimetex_object, 'headerAdditions') );

add_filter('the_title', array($mimetex_object, 'parse'), 1);
add_filter('the_content', array($mimetex_object, 'parse'), 1);
add_filter('the_excerpt', array($mimetex_object, 'parse'), 1);
// dont allow embedding stuff into comments for security reasons
//add_filter('comment_text', array($mimetex_object, 'parseTex'), 1);

?>

Hopefully this can be useful for somebody else, too.
Cheers,
Andreas

Since I've been working on another blog (http://info1.blackhc.net) for my tutorial at university, I've played around with using Microsoft Word for publishing and editing content on the blog. I gotta say that SmartArts are really nice and good way to lighten up blog posts easily.

However Word's support for the current version of WordPress is sub-optimal - that is, it totally messes up formatting when retrieving existing blog posts (that is, Word doesn't recognize any paragraphs or line breaks). I've spent some time digging around in the blog's .php code to find a way to fix it and after an hour or so, I've been able to fix it.
It's one-liner I want to share with you:

//'description' => $post['main'], // line 1939 in xmlrpc.php (function mw_getPost)
'description' => wpautop( $post['main'] ),

The function wpautop adds <p> tags to your text (and by default <br> tags, too).

This will fix the formatting bug in Word. This is a hack, of course, but it works for Word.

WordPress has a bug in its Atom Publishing Protocol code, too:
It messes up the status code header when asking for user credentials, which prevents your browser from displaying the login form.
Again only one line needs to be changed:

// header('Status: ' . $msg); // line 1014 in wp-app.php (function: auth_required)
header('Status: 401 ' . $msg);

Stay tuned for more updates in the next months..
Cheers,
Andreas

PS: University is really keeping me busy..