Storing tweets using nodejs

Since I have my own VPS I’m constantly trying out new stuff. After messing around with nodejs I have my own little script (just over 100 lines) that connects to Twitter using their streaming API, get’s all tweets that contain certain words, do a minimal amount of parsing and store it all in a SQL database.

Nodejs does not contain a lot of stuff out of the box which I need, the power lies in the huge amount of ‘modules’ (extenstions) for node. They even got their own package manager! In this example I use the modules node-mysql, ntwitter and forever.

Installing all those couldn’t be easier:

npm install mysql ntwitter forever

In my script I fetch all tweets by people who use nikeplus (app) to keep track of their running results. By storing all those tweets I can later create something cool with all the data. I use the streaming API to keep a connection open, to do this you need to register your own app at dev.twitter.com to get the required keys.

Setup:

var sys = require('util');

/* setup mysql connection */
var Client = require('mysql').Client;
var client = new Client();
client.user = 'sql username';
client.password = 'sql password';
console.log('Connected to MySQL');

Now we have a we have twit object for opening a Twitter stream and a client object for talking to our SQL server.

more

Getting started with Processing.js

Lately I’ve been reading a book about data visualization. The book goes into the thought process of visualizing data for humans and also talks a lot about visual coding. All code in the book is Processing code, an extremely easy and fast way to visualize.

Last night I started playing with the code and I’m amazed at how clean the code is. With Processing you only need to write stuff the gets on the screen in an elegant and clean way. I can imagine that if you want to take your animations to the next level (interaction wise) things would get a little difficult.

For example I haven’t crossed any type of event system yet. So you want to get the X position of the mouse? It’s stored in mouseX. This makes it extremely easy to use it but on the other hand it’s limited to the global vars Processing offers.

Here is the result of my playing last night (you should hover).

and here is the code.

// the arraylist that holds all the atoms
ArrayList atoms = new ArrayList();

// this function only runs on init
void setup() {
    size(584,500);
    frameRate(30);
    noStroke();

// create 250 atoms
for(int i = 0; i < 250; i++) {
    atoms.add(new Atom());  
}

}

// this function runs every frame
void draw() {
    background(100);

// loop through all atoms and update() and draw() them
for(int i = 0; i < atoms.size(); i++) {
    Atom a = (Atom)atoms.get(i);
    a.update();
    a.draw();
}

}

class Atom {
    // position
    int x, y, xV, yV;
    // filling
    int r, g, b, a;
    // size
    int size, xDiff, yDiff, diff, treshold;

// constructor of the class
Atom() {
    // create a random color
    r = random(50);
    g = random(255);
    b = random(255);

    x = random(width);
    y = random(height);

    // maximum distance from atom 
    // to mouse before bubbling
    treshold = width / 4;

    xV = random(-2, 2);
    yV = random(-2, 2);
}

// this function calcs
// and sets: x, y, a, size
void update() {

    x += xV;
    y += yV;

    // take care of borders right & bottom  
    x %= width;
    y %= height;

    // take care of borders left & top
    if(x < 0) {
        x += width;
    }
    if(y < 0) {
        y += height;
    }

    // determinse X and Y 
    // distance to mouse
    xDiff = mouseX - x;
    yDiff = mouseY - y;

    // if it's close enough
    if(
        xDiff < treshold && 
        xDiff > -treshold && 
        yDiff < treshold && 
        yDiff > -treshold
        ) {

        // determine distance to mouse
        // a2 + b2 = c2
        int diff = sqrt(sq(xDiff) + sq(yDiff));

        // make smaller diff always bigger
        if(diff < 0) {
            diff *= -1;
        }

        // make the size bigger
        size = treshold - diff + 20;
        if(size < 20) {
            size = 20;
        }

        // set alpha
        a = size * 1.5;

        if(a < 50) {
            a = 50;
        }

    } else {
        // it's not close enough
        size = 20;
        a = 50;
    }

}

void draw() {
    fill(r, g, b, a);
    ellipse(x, y, size, size);
}

You might notice some things being just simpler compared to languages like ActionScript (flash). For example:

  • Random is a much simpler

    random(40); VS Math.random() * 40;
    or random(-40, 40); VS Math.random() * 40 - 40;

  • drawing is much simpler

    If you want an ellipse you just call ellipse(), no need to add it to a display manager. You also just draw stuff in the order you want it, instead of needing to keep track of what items are drawn when.

  • Properties are just vars

if(y < 0) {
  y += height;
}

VS

if(this.y < 0) {
 // only accessible from certain places
 this.y += stage.stageHeight;
}

An easy way to manipulate data off dom using jQuery

When you’re working with javascript you need to take the execution speed into account since its all being processed on the client. There are lots of best practices when it comes to working with websites. You should for example always minify and concatenate your scripts, and make good use of closures and avoid polluting the global scope.

Another thing you should keep in mind is to keep as much big changes to the DOM offside the DOM. This is because every time you change something the browser needs to calculate a lot of things again (repaints and reflows) and more importantly, every query to the DOM is very slow.

If you need to change a part of the page you need to do this of course, and you can’t get around it when you want to animate stuff (when CSS3 is no possibility). But when you want to update a large amount of data on the page (at the same time) it’s best to temporary take that part off the dom, update everything you want and put it back on the dom. This way you don’t have a reflow / repaint on every manipulation.

This can easily be achieved by using jQuery.

//instead of doing:
$('#container').find('p').each(function(i) {
   $(this).append('this is p ' + i);
});

Keep in mind that you need to know the parent element to which you want to append (or prepend) it later on. Notice how I chain it all together, by using .appendTo instead of .append we can use the current working set we have already instead of saving it into a variable.

more

Posted at March 05, 2012, under jQuery.

How to ajaxify your WordPress theme

When building a custom WordPress based website for a client I needed to include AJAX based page transitions into the website. In this video I’ll explain a little about how I did it.

In the video I’ll explain some simple principles I followed like graceful degration and don’t break the back button.

Here is the code that makes it happen.

These are the ajax checks inside page.php to serve only the content to an ajax call:

if(!$GET['ajax']) 
   getheader();

// code to get the content

And here is the javascript:

Edit: This is a small revision of the code in the video, it only contains code to make the transitions happen. You can find the original code in the video here.

/* Author: Mike van Rossum */
$(function(){
  var $footer = $('#footer'),
    url = $footer.data('url'),
    $book = $('#innerBook'),
    speed = 200,
    $menu = $('.menu');
  if(Modernizr.hashchange) {
    var baseLength = url.split('/').length,
      ajaxPage = function() {
        $book.children().fadeOut(speed);
        $.get(location.href.replace(/#/,") + '?ajax=true', function(data) {
          //insert the data into the page and fade it in
          $book.html(data).children().hide().fadeIn(speed);
          if($('#pageData').data('title') === 'Portfolio') {
            $book.addClass('portfolio');
            log('a');
          } else {
            $book.removeClass('portfolio');
            log('b');
          }
        });
      }
    if(location.hash)
      ajaxPage();
    // if a direct page get's loaded we want to reload the homedir and ajax it:
    //    http://website.com/page
    // to
    //    http://website.com/#page
    var loc = location.href;
    if(loc != url && !/#/.test(loc)) {
      var redirect = location.href.split('/');
      redirect[baseLength -- 1] = '#' + redirect[baseLength -- 1];
      redirect = redirect.join('/');
      location = redirect;
    }
    // convert all links to the website from
    //    http://website.com/page
    // to
    //    http://website.com/#page
    $menu.find('a').add('.ajax').each(function() {
      $(this).attr('href', function() {
        var url = $(this).attr('href').split('/');
        url[baseLength -- 1] = '#' + url[baseLength -- 1];

    return url.join('/');
  });
});
// bind ajax to hashchange
$(window).on('hashchange', function() {
  ajaxPage(location.href.replace(/#/,'') + '?ajax=true');
});