Overfloater

Update 10/2009: The project is currently hosted at GitHub.
Update 01/2009: Created a new hoc3.zip file and some documentation.

A while ago Radiohead published their House of Cards video data in form of CSV files. Each CSV file contains information about the 3D position of the points for each frame.

I wrote a couple of previous posts that covered how to render and save that data with OpenGL and OCaml and also how to customize camera movement in OpenGL.

This post shows how to customize particle animations for Radiohead’s House of Cards video.
A proof of concept for camera movement + particle animation is shown in this youtube video:

If you want to generate the video, you have to download Radiohead’s HoC music video data here. Also, you can download the source code for this project here.

This small project is organized in a way that is easy to add new features, camera animations and particle transformations in order to easily code new videos with different effects using the HoC data.

Radiohead’s HoC data is a set of CSV files. Those files are rendered in OpenGL with OCaml and then saved in bmp or jpeg files to be merged into a video using ffmpeg. If you want to know more about this you should probably read part 1 of this “trilogy”.

The Camera Model class allows you to make custom camera movements that can be handled and defined in a Timeline object in the main.ml file. If you like to know more about this, you can read part 2 of this “trilogy”.

This last post shows how to customize particle interpolation and movement by using the Particle Model class, the ParticleTrans module and the Timeline object.

Particle Model

The particle_model class handles particle animations.
Somewhat like the camera class, particle_model stores the initial frame and the last frame along with some extra information about the timing of the animation.
The particle_model then performs an interpolation from the initial_frame to the last_frame, rendering the state of the transformation in the draw function.

A possible interface for the particle model could be something like this:

class particle_model :
  object
    (* starting frame *)
    val mutable start_frame : VertexType.depth_vertex list
    (* ending frame *)
    val mutable last_frame : VertexType.depth_vertex list
    (* currently loaded frame *)
    val mutable loaded_frame : VertexType.depth_vertex list
    (* if setted to true, it will load a new frame for each
    step of the animation *)
    val mutable refresh_frames : bool
    (* same as the camera_model -check that post *)
    val mutable time : float
    val mutable total_frames : float
    val mutable transition : Transition.trans * Transition.ease
    (* extend start_frame or last_frame in order to
    have same number of points *)
    method balance : unit
    (* equivalent to the camera methods *)
    method step : unit
    method draw : float -> unit
    (* set the type of the animation you want
    to perform *)
    method set_animation :
      float ->
      bool * bool *
      (ParticleTrans.transformation * float *
       (Transition.trans * Transition.ease)) ->
      unit
  end

Particle animations have a special type, that ressembles the camera model transition type.
This type is defined as follows:

type animation_op =
    ParticleTrans.transformation * float *
    (Transition.trans * Transition.ease)

Just to make a comparison, the camera model transition type is:

type camera_op_list = (camera_op list) * float * (trans * ease)

This type can be divided into three main parts:

• The float value is the total number of frames the animation will use.
• The (trans * ease) value allows you to customize different type of transitions, from Linear, None to Quad, EaseInOut. More information about this is in the camera_model post.
• ParticleTrans.transformation is a function that applies a transformation to a frame. You can define custom functions in that module and then apply them to the visualization.
  I only defined a couple of functions, but you can define any other animation you like. You just have to define a function that receives a frame as input and returns a frame as output.
  The interface for ParticleTrans is:

  type transformation =
  	| Idle
  	| Project of float * float * float
  	| Random
  
  val idle : 'a -> 'a
  
  val project :
    transformation ->
    VertexType.depth_vertex list ->
  	VertexType.depth_vertex list
  
  val random : VertexType.depth_vertex list ->
  	VertexType.depth_vertex list
  
  val get_trans :
    transformation ->
    VertexType.depth_vertex list ->
  	VertexType.depth_vertex list

Putting it all together

The timeline object (described in the previous post) holds information about the camera and particle transformations beeing applied at each stage of the animation.
This class-less object is defined in the main.ml file and looks like this:

let timeline =
  object (self)
    val mutable frame = 0.
    val camera_timeline = [
    (* operations defined in the
    camera model post *)
    ]

    val particle_timeline = [
      (* frame number,  (invert, refresh frames, instruction) *)
      (1., (true, true, (Random, 120., (Elastic, EaseOut))));
      (420., (false, false, (Random, 50., (Quad, EaseOut))));
      (471., (false, true, (Idle, 80., (Quad, EaseIn))))
      ]

    method get_frame = frame

    method tick =
      frame <- frame +. 1.;
      self#update_camera;
      self#update_animation

    method update_camera =
      try
        let camera_anim = List.assoc frame camera_timeline in
          cam#set_animations camera_anim;
        with
          | Not_found -> ()

    method update_animation =
      try
        let anim = List.assoc frame particle_timeline in
          part#set_animation frame anim;
      with
          | Not_found -> ()
end

The particle_timeline and camera_timeline variables hold the transformations to be performed at different stages of the animation.

Download and Use

You can download the project here.
You can compile the project by typing:

ocamlc -g str.cma -I +camlimages ci_core.cma ci_jpeg.cma ci_bmp.cma
-I +lablGL lablglut.cma lablgl.cma interpolate.ml transition.ml camera.ml
 loader.ml particleTrans.ml particle.ml main.ml -o main

Any comment about the Video or the OpenGL/OCaml implementation is very welcome!

This post is about performing advanced camera movement in OpenGL.
We’ll use the same Radiohead’s HoC dataset we used in the previous post.

Once again, the quality of the youtube video is pretty lame. You can right click here and save link as… to download a high quality version of the video (~100MB).
I strongly recommend you to see the high quality video

Camera Instructions

Camera movement is made of Translations and/or Rotations.
We want to provide our camera model with instructions of the type:

• [ Translate from to ]
• [ Rotate from to rotation_axis ]
• [ Translate ...; Rotate ... ]

As the last example shows, multiple transformations can be done at the same time (translations and rotations).

The definition for a transformation type is:

type camera_op =
  | Translate of Gl.point3 * Gl.point3
  | Rotate of float * float * Gl.vect3

A camera instruction is a list of these operations (camera_op) and a number specifying the number of frames this transformation should take (i.e the duration of the transformation).

So, for example, this instruction:
( [ Translate( (100., 100., 100.), (0., 0., 0.) ) ], 300. )
translates the camera from (100, 100, 100) to (0, 0, 0) in 300 frames, that is in 10 seconds (at 30 frames per second).

Translation is done by simple interpolation. The interpolation formula for translating from A to B is something like this:
A + (B – A) * delta with delta in (0, 1).

Transitions

It would be nice if camera movement, besides being linear, could also perform other advanced transitions, like the ones used in Fx.Transitions by Mootools.
Some of these transitions are: Quadratic, EaseIn, EaseOut, EaseInOut, Back, Sine, etc…

These effects are achieved by applying functions to the delta value, changing the way it increases or descreases its value.
A possible interface for a Transition module is:

type trans = Linear | Quart
type ease = None | EaseOut | EaseIn | EaseInOut

val linear : 'a -> 'a
val quart : float -> float

val ease_in : ('a -> 'b) -> 'a -> 'b
val ease_out : (float -> float) -> float -> float
val ease_in_out : (float -> float) -> float -> float

val get_transition : trans -> float -> float
val get_ease : ease -> (float -> float) -> float -> float

val get_animation : trans -> ease -> float -> float

By using Transition.get_animation Quad EaseInOut delta we can change the timing of our animation from this:

into this:

Our camera instructions are then defined as:

type camera_op_list = (camera_op list) * float * (trans * ease)

For example:
( [ Translate( (100., 100., 100.), (0., 0., 0.) ) ], 300. (Quad, EaseOut))

The Camera Model

A possible interface for the camera model is:

class camera_model :
  object
    val mutable animations : camera_op list
    val mutable time : float
    val mutable total_frames : float
    val mutable transition : Transition.trans * Transition.ease

    method get_time : float
    method step : unit
    method draw : unit
    method translate : Gl.point3 -> Gl.point3 -> float -> unit
    method rotate : float -> float -> Gl.vect3 -> float -> unit
    method set_animations :
      camera_op list * float * (trans * ease) -> unit
  end

The camera_model instance variables contain the destructured camera_op_list type elements: animations, total_frames and transition.
We also provide individual methods for handling translations and rotations. These methods simply compute a delta value, apply the interpolation and then call GlMat.translate3 or GlMat.rotate3.

The 40 line implementation looks like this:

class camera_model =
 object (self)
  val mutable total_frames = 0.
  val mutable time = 0.
  val mutable transition = (Linear, None)
  val mutable animations = []

  method get_time = time

  method set_animations ans =
    let (x, y, z) = ans in
      animations <- x;
      total_frames <- y;
      transition <- z;
      time <- 0.

  method step =
    if time < total_frames then
      time <- time +. 1.

  method translate start last delta =
    let (trans, ease) = transition in
    let delta_val = Transition.get_animation trans ease delta in
    let (x, y, z) = start in
    let (x', y', z') = last in
    let DVertex(a, b, c, d) = Interpolate.cartesian
                                (DVertex(x, y, z, 0.))
                                (DVertex(x', y', z', 0.))
                                delta_val
    in
      GlMat.translate3 (a, b, c)

  method rotate start last vec delta =
    let (trans, ease) = transition in
    let delta_val = Transition.get_animation trans ease delta in
    let ang = Interpolate.cartesian_float start last delta_val in
    GlMat.rotate3 ang vec 

    method draw =
      let delta = time /. total_frames in
        List.iter (fun anim ->
          match anim with
            | Translate(start, last) ->
               self#translate start last delta
            | Rotate(start, last, vec) ->
               self#rotate start last vec delta ) animations
end

Timeline

Now that we have our camera model, we need a “timeline” object that can pass intructions to the camera at different stages of the animation.
We define a class-less object timeline that holds a list of camera transformations to be executed at a specific frame of the animation:

let timeline =
  object (self)
    val mutable frame = 0.
    (* Starting frame number, camera_instructions *)
    val camera_timeline = [
      (1.,   (* camera_instructions *));
      (310., (* camera_instructions *));
      (631., (* camera_instructions *) ]

    method get_frame = frame

    method tick =
      frame <- frame +. 1.;
      self#update_camera;

    method update_camera =
      try
        let camera_anim = List.assoc frame camera_timeline in
          cam#set_animations camera_anim;
        with
          | Not_found -> ()
end

Download and Use

This is all I’ve done to handle camera movement.
I’m not an advanced OpenGL/OCaml developer, so any comment/suggestion about my understanding of OCaml/OpenGL is very welcome.
You can download the source here.
You can compile the source with:

ocamlc -g str.cma -I +camlimages ci_core.cma ci_jpeg.cma ci_bmp.cma
-I +lablGL lablglut.cma lablgl.cma interpolate.ml transition.ml
camera.ml loader.ml main.ml -o main

Last part of this “trilogy” will be about particle transformations in OpenGL.
Hope you enjoyed it!

So I was looking for some excuse to learn OCaml + OpenGL, and I run into Radiohead’s House of Cards video dataset hosted at google code.
The dataset is made of many CSV files, one for each frame of the HoC video.
The data is also shipped with an application that uses Processing to create an image for each frame of the video.

I decided to do the same program in OCaml + OpenGL: for each CSV file, the program loads it, renders it in OpenGL, and then saves that rendered data into a jpg (or bmp) image.

You can merge the generated image frames with the sample mp3 provided at google code, by using ffmpeg:

ffmpeg -f image2 -r 30 -i ./img%d.jpg -sameq -i 1.mp3 ./out.mpeg -pass 2

Anyway, the result is quite interesting, and it gives us a good ground to build better visualizations:

(This youtube video quality is pretty lame, I’d recommend you to right click here and save link as…).

This post is going to be about the making of this simple application.
Further posts on this “project” will cover advanced camera movement and particle transformations.

The Code

This app was made in one single file, but it contains two important parts:

A data object containing information about the location of the CSV and generated image files, along with some methods to load CSV files and save OpenGL rendered pictures into image files (bmp and jpeg formats).
This object uses camlimages for saving images in different formats, and the OpenGL/GLUT bindings provided by lablGL.

(* Loads csv frames and saves the rendered OpenGL image *)
let data =
  object (self)
    val path_to_file = "path_to_folder_containing_csv_files"
    val path_to_image_file = "path_to_folder_that_will_contain_imgs"
    val mutable current_frame = 1
    val total_frames = 2101
    val time_interval = 33

    method get_time_interval =   time_interval

    method load_file filename =
      let channel = open_in (path_to_file ^ filename) in
      let ans = ref [] in
        try
          while true do
            let line = input_line channel in
            let sp = split (regexp ",")
                        (sub line 0 (pred (length line))) in
            match List.map float_of_string sp with
              | [ x; y; z; d ] ->
                ans := DVertex (x, y, z, d) :: !ans
              | _ -> raise (Invalid_argument "not a depth vertex")
          done;
          !ans
        with End_of_file | Invalid_argument _ ->
                 close_in_noerr channel; !ans

    method save_image =
        let img_rgb = new OImages.rgb24 600 400 in
        let pixels = GlPix.read
          ~x:0 ~y:0
          ~width:600 ~height:400
          ~format:`rgb ~kind:`ubyte
        in
        let praw = GlPix.to_raw pixels in
        let raw = Raw.gets ~pos:0 ~len:(Raw.byte_size praw) praw in
        let w = GlPix.width pixels in
        let h = GlPix.height pixels in
        for i = 0 to pred (w * h) do
          let color_rgb = { r = raw.(i * 3 + 2);
                                  g = raw.(i * 3 + 1);
                                  b = raw.(i * 3 + 0) }
          in
            img_rgb#set (i mod w) (i / w) color_rgb;
        done;
        img_rgb#save (path_to_image_file ^ "img" ^ (string_of_int current_frame) ^ ".jpg")
                              None []

    method next_frame =
      current_frame <- (current_frame + 1) mod total_frames;
      if current_frame = 0 then
        current_frame <- total_frames;
      self#load_file ((string_of_int current_frame) ^ ".csv")
end

This object is included in the main.ml file, which is the main entry point for the OpenGL application.
This file defines functions for initializing and binding events to the main openGL app. You'll find this code familiar if you know some OpenGL.

open Str
open String
open Color
open VertexType

(* Initializes openGL scene components*)
let init width height =
    GlDraw.shade_model `smooth;
    GlClear.color (0.0, 0.0, 0.0);
    GlClear.depth 1.0;
    GlClear.clear [`color; `depth];
    Gl.enable `depth_test;
    GlFunc.depth_func `lequal;
    GlMisc.hint `perspective_correction `nicest

(*  Draws the image*)
let draw () =
  GlClear.clear [`color; `depth];
  GlMat.load_identity ();
  GlMat.translate3 (-150.0, -150.0, -400.0);
  GlDraw.begins `points;
  List.iter (fun (DVertex (x, y, z, d)) ->
    let color = d /. 255. in
      GlDraw.color (color, color, color);
      GlDraw.vertex ~x:x ~y:y ~z:z ()) data#next_frame;
  GlDraw.ends ();
  Glut.swapBuffers ();
  data#save_image

(* Handle window resize *)
let reshape_cb ~w ~h =
  let
    ratio = (float_of_int w) /. (float_of_int h)
  in
    GlDraw.viewport 0 0 w h;
    GlMat.mode `projection;
    GlMat.load_identity ();
    GluMat.perspective 45.0 ratio (0.1, 1300.0);
    GlMat.mode `modelview;
    GlMat.load_identity ()

(* Handle keyboard events *)
let keyboard_cb ~key ~x ~y =
  match key with
    | 27 (* ESC *) -> exit 0
    | _ -> ()

(* A timer function setted to draw a new frame each time_interval ms *)
let rec timer value =
  Glut.postRedisplay ();
  Glut.timerFunc ~ms:data#get_time_interval
                 ~cb:(fun ~value:x -> (timer x))
                 ~value:value

(*  Main program function*)
let main () =
  let
    width = 640 and
    height = 480
  in
    ignore (Glut.init Sys.argv);
    Glut.initDisplayMode ~alpha:true ~depth:true ~double_buffer:true ();
    Glut.initWindowSize width height;
    ignore (Glut.createWindow "Radiohead HoC");
    Glut.displayFunc draw;
    Glut.keyboardFunc keyboard_cb;
    Glut.reshapeFunc reshape_cb;
    Glut.timerFunc ~ms:data#get_time_interval
                   ~cb:(fun ~value:x -> (timer x))
                   ~value:1;
    init width height;
    Glut.mainLoop ()

let _ = main ()

You can download the source here.
You can compile the files with the bytecode compiler:

ocamlc -g str.cma -I +camlimages ci_core.cma ci_bmp.cma ci_jpeg.cma
-I +lablGL lablglut.cma lablgl.cma main.ml -o main

.
Just remember that you have to install OCaml + lablGL + camlimages to be able to use this.

Any comment about the code will be well appreciated, since I'm an OCaml beginner .