I’m currently working on a project in collaboration with Karen Kipphoff to be part of the “Tracing Spaces” exhibition opening at Bergen kunsthall next Friday. We will be using a large video projection sized 2048 × 407 pixels. It will be composited in real time using Jitter made up from several layers and with some processing of the various layers: blending cropping rotating and moving layers along the x axis as well as chroma keying. It is very demanding on the computer so I have spent a lot of time figuring ways to optimize the processing. We will be using a PC Shuttle with an ATI Radeon 9800 graphical card and split the image to two projectors.
Parts of the processing will take place on the CPU but I am also using OpenGL processing to move parts of the processing to the graphical processing unit (GPU). Here are some of the solutions I have come up with so far:
The background image will be rotated in the x-direction. I first used jit.rota but then found that chopping the image in two and mount again using srcdim and dstdim double the frame rate. I guess the reason is that jit.rota is capable of performing more complex rotations and distortions than we need. As many of the parameters are left unused in our case I suppose that there will be some redundant processing going on in jit.rota that we avoid by using only jit.matrix with sourcedim/dstdim magic applied.
Using colormode uyvy reduces the dimension of the matrix from 2048×407 to 1024×407 again doubling the frame rate. The downside is that rotation now can happen only in steps of 2 pixels in the horizontal and might appear a bit less smooth. On the other hand one pixel projected onto the wall will equal approx. 0.400 cm so I assume that we can live with it. Poor framerate would be much worse.
Yesterday Joshua Kit Clayton posted a slab for hardware processed convertion of uyvy matrix to texture on PC on the MaxMSP list. Compared to software convertion to texture on PC it is twice as fast.
Several of the layers will change sometimes but not always. An important part of the tweeking is to avoid unnecsessary processing when no changes are taking place. One way of doing this is by using “zl reg” to store the name of matrixes and gl_textures and pull these instead of the jitter objects themselves when static so that I do not force unnecsessary recalculation of matrixes or textures.
I’ve also been investigating the data flow in Jitter and OpenGL rendering to avoid processing twice. The pacth below is quite instructive. I have set all jit.gl.objects to “@automatic 0”. This way I can control the sequence of rendering to make sure I get correct layering. Check e.g. the jit.gl.videoplane-xfade example coming as part of the MaxMSP/Jitter install to see how this is working. In addition I can avoid tasks being processed several times for each frame. E.g. the patch shows that when jit.gl.texture receives a jit_matrix the matrix is copied to the memory of the object. This probably costs a bit of CPU/GPU. If the incoming frame do not change there’s no need to do it again. Also the patch illustrates that if jit.gl.texture receives a matrix it outputs the new gl_texture. In a more complex patch this might force other jit.gl.objects to update as well. I’m not sure about this but I fear that they migth end up doing it more than once per displayed frame.
A lot of these optimalisations would not be possible in the previous version of Jitter (1.2).
Working in Jitter generally implies a lot of work of a very technical mode it is not necsessarily fast and intuitive. On the other hand computers are still quite slow for dealing with real-time video processing if you carfe about resolution and quality. Let’s face it: 8 bit color resolution 25 frames or so a second and 2048 pixels wide is still a very low resolution I’m convinced that we will at some point be able to work on float32 color resolution 50 frames per second or more and ten times as detailed images. At the moment one have to find smart and creative solutions to get the maximum results of the hardware at hand. Out of necsessity this requires work on a fairly technical level dealing with and exploiting hardware and software implementation. I don’t mind. Ever since I started using MSP I have been thinking of sound as a stream of samples and dealt with the maths implied in digital sound processing. It is the same way when dealing with video I find it more interest to deal with images as a bunch of pixels rather than working on abstract image processing simulating analogue photo techniques.
One last thing: I’ve made a custom ramp object that’s differing from the Max objects line and bline. When ramping to a new value it takes place in a fixed amount of time same way as for line. But it only outputs current value in responce to a bang received sort of same way as bline. It is using the CPU clock to get current time thus avoiding problems if the Max clock friifts out of sync. The benefit of the object is that I can ramp images to new settings in a fixed amount of time. When using bline transitions typically gets slower if the framerate drops. Compared to using line I will always get the most updated value of the parameter when the frame is calculated. I have also added several ramp functions in addition to the straight line: cosine tanh sinh and exponential. I haven’t had time to include in tl.objects yet but I will do as soon as I get the time probably not untli the beginning of December.
All video processing will be organised in a numjber of custom Jamoma modules to simplify high-level control of parameters.
This must be the techiest blog entry so far….
#P user gswitch2 447 302 39 32 1 0 #P user gswitch2 447 179 39 32 0 0 #P button 476 113 15 0 #P button 458 263 15 0 #P button 457 153 15 0 #P window setfont “Sans Serif” 9.000 #P number 83 91 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0 #P window linecount 1 #P newex 387 338 52 196617 r dothedo #P newex 303 93 52 196617 r dothedo #P newex 105 158 52 196617 s dothedo #P user ubumenu 275 68 141 196617 0 1 1 0 #X add 0 – only update videoplane #X add 1- update texture #X add 2 – update movie #X prefix_set 0 0 0 #P newex 275 117 38 196617 gate 2 #P message 497 326 34 196617 reset #P newex 494 348 84 196617 jit.gl.handle test #P user jit.fpsgui 24 223 60 196617 0 #P user jit.pwindow 493 175 82 62 0 1 0 0 1 0 #P toggle 36 91 15 0 #P newex 36 136 149 196617 t b b erase #P newex 36 113 57 196617 qmetro 33 #P newex 36 199 171 196617 jit.gl.render test @ortho 2 @sync 0 #P newex 36 262 150 196617 jit.window test @size 400 300 #P newex 476 374 255 196617 jit.gl.videoplane test @scale 1.333 1.000 1.000 @automatic 0 #P newex 476 263 154 196617 jit.gl.texture test @automatic 0 #P message 538 124 30 196617 read #P newex 476 151 105 196617 jit.qt.movie 320 240 #P fasten 17 0 3 0 392 367 481 367 #P connect 2 0 20 0 #P connect 2 0 23 1 #P connect 23 1 3 0 #P connect 22 1 2 0 #P connect 0 0 19 0 #P connect 0 0 22 1 #P connect 0 0 9 0 #P connect 21 0 0 0 #P connect 18 0 6 1 #P fasten 7 0 10 0 41 158 29 158 #P connect 7 0 5 0 #P fasten 7 2 5 0 179 183 41 183 #P connect 11 0 3 0 #P fasten 13 0 2 0 280 249 481 249 #P fasten 13 1 0 0 308 142 481 142 #P connect 16 0 13 1 #P connect 7 1 15 0 #P connect 14 0 13 0 #P connect 12 0 11 0 #P fasten 1 0 0 0 543 145 481 145 #P connect 6 0 7 0 #P connect 8 0 6 0 #P window clipboard copycount 24
