Augmented Reality in Blender got some attention a while ago. Several people [here, here and here] reported successful use of the Python ARToolkit bindings, Blender and the Blender videotexture plugin. Because I needed to do some experiments with AR, I tried PyARTK but it failed to compile on any Linux platform (unlike ARToolkit itself, which works fine).
Not willing to spend several days on fixing dependencies I decided to just forget about PyARTK and connect Blender to ARToolkit directly. However, my lack of (sufficient) C skills called for a ‘creative’ solution here.
Data flow for Blender AR. From left to right: the webcam stream is split into two virtual streams. The first stream is used by ARToolkit. The second stream is sent to the Blender Game Engine.
Tool chain
The above diagram illustrates the solution.
Tools used:
- Vloopback 1.3 to split the video into two virtual streams
- ARToolkit 2.72.1 to do the marker detection and tracking
- Modified simpleTest to put matrix in buffer
- Python script to read buffer and control BGE
- VideoTexture plugin for Blender to show video stream
Firstly, I’m using vloopback to split the webcam stream into two virtual streams, both accessible through /dev/video3 (in this case). One of these streams is used by a modified version of ARToolkit’s simpleTest. This tool analyses the video, finds a marker and outputs the resulting orientation matrix to a buffer in RAM (simply write the matrix to a tmpfs file). This buffer is read by a Python script in the Blender Game Engine, allowing it to align the camera and objects properly. The Game Engine uses the videoTexture plugin to show the webcam’s virtual stream in real-time.
Performance
The performance of this setup is better than I originally expected. I’ve tested it on two laptops (two dual cores, 2,5 Ghz and 1,6 Ghz) with NVidia graphic cards (G96M Quadro FX 770M, the other one is older). For now it’s Linux only (Tested on Ubuntu Karmic), mainly because of the vloopback trick. The other tools are pretty much platform independent I think. Compared to the standalone ‘simpleTest’ program there is some lag, but it doesn’t really affect the AR interaction.
Todo
- I’m working on a multi-marker setup
- Write some supporting python scripts for the BGE
This is really great. And I have a use for it too.
I have this shoot where there is a guy holding a prop that I am going to replace with an animated character in post.
It seems a little far fetched for my resources, but I could have a camera feed going into the blender game engine so I could get a nice on set preview, while recording a clean signal directly to the camera.
Very theoretical, in the sense of “who’s got the time”, but if you have any suggestions, I would love to hear them.
Luke
Posted by Luke Haddock | September 16, 2010, 21:27what is the script you use? You can send me it? Thanks!
Posted by daniel | November 22, 2010, 14:06what is the script you use? You can give me some tutorial done for you? Thanks!
Posted by daniel | November 22, 2010, 14:13Very good! Could you send me the script or any tutorial on Blender and Augmented Reality? Thanks!
e-mail: danieldhc86@hotmail.com
Posted by Daniel Henrique | November 29, 2010, 19:44How did you get the webcam stream into VideoTexture? I’ve been looking for days, and stil haven’t found it. Could you show some code of the VideoTexture part?
Posted by Izaak Van Crombrugge | January 23, 2011, 22:30Have a look here for a more thorough explanation and sample code.
Gr.
Jos
Posted by Jos Thalen | February 11, 2011, 00:09