Week 2 for Google Summer of Code 2013 is mainly about model management and getting 3D models exported into Three.js. This is necessary because the final application will have multiple markers, with different 3D models mapped to each marker. The 3D models obtained come in a variety of formats, so I am using Blender to read these files and then exporting them into Three.js formats.

I have also created a Javascript framework skarf.js that will handle arbitrary augmented reality and rendering libraries and simplify the setup code.

Results

I am using JSARToolKit and Three.js in conjunction with skarf.js to get the above demo working in Google Chrome. A JSON model file is supplied which maps marker IDs to model specs (name, url etc). Marker 22 maps to the US police car while marker 64 maps to the bench.

The models were imported into Blender and exported to Three.js-compatible formats. I have created .obj, .js and binary .js formats and imported them into Three.js successfully, using my ModelManager, ModelLoaderFactory and corresponding ModelLoader wrapper classes, as described in sections below.

I am using my webcam for this demo. Marker 49 maps to the US police car, marker 81 maps to the bench and marker 63 maps to the house.

I was a little surprised that the models were showing jitters once in a while even though the webcam was stationary. Perhaps there was too much noise in the video stream. This is something to investigate in later weeks.

Back To Top

skarf.js

This week, I decided to write skarf.js, a generic augmented reality framework that is able to take in any augmented reality library and rendering library. I will be trying out different JavaScript augmented libraries in future weeks, so a generic framework like this will also allow me to easily swap between the different libraries. This framework should also help to simplify some of the setup codes needed.

To use the library, create an instance of the augmented reality library and an instance of the renderer, then create a SkArF instance:

//get canvas element e.g.
var canvas = document.getElementById('myCanvas');

//get video element
var video = document.getElementById('myVideo');

//create an AR library
var jsArToolKitArLib = new JsArToolKitArLib...

//create a renderer
var threeJsRenderer = new ThreeJsRenderer...

//create an AR framework (SkArF)
var skarf = new SkArF({
    canvasElem: canvas,
    videoElem: video,
    arLib: jsArToolKitArLib,
    renderer: threeJsRenderer
});

Then within the main loop in your application, call:

skarf.update();

Take a look at threejs_skarf.html for a complete example.

If you wish to use your own augmented reality library, you can do so by:

• Subclassing ArLib
• Override the init() and loop() methods

You can do similar things to create your own renderer.

Unfortunately, it only works with Three.js and JSARToolKit now. I will need to remove the coupled codes in future weeks when I get back to this topic, and also set up example files and create the min file.

Back To Top

Exporting Models From Blender

Most of the models found online come in a variety of formats, such as .dae (COLLADA), .3ds and .obj. I am using Blender to read in these formats. It is a powerful open-source 3D package that can not only do complete animation and visual effects work, but also read in a variety of file formats. The latter is just what I need!

Before exporting, I made sure that the model is facing the -Y direction in Blender (more explanation in the Coordinate Systems section).

As a side note, I found some free models online and had trouble getting the textures and materials to show up correctly in Three.js for these models. Then I realized that the model itself is problematic to start with: textures not assigned, missing UVs etc. Thus, if you have tried all ways to export and the mesh simply does not work as intended, then perhaps the mesh itself is problematic.

Exporting to .obj

Blender comes with an obj exporter by default, so there’s no need to install something special from Three.js repository.

These are the settings that I was using to get my static non-deforming models to .obj:

Blender .obj export settings that I was using

Some things to note:

• Forward should be -Z and Up should be Y to match Three.js coordinate system (see the Coordinate Systems section for more information)
• I had to check “Triangulate Faces” to prevent some missing faces from disappearing
• Check “Copy Textures” so that the texture paths in the .mtl are not relative

One minor bug that I have found: .obj models exported from Blender into Three.js look darker than expected. I got a hint online that this has something to do with the exported ambient value using the world ambient color rather than the material ambient color. I tried to follow the codes in the forum post but the suggested edit didn’t seem to work for me. What I ended up doing was to change line 84 in Blender’s obj exporter (C:\Program Files\Blender Foundation\Blender\2.67\scripts\addons\io_scene_obj\export_obj.py) from:

fw('Ka %.6f %.6f %.6f\n' % (mat.ambient * world_amb)[:])

to

fw('Ka %.6f %.6f %.6f\n' % (mat.ambient * mat.diffuse_intensity * mat.diffuse_color)[:])

This new equation is based on the calculations found in the Three.js exporter (C:\Program Files\Blender Foundation\Blender\2.67\scripts\addons\io_mesh_threejs\export_threejs.py). With this edit to the OBJ exporter, the exported .obj model now looks the same as the same .js model when loaded into Three.js. Sweet!

Exporting to .js

In order to export to a Three.js-compatible .js model file, I installed the Three.js exporter for Blender. If you are following along, please make sure that you are getting the exporter from the same release of Three.js that you are using.

These are the settings that I was using to get my static non-deforming models to .js:

Blender .js export settings that I was using

Some things to note:

• “Flip YZ” has to be checked (see the Coordinate Systems section for more information)
• Check “Copy Textures” so that the texture paths in the .js file are not relative

Exporting to binary .js

Saving and loading large data ASCII files is never a good idea due to the large file size and slow reading/writing times, so I was looking for a way to get a binary model format that Three.js can read in. It turns out that there is a binary .js format, which is great. The catch is that only a python converter can create this format (as far as I know), and I need a .obj file as the starting point.

I converted the .obj file to binary .js file using the “-t binary” flag, like so:

python.exe C:\path\to\convert_obj_three.py -i myFile.obj -o myFile_bin.js -t binary

After running the command, two files were created: a .bin file and a normal .js file which points to it.

As expected, binary files are much smaller. For example, these are the file sizes of the house_15 asset that I have exported:

Back To Top

Importing Models into Three.js

Three.js comes with a variety of loaders that can load in models of different formats:

• BinaryLoader.js for loading binary .js files
• OBJMTLLoader.js and MTLLoader.js for loading .obj files that come coupled with .mtl files

However, they do not have a common interface and it becomes troublesome to dynamically load in different types of models. So I created a bunch of ModelLoaders and a ModelLoaderFactory that creates these instances. These instances are managed by a ModelManager, which takes in a model JSON file that specifies attributes of each model, such as the marker ID it is attached to, the file url, the file type and any additional transforms. These classes are currently in skarf.js (I will most probably move the codes out into their own .js file soon once I start code cleanup in future weeks). Note that you will still need to include the loader .js files in your application.

As a side note, I found a bug in Three.js’s MTLLoader.js which causes this error: “Object [object Object] has no method ‘loadTexture’”. Made a pull request to the GitHub respository.

Back To Top

Coordinate Systems

I imported some models into Three.js and realised that they do not appear in the right orientation. I was not sure whether the model was exported in the wrong orientation or whether the coordinate systems were different in Blender, Three.js, JSARToolKit etc, so I did a little investigation.

I created a simple dice in Blender (pardon the quick paint job!) and exported it into Three.js. I then attached a local axis to the different objects using THREE.AxisHelper.

These are the results:

Coordinate system in Blender: right-handed, +Z up

Coordinate system in Three.js: right-handed, +Y up

Coordinate system in JSARToolKit: left-handed, -Z up

To get a matching coordinate system from Blender to Three.js is not an issue. The exporters in Blender come with an option to get a Y-up coordinate system for the exported model. For example:

• .obj exporter comes with options of specifying the forward and up axes (which should be -Z and +Y respectively)
• Three.js .js exporter comes with a “Flip YZ” option

Additionally, the object should face the right direction (-Y) in Blender so that it faces the forward direction +Z when exported into Three.js.

However, to get a matching coordinate system from JSARToolKit to Three.js, I needed to do some additional work. JSARToolKit is in the left-handed coordinate system, as opposed to the right-handed coordinate system used in Blender and Three.js, and its up axis is -Z. I resolved this by scaling the JSARToolKit matrix in -Z to get it from left-handed coordinate system to right-handed. Then rotate +90 degrees in the X axis to get +Y to point upwards.

var m = new THREE.Matrix4();

//get marker transform
var markerTransform = this.markerTransforms[markerId];

//scale in -z to swap from LH-coord to RH-coord
m.makeScale(1,1,-1);
markerTransform.matrix.multiply(m);

//rotate 90deg in X to get Y-up
m.makeRotationX(THREE.Math.degToRad(90));
markerTransform.matrix.multiply(m);

//mark the world matrix for updating
markerTransform.matrixWorldNeedsUpdate = true;

So now everything is nice and neat in the right-handed Y-up world:

Corrected coordinate system for JSARToolKit

Back To Top

Misc

As part of trying to get a more complete web application, I started to add in some miscellaneous enhancements to the HTML pages.

The first thing that you might notice is a nifty frame rate counter using stats.js in the pages:

A nifty frame rate counter using stats.js

Time elapsed counter (after clicking on the frame rate counter once)

Now that I can see the frame rate, I realized that if I run Three.js on its own, I can get 60 fps. However, once I start mixing in JSARToolKit, the frame rate immediately drops to 20 fps. There’s probably something in the JSARToolKit code that restricts the sampling rate. To investigate in future weeks.

There’s also a nice user interface, thanks to the excellent dat.gui:

A nice GUI for user options using dat.gui

This should help when I have more and more options in future weeks for the user to adjust.

Lastly, I’m using Detector.js to detect WebGL on the browser. This will elegantly fail when WebGL is not available and show a message to the user:

The message shown when WebGL is not available

Back To Top

Credits

• Video of markers on mobile device by JSARToolKit
• US Police Car model by Antics 3D
• Bench model by dandis
• House model by Doormatt

Back To Top

Source Codes

The source codes at the end of week 2 can be obtained from my ifc-ar-flood GitHub repository. Checkout the “GSoC_wk_02_end” tag to get the state of the source codes at the end of week 2.

Please remember that you need to serve the HTML pages using a http server before you can see them as intended.

Back To Top