PoC - Master Thesis

Optimierung von Augmented Reality Anwendungen durch die Berücksichtigung von Tiefeninformationen mit Googles Project Tango

Optimization of AR applications considering the depth information with Googles project tango

Table of Content

• Final Prototoype
• Augmented Reality Demo
• Pointcloud Extraction
• Plane based Reconstruction
• PCL based native Reconstruction (Greedy Projection Triangulation)
• CHISEL based native reconstruction
• Unity Marching Cubes AR Demo
• OpenCV Depthmap Generation and Filtering
• OpenCV Depthmap Filtering for AR

Final Prototype

• depthmap rendering with 3 different reconstruction mechanisms
  • pointcloud projection
  • tsdf reconstruction
  • plane reconstruction
• using depthmap as Z-Buffer (GL_DEPTH_COMPONENT) for augmented reality occlustion

• optional OpenCV guided filtering with settings possible

Augmented Reality (ar/)

• AR Camera matching intrinsics
• motion tracking with correct gl positioning
• visualization of the current pointcloud
• ray-intersection for pointcloud interaction
• simple tower defense game with ray intersection
• simple direct pointcloud z-buffer occlusion

Pointcloud App (pc/)

• collects points into a large pointcloud using an octree
• exports to pts file
• plane based reconstruction applicable
• marching cube reconstruction applicable (usualy to slow here!)

Plane Based Reconstruction (construct/)

1. OctTree clustering of the global pointcloud
2. on incoming depth frame
	* update random n global clusters with points from depth frame
	* use RANSAC to detect 3 planes per cluster
	* project points from 3D space to 2D space (based on detected planes)
	* use Graham Scan to compute comvex hull for each plane
	* use Sweep‐line for triangulation with poly2tri
	* project polygon vertices back to 3D space (based on detected planes)

Video Demonstration

Old Screenshots

Native PCL based Reconstruction (construct-native/)

• VoxelGrid downsampling
• Greedy Triangulation with PCL
• Build PCL Library not included!

CHISEL open_chisel implementation (chisel/)

• online reconstruction using open_chisel library
• transformation and interfaces to native environment
• PLY exporter
• Build open_chisel Library not included!

OpenCV Depthmap Generation and Filtering

• generate depthmap using this approach
• applying inpaint and guided filtering for depthmap improvements
• Build opencv+contrib Library not included!

Video Demonstration

OpenCV Depthmap Filtering for AR

• using the previous approach for a pointcloud reprojection

Unity Implementierung (unity/)

• Reconstruction using the MeshBuilder with an simple TSDF by Google
• Implementation of a Clipping Depth Shader
• simple ball moving interaction
• extension of shader for shadowing effects

Video Demonstration

Old Screenshots

Verwendete Librarys

• Rajawali
• tango-examples-java
• material-dialogs
• EventBus
• commons-math3
• jama
• Poly2Tri
• PCL
• Unity Engine
• OpenCV
• tango-examples-unity