Computer Vision Team
3D computational geometry, sensor optics, real-time point cloud processing, and physics-grounded algorithms for 99.99%+ industrial automation reliability.
Team Overview
The Computer Vision team builds the sensory intelligence layer of MujinOS. In high-speed logistics facilities and factories processing hundreds of thousands of packages daily, robots cannot rely on black-box probabilistic models that achieve academic 95% benchmarks. On an automated production line, a 5% error rate translates to thousands of operational halts and catastrophic downtime.
Our mission is Vision Engineering: uniting optical physics, 3D computational geometry, calibrated sensor arrays, and deterministic algorithms to conquer the “Last 5%” and achieve 99.99%+ continuous operational reliability. Rather than treating perception as an opaque statistical black box, we engineer deterministic spatial intelligence that models physical reality directly—evaluating surface normals, computing 6-DoF grasp affordances, and rejecting optical noise in real time.
Products & Applications Powered by This Team
Mujin Robot Controller
Drives 3D point cloud perception, segmentation, and 6-DoF pose estimation across Depalletizing, Palletizing & Mixed-SKU, Piece Picking, and Bin Picking.
Real-Time Digital Twin
Feeds real-time workpiece point clouds and calibrated sensor geometry directly into the active virtual cell for collision verification.
Tech Stack
Core Languages & Execution
- Languages: C++ (low-latency execution), Python 3.11+ (geometric algorithm design & rapid prototyping)
- Environment: Ubuntu LTS, Linux with real-time kernel optimizations
- Build Toolchains: CMake, Docker
3D Perception & Spatial Engines
- Point Cloud Processing: PCL (Point Cloud Library), Open3D, custom SIMD-vectorized spatial filters
- Math & Spatial Representation: Eigen3, KD-trees, Octrees, Lie algebra ($SO(3)$, $SE(3)$)
- Geometric Alignment: Point-to-Plane ICP, RANSAC outlier rejection, robust surface normal estimation
Sensors, Optics & Pipeline
- Sensors: High-speed stereoscopic rigs, structured light 3D cameras, active range sensors (stationary & Eye-in-Hand)
- Optics & Calibration: Epipolar geometry, intrinsic/extrinsic camera calibration, bundle adjustment
- Data Pipeline: Low-latency shared memory IPC (
shm_open, lock-free ring buffers) for multi-megapixel 3D point cloud feeds
Simulation & Verification
- Physical Testbed: Tokyo HQ testing floor with calibrated camera test cells and multi-axis industrial manipulators
- Sensor Simulation: Real-time 3D optical simulation and ray-traced geometric validation
- Analysis Tooling: Tracy Profiler, Perf, AddressSanitizer (ASan), ThreadSanitizer (TSan)
Technical Challenges
Adaptive Grasp Calculation for Deformed Packaging
Challenge: Cardboard cartons with crushed corners or surface creases caused vacuum suction leakage and dropped items, bringing automated lines to an immediate halt. Brute-force data collection of destroyed products was unviable.
Approach: Formulated a physics-grounded geometric evaluation analyzing 3D surface normal deviation and planar flatness, dynamically relocating suction points toward rigid zones closer to the item’s center of mass.
Impact: Eliminated vacuum drop failures, maintaining 99.99%+ pick reliability across millions of irregular commercial package cycles.
Continuous Autonomous Camera Calibration
Challenge: Ambient temperature swings between day and night in factory and warehouse environments cause microscopic thermal expansion of camera mounting frames, causing spatial calibration drift over extended shifts.
Approach: Architected an online self-calibration routine using stationary cell fiducials and asynchronous bundle adjustment running during robot dwell or motion intervals.
Impact: Maintains sub-millimeter 3D spatial calibration accuracy across continuous 24/7 operations without interrupting production throughput or requiring manual technician recalibration.
Key Responsibilities
- Target Object 3D Perception: Design geometric algorithms that selectively capture and process point clouds of pick target objects without requiring CAD models or prior SKU master registration.
- Deterministic 6-DoF Pose Calculation: Formulate robust spatial algorithms that calculate exact 6-DoF workpiece coordinates from raw point clouds in real time.
- Optical Physics & Reflection Rejection: Develop adaptive optical filters that eliminate specular reflections, transparent film glare, and ambient factory lighting swings.
- Continuous Sensor Calibration: Build autonomous intrinsic and extrinsic calibration pipelines that automatically compensate for camera mount thermal expansion and mechanical vibration.
- Deformable & Bag Perception: Calculate optimal pick centroids and vacuum suction distributions on soft sacks, mail bags, and wrapped goods with variable geometry.
- Gemba Verification on Iron: Validate vision algorithms on physical test cells at Tokyo HQ across thousands of packaging variations, bridging mathematical design with factory reality.
Who Will Enjoy and Excel on This Team
If you want your computer vision code to interact with the physical world rather than just label 2D bounding boxes on benchmark datasets, this team is an incredible place to be. You will excel and have the most fun here if you:
- Love Solving Real-World Optical Chaos: You get a genuine rush when your algorithms conquer messy factory reality—extreme reflections on metallic parts, transparent plastic film, dust, and crushed boxes.
- Enjoy Thinking in 3D Geometry & Physics: You reason naturally in 3D coordinate frames, camera projection matrices, point cloud surface normals, and transformation matrices rather than treating perception as an opaque black box.
- Take Pride in High-Performance Code & Determinism: You care deeply about cache efficiency, SIMD vectorization, low-latency shared memory, and writing deterministic code that runs 24/7 with 99.99%+ reliability.
- Like Hands-on Testing (The Gemba): You prefer walking down to the Tokyo HQ testing floor, setting up camera rigs, observing real robot manipulation, and diagnosing anomalies directly on live test cells.
- Want High Autonomy in a Global Team: You thrive when given the freedom to own architectural choices, test new ideas quickly, and collaborate with passionate engineers from 26+ countries.
Open Roles & Requirements
Required Skills vs. What You Can Learn
We look for strong first-principles thinkers rather than domain perfectionists. You do not need to check every box on day one.
- Applied Mathematics & 3D Geometry: Deep comfort with coordinate transformations, 3D rotation representations (matrices, quaternions), linear algebra, and projective geometry.
- Solid Programming: Strong proficiency in C++ (memory safety, pointer mechanics, performance) or Python (numerical and scientific stack).
- Data Structures & Algorithms: Experience with spatial indexing (KD-trees, Octrees, voxel grids) and computational geometry fundamentals.
- Practical Debugging Mindset: Ability to trace why a vision algorithm fails on an edge-case workpiece rather than treating perception as an opaque system.
- Multi-View Calibration: Epipolar geometry, structured light calibration, and markerless bundle adjustment.
- SIMD Point Cloud Optimization: Low-latency vectorization and cache-conscious point cloud data structures.
- Industrial Packaging Optics: Mitigating specular reflections from plastic wrap and metallic surfaces under variable factory lighting.
- Robot & Sensor Integration: Direct synchronization of vision perception pipelines with real-time motion planning engines.
Interview Process
This team evaluates candidates through our standardized Engineering & Systems Track:
- Recruiter Screen (30 min): Career background, motivations, and culture alignment.
- Technical Assessment (60–90 min): Hands-on problem solving, 3D geometry fundamentals, and practical design.
- System Architecture Deep Dive (60 min): Technical tradeoffs, real-time determinism, and perception pipeline design.
- Gemba Tour & Leadership Conversation (60 min): Live tour of Tokyo HQ labs/testing cells and mutual expectation alignment.
For interview preparation tips, sample geometry questions, and domain deep dives, read Ace Your Computer Vision Job Interview at Mujin.