Computer Vision Team
Uniting optical physics, 3D geometry, sensor arrays, and deterministic algorithms for 99.99%+ continuous industrial automation perception.
Team Overview
The Computer Vision team builds the sensory intelligence layer of MujinOS. In high-speed logistics facilities and automated factory production lines processing hundreds of thousands of packages daily, robots cannot rely on black-box probabilistic models that achieve academic benchmarks. On an automated production line, even small error rates translate to frequent operational halts and severe downtime.
Our mission is to unite optical physics, 3D computational geometry, calibrated sensor arrays, and deterministic algorithms to conquer the “Last 5%” and achieve 99.99%+ continuous operational reliability in industrial automation, moving beyond black-box probabilistic models.
We provide sensory intelligence for the Mujin Robot Controller (Depalletizing, Palletizing & Mixed-SKU, Piece Picking, Bin Picking) and feed real-time point clouds into the Real-Time Digital Twin for active collision verification. If the vision system fails, operations halt and cause catastrophic downtime.
Products & Applications Powered by This Team
Mujin Robot Controller
Drives 3D point cloud perception, segmentation, and 6-DoF pose estimation across Depalletizing, Palletizing & Mixed-SKU, Piece Picking, and Bin Picking.
Real-Time Digital Twin
Feeds real-time workpiece point clouds and calibrated sensor geometry directly into the active virtual cell for collision verification.
Tech Stack
Languages & Runtimes
- C++ (low-latency execution)
- Python (geometric algorithm design & rapid prototyping)
- Debian Linux with real-time kernel optimizations
- CMake, Docker, CI
Perception Libraries & Frameworks
- OpenCV, PyTorch, TensorFlow, PCL, Open3D, scikit
- 2D and 3D data processing to extract features, classify and verify
- 2D and 3D spatial representation of information and their transformations
Applied Mathematics & Spatial Geometry
- Coordinate transformations
- 3D rotation representations (rotation matrices, quaternions)
- Projective geometry and camera projection models
- Linear algebra for real-time spatial calculations
Hardware, Testbeds & Tools
- High-speed stereoscopic rigs, structured light 3D cameras, active range sensors (stationary & Eye-in-Hand)
- Low-latency shared memory IPC for multi-megapixel 3D point cloud feeds
- Tokyo HQ testing floor with calibrated camera test cells and multi-axis industrial manipulators
- Real-time 3D optical simulation and ray-traced geometric validation
- Performance and memory profiling tools
Technical Challenges
Adaptive Detection for Deformed Packaging
Challenge: Cardboard cartons are generally considered to have a cuboid shape and most detection algorithms describe these with an oriented bounding box seen from the top. However, boxes can be slightly crushed due to stacked weight on top, bulging for content etc. causing them to be not cuboid in shape. This causes degraded performance on creating stable pallets when placing/stacking these items.
Approach: Formulated a physics-grounded geometric evaluation analyzing 3D surface and the contour, finding a more optimal description of the shape.
Impact: Improved packing stability and efficiency, while maintaining 99.99%+ pick reliability across millions of irregular commercial package cycles.
Stacking Order Detection of Deformed Paper Bags
Challenge: A pallet stacked with paper bags (for example containing sugar or flour) becomes tightly compressed over time. Depalletizing these items is a challenge because only the top bag can be picked. A small overlap of a bag can cause the bag to slip and fall off the pallet.
Approach: Created a two-pronged approach: first, an algorithmic approach that allows running the system in a supervised way; second, a data-driven approach that learns from the data collected during supervised production, improving performance to >99.9% over time. Collecting data for the data-driven approach was initially challenging because manually stacked pallets are not compressed as tightly as production pallets, and any mistake had to be prevented to avoid dropping or tearing bags.
Impact: Heavy paper bags can now efficiently be handled without causing a mess in the factory.
Key Responsibilities
- Design geometric algorithms that selectively process images and point clouds of target objects and determine their 6 DoF pose, shape and other properties.
- Enhance geometric algorithms to handle uncertainty and unknown target objects.
- Develop adaptive algorithms that can handle environmental and workpiece variations needed to make a system perform at production level.
- Investigate and understand problems quickly that are encountered in testing and production to find the right solutions to be implemented.
- Validate vision algorithms through testing pipelines to ensure continued quality of the system.
- Validate vision algorithms on physical test cells at Tokyo HQ across thousands of packaging variations, bridging mathematical design with factory reality.
Weekly Engineering Rhythm
- 50% Algorithmic design, 3D computational geometry, and C++/Python software implementation
- 25% Simulation, pipeline optimization, and testing
- 25% Hands-on testing and validation on live test cells / real robot arms at Tokyo HQ
Who Will Enjoy and Excel on This Team
If you want your computer vision algorithms to interact with physical reality rather than just benchmark datasets, this team is an incredible place to be. You will excel and have the most fun here if you:
- Love Solving Real-World Optical Chaos: Excited by conquering issues in reality encountered in production settings, and ensuring that these systems can work reliably 24/7.
- Enjoy Thinking in 3D Geometry & Physics: Naturally reason in 3D coordinate frames, camera projection matrices, point cloud surface normals, and transformation matrices rather than treating perception as an opaque black box.
- Take Pride in High-Performance Code & Determinism: Passionate about cache efficiency, SIMD vectorization, low-latency shared memory, and writing deterministic code that runs 24/7 with 99.99%+ reliability.
- Like Hands-on Testing: Prefer walking down to the Tokyo HQ testing floor, using our vision-team testing setup with various camera rigs, observing real robot manipulation, and diagnosing anomalies directly on live test cells.
- Want High Autonomy in a Global Team: Thrive with freedom to own architectural choices, test new ideas quickly, and collaborate across 26+ nationalities in English.
Who Might Struggle on This Team
- Engineers who prefer working solely with 2D bounding boxes on benchmark datasets or academic black-box models and dislike dealing with physical hardware, optical physics, or live test-cell debugging.
Join the Team
Qualifications & Growth
- Understanding of both traditional and data-driven computer vision algorithms, knowing how and when (or not) to apply them.
- Applied Mathematics & 3D Geometry: Coordinate transformations, 3D rotation representations (matrices, quaternions), linear algebra, projection, 3D geometry.
- Solid Programming: Strong proficiency in C++ (memory safety, pointer mechanics, performance) or Python (numerical and scientific stack).
- Data Structures & Algorithms: Spatial algorithms and computational geometry fundamentals.
- Practical Debugging Mindset: Ability to trace edge-case workpiece failures deterministically.
- A deeper understanding of how vision algorithms work and how they can be applied effectively in different situations.
- A deeper understanding of cameras, optics, calibration, illumination, and how to address the issues that arise from them.
- Deep application of C++ optimization for efficient computations to achieve low-latency edge computing.
- Direct synchronization of vision perception pipelines with real-time robot motion planning engines.
- An understanding of how your work impacts and hinges with real production in factories and warehouses, where integrators and operators use these on a daily basis.
Open Positions
View Computer Vision Openings
Explore active 3D computer vision and perception engineering openings on Lever.
Vision Engineering Beyond Deep Learning
Read our engineering deep dive on conquering the ‘Last 5%’ with computational geometry.
Ace Your CV Interview
Preparation tips, sample 3D geometry questions, and technical deep dives.