This is a stable fork of utils3d maintained as a dependency for microsoft/MoGe.
A pure-Python collection of useful functions for 3D computer vision and graphics research.
- NumPy / PyTorch pairs: most functions have both implementations.
- Flat & non-modular: standalone functions only, no classes, no hierarchies.
- Native types: always use native Python / NumPy / PyTorch types.
- Vectorized only: no Python loops beyond O(log N).
-
If you are looking for one or two specific functions, feel free to copy them directly from the source code. This repo may serve a better purpose as a reference implementation.
-
If installing
utils3das a dependency, use commit id or fork one if you need stability.pip install git+https://github.com/EasternJournalist/utils3d.git@<commit_id>
-
If you are willing to accept the latest changes, fork this repo and join me in making 3D research easier!
- Use
utils3d.{function}to call the function automatically selecting the backend based on the input type (Numpy ndarray or Pytorch tensor). - Use
utils3d.{np/pt}.{function}to specifically call the Numpy or Pytorch version.
The links below will take you to the source code of each function with detailed documentation and type hints.
The PyTorch-only pose_graph_optimization and pose_graph_optimization_gnc
promote compact edge moments and weights to float64 before graph assembly.
Rotation synchronization, scale synchronization and translation solves run in
float64, including under autocast; GNC also keeps residuals and reweighting in
float64 between iterations. This avoids discarding weak nonzero graph modes
through float32 pseudoinverse rank thresholds. Results retain the input
cross-covariance's dtype and device. Raw point-map reduction, index dtypes,
solver objectives and robust-rejection settings are unchanged.
Run the precision regression checks from the repository root with
PYTHONPATH=. python test/test_pose_graph_precision.py.
utils3d.pt.pose_graph_optimization_huber (also available through automatic
dispatch) provides the harmonic-distance Huber IRLS solver. Like the other
PGO implementations it is PyTorch-only; no NumPy PGO backend exists.
moments = utils3d.pt.pose_graph_edge_moments(x, y, q)
Q = moments[-1]
harmonic_ranges = torch.stack((Q / (q / range_source).sum(-1), Q / (q / range_target).sum(-1)), dim=-1)
poses, weights, info = utils3d.pt.pose_graph_optimization_huber(
num_nodes, edges, *moments, harmonic_ranges=harmonic_ranges,
mode="similar", threshold=0.3,
)Use positive, guarded ranges from actual observer origins and the same point
weights as the original moments. Only the outer score is normalized by
s_ij * H_source * H_target; inner point moments/information remain unchanged.
Opposite queries share one Huber weight, clamped below by 1e-3. Defaults are
128 outer updates, weight-change tolerance 1e-4, and ten inner rotation
refinement steps. All compact calculations use FP64, preserving input dtype
and device at the output boundary. info reports iterations and convergence;
nonconvergence emits a RuntimeWarning. This solver does not add graph edges
or repair initially disconnected graphs. Pixel weight totals and optional
edge priors remain coefficients of the inner solver, not the percentage score.
No per-iteration CPU graph diagnostics or raw point decoding are performed. The ordinary PGO and existing GNC APIs retain their behavior.
pose_graph_edge_moments already returns the total point weight as its sixth
tensor, so two calls can supply all statistics needed by a relative RMS score:
inner = utils3d.pt.pose_graph_edge_moments(x, y, q)
scoring = utils3d.pt.pose_graph_edge_moments(x, y, q / range_source / range_target)
poses, weights = utils3d.pt.pose_graph_optimization_gnc(
num_nodes, edges, *inner, mode="similar", threshold=0.05,
residual_moments=scoring, residual_scale="geometric_mean",
)Here ranges are fixed distances to the actual camera origins (include any
calibration shifts), with positive, unit-consistent near-zero guards. Both calls
must use the same point correspondences and validity mask. The inner PGO always
uses inner; only the outer residual uses scoring. If the weight sums are
Q=sum(q) and A=sum(q/(range_source*range_target)), the squared outer score is
A/Q times the scoring-moment mean squared residual, additionally divided by
relative Sim(3) scale in "geometric_mean" mode. The resulting RMS is invariant
to source/target exchange of the same correspondences and independent local
unit changes; 0.05 means 5%. Omitting both new keywords preserves the previous
metric and GNC behavior. This change does not alter the continuation schedule,
pair directional weights, or add connectivity safeguards.
No point-count output is needed for this weighted metric: its denominator is
Q, not the sample count. Callers may separately retain point counts for
diagnostics. The cached moments remain valid while point predictions, weights,
calibration shifts and range guards stay fixed.
Both backends return original point indices, so the same selection can be applied to coordinates, colors, normals, or other attributes:
indices = utils3d.voxel_grid_sampling(points, voxel_size=0.01)
indices = utils3d.hierarchical_voxel_sampling(points, num_samples=100_000)
sampled_points = points[indices]Inputs are finite floating-point arrays/tensors of shape (N, 3); outputs are
one-dimensional int64 indices, on the input device for PyTorch.
voxel_grid_samplingkeeps the first input point in each occupied cubic voxel. The grid is anchored at the coordinate-wise minimum. Every input point is withinsqrt(3) * voxel_sizeof a selected point, up to numerical rounding.hierarchical_voxel_samplingsorts 21-bit-per-axis Morton codes once, chooses an octree level, and refines a spatially dispersed subset of its cells. Final cells differ by at most one level. Each is represented by the first point in Morton order, with original index breaking ties. For0 < num_samples < N, ifUfinest-grid cells are occupied andT = min(num_samples, U), the output count is betweenmax(1, T - 6)andT. Duplicate coordinates or finest-grid quantization can reduceU; outliers can limit effective resolution. A zero target gives empty indices, and a target at leastNreturns all indices, including coincident points.
These are grid-based coverage samplers, not FPS or Poisson-disk sampling: there is no minimum point-separation guarantee, no progressive FPS ordering, and selections need not be nested across targets. Both use float64 grid arithmetic, O(N) workspace, and sorting (O(N log N) in the comparison model), without loops over input points or output samples. PyTorch selection runs without autograd; gathering the returned indices can still propagate gradients. The PyTorch implementations require a device supporting float64, read bounds and (for the hierarchical sampler) at most five level counts on the CPU, and otherwise keep point data on the input device.
Invalid shapes, dtypes, parameters, nonfinite coordinates, and unrepresentable coordinate ranges raise errors. Single-scale voxel coordinates must fit int64; if only the packed key would overflow, lexicographic sorting preserves distinct voxels instead of allowing collisions.
Run the NumPy / PyTorch CPU and available-CUDA regression checks with
PYTHONPATH=. python test/test_sampling.py.
utils3d.pt.rasterize_triangles, utils3d.pt.rasterize_lines, and
utils3d.pt.rasterize_point_cloud share a small, point-sampled Triton rasterizer.
It implements homogeneous view-frustum clipping, backface culling, a top-left
shared-edge rule with eight subpixel bits, and perspective-correct interpolation.
Unsigned 64-bit atomic depth/face keys select the visible triangle; equal depths
choose the lowest face ID. It scans triangle bounding boxes, without a scene
graph, binning queues, or cached graphics contexts. Large overlapping or thin
triangles can be slow; this is a reference implementation, not a replacement for
a full graphics engine.
Install compatible GPU builds of PyTorch and Triton separately
(pip install torch triton). No nvdiffrast, OpenGL context, or custom CUDA/C++
extension is needed for the Torch path. NVIDIA execution still requires a GPU
driver and the PyTorch/Triton runtime; it does not require compiling .cu sources
or configuring a CUDA toolkit. Triton is loaded only when rendering is requested.
The NumPy/ModernGL backend is unchanged.
Image height and width are runtime kernel parameters, not tl.constexpr.
Their automatic value specializations (and that of the derived pixel count)
are disabled, so changing only resolution reuses the same compiled kernels.
Shape/mode switches such as point shape, culling, peeling and requested outputs
can still select different variants, as can argument types, other specialization
categories or GPU targets. The first use of a new variant still incurs JIT cost.
import torch
import utils3d
vertices = torch.tensor([
[-0.8, -0.8, 0.0], [0.8, -0.8, 0.0], [0.0, 0.8, 0.0],
], device="cuda")
faces = torch.tensor([[0, 1, 2]], device="cuda")
colors = torch.eye(3, device="cuda", requires_grad=True)
result = utils3d.pt.rasterize_triangles(
(256, 256), vertices=vertices, faces=faces, attributes=colors,
return_interpolation=True,
)
result["image"].sum().backward() # Gradients flow to colors, not geometry.Coordinates and camera matrices are float32 GPU tensors. Faces are shared
(T, 3) int32/int64 indices; omit them for (..., T, 3, D) triangle soup.
Vertex coordinates may have 2, 3, or 4 components. Vertex, camera and attribute
batch dimensions broadcast. Attribute tensors support any number of channels
and retain their floating-point dtype. Use attributes_domain="face" for
constant per-face attributes.
Outputs follow NumPy's top-left image convention: image has shape
(..., H, W, C), and mask has shape (..., H, W). Optional depth is
camera-space -z/w, with inf for background; with identity transforms this
is simply negative input z. Optional interpolation_id contains original face
IDs (-1 for background), and interpolation_uv contains the first two
perspective-correct barycentric weights (zero for background). Visibility uses
float32 camera-space linear depth, computed directly from view-space vertices
with perspective-correct interpolation, not reconstructed from NDC depth. The
64-bit atomic key packs the losslessly ordered float32 depth bits and a face ID;
it is not a float64 depth buffer. Returned depth is decoded from the winning key,
so visibility, peeling, and background comparisons use the same values.
Point billboards use their center's camera-space depth. There is no logarithmic
encoding in the Torch backend; NumPy's logarithmic depth buffer is unchanged.
NDC is used only for clipping, with the near/far planes included. Unlike the old
NDC depth test, fragments on the far boundary (or whose float32 projection rounds
to that boundary with an infinite far plane) are not discarded just because their
NDC depth equals 1. Signed linear depths are supported: with identity or custom
projections, the smallest -view_z / view_w wins, not necessarily the smallest
NDC z. This intentionally fixes the former inconsistency with background
compositing. Floating-point input/transform/interpolation error and genuinely
coplanar surfaces still limit precision; linear depth does not remove all causes
of z-fighting.
Geometry outputs are detached; only attributes receive gradients through native PyTorch indexing and weighted sums. Validation checks finite transformed coordinates and index bounds with one GPU-to-CPU synchronization. Background attributes are zero unless supplied explicitly.
Triangle, line and point-cloud rendering accept a background dictionary.
Compositing uses PyTorch operations after rasterization, without changing the
JIT core:
result = utils3d.pt.rasterize_triangles(
(256, 256), vertices=vertices, faces=faces, attributes=colors,
background={
"image": background_image, # (..., H, W, C), same dtype/channels as colors
"depth": background_depth, # (..., H, W), float32 camera-space -z/w
},
return_depth=True,
)Every entry is optional. The image uses the same top-left origin as the outputs.
Background depth is linear camera-space depth, not NDC, ray distance or an
alpha channel. Finite values (including signed depths for identity transforms)
and +inf are accepted; NaN and -inf raise errors. A covered foreground pixel
wins only when its depth is strictly smaller; ties retain the background.
Without background depth, foreground coverage alone selects the winner.
Depth occlusion also works with return_depth=False: depth is computed internally
but not returned. Point billboards use their center's camera depth for this test.
Background dictionaries may additionally carry mask (bool), interpolation_id
(int32) and interpolation_uv (float32, last dimension 2) for triangles/lines,
or point_id (int32) for points. All requested output maps follow the same winner.
Missing fields default to zero image/UV, inf depth, -1 ID and false mask,
matching NumPy's independent background-mask semantics. Mask is not inferred
from an image or finite depth, and a false mask does not disable depth occlusion;
supply a mask explicitly if background coverage should be reported. Metadata is
returned only when its corresponding return flag is enabled. IDs are preserved
without rebasing, so use a consistent ID convention when chaining render passes.
All tensors must share the geometry's GPU; spatial dimensions must match the requested image, while leading batch dimensions broadcast with geometry, cameras and attributes. Background images retain gradients, along with foreground attributes. Depth, selection and metadata remain detached. Inputs are not mutated. Without foreground attributes, a background image still produces an image output, with zero-valued foreground pixels. This is depth-based selection, not alpha blending. Depth peeling retains its separate background-free layer interface.
Point billboards and line rectangles are expanded on the GPU and reuse the triangle coverage/depth core, rather than maintaining separate depth algorithms. Attribute lookup and interpolation remain native PyTorch operations.
point_result = utils3d.pt.rasterize_point_cloud(
(256, 256), points=vertices, attributes=colors,
point_sizes=12, point_shape="circle", return_point_id=True,
)
line_result = utils3d.pt.rasterize_lines(
(256, 256), vertices=vertices, attributes=colors,
lines=torch.tensor([[0, 1], [1, 2], [2, 0]], device="cuda"),
line_width=2.5, return_interpolation=True,
)- Points: all five NumPy shapes are supported: triangle, square, pentagon,
hexagon and circle. Sizes can be a scalar or a broadcasting
(..., N)/(..., 1)GPU tensor.point_size_in="2d"uses pixels;"3d"uses camera-plane units, matching NumPy's billboard convention. Square size is its side length; circle size is its diameter; other polygons use the circumcircle diameter. Zero sizes are invisible. The full billboard is clipped, so centers outside the image can still contribute. Circles discard outside fragments before depth testing. Optionalpoint_idstores original IDs with background -1; depth is the point center's camera-space depth. - Lines: supply shared
(L, 2)indices or unindexed(..., L, 2, D)endpoints.attributes_domain="line"selects constant per-line attributes; the default interpolates endpoint attributes perspective-correctly. Centerlines are clipped against all six frustum planes, then expanded to flat-capped rectangles with the requested perpendicular width in pixels. Widths need not be integers. Projected zero-length lines are invisible.interpolation_idstores original line IDs, whileinterpolation_uvstores the two original endpoint weights, including after clipping. - Coverage is intentionally not bit-identical to OpenGL lines: the Torch implementation uses geometric rectangles, not driver-specific aliased line/diamond-exit rules. Diagonal edges and endpoints may cover different pixels. There is no antialiasing or automatic joining of adjacent segments.
Positions, cameras and point sizes are detached; only attributes receive gradients. The same batching, dtype, background, and optional-dependency rules apply to all three renderers. Expanded primitives require O(N) geometry storage, not an image-by-primitive tensor.
utils3d.pt.rasterize_triangles_peeling returns a context-managed layer iterator.
It accepts the same geometry, camera, attribute-domain and output options as
triangle rendering, without a graphics context:
with utils3d.pt.rasterize_triangles_peeling(
(256, 256), vertices=vertices, faces=faces, attributes=colors,
return_depth=True, return_interpolation=True,
) as layers:
for i, layer in enumerate(layers):
image = layer["image"] # (..., H, W, C), including the final empty layer.
if i == 3:
breakThe first layer matches ordinary rendering. Each next layer selects the closest
fragment at strictly greater float32 camera-space linear depth per pixel. Equal-depth fragments
are skipped together; face ID breaks ties within a layer. Exhausted pixels stay
empty, and iteration ends after yielding one entirely empty layer across the
batch. Geometry is prepared once, then rerasterized per layer, with one CPU
coverage check per layer. There is no fixed layer limit; leaving the with block
closes the iterator. Outputs can be retained, and attribute losses can be summed
over layers before backward. Do not mutate inputs while iterating.
Migration from the nvdiffrast wrapper: the Torch RastContext,
sample_texture, and texture_composite APIs
have been removed, as have the ctx, antialiasing, and
return_image_derivatives rendering parameters. There is no geometry gradient,
screen-space derivative, mipmap or antialiasing path. Texture sampling can be
composed separately with PyTorch operations. Background image/depth compositing
is supported as described above.
Triangle peeling is now available through the Triton layer iterator above.
NumPy-only functions remain available through utils3d.np and automatic dispatch.
Run the GPU, independent CPU-reference, optional NumPy/OpenGL, attribute-gradient
and lazy-import regression checks with
PYTHONPATH=. python test/test_rasterization.py.
- Camera & Projection & Coordinate Transforms
- Pose Solver
- Image & Maps
- Mesh
- Point Sampling
- Rasterization
- Array Utils
- Segment Array Operations
- IO
| Function | Numpy | Pytorch |
|---|---|---|
utils3d.hierarchical_voxel_samplingSelect approximately a target number of spatially distributed original points. |
utils3d.np.hierarchical_voxel_sampling(points, num_samples) |
utils3d.pt.hierarchical_voxel_sampling(points, num_samples) |
utils3d.voxel_grid_samplingSelect one original point per occupied cubic voxel. |
utils3d.np.voxel_grid_sampling(points, voxel_size) |
utils3d.pt.voxel_grid_sampling(points, voxel_size) |
| Function | Numpy | Pytorch |
|---|---|---|
utils3d.csr_eliminate_zerosRemove zero elements from a sparse CSR tensor. |
- | utils3d.pt.csr_eliminate_zeros(input) |
utils3d.csr_matrix_from_dense_indicesConvert a regular indices array to a sparse CSR adjacency matrix format |
utils3d.np.csr_matrix_from_dense_indices(indices, n_cols) |
utils3d.pt.csr_matrix_from_dense_indices(indices, n_cols) |
utils3d.groupSplit the data into groups based on the provided labels. |
utils3d.np.group(labels, data) |
utils3d.pt.group(labels, data) |
utils3d.index_reducePut values into the input tensor at the specified indices (like index_put), with reduction support. |
- | utils3d.pt.index_reduce(input, indices, values, reduce, include_self) |
utils3d.index_reduce_In-place put values into the input tensor at the specified indices (like index_put_), with reduction support. |
- | utils3d.pt.index_reduce_(input, indices, values, reduce, include_self) |
utils3d.large_multinomial |
- | utils3d.pt.large_multinomial(weights, num_samples, replacement) |
utils3d.lexsortPerform lexicographical sort on multiple keys. Like numpy.lexsort. |
- | utils3d.pt.lexsort(keys, dim) |
utils3d.lookupLook up query in key like a dictionary. Useful for COO indexing. |
utils3d.np.lookup(key, query) |
utils3d.pt.lookup(key, query) |
utils3d.lookup_getDictionary-like get for arrays |
utils3d.np.lookup_get(key, value, get_key, default_value) |
utils3d.pt.lookup_get(key, value, get_key, default_value) |
utils3d.lookup_setDictionary-like set for arrays. |
utils3d.np.lookup_set(key, value, set_key, set_value, append, inplace) |
utils3d.pt.lookup_set(key, value, set_key, set_value, append, inplace) |
utils3d.masked_maxSimilar to torch.max, but with mask |
- | utils3d.pt.masked_max(input, mask, dim, keepdim) |
utils3d.masked_minSimilar to torch.min, but with mask |
- | utils3d.pt.masked_min(input, mask, dim, keepdim) |
utils3d.matrix_traceCompute the trace of a batch of matrices |
- | utils3d.pt.matrix_trace(input, dim1, dim2) |
utils3d.max_pool_2d |
utils3d.np.max_pool_2d(x, kernel_size, stride, padding, axis) |
- |
utils3d.poolingCompute the pooling of the input array. |
utils3d.np.pooling(x, kernel_size, stride, padding, axis, mode) |
- |
utils3d.reverse_permutationCompute the reverse of a permutation array. |
utils3d.np.reverse_permutation(perm, axis) |
utils3d.pt.reverse_permutation(perm, dim) |
utils3d.scatter_argmaxScatter src into input at index along dim with min reduction. Return the indices of the winners in src. |
- | utils3d.pt.scatter_argmax(input, dim, index, src, include_self) |
utils3d.scatter_argminScatter src into input at index along dim with min reduction. Return the indices of the winners in src. |
- | utils3d.pt.scatter_argmin(input, dim, index, src, include_self) |
utils3d.sliding_windowGet a sliding window of the input array. Window axis(axes) will be appended as the last dimension(s). |
utils3d.np.sliding_window(x, window_size, stride, dilation, pad_size, pad_mode, pad_value, axis) |
utils3d.pt.sliding_window(x, window_size, stride, dilation, pad_size, pad_mode, pad_value, dim) |
utils3d.vector_outerCompute the outer product of two arrays. |
utils3d.np.vector_outer(x, y) |
utils3d.pt.vector_outer(x, y) |
| Function | Numpy | Pytorch |
|---|---|---|
utils3d.group_as_segmentsGroup as segments by labels |
utils3d.np.group_as_segments(labels, data, return_inverse, return_group_ids) |
utils3d.pt.group_as_segments(labels, data, return_inverse, return_group_ids) |
utils3d.segment_argmaxCompute the argmax of each segment in the segmented data. |
utils3d.np.segment_argmax(data, offsets, axis) |
utils3d.pt.segment_argmax(data, offsets, dim) |
utils3d.segment_argminCompute the argmin of each segment in the segmented data. |
utils3d.np.segment_argmin(data, offsets, axis) |
utils3d.pt.segment_argmin(data, offsets, dim) |
utils3d.segment_argsortCompute the argsort indices within each segment. |
- | utils3d.pt.segment_argsort(input, offsets, descending, dim) |
utils3d.segment_chainConcatenate segmented arrays in sequence. The number of segments are summed. |
utils3d.np.segment_chain(segments, axis) |
utils3d.pt.segment_chain(segments, axis) |
utils3d.segment_combinationsGenerate all combinations of elements within each segment. Vectorized implementation. |
- | utils3d.pt.segment_combinations(input, offsets, r, with_replacement) |
utils3d.segment_concat(Alias for segment_concatenate). |
utils3d.np.segment_concat(segments, axis) |
utils3d.pt.segment_concat(segments, axis) |
utils3d.segment_concatenateConcatenate segmented arrays within each segment. All numbers of segments remain the same. |
utils3d.np.segment_concatenate(segments, axis) |
utils3d.pt.segment_concatenate(segments, dim) |
utils3d.segment_cumsumCompute the sum of each segment in the segmented data. Workaround supports for dtypes other than float32. |
- | utils3d.pt.segment_cumsum(input, offsets, dim) |
utils3d.segment_medianCompute the median of each segment. |
- | utils3d.pt.segment_median(input, offsets, dim) |
utils3d.segment_multinomialPerform multinomial sampling within each segment. |
- | utils3d.pt.segment_multinomial(weights, offsets, num_samples, eps, replacement) |
utils3d.segment_rollRoll the data within each segment. |
utils3d.np.segment_roll(data, offsets, shift, axis) |
utils3d.pt.segment_roll(data, offsets, shift, dim) |
utils3d.segment_searchsortedPer-segment searchsorted operation implemented with triton. |
- | utils3d.pt.segment_searchsorted(sorted_sequence, offsets, input, segment_ids, side) |
utils3d.segment_sortSort the data within each segment. |
- | utils3d.pt.segment_sort(input, offsets, descending, dim) |
utils3d.segment_sumCompute the sum of each segment in the segmented data. Workaround supports for dtypes other than float32. |
- | utils3d.pt.segment_sum(input, offsets, dim) |
utils3d.segment_takeTake some segments from a segmented array |
utils3d.np.segment_take(data, offsets, taking, axis) |
utils3d.pt.segment_take(data, offsets, taking, dim) |
utils3d.segment_topkSelect the top-k values and indices within each segment. |
- | utils3d.pt.segment_topk(input, offsets, k, largest, dim) |
utils3d.stack_segmentsStack segments into a padded tensor. |
- | utils3d.pt.stack_segments(input, offsets, max_length, padding_value, dim) |
| Function | Numpy | Pytorch |
|---|---|---|
utils3d.read_extrinsics_from_colmapRead extrinsics from colmap images.txt file. |
utils3d.np.read_extrinsics_from_colmap(file) |
- |
utils3d.read_intrinsics_from_colmapRead intrinsics from colmap cameras.txt file. |
utils3d.np.read_intrinsics_from_colmap(file, normalize) |
- |
utils3d.read_objRead wavefront .obj file. |
utils3d.np.read_obj(file, encoding, ignore_unknown) |
- |
utils3d.read_plyRead a PLY file. Supports arbitrary properties, polygonal meshes. Very fast. |
utils3d.np.read_ply(file) |
- |
utils3d.write_extrinsics_as_colmapWrite extrinsics to colmap images.txt file. |
utils3d.np.write_extrinsics_as_colmap(file, extrinsics, image_names, camera_ids) |
- |
utils3d.write_intrinsics_as_colmapWrite intrinsics to colmap cameras.txt file. Currently only support PINHOLE model (no distortion) |
utils3d.np.write_intrinsics_as_colmap(file, intrinsics, width, height, normalized) |
- |
utils3d.write_obj |
utils3d.np.write_obj(file, obj, encoding) |
- |
utils3d.write_plyWrite a PLY file. Supports arbitrary properties, polygonal meshes. |
utils3d.np.write_ply(file, data, format_) |
- |