Generating¶

Save a datapoint per call: the render, passes, AOVs, masks, preview images and a JSON label, <index>.json.

The steps can be listed in any order; they always run as: label steps (BBox, RotationMatrix, OutputField, CameraData, Keypoints), so a datapoint skipped by BBox (iou_deconflict, max_truncation, max_occlusion) is never rendered, then Render, then AOVToImage, Passes and BBoxImage, then Segmentation, then SegmentationImage. All steps except Segmentation share one render; Segmentation reuses the id render of BBox(max_occlusion=...) when both have the same classes.

BBox and Segmentation take classes as {class name: [instances]}, where an instance is an object, or a sublist of objects labeled as one: {"car": [car_1, car_2], "table": [[table_top, table_legs]]}. A class can also be a dict with its own BBox skip settings, {"instances": [...], "max_truncation": 0.5}, which win over the BBox arguments; Segmentation ignores them, so both steps can share one classes dict.

Compose ¶

Compose(
    steps: Sequence[Any],
    path: str,
    resolution: tuple[int, int],
)

Generates one datapoint per call from a list of steps.

Every call saves <index>.png, <index>.json with the labels, and the files of the other steps to path. The index is the next free number in the folder, so generating can be stopped and resumed. The settings of every step are checked before anything renders, and the scene's resolution and output settings are restored afterwards.

Parameters:

steps (Sequence[Any]) –

generating steps, in any order.
path (str) –

folder to save the files to. // paths are relative to the .blend file, and "" is its folder.
resolution (tuple[int, int]) –

(width, height) of the images in pixels.

Raises:

ValueError –

a BBoxImage without a BBox step, or a SegmentationImage without a Segmentation step.

Example

classes = {"car": [car_1, car_2], "table": [[table_top, table_legs]]}
generator = generating.Compose(
    [generating.Render(), generating.BBox(classes), generating.Segmentation(classes)],
    path="//dataset",
    resolution=(640, 480),
)
generator()

preview ¶

preview(
    scaling_factor: float,
    custom_dict: Optional[dict[str, Any]] = None,
) -> bool

Generates one datapoint at a reduced resolution, for a quick look.

Parameters:

scaling_factor (float) –

the resolution is divided by this factor.
custom_dict (Optional[dict[str, Any]], default: None ) –

extra keys and values to save in the label.

Returns:

bool –

False if the datapoint was skipped, otherwise True.

BBox ¶

BBox(
    classes: Classes,
    iou_deconflict: Optional[float] = None,
    max_truncation: Optional[float] = None,
    max_occlusion: Optional[float] = None,
)

Adds the 2D bounding box of every instance to the label.

Boxes are in pixels as [x_min, y_min, x_max, y_max] from the top-left corner, computed from the evaluated geometry (modifiers included). Occlusion is not taken into account, so hidden parts are inside the box. An instance out of frame gets None.

In the label JSON: "bboxes": [{"class", "objects", "bbox"}].

Parameters:

classes (Classes) –

{class name: [instances]}, an instance is an object or a sublist of objects labeled as one (its box is the union of the members). A class can instead be {"instances": [instances], "iou_deconflict": ..., "max_truncation": ..., "max_occlusion": ...} to set its own skip settings (all keys but instances optional).
iou_deconflict (Optional[float], default: None ) –

skip the datapoint, before rendering, when any two boxes overlap more than this IoU. None = never skip.
max_truncation (Optional[float], default: None ) –

skip the datapoint, before rendering, when more than this fraction (0-1) of any instance's box is out of frame, measured on its unclipped box. An instance partly behind the camera or fully out of frame counts as 1, so max_truncation=0 keeps only datapoints with every instance fully in frame. None = never skip.
max_occlusion (Optional[float], default: None ) –

skip the datapoint, before the beauty render, when objects in no class (clutter, distractors) cover more than this fraction (0-1) of any instance: 1 - its visible pixels / its pixels with those objects hidden. Class objects covering each other don't count (that is iou_deconflict), and an instance with no pixels even then (out of frame, or behind another class object) counts as 0. None = never skip.

Note

The BBox arguments are the defaults for every class. A setting a class sets in its dict wins over them, even when it is None (no limit for that class). A class's max_truncation and max_occlusion apply to each of its instances. max_occlusion is measured with two flat Workbench renders, like Segmentation, which only run when some class has a limit and the cheaper checks passed; Workbench draws every object solid, so transparent objects occlude fully and shadows don't count. For iou_deconflict, two boxes conflict when their IoU is over the lower of their two classes' limits, so a strict class can't overlap anything, and a pair is only free when both classes have no limit.

Example

classes = {
    "car": [car_1, car_2],                                       # uses the BBox arguments
    "table": {"instances": [[top, legs]], "max_truncation": 0.8},  # may be cut more
}
generating.BBox(classes, iou_deconflict=0.5, max_truncation=0.3, max_occlusion=0.5)

RotationMatrix ¶

RotationMatrix(objects: Sequence[Object])

Adds the rotation of every object relative to the camera to the label, as a 3x3 matrix with the scale removed.

In the label JSON: "rotation_matrices": [{"object", "rotation_matrix"}].

Parameters:

objects (Sequence[Object]) –

objects to save the rotation of.

Example

generating.RotationMatrix([car_1, car_2])

OutputField ¶

OutputField(
    name: str,
    data_path: str,
    objects: Optional[Sequence[Object]] = None,
)

Saves the value at a data path to the label.

Values are converted to JSON: datablocks become their name, menus the option name, and vectors and matrices nested lists.

In the label JSON: name: value for an absolute path, name: {object name: value} for a relative one.

Parameters:

name (str) –

key in the label.
data_path (str) –

path to the value. Paths starting with bpy. are absolute (right click > Copy Full Data Path) and saved once, e.g. 'bpy.data.lights["Light"].energy'. Other paths are relative to objects.
objects (Optional[Sequence[Object]], default: None ) –

objects a relative path is resolved on.

Example

generating.OutputField("light_energy", 'bpy.data.lights["Light"].energy')
generating.OutputField("smile", 'data.shape_keys.key_blocks["Smile"].value', objects=[face])

CameraData ¶

Adds the active camera to the label.

Saves the name, type, matrix_world, extrinsics_opencv (3x4 world-to-camera [R|t] with OpenCV axes: x right, y down, z forward), clip_start, clip_end and depth_of_field (use_dof, focus_object, focus_distance, f_stop, aperture_blades, aperture_rotation in radians and aperture_ratio). focus_distance is the distance to the focal plane Blender uses, so with a focus object it is the distance to that object along the view axis. A perspective camera adds the lens, sensor size and fit and intrinsics (3x3 K in pixels, top-left image origin, with sensor fit, lens shift and pixel aspect), an orthographic one ortho_scale.

In the label JSON: "camera": {...}.

Example

generating.CameraData()

Keypoints ¶

Keypoints(points: dict[str, KeypointSource])

Projects 3D points to the image and adds them to the label.

Vertices include deformations from armatures and modifiers. A point is visible when it is in frame and a ray cast from the camera reaches it.

In the label JSON: "keypoints": [{"name", "position", "depth", "in_frame", "visible"}], with position in pixels from the top-left corner and depth along the view axis.

Parameters:

points (dict[str, KeypointSource]) –

{name: source}, where a source is an object (its origin), (mesh object, vertex index), (mesh object, "vertex group") for the center of the group, (armature, "bone") for the bone head, or a point (x, y, z).

Example

generating.Keypoints({"car_origin": car_1, "car_corner": (car_1, 0), "hand": (rig, "hand.L")})

Render ¶

Render(
    file_format: Literal["PNG", "JPEG", "OPEN_EXR"] = "PNG",
)

Renders the image with the scene's engine to <index>.<ext>.

In the label JSON: "image": file name.

Parameters:

file_format (Literal['PNG', 'JPEG', 'OPEN_EXR'], default: 'PNG' ) –

"PNG", "JPEG" or "OPEN_EXR".

Example

generating.Render("JPEG")

AOVToImage ¶

AOVToImage(
    names: Sequence[str],
    file_format: Literal["OPEN_EXR", "PNG"] = "OPEN_EXR",
    skip_empty: bool = False,
)

Saves shader AOVs as images <index>_<aov>.<ext>.

Uses the same render as Render; without Render, the scene is still rendered once, but no image is saved. Each AOV must be added in View Layer Properties > Passes > Shader AOV, and the engine must be Cycles or EEVEE.

In the label JSON: "aovs": {name: file name}, None for an AOV skipped by skip_empty.

Parameters:

names (Sequence[str]) –

names of the AOVs.
file_format (Literal['OPEN_EXR', 'PNG'], default: 'OPEN_EXR' ) –

"OPEN_EXR" (32-bit float) or "PNG" (8-bit, clamped to 0-1).
skip_empty (bool, default: False ) –

don't write an AOV that is empty, every pixel 0 (black) in the render, alpha ignored; its file name in the label is None. The datapoint is still generated.

Example

generating.AOVToImage(["Albedo"])

Passes ¶

Passes(
    names: Sequence[str],
    file_format: Literal["OPEN_EXR", "PNG"] = "OPEN_EXR",
    skip_empty: bool = False,
)

Saves built-in render passes as images <index>_<pass>.<ext>.

Uses the same render as Render and AOVToImage. Each pass is enabled in the view layer for that render only. Cycles has all passes, EEVEE no UV or index passes, and Workbench only Depth; in Cycles, Vector also needs motion blur off.

In the label JSON: "passes": {name: file name}, None for a pass skipped by skip_empty.

Parameters:

names (Sequence[str]) –

any of "Depth", "Mist", "Normal", "Position", "Vector", "UV", "ObjectIndex" and "MaterialIndex".
file_format (Literal['OPEN_EXR', 'PNG'], default: 'OPEN_EXR' ) –

"OPEN_EXR" (32-bit float) or "PNG" (8-bit, clamped to 0-1, so only useful for some passes like Normal).
skip_empty (bool, default: False ) –

don't write a pass that is empty, every pixel 0 (black) in the render, alpha ignored; its file name in the label is None. The datapoint is still generated. Depth is never empty, its background is far away.

Example

generating.Passes(["Depth", "Normal"])

BBoxImage ¶

BBoxImage(
    file_format: Literal["PNG", "JPEG"] = "PNG",
    line_width: int = 2,
    show_class: bool = True,
)

Saves a copy of the rendered image with the boxes of the BBox step drawn on it, to <index>_bboxes.<ext>, for checking the labels.

Uses the same render as Render, and the main image stays clean. Each class has its own color, and its name is written above each box in capitals. Needs a BBox step in the same Compose.

In the label JSON: "bbox_image": file name.

Parameters:

file_format (Literal['PNG', 'JPEG'], default: 'PNG' ) –

"PNG" or "JPEG".
line_width (int, default: 2 ) –

outline width in pixels, at least 1.
show_class (bool, default: True ) –

write the class name above each box.

Example

generating.BBoxImage(line_width=3)

Segmentation ¶

Segmentation(
    classes: Classes,
    per: Literal["instance", "class", "both"] = "instance",
    skip_empty: bool = False,
)

Saves black-and-white masks of the visible pixels of every instance or class.

Uses one extra, fast Workbench render whatever the engine. Objects that are not in classes still hide what is behind them. Files are <index>_mask_<n>.png per instance, where n counts instances across all classes, and <index>_mask_<class>.png per class.

In the label JSON: "masks": [{"class", "objects", "mask", "per"}], with per "instance" or "class", and mask None for a mask skipped by skip_empty.

Parameters:

classes (Classes) –

{class name: [instances]}, an instance is an object or a sublist of objects labeled as one (one mask). A class given as a dict with "instances" (see BBox) works too; its skip settings are ignored here.
per (Literal['instance', 'class', 'both'], default: 'instance' ) –

"instance", "class" or "both".
skip_empty (bool, default: False ) –

don't write empty masks (an instance or class with no visible pixel: out of frame or fully hidden); its mask in the label is None. The datapoint is still generated, and export skips these instances like empty masks.

Example

generating.Segmentation({"car": [car_1, car_2]}, per="both", skip_empty=True)

SegmentationImage ¶

SegmentationImage(
    file_format: Literal["PNG", "JPEG"] = "PNG",
    opacity: float = 0.5,
    line_width: int = 2,
    show_class: bool = True,
)

Saves a copy of the rendered image with the masks of the Segmentation step drawn over it, to <index>_segmentation.<ext>, for checking the labels.

Uses the same render as Render, and the main image stays clean. Each mask is laid over in its class color (the same as in BBoxImage), each instance outlined, and its class name written above. Instance masks are used when there are any, otherwise the class masks. Needs a Segmentation step in the same Compose.

In the label JSON: "segmentation_image": file name.

Parameters:

file_format (Literal['PNG', 'JPEG'], default: 'PNG' ) –

"PNG" or "JPEG".
opacity (float, default: 0.5 ) –

how much the class color covers the image, 0-1.
line_width (int, default: 2 ) –

outline width in pixels, 0 = no outline.
show_class (bool, default: True ) –

write the class name above each mask.

Example

generating.SegmentationImage(opacity=0.4)