Sample Transform

Download this notebook from GitHub

Preliminaries

Imports

from pathlib import Path

import holoviews as hv
import hvplot.pandas  # noqa
import panel as pn

hv.extension("bokeh")
pn.extension()
# If in google colab, run hack that allows holoviews to work properly
try:
    import google.colab  # noqa

    def _render(self, **kwargs):
        hv.extension("bokeh")
        return hv.Store.render(self)

    hv.core.Dimensioned._repr_mimebundle_ = _render
except ModuleNotFoundError:
    pass

TMP_NOTEBOOK_ROOT = Path("/tmp/bridge-ds/tutorials")

%opts magic unavailable (pyparsing cannot be imported)
%compositor magic unavailable (pyparsing cannot be imported)

Loading the dataset

from bridge.providers.vision import Coco2017Detection

root_dir = TMP_NOTEBOOK_ROOT / "coco"

provider = Coco2017Detection(root_dir)
ds = provider.build_dataset()
ds

Annotations file /tmp/bridge-ds/tutorials/coco/annotations/instances_train2017.json already exists, skipping download.
loading annotations into memory...
Done (t=13.51s)
creating index...
index created!

Dataset: {'n_samples': 118287, 'n_bbox': 860001, 'n_image': 118287}

SampleTransforms

Manipulating data in Bridge is done through SampleTransforms. If you recall, data is stored in Bridge Elements rather than Samples, but in many cases we want to transform all Elements in a Sample together (for example, crop an image and remove all bboxes outside of the crop).

Bridge utilizes SampleTransforms in two contexts:

• new_sample = sample.transform(sample_transform)

• new_ds = ds.transform_samples(sample_transform) - iterate over all samples and transform each one.

An Example

We will use TorchvisionV2Transform, a subclass of SampleTransform to take a sample from our dataset and flip it:

import holoviews as hv
from torchvision.transforms import v2

from bridge.primitives.sample.transform.vision import TorchvisionV2Transform


def flip_sample(sample):
    cmp = v2.Compose([v2.RandomHorizontalFlip(p=1)])
    transform = TorchvisionV2Transform(cmp)
    flipped_sample = sample.transform(transform)
    return flipped_sample


sample = ds.iget(1)
flipped = flip_sample(sample)

opts = dict(frame_width=300)
hv.Layout([sample.show(sample_plot_kwargs=opts), flipped.show(sample_plot_kwargs=opts)])

We would like to apply this transformation to the entire dataset; how can we do this?

At first glance, this may seem rather straightforward: Perform sample.transform() iteratively (i.e. call ds.transform_data()) over the entire dataset, and we’ve successfully transformed our dataset.

Well… not exactly. Observe the following:

print("Original Sample:")
print(sample._element._load_mechanism.url_or_data)
print()
print("Flipped Sample:")
print(flipped._element._load_mechanism.url_or_data)

Original Sample:
http://images.cocodataset.org/train2017/000000000025.jpg

Flipped Sample:
[[[ 71  69  80]
  [ 58  57  65]
  [ 59  60  64]
  ...
  [  7   8   3]
  [  8   9   4]
  [  8   9   4]]

 [[150 149 157]
  [148 147 153]
  [151 150 155]
  ...
  [  7   8   3]
  [  8   9   4]
  [  8   9   4]]

 [[172 171 176]
  [180 179 184]
  [187 187 189]
  ...
  [  7   8   3]
  [  7   8   3]
  [  8   9   4]]

 ...

 [[133 100  69]
  [140 106  78]
  [154 120  93]
  ...
  [148 132  44]
  [174 166  68]
  [191 189  86]]

 [[139 109  83]
  [145 115  91]
  [126  96  72]
  ...
  [137 134  39]
  [154 159  69]
  [192 186  92]]

 [[139 109  83]
  [145 115  91]
  [127  97  73]
  ...
  [120 117  24]
  [119 123  36]
  [136 129  38]]]

See how url_or_data has changed for the flipped sample? Consider that the LoadMechanism for the original sample is just configured to just load an image from a URL; when we apply an augmentation, our new image is not the same as the source. To keep the new image we need to store it somewhere, detached from the original source.

The default implementation for sample.transform() is to save the new data to RAM, but keeping the entire transformed dataset in RAM cannot scale.

To understand how we can solve this issue, and how did the url_or_data property change to begin with, proceed to the next tutorial where we talk about CacheMechanisms.