bugs in object detection kpl

[innat]

There are many issues are reported regarding object detection specific layers and model, few tickets are opened or few closed without fix. Here is list of some of those bugs. I will make gist for specific issue later.

🐛 1. mixed precisoin #1784
There are many KPL still remains that doesn't work for mixed precision. Compare to keras-cv, the keras-aug does work, which builts on keras-cv - weird!. Further, the output dtype of YOLO-v8 need to be set explicitly as float32 for mixed precision, which is absent currently.

🐛 2. random apply layer #1783

consider the following code

preprocessor = keras.Sequential(layers.Resizing(_, _)])
augmenter = keras.Sequential(
    layers=[
        preprocessor,
        
        # option 1
        keras_cv.layers.RandomFlip(mode="horizontal")

        # option 2
        keras_cv.layers.RandomApply(
            keras_cv.layers.RandomFlip(mode="horizontal"),
            rate=0.7
        ),
    ]
)

BATCH_SIZE = 8
eval_ds = load_pascal_voc(split="test", dataset="voc/2007",)
eval_ds = eval_ds.shuffle(BATCH_SIZE * 4)
eval_ds = eval_ds.ragged_batch(BATCH_SIZE, drop_remainder=True)
eval_ds_ = eval_ds.map(
    augmenter, num_parallel_calls=tf.data.AUTOTUNE
)

With above setup, option 1 works but option 2 gives following error.

ValueError: Either both boxes and classes should be Ragged, or neither should be ragged. Got type(boxes)=<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>, type(classes)=<class 'tensorflow.python.framework.ops.SymbolicTensor'>.

🐛 3. random chioce layer #1782

consider the following code

preprocessor = keras.Sequential(layers.Resizing(_, _)])
augmenter = keras.Sequential(
    layers=[
        preprocessor,
        
        # option 1
        keras_cv.layers.MixUp(alpha=0.4),
        keras_cv.layers.Mosaic(bounding_box_format="xywh")

        # option 2
        keras_cv.layers.RandomChoice(
            layers=[
                keras_cv.layers.Mosaic(bounding_box_format="xywh"),
                keras_cv.layers.MixUp(alpha=0.4)
            ]
        )
    ]
)

BATCH_SIZE = 8
eval_ds = load_pascal_voc(split="test", dataset="voc/2007",)
eval_ds = eval_ds.shuffle(BATCH_SIZE * 4)
eval_ds = eval_ds.ragged_batch(BATCH_SIZE, drop_remainder=True)
eval_ds_ = eval_ds.map(
    augmenter, num_parallel_calls=tf.data.AUTOTUNE
)

With above setup, option 1 works but option 2 gives following error.

ValueError: Mosaic received a single image to call. The layer relies on combining multiple examples, and as such will not behave as expected. Please call the layer with 4 or more samples.

🐛 4. mAP gives -1 #2212

It is probably because the model is not learning and need better training strategies but loading pre trained weight, it seems unlikely the weight is damaged for custom training. It is reported many times. I've also noticed in my case. More ticket on this: #2095 (comment), This probably need some investigation.

🐛 5. yolo-v8 optimization strategies: (a). what are the main differences between official yolo-v8 anad keras version? (what are the further to-do list? cc. @ianstenbit ) #2333

How the pretrained weights are achieved (ported or trained)? Is the validation score on COCO reproduced with it? If so, Is the evaluation scripts available?

🐛 6. KPL order:

# option 1
keras_cv.layers.JitteredResize(
    target_size=(input_shape, input_shape), 
    scale_factor=(0.75, 1.6),
    bounding_box_format=bbox_format,
)
keras_cv.layers.Mosaic(bounding_box_format=bbox_format),

# option 2
keras_cv.layers.Mosaic(bounding_box_format=bbox_format),
keras_cv.layers.JitteredResize(
    target_size=(input_shape, input_shape), 
    scale_factor=(0.75, 1.6),
    bounding_box_format=bbox_format,
)

This is one example, as shown above, option 1 works but option 2 doesn't.

🐛 7. Potential memory leak

I've found that even using yolov8 _s_backbone_coco and with basic augmentation setup, many times the system memory gets overflowing. Additionally, few augmentation, like Random Color Jitter causes terrible slow and makes training time abnormally large.

🐛 8. Callback

Consider the following setup

coco_metrics_callback = keras_cv.callbacks.PyCOCOCallback(
    test_dataloader
)

history = model.fit(
    train_dataloader,
    validation_dataloader
    epochs=20,
    callbacks=[coco_metrics_callback],
)

The callback gets the test set, and fit gets the training and validation set. But in the log, the callback will log the test set output as val_ key. Probably, this arugment should be renamed generally.

🐛 9. Scope of adding augmentation layer with mdoel instead of in the data loader. When the model is small, it would be effective if augmentation layers is added to the model - same as other image-net models. But currently with the given set up, it looks unlikely to do that with object det models.