[Semantic Segmentation] - Add SegFormer Architecture, Weight Conversion Script and Presets #1883

DavidLandup0 · 2024-09-26T13:05:08Z

This PR adds:

SegFormerBackbone and presets
SegFormerImageSegmenter and presets for Cityscapes and ADE20k (B0...B5 each)
Conversion script
Tests

Basic Usage

preprocessor = keras_hub.models.ImageSegmenterPreprocessor.from_preset("segformer_b0_ade20k_512")
segmenter = keras_hub.models.SegFormerImageSegmenter.from_preset("segformer_b0_ade20k_512")
segmenter(np.random.rand(1, 512, 512, 3))

End-to-end example with preprocessor:

import urllib.request 
from PIL import Image 
import numpy as np
import keras_hub

preprocessor = keras_hub.models.ImageSegmenterPreprocessor.from_preset("segformer_b0_ade20k_512")
segmenter = keras_hub.models.SegFormerImageSegmenter.from_preset("segformer_b0_ade20k_512")

img_url = "https://www.vanorohotel.com/wp-content/uploads/2021/07/drz-vanoro_6737.jpg"  
urllib.request.urlretrieve(img_url, "image.png") 
  
img = np.array(Image.open("image.png").resize((512, 512)))
img = np.expand_dims(img, 0)
inputs = preprocessor(img)
outs = segmenter(inputs)

Training Pipeline Example

A few examples in the notebook below:

Instantiation of Backbone and Segmenter with MiT Encoder
Running on input images
Training pipeline with TFDS on Oxford IIIT Pets as an example

https://colab.research.google.com/drive/1EBNg6nPKx_KzyRuQQtHZ_PG_Nsf2pAg2#scrollTo=V9Ub4NHKCx9e

After a few minutes of training from scratch (both encoder and segmenter):

keras_hub/src/models/segformer/segformer_backbone.py

keras_hub/src/models/segformer/segformer_image_segmenter.py

DavidLandup0 · 2024-09-30T00:14:18Z

keras_hub/src/models/segformer/segformer_image_segmenter.py

+    )
+
+    segformer_backbone = keras_hub.models.SegFormerBackbone(backbone=encoder)
+    segformer = keras_hub.models.SegFormerImageSegmenter(backbone=segformer_backbone, num_classes=4)


To be updated to SegFormerImageSegmenter.from_preset() once the config.json files are uploaded to Kaggle

divyashreepathihalli · 2024-10-11T06:41:44Z

Is this PR ready for review?

DavidLandup0 · 2024-10-15T18:58:41Z

@divyashreepathihalli In essence - yes, but there's some noise when running predictions using the converted weights (example image below of huggingface outputs vs our outputs). Looking into the numerics again, but the rest of the PR is ready for review :)

DavidLandup0 · 2024-10-16T04:55:44Z

Found the issue - a transpose call shuffling the order of a latent in the encoder incorrectly. I'll get the presets up on Kaggle now

divyashreepathihalli

Amazing!! the PR looks great. Just a few comments.

divyashreepathihalli · 2024-10-02T20:52:27Z

keras_hub/src/models/segformer/segformer_backbone_tests.py

@@ -0,0 +1,99 @@
+# Licensed under the Apache License, Version 2.0 (the "License");


remove copyright banner

divyashreepathihalli · 2024-10-02T20:52:48Z

keras_hub/src/models/segformer/segformer_image_segmenter.py

+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#


remove copyright

divyashreepathihalli · 2024-10-02T20:54:15Z

keras_hub/src/models/segformer/segformer_image_segmenter.py

+    """
+
+    backbone_cls = SegFormerBackbone
+


please also add the preprocessor flow : follow SAM for example.
this will involve adding an image converter, task preprocessor and then passing the preprocessor here to the task model.

divyashreepathihalli · 2024-10-16T20:18:50Z

keras_hub/src/models/segformer/__init__.py

@@ -0,0 +1,34 @@
+# Licensed under the Apache License, Version 2.0 (the "License");


remove copyright banner

divyashreepathihalli · 2024-10-16T20:19:01Z

keras_hub/src/models/segformer/segformer_backbone.py

@@ -0,0 +1,191 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.


remove copyright banner

divyashreepathihalli · 2024-10-16T20:20:26Z

keras_hub/src/models/segformer/segformer_backbone.py

+
+    """
+
+    backbone_cls = MiTBackbone


divyashreepathihalli · 2024-10-16T20:21:13Z

keras_hub/src/models/segformer/segformer_backbone_presets.py

@@ -0,0 +1,83 @@
+# Licensed under the Apache License, Version 2.0 (the "License");


remove copyright on all files. I will stop commenting this

divyashreepathihalli · 2024-10-16T20:23:31Z

keras_hub/src/models/segformer/segformer_image_segmenter.py

+        self.backbone = backbone
+        self.preprocessor = preprocessor
+        self.dropout = keras.layers.Dropout(0.1)
+        self.output_segmentation = keras.layers.Conv2D(


should we call this the output_segmentation_head

divyashreepathihalli · 2024-10-16T20:24:26Z

keras_hub/src/models/segformer/segformer_image_segmenter_preprocessor.py

+    image_converter_cls = SegFormerImageConverter
+
+    @preprocessing_function
+    def call(self, x, y=None, sample_weight=None):


how about transformations to y - which would be masks in the training set. If the images are resized/transformed, so should the masks

initial commit - tf-based, kcv

716ae64

DavidLandup0 marked this pull request as draft September 26, 2024 13:05

DavidLandup0 and others added 6 commits September 27, 2024 17:35

porting to keras_hub structure - removing aliases, presets, etc.

71bd40b

enable instantiation of segformer backbone with custom MiT backbone

8894a86

remove num_classes from backbone

b66c659

fix input

392ec36

add imports to __init__

d80d8d0

Merge branch 'master' into feature/segformer

538adf7

DavidLandup0 marked this pull request as ready for review September 29, 2024 09:52

DavidLandup0 requested review from mattdangerw, divyashreepathihalli and fchollet September 29, 2024 09:52

DavidLandup0 changed the title ~~[Semantic Segmentation] - SegFormer (and MiTs)~~ [Semantic Segmentation] - SegFormer (MixTransformer-based) Sep 29, 2024

DavidLandup0 added 4 commits September 29, 2024 18:55

update preset

1571677

update docstrings

4b82a16

add basic tests

9b260e7

remove redundant imports

b93954f

DavidLandup0 changed the title ~~[Semantic Segmentation] - SegFormer (MixTransformer-based)~~ [Semantic Segmentation] - Add SegFormer Sep 29, 2024

DavidLandup0 added 7 commits September 29, 2024 19:20

update docstrings

159dca5

remove unused import

3ec02dd

running api_gen.py

7b6286e

undo refactor of mit

c40fdcd

update docstrings

9a13544

add presets for mit

4dc3fff

add standin paths

191656c

DavidLandup0 commented Sep 30, 2024

View reviewed changes

keras_hub/src/models/segformer/segformer_backbone.py Show resolved Hide resolved

DavidLandup0 commented Sep 30, 2024

View reviewed changes

keras_hub/src/models/segformer/segformer_image_segmenter.py Show resolved Hide resolved

DavidLandup0 commented Sep 30, 2024

View reviewed changes

DavidLandup0 added 2 commits September 30, 2024 09:36

add presets for segformer backbone

9e47564

register presets in __init__.py

98bb69d

DavidLandup0 added 11 commits October 8, 2024 18:39

merge master into feature branch

d1bc073

conform to new image classifier style

000d7d0

format

fa34f9e

resizing image converter -> ImageConverter

e3c6dc6

Merge branch 'feature/mit_presets' into feature/segformer

3483c10

merge mit branch into segformer branch

38dadef

add preprocessor and converter

983642c

address comments

2a8ffcb

Merge branch 'feature/mit_presets' into feature/segformer

34e8701

clarify backbone usage

5b5eb93

add conversion script

fcdadb3

DavidLandup0 added 2 commits October 14, 2024 19:58

numerical equivalence changes

68cef65

fix numerical inaccuracies

205ae4a

DavidLandup0 added 4 commits October 16, 2024 03:59

update conversion script

8b5fa44

merge master branch into feature branch

79e05f8

update conversion script

de17c03

remove transpose

04ba1eb

DavidLandup0 added 6 commits October 16, 2024 14:13

add preprocessor to segformer class

9e04b6e

fix preset path

e9e8ed5

update test shape

a7a21f6

update presets

28e1297

update test shape

fc8fffe

expand docstrings

fa89a09

DavidLandup0 requested a review from divyashreepathihalli October 16, 2024 06:31

DavidLandup0 changed the title ~~[Semantic Segmentation] - Add SegFormer Architecture (+ configs for random initialization)~~ [Semantic Segmentation] - Add SegFormer Architecture, Weight Conversion Script and Presets Oct 16, 2024

add rescaling and normalization to preprocessor

6e3d3d1

divyashreepathihalli reviewed Oct 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Semantic Segmentation] - Add SegFormer Architecture, Weight Conversion Script and Presets #1883

[Semantic Segmentation] - Add SegFormer Architecture, Weight Conversion Script and Presets #1883

DavidLandup0 commented Sep 26, 2024 •

edited

Loading

DavidLandup0 Sep 30, 2024

divyashreepathihalli commented Oct 11, 2024

DavidLandup0 commented Oct 15, 2024

DavidLandup0 commented Oct 16, 2024

divyashreepathihalli left a comment

divyashreepathihalli Oct 2, 2024

divyashreepathihalli Oct 2, 2024

divyashreepathihalli Oct 2, 2024

divyashreepathihalli Oct 16, 2024

divyashreepathihalli Oct 16, 2024

divyashreepathihalli Oct 16, 2024

divyashreepathihalli Oct 16, 2024

divyashreepathihalli Oct 16, 2024

divyashreepathihalli Oct 16, 2024

		@@ -0,0 +1,99 @@
		# Licensed under the Apache License, Version 2.0 (the "License");

		@@ -0,0 +1,34 @@
		# Licensed under the Apache License, Version 2.0 (the "License");

		@@ -0,0 +1,191 @@
		# Licensed under the Apache License, Version 2.0 (the "License");
		# you may not use this file except in compliance with the License.

		@@ -0,0 +1,83 @@
		# Licensed under the Apache License, Version 2.0 (the "License");

[Semantic Segmentation] - Add SegFormer Architecture, Weight Conversion Script and Presets #1883

Are you sure you want to change the base?

[Semantic Segmentation] - Add SegFormer Architecture, Weight Conversion Script and Presets #1883

Conversation

DavidLandup0 commented Sep 26, 2024 • edited Loading

Basic Usage

Training Pipeline Example

Choose a reason for hiding this comment

divyashreepathihalli commented Oct 11, 2024

DavidLandup0 commented Oct 15, 2024

DavidLandup0 commented Oct 16, 2024

divyashreepathihalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DavidLandup0 commented Sep 26, 2024 •

edited

Loading