Skip to content

ReXNet

The ResNet model is based on the "ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network" paper.

Architecture overview

This paper investigates the effect of channel configuration in convolutional bottlenecks.

The key takeaways from the paper are the following:

  • increasing the depth ratio of conv 1x1 and inverted bottlenecks
  • replace ReLU6 with SiLU

Model builders

The following model builders can be used to instantiate a ReXNet model, with or without pre-trained weights. All the model builders internally rely on the ReXNet base class.

ReXNet 

ReXNet(width_mult: float = 1.0, depth_mult: float = 1.0, num_classes: int = 1000, in_channels: int = 3, in_planes: int = 16, final_planes: int = 180, use_se: bool = True, se_ratio: int = 12, dropout_ratio: float = 0.2, bn_momentum: float = 0.9, act_layer: Module | None = None, norm_layer: Callable[[int], Module] | None = None, drop_layer: Callable[..., Module] | None = None)

Bases: Sequential

Mostly adapted from https://github.com/clovaai/rexnet/blob/master/rexnetv1.py

Source code in holocron/models/classification/rexnet.py 
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
def __init__(
    self,
    width_mult: float = 1.0,
    depth_mult: float = 1.0,
    num_classes: int = 1000,
    in_channels: int = 3,
    in_planes: int = 16,
    final_planes: int = 180,
    use_se: bool = True,
    se_ratio: int = 12,
    dropout_ratio: float = 0.2,
    bn_momentum: float = 0.9,
    act_layer: nn.Module | None = None,
    norm_layer: Callable[[int], nn.Module] | None = None,
    drop_layer: Callable[..., nn.Module] | None = None,
) -> None:
    """Mostly adapted from https://github.com/clovaai/rexnet/blob/master/rexnetv1.py"""
    super().__init__()

    if act_layer is None:
        act_layer = nn.SiLU(inplace=True)
    if norm_layer is None:
        norm_layer = nn.BatchNorm2d

    num_blocks = [1, 2, 2, 3, 3, 5]
    strides = [1, 2, 2, 2, 1, 2]
    num_blocks = [ceil(element * depth_mult) for element in num_blocks]
    strides = functools.reduce(
        operator.iadd, [[element] + [1] * (num_blocks[idx] - 1) for idx, element in enumerate(strides)], []
    )
    depth = sum(num_blocks)

    stem_channel = 32 / width_mult if width_mult < 1.0 else 32
    inplanes = in_planes / width_mult if width_mult < 1.0 else in_planes

    # The following channel configuration is a simple instance to make each layer become an expand layer
    chans = [round(width_mult * stem_channel)]
    chans.extend([round(width_mult * (inplanes + idx * final_planes / depth)) for idx in range(depth)])

    ses = [False] * (num_blocks[0] + num_blocks[1]) + [use_se] * sum(num_blocks[2:])

    layers = conv_sequence(
        in_channels,
        chans[0],
        act_layer,
        norm_layer,
        drop_layer,
        kernel_size=3,
        stride=2,
        padding=1,
        bias=(norm_layer is None),
    )

    t = 1
    for in_c, c, s, se in zip(chans[:-1], chans[1:], strides, ses, strict=True):
        layers.append(ReXBlock(in_channels=in_c, channels=c, t=t, stride=s, use_se=se, se_ratio=se_ratio))
        t = 6

    pen_channels = int(width_mult * 1280)
    layers.extend(
        conv_sequence(
            chans[-1],
            pen_channels,
            act_layer,
            norm_layer,
            drop_layer,
            kernel_size=1,
            stride=1,
            padding=0,
            bias=(norm_layer is None),
        )
    )

    super().__init__(
        OrderedDict([
            ("features", nn.Sequential(*layers)),
            ("pool", GlobalAvgPool2d(flatten=True)),
            ("head", nn.Sequential(nn.Dropout(dropout_ratio), nn.Linear(pen_channels, num_classes))),
        ])
    )

    # Init all layers
    init.init_module(self, nonlinearity="relu")

rexnet1_0x 

rexnet1_0x(pretrained: bool = False, checkpoint: Checkpoint | None = None, progress: bool = True, **kwargs: Any) -> ReXNet

ReXNet-1.0x from "ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network"

PARAMETER DESCRIPTION
pretrained

If True, returns a model pre-trained on ImageNette

TYPE: bool DEFAULT: False

checkpoint

If specified, the model's parameters will be set to the checkpoint's values

TYPE: Checkpoint | None DEFAULT: None

progress

If True, displays a progress bar of the download to stderr

TYPE: bool DEFAULT: True

kwargs

keyword args of ReXNet

TYPE: Any DEFAULT: {}

RETURNS DESCRIPTION
ReXNet

classification model

ReXNet1_0x_Checkpoint

Bases: Enum

IMAGENET1K class-attribute instance-attribute 
IMAGENET1K = _checkpoint(arch='rexnet1_0x', url='https://github.com/frgfm/Holocron/releases/download/v0.1.2/rexnet1_0x_224-ab7b9733.pth', dataset=IMAGENET1K, acc1=0.7786, acc5=0.9387, sha256='ab7b973341a59832099f6ee2a41eb51121b287ad4adaae8b2cd8dd92ef058f01', size=14351299, num_params=4796186)
IMAGENETTE class-attribute instance-attribute 
IMAGENETTE = _checkpoint(arch='rexnet1_0x', url='https://github.com/frgfm/Holocron/releases/download/v0.2.1/rexnet1_0x_224-7c19fd53.pth', acc1=0.9439, acc5=0.9962, sha256='7c19fd53a5433927e9b4b22fa9cb0833eb1e4c3254b4079b6818fce650a77943', size=14351299, num_params=3527996, commit='d4a59999179b42fc0d3058ac6b76cc41f49dd56e', train_args='./imagenette2-320/ --arch rexnet1_0x --batch-size 64 --mixup-alpha 0.2 --amp --device 0 --epochs 100 --lr 1e-3 --label-smoothing 0.1 --random-erase 0.1 --train-crop-size 176 --val-resize-size 232 --opt adamw --weight-decay 5e-2')
DEFAULT class-attribute instance-attribute 
DEFAULT = IMAGENET1K
Source code in holocron/models/classification/rexnet.py 
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
def rexnet1_0x(
    pretrained: bool = False,
    checkpoint: Checkpoint | None = None,
    progress: bool = True,
    **kwargs: Any,
) -> ReXNet:
    """ReXNet-1.0x from
    ["ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network"](https://arxiv.org/pdf/2007.00992.pdf)

    Args:
        pretrained: If True, returns a model pre-trained on ImageNette
        checkpoint: If specified, the model's parameters will be set to the checkpoint's values
        progress: If True, displays a progress bar of the download to stderr
        kwargs: keyword args of [`ReXNet`][holocron.models.classification.rexnet.ReXNet]

    Returns:
        classification model

    ::: holocron.models.ReXNet1_0x_Checkpoint
        options:
            heading_level: 4
            show_if_no_docstring: true
    """
    checkpoint = _handle_legacy_pretrained(
        pretrained,
        checkpoint,
        ReXNet1_0x_Checkpoint.DEFAULT.value,
    )
    return _rexnet(checkpoint, progress, 1, 1, **kwargs)

rexnet1_3x 

rexnet1_3x(pretrained: bool = False, checkpoint: Checkpoint | None = None, progress: bool = True, **kwargs: Any) -> ReXNet

ReXNet-1.3x from "ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network"

PARAMETER DESCRIPTION
pretrained

If True, returns a model pre-trained on ImageNet

TYPE: bool DEFAULT: False

checkpoint

If specified, the model's parameters will be set to the checkpoint's values

TYPE: Checkpoint | None DEFAULT: None

progress

If True, displays a progress bar of the download to stderr

TYPE: bool DEFAULT: True

kwargs

keyword args of ReXNet

TYPE: Any DEFAULT: {}

RETURNS DESCRIPTION
ReXNet

classification model

ReXNet1_3x_Checkpoint

Bases: Enum

IMAGENET1K class-attribute instance-attribute 
IMAGENET1K = _checkpoint(arch='rexnet1_3x', url='https://github.com/frgfm/Holocron/releases/download/v0.1.2/rexnet1_3x_224-95479104.pth', dataset=IMAGENET1K, acc1=0.795, acc5=0.9468, sha256='95479104024ce294abbdd528df62bd1a23e67a9db2956e1d6cdb9a9759dc1c69', size=14351299, num_params=7556198)
IMAGENETTE class-attribute instance-attribute 
IMAGENETTE = _checkpoint(arch='rexnet1_3x', url='https://github.com/frgfm/Holocron/releases/download/v0.2.1/rexnet1_3x_224-cf85ae91.pth', acc1=0.9488, acc5=0.9939, sha256='cf85ae919cbc9484f9fa150106451f68d2e84c73f1927a1b80aeeaa243ccd65b', size=23920480, num_params=5907848, commit='d4a59999179b42fc0d3058ac6b76cc41f49dd56e', train_args='./imagenette2-320/ --arch rexnet1_3x --batch-size 64 --mixup-alpha 0.2 --amp --device 0 --epochs 100 --lr 1e-3 --label-smoothing 0.1 --random-erase 0.1 --train-crop-size 176 --val-resize-size 232 --opt adamw --weight-decay 5e-2')
DEFAULT class-attribute instance-attribute 
DEFAULT = IMAGENET1K
Source code in holocron/models/classification/rexnet.py 
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
def rexnet1_3x(
    pretrained: bool = False,
    checkpoint: Checkpoint | None = None,
    progress: bool = True,
    **kwargs: Any,
) -> ReXNet:
    """ReXNet-1.3x from
    ["ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network"](https://arxiv.org/pdf/2007.00992.pdf)

    Args:
        pretrained: If True, returns a model pre-trained on ImageNet
        checkpoint: If specified, the model's parameters will be set to the checkpoint's values
        progress: If True, displays a progress bar of the download to stderr
        kwargs: keyword args of [`ReXNet`][holocron.models.classification.rexnet.ReXNet]

    Returns:
        classification model

    ::: holocron.models.ReXNet1_3x_Checkpoint
        options:
            heading_level: 4
            show_if_no_docstring: true
    """
    checkpoint = _handle_legacy_pretrained(
        pretrained,
        checkpoint,
        ReXNet1_3x_Checkpoint.DEFAULT.value,
    )
    return _rexnet(checkpoint, progress, 1.3, 1, **kwargs)

rexnet1_5x 

rexnet1_5x(pretrained: bool = False, checkpoint: Checkpoint | None = None, progress: bool = True, **kwargs: Any) -> ReXNet

ReXNet-1.5x from "ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network"

PARAMETER DESCRIPTION
pretrained

If True, returns a model pre-trained on ImageNet

TYPE: bool DEFAULT: False

checkpoint

If specified, the model's parameters will be set to the checkpoint's values

TYPE: Checkpoint | None DEFAULT: None

progress

If True, displays a progress bar of the download to stderr

TYPE: bool DEFAULT: True

kwargs

keyword args of ReXNet

TYPE: Any DEFAULT: {}

RETURNS DESCRIPTION
ReXNet

classification model

ReXNet1_5x_Checkpoint

Bases: Enum

IMAGENET1K class-attribute instance-attribute 
IMAGENET1K = _checkpoint(arch='rexnet1_5x', url='https://github.com/frgfm/Holocron/releases/download/v0.1.2/rexnet1_5x_224-c42a16ac.pth', dataset=IMAGENET1K, acc1=0.8031, acc5=0.9517, sha256='c42a16ac73470d64852b8317ba9e875c833595a90a086b90490a696db9bb6a96', size=14351299, num_params=9727562)
IMAGENETTE class-attribute instance-attribute 
IMAGENETTE = _checkpoint(arch='rexnet1_5x', url='https://github.com/frgfm/Holocron/releases/download/v0.2.1/rexnet1_5x_224-4b9d7a59.pth', acc1=0.9447, acc5=0.9962, sha256='4b9d7a5901da6c2b9386987a6120bc86089d84df7727e43b78a4dfe2fc1c719a', size=31625286, num_params=7825772, commit='d4a59999179b42fc0d3058ac6b76cc41f49dd56e', train_args='./imagenette2-320/ --arch rexnet1_5x --batch-size 64 --mixup-alpha 0.2 --amp --device 0 --epochs 100 --lr 1e-3 --label-smoothing 0.1 --random-erase 0.1 --train-crop-size 176 --val-resize-size 232 --opt adamw --weight-decay 5e-2')
DEFAULT class-attribute instance-attribute 
DEFAULT = IMAGENET1K
Source code in holocron/models/classification/rexnet.py 
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
def rexnet1_5x(
    pretrained: bool = False,
    checkpoint: Checkpoint | None = None,
    progress: bool = True,
    **kwargs: Any,
) -> ReXNet:
    """ReXNet-1.5x from
    ["ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network"](https://arxiv.org/pdf/2007.00992.pdf)

    Args:
        pretrained: If True, returns a model pre-trained on ImageNet
        checkpoint: If specified, the model's parameters will be set to the checkpoint's values
        progress: If True, displays a progress bar of the download to stderr
        kwargs: keyword args of [`ReXNet`][holocron.models.classification.rexnet.ReXNet]

    Returns:
        classification model

    ::: holocron.models.ReXNet1_5x_Checkpoint
        options:
            heading_level: 4
            show_if_no_docstring: true
    """
    checkpoint = _handle_legacy_pretrained(
        pretrained,
        checkpoint,
        ReXNet1_5x_Checkpoint.DEFAULT.value,
    )
    return _rexnet(checkpoint, progress, 1.5, 1, **kwargs)

rexnet2_0x 

rexnet2_0x(pretrained: bool = False, checkpoint: Checkpoint | None = None, progress: bool = True, **kwargs: Any) -> ReXNet

ReXNet-2.0x from "ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network"

PARAMETER DESCRIPTION
pretrained

If True, returns a model pre-trained on ImageNet

TYPE: bool DEFAULT: False

checkpoint

If specified, the model's parameters will be set to the checkpoint's values

TYPE: Checkpoint | None DEFAULT: None

progress

If True, displays a progress bar of the download to stderr

TYPE: bool DEFAULT: True

kwargs

keyword args of ReXNet

TYPE: Any DEFAULT: {}

RETURNS DESCRIPTION
ReXNet

classification model

ReXNet2_0x_Checkpoint

Bases: Enum

IMAGENET1K class-attribute instance-attribute 
IMAGENET1K = _checkpoint(arch='rexnet2_0x', url='https://github.com/frgfm/Holocron/releases/download/v0.1.2/rexnet2_0x_224-c8802402.pth', dataset=IMAGENET1K, acc1=0.8031, acc5=0.9517, sha256='c8802402442551c77fe3874f84d4d7eb1bd67cce274375db11a869ed074a1089', size=14351299, num_params=16365244)
IMAGENETTE class-attribute instance-attribute 
IMAGENETTE = _checkpoint(arch='rexnet2_0x', url='https://github.com/frgfm/Holocron/releases/download/v0.2.1/rexnet2_0x_224-3f00641e.pth', acc1=0.9524, acc5=0.9957, sha256='3f00641e48a6d1d3c9794534eb372467e0730700498933c9e79e60c838671d13', size=55724412, num_params=13829854, commit='d4a59999179b42fc0d3058ac6b76cc41f49dd56e', train_args='./imagenette2-320/ --arch rexnet2_0x --batch-size 32 --grad-acc 2 --mixup-alpha 0.2 --amp --device 0 --epochs 100 --lr 1e-3 --label-smoothing 0.1 --random-erase 0.1 --train-crop-size 176 --val-resize-size 232 --opt adamw --weight-decay 5e-2')
DEFAULT class-attribute instance-attribute 
DEFAULT = IMAGENET1K
Source code in holocron/models/classification/rexnet.py 
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
def rexnet2_0x(
    pretrained: bool = False,
    checkpoint: Checkpoint | None = None,
    progress: bool = True,
    **kwargs: Any,
) -> ReXNet:
    """ReXNet-2.0x from
    ["ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network"](https://arxiv.org/pdf/2007.00992.pdf)

    Args:
        pretrained: If True, returns a model pre-trained on ImageNet
        checkpoint: If specified, the model's parameters will be set to the checkpoint's values
        progress: If True, displays a progress bar of the download to stderr
        kwargs: keyword args of [`ReXNet`][holocron.models.ReXNet]

    Returns:
        classification model

    ::: holocron.models.ReXNet2_0x_Checkpoint
        options:
            heading_level: 4
            show_if_no_docstring: true
    """
    checkpoint = _handle_legacy_pretrained(
        pretrained,
        checkpoint,
        ReXNet2_0x_Checkpoint.DEFAULT.value,
    )
    return _rexnet(checkpoint, progress, 2, 1, **kwargs)

rexnet2_2x 

rexnet2_2x(pretrained: bool = False, checkpoint: Checkpoint | None = None, progress: bool = True, **kwargs: Any) -> ReXNet

ReXNet-2.2x from "ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network"

PARAMETER DESCRIPTION
pretrained

If True, returns a model pre-trained on ImageNet

TYPE: bool DEFAULT: False

checkpoint

If specified, the model's parameters will be set to the checkpoint's values

TYPE: Checkpoint | None DEFAULT: None

progress

If True, displays a progress bar of the download to stderr

TYPE: bool DEFAULT: True

kwargs

keyword args of ReXNet

TYPE: Any DEFAULT: {}

RETURNS DESCRIPTION
ReXNet

classification model

ReXNet2_2x_Checkpoint

Bases: Enum

IMAGENETTE class-attribute instance-attribute 
IMAGENETTE = _checkpoint(arch='rexnet2_2x', url='https://github.com/frgfm/Holocron/releases/download/v0.2.1/rexnet2_2x_224-b23b2847.pth', acc1=0.9544, acc5=0.9946, sha256='b23b28475329e413bfb491503460db8f47a838ec8dcdc5d13ade6f40ee5841a6', size=67217933, num_params=16694966, commit='d4a59999179b42fc0d3058ac6b76cc41f49dd56e', train_args='./imagenette2-320/ --arch rexnet2_2x --batch-size 32 --grad-acc 2 --mixup-alpha 0.2 --amp --device 0 --epochs 100 --lr 1e-3 --label-smoothing 0.1 --random-erase 0.1 --train-crop-size 176 --val-resize-size 232 --opt adamw --weight-decay 5e-2')
DEFAULT class-attribute instance-attribute 
DEFAULT = IMAGENETTE
Source code in holocron/models/classification/rexnet.py 
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
def rexnet2_2x(
    pretrained: bool = False,
    checkpoint: Checkpoint | None = None,
    progress: bool = True,
    **kwargs: Any,
) -> ReXNet:
    """ReXNet-2.2x from
    ["ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network"](https://arxiv.org/pdf/2007.00992.pdf)

    Args:
        pretrained: If True, returns a model pre-trained on ImageNet
        checkpoint: If specified, the model's parameters will be set to the checkpoint's values
        progress: If True, displays a progress bar of the download to stderr
        kwargs: keyword args of [`ReXNet`][holocron.models.classification.rexnet.ReXNet]

    Returns:
        classification model

    ::: holocron.models.ReXNet2_2x_Checkpoint
        options:
            heading_level: 4
            show_if_no_docstring: true
    """
    checkpoint = _handle_legacy_pretrained(
        pretrained,
        checkpoint,
        ReXNet2_2x_Checkpoint.DEFAULT.value,
    )
    return _rexnet(checkpoint, progress, 2.2, 1, **kwargs)