Skip to content

MMDetection

OpenMMLab Detection Toolbox and Benchmark

MMDetection은, object detection과 instance segmentation을 다루는 유명하고 다양한 모델을 하나의 toolbox로 구현한 일종의 플랫폼이다.

많은 사람들이 사용하여, 기존 디텍션 모델을 다양한 분야에 응용하고, 새로운 모델을 만드는데 도움이 되었으면 한다고 한다.

Categories

Features

  • Modular design 모델이 모듈화 되어있어 사용자 제작이 간편하다.
  • Supported Framwork 독창적으로 여러 프레임워크를 지원한다.
  • High efficiency 모든 box와 mask 연산은 GPU에서 동작한다. Detectron(facebook), maskrcnn-benchmark, SimpleDet 등 다른 플랫폼 보다 학습속도가 빠르다.
  • State of the art 2018년 COCO Detection Challenge 에서 우승한 MMDet team의 코드를 토대로 만들었다.

Supported Framworks

MMDetection 안에 구성된 모델들이다.

  • Single-state Method
  • Two-state Methods
  • Multi-state Methods
    • Cascade R-CNN
    • Hybrid Task Cascade - 가장 높은 성능을 보인 모델
  • General Modules and Methods
    • Mixed Precision Training
    • Soft NMS
    • OHEM
    • DCN
    • DCNv2
    • Train from Scratch
    • ScratchDet
    • M2Det - 2018년, 더 효율적인 피처 피라미드를 구성하기 위한 새로운 피처 피라미드 네크워크
    • GCNet
    • Generalized Attention
    • SyncBN
    • Group Normalization
    • Weight Standardization
    • HRNet - 2019년, 고해상도 이미지를 학습하는 데 집중한 새로운 뼈대(backbone)이다.
    • Guided Anchoring
    • Libra R-CNN

How to install

2021-10-20 기준, 설치 버전은 다음과 같다:

CUDA 설치

Ubuntu 18.04:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.4.2/local_installers/cuda-repo-ubuntu1804-11-4-local_11.4.2-470.57.02-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-11-4-local_11.4.2-470.57.02-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu1804-11-4-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

Ubuntu 20.04:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.4.2/local_installers/cuda-repo-ubuntu2004-11-4-local_11.4.2-470.57.02-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-4-local_11.4.2-470.57.02-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu2004-11-4-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

설치 후 재부팅

PyTorch 설치

PyTorch 1.8.2 버전, PIP로 설치 방법:

WARNING

PyTorch 1.9.1 은 MMDetection 에서 지원하지 않아서 1.8.2 로 설치했다.

CUDA 11.1:

pip3 install torch==1.8.2+cu111 torchvision==0.9.2+cu111 torchaudio==0.8.2 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html

CUDA 10.2:

pip3 install torch==1.8.2+cu102 torchvision==0.9.2+cu102 torchaudio==0.8.2 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html

mmcv 설치

2021-10-20 기준, 1.3.15 버전에서 다음과 같은 조건으로 받을 수 있다.

  • PyTorch 1.3.0 ~ 1.8.0
  • CUDA 10.1, 10.2, 11.1
  • Python 3.6, 3.7, 3.8

MMDetection 2.17.0 버전은 mmcv-full>=1.3.8, <1.4.0 와 호환된다.

lite 버전이 아닌, full 버전으로 받아야 한다. 다음과 같은 포맷으로 입력하면 된다.

pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/{cu_version}/{torch_version}/index.html

CUDA 11.1, PyTorch 1.8.0:

opy -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html

CUDA 10.2, PyTorch 1.8.0:

opy -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.8.0/index.html

패키지 목록은 https://download.openmmlab.com/mmcv/dist/ 페이지 참조.

MMDetection 설치

pip install mmdet

2021-10-20 기준, 2.17.0 버전이 받아지더라.

[Optional] 확장 종속성은 다음과 같다

Install extra dependencies for Instaboost, Panoptic Segmentation, LVIS dataset, or Albumentations.

# for instaboost
pip install instaboostfast
# for panoptic segmentation
pip install git+https://github.com/cocodataset/panopticapi.git
# for LVIS dataset
pip install git+https://github.com/lvis-dataset/lvis-api.git
# for albumentations
pip install albumentations>=0.3.2 --no-binary imgaug,albumentations

MIM을 사용한 설치 방법

쉬운 OpenMMLab Packages 설치를 위한 mim을 설치한다.

opy 스크립트로 venv 공간 설치 완료 후 진행:

mkdir test
cd test
cp {나의 opy python 단독 스크립트} ./python

## CUDA 12.2 에서 테스트진행
./python -m pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu121
./python -m pip install -U openmim
./python -m mim install mmengine

./python -m mim install "mmcv>=2.0.0"
## 만약 mmdet 3.3.0 을 설치할 예정이라면 `mmcv<2.2.0,>=2.0.0rc4` 조건이 걸려있다. 다음과 같이 설치해야 한다.
./python -m mim install "mmcv==2.1.0"

참고로, MMCV-v2.x 에서 mmcv-full은 mmcv로 이름이 바뀌었습니다. CUDA ops 없이 mmcv를 설치하려면 mim install "mmcv-lite>=2.0.0rc1"을 사용하여 라이트 버전을 설치할 수 있습니다.

mmdet 소스코드에서 바로 설치

git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection
pip install -v -e .
# "-v" means verbose, or more output
# "-e" means installing a project in editable mode,
# thus any local modifications made to the code will take effect without reinstallation.

mim 으로 설치

mmdet 가 종속성이며 third-party package 로 설치하고 싶다면

./python -m mim install mmdet

만약 mmdet 3.3.0 requires mmcv<2.2.0,>=2.0.0rc4; extra == "mim", but you have mmcv 2.2.0 which is incompatible. 에러가 난다면 mmcv==2.1.0를 설치하자.

mmcv의 버전 히스토리는 https://pypi.org/project/mmcv/#history 페이지 참조.

Build dockerfile

ARG PYTORCH="2.4.0"
ARG CUDA="12.4"
ARG CUDNN="9"

FROM pytorch/pytorch:${PYTORCH}-cuda${CUDA}-cudnn${CUDNN}-runtime as builder

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

RUN apt-get update && \
    apt-get install -y --no-install-recommends gcc g++

WORKDIR /app
COPY . .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /app/.wheels -r requirements.docker.txt && \
    pip wheel --no-cache-dir --no-deps --wheel-dir /app/.wheels "openmim>=0.3.9" && \
    python setup.py bdist_wheel && \
    cp dist/*.whl /app/.wheels

# --[[ NEXT STAGE
FROM pytorch/pytorch:${PYTORCH}-cuda${CUDA}-cudnn${CUDNN}-runtime

WORKDIR /app
COPY --from=builder /app/.wheels /app/.wheels
RUN apt-get update && \
    apt-get install -y --no-install-recommends libgl1-mesa-dev gcc g++ ffmpeg tk vim && \
    pip install --no-cache /app/.wheels/* && \
    mim install "mmengine==0.10.4" && \
    mim install "mmcv==2.1.0" && \
    mim install "mmdet==3.3.0" && \
    rm -rf /app/.wheels && \
    addgroup --gid 1001 --system app && \
    adduser --no-create-home \
        --shell /bin/false \
        --disabled-password \
        --uid 1001 \
        --system \
        --group app
USER app

ENTRYPOINT ["python", "-OO", "-m", "ddrm"]

Import the custom module

If we customize a module and register it into the corresponding registry, could we directly build it from the configuration file as the previous section does? The answer is “I don’t know” since I’m not sure the registration process has been triggered. To solve this “unknown” case, Config provides the custom_imports function, to make sure your module could be registered as expected.

For example, we customize an optimizer:

from mmengine.registry import OPTIMIZERS

@OPTIMIZERS.register_module()
class CustomOptim:
    pass

CocoDataset Example

# -*- coding: utf-8 -*-

from mmdet.datasets.coco import CocoDataset
from mmdet.registry import DATASETS


@DATASETS.register_module()
class PlateCocoDataset(CocoDataset):
    """Dataset for COCO"""

    METAINFO = {
        "classes": ("Plate",),
        # palette is a list of color tuples, which is used for visualization.
        "palette": [
            (106, 0, 228),
        ],
    }

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._metainfo["dataset_type"] = None

XMLDataset Example

# -*- coding: utf-8 -*-

from mmdet.datasets.xml_style import XMLDataset
from mmdet.registry import DATASETS


@DATASETS.register_module()
class SeahVOCDataset(XMLDataset):
    """Dataset for PASCAL VOC."""

    METAINFO = {
        "classes": ("Bloom", "Ingot", "Billet", "Slab", "Rod", "Forklift", "Support"),
        # palette is a list of color tuples, which is used for visualization.
        "palette": [
            (106, 0, 228),
            (119, 11, 32),
            (165, 42, 42),
            (0, 0, 192),
            (197, 226, 255),
            (0, 60, 100),
            (0, 0, 142),
        ],
    }

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._metainfo["dataset_type"] = None

demo/image_demo.py

MMDetection:ImageDemo 항목 참조.

Simple Example

# -*- coding: utf-8 -*-

import os
from datetime import datetime
from glob import glob
from pathlib import Path
from sys import exit as sys_exit
from typing import List, Tuple, Union

import cv2
from mmdet.apis import init_detector, inference_detector
from mmdet.datasets.coco import CocoDataset
from mmdet.registry import DATASETS, VISUALIZERS
from mmdet.structures.det_data_sample import DetDataSample
from numpy import uint8, ndarray
from numpy.typing import NDArray

NumberType = Union[int, float]


@DATASETS.register_module()
class PlateCocoDataset(CocoDataset):
    """Dataset for COCO"""

    METAINFO = {
        "classes": ("Plate",),
        # palette is a list of color tuples, which is used for visualization.
        "palette": [
            (106, 0, 228),
        ],
    }

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._metainfo["dataset_type"] = None


class ObjectSample:
    def __init__(
        self,
        index: int,
        label: str,
        score: float,
        x1: NumberType,
        y1: NumberType,
        x2: NumberType,
        y2: NumberType,
        color: Tuple[int, int, int],
    ):
        self.index = index
        self.label = label
        self.score = score
        self.x1 = x1
        self.y1 = y1
        self.x2 = x2
        self.y2 = y2
        self.color = color

def convert_det_to_obj(classes: List[str], palette: List[Tuple[int, int, int]], sample: DetDataSample) -> List[ObjectSample]:
    result = list()
    pred = sample.pred_instances.numpy()
    for i in range(len(pred)):
        x1, y1, x2, y2 = pred.bboxes[i].tolist()  # noqa
        label_index = pred.labels[i].tolist()  # noqa
        label_name = classes[label_index]
        color = palette[label_index]
        score = pred.scores[i].tolist()  # noqa
        obj = ObjectSample(label_index, label_name, score, x1, y1, x2, y2, color)
        result.append(obj)
    return result

def visualize(visualizer, score_threshold: float, image: NDArray[uint8], data_sample: DetDataSample) -> NDArray[uint8]:
    rgb_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    visualizer.add_datasample(
        name="Preview",
        image=rgb_image,
        data_sample=data_sample,
        draw_gt=False,
        pred_score_thr=score_threshold,
        show=False,
    )
    visualized_image = visualizer.get_image()
    return cv2.cvtColor(visualized_image, cv2.COLOR_RGB2BGR)

def main() -> None:
    config_file = "./rtmdet-ins_x_8xb16-300e_coco_dongkuk_rolling_plate_cam1_2nd.py"
    checkpoint_file = "./epoch_300.pth"
    model = init_detector(config_file, checkpoint_file, device="cuda:0")  # or device="cpu"

    files = glob("./input/*.jpg")
    files.sort()

    output_dir = Path("output")
    if not output_dir.is_dir():
        output_dir.mkdir()

    visualizer = VISUALIZERS.build(model.cfg.visualizer)
    visualizer.dataset_meta = model.dataset_meta

    for filename in files:
        frame = cv2.imread(filename)

        begin = datetime.now()
        result = inference_detector(model, frame)
        assert isinstance(result, DetDataSample)
        duration = (datetime.now() - begin).total_seconds()
        print(f"Predict duration: {duration:.04f}s")

        # print(result)

        begin = datetime.now()
        try:
            visualized_image = visualize(visualizer, 0.5, frame, result)
        except BaseException as e:
            print(f"Visualizer error: {e}")
            continue
        duration = (datetime.now() - begin).total_seconds()
        print(f"Visualizer duration: {duration:.04f}s")

        file_prefix, ext = os.path.splitext(os.path.basename(filename))
        output_filename = file_prefix + ".png"
        output_filepath = str(output_dir / output_filename)
        print(output_filepath)
        cv2.imwrite(output_filepath, visualized_image)

        classes = visualizer.dataset_meta["classes"]
        palette = visualizer.dataset_meta["palette"]
        objs = convert_det_to_obj(classes, palette, result)
        # print(objs)

if __name__ == "__main__":
    sys_exit(main())

Troubleshooting

unexpected key in source state_dict: data_preprocessor.mean, data_preprocessor.std

훈련 결과(*.pth파일)에 사용하지 않는 키가 있다. 그냥 무시하자.

dependency conflicts

Python 3.11.4 에서 다음 pytorch requirements 설치:

--index-url https://download.pytorch.org/whl/cu124
torch==2.4.0+cu124
torchvision==0.19.0+cu124
torchaudio==2.4.0+cu124

이후, mmdet을 위한 다음 bash 스크립트 실행:

#!/usr/bin/env bash

ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" || exit; pwd)

"$ROOT_DIR/python" -m pip install "openmim>=0.3.9"
"$ROOT_DIR/python" -m mim install "mmengine>=0.10.3"
"$ROOT_DIR/python" -m mim install "mmcls>=0.25.0"
"$ROOT_DIR/python" -m mim install "mmdet>=3.3.0"
"$ROOT_DIR/python" -m mim install "mmcv>=2.1.0"

# "$ROOT_DIR/python" -m mim install "mmcv-full>=1.7.2"  # Error

설치 중 다음 에러 발생:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mmdet 3.3.0 requires mmcv<2.2.0,>=2.0.0rc4; extra == "mim", but you have mmcv 2.2.0 which is incompatible.

번역:

오류: pip의 종속성 리졸버는 현재 설치된 모든 패키지를 고려하지 않습니다. 이 동작은 다음 종속성 충돌의 원인입니다.
mmdet 3.3.0에는 mmcv<2.2.0,>=2.0.0rc4; extra == "mim"이 필요하지만 호환되지 않는 mmcv 2.2.0이 있습니다.

결론은 mmcv<2.2.0,>=2.0.0rc4 조건에 맞춰서 mmcv 다시 설치하자. 참고로 이 조건에 맞는 버전은 2023년 10월 17일 에 업로드된 2.1.0 버전이 최신이다.

./python -m pip uninstall mmcv
./python -m pip install mmcv==2.1.0

다음 #AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0 항목도 참조.

=== AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0 === 실행했는데 에러가 출력...

/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
Traceback (most recent call last):
  File "/home/user/Projects/rtmdet/./sample.py", line 9, in <module>
    from mmdet.apis import init_detector, inference_detector
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/__init__.py", line 17, in <module>
    and mmcv_version < digit_version(mmcv_maximum_version)), \
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.

mmcv 정보 확인해보자:

./python -m pip show mmcv

다음과 같이 출력됨:

Name: mmcv
Version: 2.1.0
Summary: OpenMMLab Computer Vision Foundation
Home-page: https://github.com/open-mmlab/mmcv
Author: MMCV Contributors
Author-email: openmmlab@gmail.com
License:
Location: /home/user/Project/ddrm/.venv/lib/python3.11/site-packages
Requires: addict, mmengine, numpy, opencv-python, packaging, Pillow, pyyaml, yapf
Required-by:

왜이러지? 했는데... 상세히 읽어보자:

/home/user/Project/ddrm/.venv/lib/python3.11/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.

문제는 mmcv-full을 설치해서 그렇다. mmcv-full 의 버전이 1.7.2 더라...

mmcv-full을 지우고 mmcv를 설치하니 잘되더라

강제 실행 방법

NVIDIA GeForce RTX 4090 에서 테스트했는데 #ModuleNotFoundError: No module named 'mmcv._ext' 이슈가 발생되어 이슈가 반복되더라...

일단 편법으로 강제 실행하게 하고 싶다면 mmcv/__init__.py 의 버전체크 Assertion 을 제거하면 된다.

ModuleNotFoundError: No module named 'mmcv._ext'

전체 Traceback:

Traceback (most recent call last):
  File "/home/user/Projects/rtmdet/./sample.py", line 9, in <module>
    from mmdet.apis import init_detector, inference_detector
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/apis/__init__.py", line 2, in <module>
    from .det_inferencer import DetInferencer
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/apis/det_inferencer.py", line 22, in <module>
    from mmdet.evaluation import INSTANCE_OFFSET
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/evaluation/__init__.py", line 4, in <module>
    from .metrics import *  # noqa: F401,F403
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/evaluation/metrics/__init__.py", line 5, in <module>
    from .coco_metric import CocoMetric
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/evaluation/metrics/coco_metric.py", line 16, in <module>
    from mmdet.datasets.api_wrappers import COCO, COCOeval, COCOevalMP
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/datasets/__init__.py", line 31, in <module>
    from .utils import get_loading_pipeline
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/datasets/utils.py", line 5, in <module>
    from mmdet.datasets.transforms import LoadAnnotations, LoadPanopticAnnotations
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/datasets/transforms/__init__.py", line 6, in <module>
    from .formatting import (ImageToTensor, PackDetInputs, PackReIDInputs,
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/datasets/transforms/formatting.py", line 11, in <module>
    from mmdet.structures.bbox import BaseBoxes
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/structures/bbox/__init__.py", line 2, in <module>
    from .base_boxes import BaseBoxes
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/structures/bbox/base_boxes.py", line 9, in <module>
    from mmdet.structures.mask.structures import BitmapMasks, PolygonMasks
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/structures/mask/__init__.py", line 3, in <module>
    from .structures import (BaseInstanceMasks, BitmapMasks, PolygonMasks,
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmdet/structures/mask/structures.py", line 12, in <module>
    from mmcv.ops.roi_align import roi_align
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmcv/ops/__init__.py", line 3, in <module>
    from .active_rotated_filter import active_rotated_filter
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in <module>
    ext_module = ext_loader.load_ext(
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Projects/rtmdet/.venv/lib/python3.11/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
    ext = importlib.import_module('mmcv.' + name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/3.11.4/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'mmcv._ext'

요약하면 mmcv-full를 설치하라...

_ext는 CUDA 의 opcode 와 관계있는듯? <- 확인 필요.

KeyError: 'PlateCocoDataset is not in the mmdet::dataset registry

KeyError: 'PlateCocoDataset is not in the mmdet::dataset registry. Please check whether the value of `PlateCocoDataset` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'

#Import the custom module 항목 참조.

만약 추가했는데도 그렇다면 해당 파일이 import 되었는지 확인해 보자.

See also

Favorite site