Future work

how to handle misalignment?

每个Bank多大？速度慢吧？

Abstract

1. unsupervised multimodal AD
2. but large memory and slow inference
   1. rely on large memory banks of multimodal features
4. Hybird fusion: feature fusion + decision fusion
5. classifiers: One-Class SVM

1. Point Feature Alignment (PFA)
   1. Fpt is a pretrained Point Transformer from PointMAE [21]
   2. projection: by camera parameters
      1. to better align the point cloud and RGB features 没体现呀？
2. Frgb is a Vision Transformer from DINO [6] to extract color info
3. Unsupervised Feature Fusion (UFF)
   1. with patch-wise contrastive learning
      1. has loss only for positive pairs, seems reasonable.
   2. χrgb, χpt are MLP layers and σr , σp are single fully connected layers;
4. 𝑫𝒆𝒄𝒊𝒔𝒊𝒐𝒏 𝑳𝒂𝒚𝒆𝒓 𝑭𝒖𝒔𝒊𝒐𝒏
   1. ϕ, ψ are score function for single memory bank detection and segmentation
      1. introduced by PatchCore [26]
   2. P is the memory bank building algorithm: PatchCore [26]
   3. Da, Ds: two learnable One-Class Support Vector Machines (OCSVM) [29] for anomaly detection and segmentation;
      1. 所以不需要synthetic data？
5. two-stage training procedure
   1. construct memory banks
   2. train the decision layer.

Method

Unsupervised Feature Fusion (UFF)

encourage the features from different modalities at the same position to have more corresponding information, while the features at different positions have less corresponding information.

这没有抗disalignment的效果呀？做个实验试试