Darius Knowledge Hub

AI Lab Sharing - Concept Bottleneck Models

3 min read

AI Lab Sharing - Concept Bottleneck Models

Created: 04 Apr 2023, 06:49 PM | Modified: =dateformat(this.file.mtime,"dd MMM yyyy, hh:mm a") Tags: knowledge,

  • Need for monitoring to verify the output of DNN

Training and test dataset come from same distribution. In deployment it is out of distribution, aspect ratio different, different camera parameter, different prepro

Plausibility - required by regulatory acts - “post market monitoring” to understand problems, understand implausible behaviour during operation

  • Local try to unds output while looking at specific samples
    • Limited, hard to understand (e.g. in saliency map)
    • Hard to generalise
  • Global look at whole dataset
    • Hard to implement in realtime

CBM

  • Another layer that describes input x after input, before output
  • Predict concept that identifies the input
  • Look at the concept layer weights and will be able to understand the concept
  • Use GPT to create the concepts - in newer methods

Their solution

  • Conepts learnt from other transfer learning

  • Their method - Domain randomisation to increase acc and robustness
  • CIConv (learn shape based concepts, not colour based concepts since their heuristic is that shape based are more generalisable)

  • Ignore colour differences, represents the shape

  • OD produces Bbox
  • Crop Bbox, throw into CBM to verify what it is - give human interp rep

  • Good at 0 shot fp detection

  • Good at object hallucination, fail at localisation

  • Since concept is steill present in the localisation set

  • Testing on kitti easy (just car, pedestrian)

  • Interpretable and robust plausibility detector

  • What extent do u identify “what an explanation should be?” is there any definition?
    • For the legislation
      • Is it post mortem? Or is it while driving?
      • When deploy hard to predict the distribution shift
  • Idea here is to have interpretable bottleneck
    • If want to detect person, attribute is a head ⇒ direct correspondence
    • So if detect head == detect person
    • What is the explanatory power here?
      • Detector say yes is a person
      • CBM say yes is person cos saw arm and leg that makes it a person
      • CIConv layer - focus on shape based attributes that are robust in scene perturbations
    • System could cheat - cos bbox is here I just hardcode that there will be a head in upper part of bbox, all the system did is detect bbox and say head is somewhere there - based on predefined rule. The way info is obtained - it is bogus. How to say output is more genuine than if the system cheats? How can the user determine if the user is cheating?
    • Never clear if the decision was made globally, or is it really cos of the concept
    • Difficult to have a “true explanation”
    • Analogy - chatgpt said I arrived at the answer cos of this this and this, but never know if it is really cos of that

Graph View