Describe

---

Created: 23 Dec 2022, 02:27 PM | Modified: =dateformat(this.file.mtime,"dd MMM yyyy, hh:mm a") Tags: knowledge,

---

Then describe

• generate_candidate_descriptions
• With candidate descriptions → embed again

phrase_templates = [

“a photo of [MASK].”,

“a photo of {} [MASK].”,

“a photo of [MASK] {}.”,

“a photo of [MASK] {} [MASK].”,

]

text_dp = generate_candidate_descriptions(

templates=phrase_templates

)

text_dp = embed(

text_dp,

input_col=“output_phrase”,

encoder=“clip”,

device=0

)

• Then plot with explore()

explore(

data=dp,

embeddings=“clip(image)”,

pred_probs=“prob”,

targets=“target”,

slices=“domino_slices”,

text=text_dp,

text_embeddings=“clip(output_phrase)”,

)

What does generate_candidate_descriptions do?

• Calls _get_wiki_words, that will:
  • get IlyaSemenov/wikipedia-word-frequency/master/results/enwiki-2022-08-29.txt
  • get nltk.corpus “words”
  • merge together, sort based on frequency and remove duplicates
  • Returns top 10k words into words_dp
• Based on the returned wiki words in words_dp, will call _forward_mlm that will:
  • Use tokenizer - BertTokenizer.from_pretrained(f”bert-base-uncased”)
  • Use model - BertForMaskedLM.from_pretrained(f”bert-base-uncased”).to(device).eval()
  • Pass input phrase_templates and wiki words words_dp into tokenizer to get input tokens
  • Pass input tokens to pretrained model
  • Output from model will be processed until can be decoded by tokenizer into output_phrases and output_probs, returned as candidate_phrases and then returned into text_dp
  • What is text_dp[“prob”]? Try on aftershock

Output text embeddings will be obtained via embed() as per Embed

Plotting of everything is then done in explore():

• plot_slice() ⇒ plot_output

  • Plotting of graphs

• show_descriptions() ⇒ description_output

  • Calls describe():
    • slice_mask = slices[:, slice_idx] > slice_threshold

slice_proto = embeddings[slice_mask].mean(axis=0)

mode_target = mode(targets[slice_mask]).mode[0]

ref_proto = embeddings[targets == mode_target].mean(axis=0)

scores = np.dot(text_embeddings, (slice_proto - ref_proto))

return mk.DataPanel({“score”: scores, “phrase”: phrases})

• Main point is to calculate the score for description_output

• The highest ranked 5 scores and their phrase will be displayed

• show_dp() ⇒ dp_output

  • Meerkat datapanel plotting

This part of the paper is about describe():