Code overview (ImageNet example)

---

Created: 23 Dec 2022, 02:09 PM | Modified: =dateformat(this.file.mtime,"dd MMM yyyy, hh:mm a") Tags: knowledge,

---

Embed images first

dp = embed(

data=dp, input_col=“image”, encoder=“clip”, device=0

)

Then run slice == mixture model

domino = DominoSlicer(

y_log_likelihood_weight=40,

y_hat_log_likelihood_weight=40,

n_mixture_components=60,

n_slices=10,

max_iter=10,

n_pca_components=128,

init_params=“confusion”,

confusion_noise=3e-3

)

domino.fit(data=dp, embeddings=“clip(image)”, targets=“target”, pred_probs=“prob”)

dp[“domino_slices”] = domino.predict_proba(

data=dp, embeddings=“clip(image)”, targets=“target”, pred_probs=“prob”

)

Then describe

• generate_candidate_descriptions
• With candidate descriptions → embed again

text_dp = embed(

text_dp,

input_col=“output_phrase”,

encoder=“clip”,

device=0

)

• Then plot with explore()

explore(

data=dp,

embeddings=“clip(image)”,

pred_probs=“prob”,

targets=“target”,

slices=“domino_slices”,

text=text_dp,

text_embeddings=“clip(output_phrase)”,

)