I don’t own a TV, but there’s one at my gym. This morning, there was this… infomercial? I don’t know how to call it, it lasted at least an hour and had the tone of the usual off-hour Japanese TV. Programs that try, not very hard, to be both entertainment and journalistic news. Of course it was interspaced by plenty of ads. The topic was shark fishing, and how eating them keeps you young.

TLDR: I wanted to make a fast and low complexity “search engine”, and got good results with neural network embeddings combined with an approximate vector search. Recent advances in large deep learning models have brought interesting multi-modal capabilities, notably for combined text and images, like Imagen and DALL-E 2 for image synthesis from raw text. CLIP, which is also used in DALL-E for image ranking, allow projecting both text and images into a coherent space.

The other day at work I noticed a slowdown in runtime between a model with random weights compared to tuned ones. It turned out to be due to denormal numbers computation on the cpu being much slower than the normal arithmetic. Denormal numbers are very low magnitude floats, treated differently to keep precision. For deep learning, those are largely below significance and can be flushed to zero without accuracy loss. To do so for example in PyTorch, either set the appropriate flag: torch.

The general problem of completing images and videos depending on a mask is called inpainting, and numerous methods exists to tackle this problem in ingenuous ways. However most approaches are relatively costly to run, especially without a graphic card, so I wanted to see what result we could get with simple and fast methods. You can find the code for the steps below in this repository. To start, we need to find the mask, which correspond to the location of pixels to remove.

👷🏻 this is a work in progress 🔧 Sometimes by exploring incongruous ideas, one can make interesting discoveries. This is know as serendipity in science, with penicillin being a famous example of an unplanned discovery. What I experimented with this week-end… is no such case. 😅 In the spirit of not holding back negative results though, and for the unlikely possibility that it turns out helpful for someone, I’ll share it regardless!