A herd of enthusiastic oxen running towards the future

Arxiv Dives

Every Friday at Oxen.ai we host a paper club called "Arxiv Dives" to make us smarter Oxen 🐂 🧠. We believe diving into the details of research papers is the best way to build fundamental knowledge, spot patterns and keep up with the bleeding edge.

In Arxiv Dives, we cover state of the art research papers, and dive into the gnitty gritty details of how AI models work. From the math to the data to the model architecture, we cover it all.

ArXiv Dives: Text Diffusion with SEDD
ArXiv Dives: Text Diffusion with SEDD

Diffusion models have been popular for computer vision tasks. Recently models such as Sora show how you can apply Diffusion + Transformers to generate state of the art videos with ...

Greg Schoeninger
Greg Schoeninger
Apr 16, 2024
11 min read
ArXiv Dives: The Era of 1-bit LLMs, All Large Language Models are in 1.58 Bits
ArXiv Dives: The Era of 1-bit LLMs, All Large Language Models are in 1.58 Bits

This paper presents BitNet b1.58 where every weight in a Transformer can be represented as a {-1, 0, 1} instead of a floating point number. The model matches full precision transfo...

Greg Schoeninger
Greg Schoeninger
Apr 8, 2024
9 min read
ArXiv Dives: Evolutionary Optimization of Model Merging Recipes
ArXiv Dives: Evolutionary Optimization of Model Merging Recipes

Today, we’re diving into a fun paper by the team at Sakana.ai called “Evolutionary Optimization of Model Merging Recipes”. The high level idea is that we have so many open weights ...

Greg Schoeninger
Greg Schoeninger
Apr 1, 2024
10 min read
ArXiv Dives: I-JEPA
ArXiv Dives: I-JEPA

Today, we’re diving into the I-JEPA paper. JEPA stands for Joint-Embedding Predictive Architecture and if you have been following Yann LeCunn, is a technique he has been hyping up ...

Greg Schoeninger
Greg Schoeninger
Mar 26, 2024
13 min read
ArXiv Dives - Diffusion Transformers
ArXiv Dives - Diffusion Transformers

Diffusion transformers achieve state-of-the-art quality generating images by replacing the commonly used U-Net backbone with a transformer that operates on latent patches. They rec...

Greg Schoeninger
Greg Schoeninger
Mar 12, 2024
14 min read
"Road to Sora" Paper Reading List
"Road to Sora" Paper Reading List

This post is an effort to put together a reading list for our Friday paper club called ArXiv Dives. Since there has not been an official paper released yet for Sora, the goal is fo...

Greg Schoeninger
Greg Schoeninger
Mar 5, 2024
21 min read
ArXiv Dives - Medusa
ArXiv Dives - Medusa

Abstract In this paper, they present MEDUSA, an efficient method that augments LLM inference by adding extra decoding heads to predict multiple subsequent tokens in parallel. The ...

Greg Schoeninger
Greg Schoeninger
Mar 4, 2024
5 min read
ArXiv Dives - Lumiere
ArXiv Dives - Lumiere

This paper introduces Lumiere – a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion – a pivotal challenge in video ...

Greg Schoeninger
Greg Schoeninger
Feb 27, 2024
11 min read
ArXiv Dives - Depth Anything
ArXiv Dives - Depth Anything

This paper presents Depth Anything, a highly practical solution for robust monocular depth estimation. Depth estimation traditionally requires extra hardware and algorithms such as...

Greg Schoeninger
Greg Schoeninger
Feb 19, 2024
16 min read
Arxiv Dives - Toolformer: Language models can teach themselves to use tools
Arxiv Dives - Toolformer: Language models can teach themselves to use tools

Large Language Models (LLMs) show remarkable capabilities to solve new tasks from a few textual instructions, but they also paradoxically struggle with basic functionality such as ...

Greg Schoeninger
Greg Schoeninger
Feb 12, 2024
10 min read