A herd of enthusiastic oxen running towards the future

Arxiv Dives

Every Friday at Oxen.ai we host a paper club called "Arxiv Dives" to make us smarter Oxen 🐂 🧠. We believe diving into the details of research papers is the best way to build fundamental knowledge, spot patterns and keep up with the bleeding edge.

In Arxiv Dives, we cover state of the art research papers, and dive into the gnitty gritty details of how AI models work. From the math to the data to the model architecture, we cover it all.

Recent
ArXiv Dives - Depth Anything
Feb 19, 2024

This paper presents Depth Anything, a highly practical solution for robust monocular depth estimation. Depth estimation traditionally requires extra hardware and algorithms such as stereo cameras, lidar, or structure from motion. In this paper they c...

Arxiv Dives - Toolformer: Language models can teach themselves to use tools
Feb 12, 2024

Large Language Models (LLMs) show remarkable capabilities to solve new tasks from a few textual instructions, but they also paradoxically struggle with basic functionality such as math, dates on a calendar, or replying with up to date information abo...

Arxiv Dives - Self-Rewarding Language Models
Feb 6, 2024

The goal of this paper is to see if we can create a self-improving feedback loop to achieve “superhuman agents”. Current language models are bottlenecked by labeled data from humans. Not only is the quantity of labels a bottleneck, but also the quali...

Arxiv Dives - Direct Preference Optimization (DPO)
Jan 30, 2024

This paper provides a simple and stable alternative to RLHF for aligning Large Language Models with human preferences called "Direct Preference Optimization" (DPO). They reformulate the loss function as a classification task between prompt completion...

Arxiv Dives - Efficient Streaming Language Models with Attention Sinks
Jan 20, 2024

This paper introduces the concept of an Attention Sink which helps Large Language Models (LLMs) maintain the coherence of text into the millions of tokens while also maintaining a finite memory footprint and latency. Transformer based language model...

Arxiv Dives - How Mixture of Experts works with Mixtral 8x7B
Jan 13, 2024

Mixtral 8x7B is an open source mixture of experts large language model released by the team at Mistral.ai that outperforms Llama-2 70B and GPT-3.5 on a variety natural language understanding tasks. The magic of the model is that it only uses 13B par...

Arxiv Dives - LLaVA 🌋 an open source Large Multimodal Model (LMM)
Jan 7, 2024

What is LLaVA? LLaVA is a Multi-Modal model that connects a Vision Encoder and an LLM for general purpose visual and language understanding. Paper: https://arxiv.org/abs/2304.08485 Team: Wisconsin-Madison, Microsoft Research, Columbia University ...

Arxiv Dives - How Mistral 7B works
Dec 23, 2023

What is Mistral 7B? Mistral 7B is an open weights large language model by Mistral.ai that was build for performance and efficiency. It outshines models that are twice it's size, including Llama-2 13B and Llama-1 34B on both automated benchmarks and ...

Mamba: Linear-Time Sequence Modeling with Selective State Spaces - Arxiv Dives
Dec 15, 2023

What is Mamba 🐍? Mamba at it's core is a recurrent neural network architecture, that outperforms Transformers with faster inference and improved handling of long sequences of length up to 1 million. This post dives into how it works and will give y...

Practical ML Dive - How to customize a Vision Transformer on your own data
Dec 14, 2023

Welcome to Practical ML Dives, a series spin off of Arxiv Dives. In Arxiv Dives, we cover state of the art research papers, and dive into the gnitty gritty details of how AI models work. From the math to the data to the model architecture, we cover ...