SAM, explained · Part 1 of 1 · Covers Title, Figure 1, Abstract, §1, Figure 2

The Big Idea: Segment Anything

Meet SAM, a model that cuts one object out of a photo after one click. We read the first two pages of the paper, and we run SAM on a photo of two cats.

Segment Anything. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick. ICCV 2023, 2023. arXiv:2304.02643

On 5 April 2023, twelve researchers at Meta AI Research (FAIR) put a paper with a short title on arXiv: Segment Anything. The paper gives a model that cuts out an object in a picture when you show it the object. You click on the object, or you draw a box around it. The model then returns the exact pixels of the object. The model is the Segment Anything Model (SAM). The team also released 1.1 billion masks on 11 million pictures, the largest segmentation dataset at that time.

Each piece below shows the exact lines of the paper, then explains them.

The title#

Computer vision has many types of output for one picture. The figure shows four of them on a small toy picture with two cats on a sofa.

Think of the photo as a sheet of squared paper, with one square for each pixel. Take a pen. Write 1 in each square that the object covers, and write 0 in all the other squares. This sheet of numbers is the mask.

Mathematics writes the same sheet in one line. For a picture with $H$ rows and $W$ columns of pixels:

Read the line from left to right:

• $M$ is the name of the sheet: the mask.
• $\in$ means "is a". So the first part says: "$M$ is a grid of $H$ rows and $W$ columns, and each square holds 0 or 1". For our cat photo, $H = 480$ and $W = 640$.
• $M_{ij}$ is the number in row $i$ and column $j$. The large bracket gives the rule for this number: 1 inside the object, 0 outside.

Here is a very small example: a mask of 4 rows and 5 columns. The object is the group of 1s.

$M_{2,3} = 1$ because the square in row 2, column 3 is part of the object. $M_{1,1} = 0$ because the top-left square is not.

How large is the object? Count the 1s. Each 1 is one pixel of the object, so the count is the area:

The symbol $\sum$ (the Greek capital letter sigma) means "add all of these". There are two of them, because the grid has two directions. The inner sum adds the numbers along one row, from column 1 to column $W$. The outer sum does this for each row, from row 1 to row $H$. For the small example, the four rows give:

For SAM's smallest mask of the cat's ear (below), the same count gives 427.

How did the field get here, and how is SAM different from a detector such as YOLO? The Background page tells that story.

Figure 1: the project in one picture#

The abstract, sentence by sentence#

The abstract has five sentences. We read it in two halves.

Four words in this half are new. The yellow boxes explain them.

Let us run it#

Before we read the introduction, we look at the model in operation. We loaded the released SAM with the largest image encoder, ViT-H (facebook/sam-vit-huge). The paper uses this model in its experiments. We gave the model one picture, the test photo from the huggingface/cats-image dataset, and two clicks.

The code is in sam_part1.py. These are its main lines:

proc = SamProcessor.from_pretrained('facebook/sam-vit-huge')
sam = SamModel.from_pretrained('facebook/sam-vit-huge').to('mps').eval()
inp = proc(images=image, return_tensors='pt')
emb = sam.get_image_embeddings(inp['pixel_values'].to('mps'))       # the heavy part: once per picture

p = proc(images=image, input_points=[[[[275, 62]]]], input_labels=[[[1]]], return_tensors='pt')
out = sam(image_embeddings=emb, input_points=p['input_points'].float().to('mps'),
          input_labels=p['input_labels'].to('mps'), multimask_output=True)   # the light part: once per prompt
masks = proc.image_processor.post_process_masks(out.pred_masks.cpu(), p['original_sizes'], p['reshaped_input_sizes'])

== the input pipeline (facebook/sam-vit-huge) ==
resize: longest side 640 -> 1024, scale 1.6000; picture becomes 1024 x 768
pad: 1024 x 768 -> 1024 x 1024 (zeros added below: 256 rows)
pixel tensor: (1, 3, 1024, 1024) = 3,145,728 numbers
image embedding: (1, 256, 64, 64) = 1,048,576 numbers (each of 64 x 64 = 4,096 cells covers 16 x 16 pixels)

== one click at (275, 62) on the left cat's ear ==
low-resolution mask logits from the decoder: (1, 1, 3, 256, 256)  (batch, prompts, 3 masks, 256, 256)
  mask 1: predicted IoU 0.946       427 pixels =   0.14% of the picture
  mask 2: predicted IoU 0.944       470 pixels =   0.15% of the picture
  mask 3: predicted IoU 0.980    49,520 pixels =  16.12% of the picture
  SAM ranks mask 3 first (highest predicted IoU)

Each line of that output is simple arithmetic. We do the arithmetic once by hand.

Step 1: resize. SAM's image encoder accepts only one size: a square of 1024 × 1024 pixels. Our photo is 640 × 480. SAM must make the photo larger, but it must not change its shape, because a stretched cat is a different picture. So SAM multiplies both sides by the same number $s$. It chooses $s$ so that the longer side becomes exactly 1024:

Read it in three steps:

• $W = 640$ is the width and $H = 480$ is the height of the photo. $\max(W, H)$ means "the larger of the two numbers". Here $\max(640, 480) = 640$.
• $s = 1024 / 640 = 1.6$. Each side becomes 1.6 times longer.
• The new width is $W' = 1.6 \times 640 = 1024$ and the new height is $H' = 1.6 \times 480 = 768$. The shape does not change, because $640 / 480 = 1024 / 768$.

Why the longer side? If SAM made the shorter side 1024, the longer side would become $640 \times 1024 / 480 \approx 1365$. That is too wide for the square.

Step 2: pad. The picture is now 1024 pixels wide but only 768 pixels high. A square needs 1024 rows, so SAM adds $1024 - 768 = 256$ rows of zeros at the bottom. Think of a photo in a square frame, with a black strip below it. The input then has 3 colour numbers for each of the $1024 \times 1024$ pixels: $3 \times 1024 \times 1024 = 3{,}145{,}728$ numbers.

Step 3: encode. Put a grid of $64 \times 64$ squares over the picture, as a large chessboard. Each square covers $1024 / 64 = 16$ pixels on each side. The image encoder writes a list of 256 numbers for each square. The list describes what is in that square and around it: fur, the edge of an ear, the sofa. All the lists together are the image embedding: $256 \times 64 \times 64 = 1{,}048{,}576$ numbers.

Step 4: decode. The prompt and the image embedding go into the mask decoder. The decoder makes three masks of 256 × 256 and three scores. SAM then scales the masks back to the size of the photo, 640 × 480.

Now look at the three masks. One click on the ear gave three different answers.

All three masks are correct answers. The click was on the ear, but a person who clicks there can want the ear or the cat. The paper calls this ambiguity.

SAM does not choose one answer. It gives a small set of valid answers and a score for each one. The score is SAM's own estimate of the quality of the mask. Part 3 explains how SAM learns the score.

The second click shows why the score is important. The click is on the buttons of a remote control.

Here, the three masks are not all valid. Mask 3 covers the back of the sofa and has many holes. SAM gives it the score 0.793, which is lower than the score 0.989 of the good mask. So the score helps an application to choose. An application can keep the mask with the highest score, or it can show all three masks to a person. The score is not perfect: here, the bad mask 3 has a higher score than the good but small mask 1.

To compare two masks, the paper uses a number called IoU. You will see it on almost every page of the paper, so we build it slowly.

Draw one shape on a sheet of tracing paper, and a second shape on a second sheet. Put one sheet on the other and hold them up to a lamp. Some area is dark on both sheets: this is the overlap. Some area is dark on at least one sheet: this is the total area of the two shapes together. IoU asks one question: which fraction of the total area is overlap?

Why do we divide by the total area, and not by the size of the object? Imagine a lazy model that marks the whole picture as the cat. Its mask covers the cat completely. If we divided by the size of the cat, the lazy mask would get a perfect score. With IoU, the total area is now the whole picture, so the score is low: for our cat, $49{,}520 / 307{,}200 = 0.16$. IoU punishes missing pixels, and it also punishes extra pixels.

Now the formula. For two masks $A$ and $B$:

Read the middle part first:

• $A \cap B$ (say "A and B") is the overlap: the pixels that are in both masks.
• $A \cup B$ (say "A or B") is the pixels that are in at least one mask.
• The bars $| \ldots |$ mean "count the pixels".

The right part gives the same counts with the 0s and 1s of the two masks. It looks at each pixel $(i, j)$ one at a time. There are only four possible cases:

The product is 1 only in the last row, where the pixel is in both masks. The maximum is 1 in the last three rows, where the pixel is in at least one mask. So the top sum counts the overlap, and the bottom sum counts the total area.

First, a small example. Mask $A$ has 4 pixels and mask $B$ has 4 pixels. 2 pixels are in both masks. The total area is not $4 + 4 = 8$, because then we count the 2 shared pixels two times: once in $A$ and once in $B$. So we subtract them one time:

Now the real masks. We computed the IoU of the two ear masks:

Mask 1 has 427 pixels and mask 2 has 470 pixels. 424 pixels are in both masks, and 473 pixels are in at least one mask. The check: $427 + 470 - 424 = 473$, because the union counts each pixel in the intersection once, not two times.

How do you read an IoU value? 1.0 means that the two masks are identical. Two careful people who draw the same object usually agree at an IoU of 0.85 to 0.91 (§5 of the paper). So above 0.9, two masks are "the same" for most uses. 0.5 means that only half of the total area is overlap. 0 means that the masks have no pixel in common.

Now we can understand SAM's scores. They are predicted IoU values. To compute a real IoU, you need the correct mask. SAM does not have it: it has only the picture and the click. So SAM estimates the IoU of its own mask.

A score of 0.98 means: "I think that my mask and the object that you want overlap with an IoU of about 0.98". Think of a student who writes an answer and also writes how sure they are. Part 3 explains how SAM learns to make this estimate.

A box is a less ambiguous prompt than a point. A box around the right cat gives one mask:

More points also remove the ambiguity. With two clicks, one on the ear and one on the body, SAM gives one mask of the whole cat (50,214 pixels, score 0.969). Part 2 explains these cases.

The output also shows where the weights of SAM are. We counted the weights of the three released models:

model                  image encoder  prompt encoder  mask decoder           total
facebook/sam-vit-base      89,670,912           6,476     4,058,340      93,735,728
facebook/sam-vit-large    308,278,272           6,476     4,058,340     312,343,088
facebook/sam-vit-huge     637,026,048           6,476     4,058,340     641,090,864

The three models have the same prompt encoder and the same mask decoder. Only the image encoder changes: ViT-B (Base), ViT-L (Large) or ViT-H (Huge).

The paper's Figure 13 gives the sizes 91M, 308M and 636M. Our counts of the released image encoders are 89.7M, 308.3M and 637.0M. The small differences can come from which layers each count includes. In all three models, almost all the weights are in the image encoder. This split is the reason for the "amortized real-time" claim in §1, as we will see.

Foundation models in language#

Now we read the introduction. It starts with language models.

The paragraph continues in the right column:

The paper cites GPT-3 here as reference [10]. The GPT-3 abstract states the zero-shot and few-shot idea clearly:

The figure puts the two kinds of prompt side by side.

Foundation models in vision#

The CLIP abstract gives the method in one sentence:

The paragraph also says that CLIP "composes" with other modules: DALL·E uses CLIP as one part of an image generator. Composition is an important idea in SAM. In Part 2 and Part 5 you will see SAM as one part of a larger system. For example, a detector finds the boxes, and SAM makes the masks.

The goal: a foundation model for segmentation#

The term "foundation model" comes from a 2021 report from Stanford. The SAM paper cites it as [8] and uses its definition in §8:

The three questions are:

1. What task will enable zero-shot generalization? The task must teach a skill that is useful for many other tasks. For language, the task is "predict the next word". For SAM, the task is "give a valid mask for any prompt" (§2, Part 2).
2. What is the corresponding model architecture? The model must accept many types of prompt. It must be fast enough for a person to use it, and it must deal with ambiguity (§3, Part 3).
3. What data can power this task and model? The data must be very large and very diverse. No such data existed, so the team made it (§4 and §5, Part 4).

The three answers depend on each other#

The task#

The next paragraphs of §1 give a short preview of each section of the paper. The first preview is the task.

Our ear click is an example. "The ear" and "the cat" are both valid masks. A mask of half the ear and half the sofa is not valid. SAM gave the ear and the cat, so its answer is valid for this click.

The model#

The word amortized needs an explanation. The image encoder is slow, but it runs only one time for each picture. The decoder is fast, and it runs one time for each prompt. When a person clicks many times on the same picture, the cost of the encoder is shared among all the clicks.

We measured the two times on our computer:

facebook/sam-vit-base: image encoder   756.2 ms (GPU)    1444.3 ms (CPU)   prompt encoder + mask decoder   5.7 ms (GPU)   17.6 ms (CPU)
facebook/sam-vit-huge: image encoder  3069.4 ms (GPU)    6809.4 ms (CPU)   prompt encoder + mask decoder   5.8 ms (GPU)   17.5 ms (CPU)

For ViT-H, the image encoder takes 3,069 ms on the laptop GPU. The prompt encoder and the mask decoder take 17.5 ms on the CPU. The paper's number is "∼50 ms in a web browser", also on a CPU. A web browser is slower than native PyTorch, so the two numbers agree well.

Think of a coffee machine. You pay a large amount one time for the machine: this is the encoder, 3,069 ms for each picture. Then each cup costs a small amount: this is the decoder, 17.5 ms for each click. If you make only one cup, that cup costs the full machine plus 17.5. If you make 100 cups, the 100 cups share the cost of the machine.

With $k$ prompts on one picture, the time for each prompt is the total time divided by $k$:

where:

• $T_{\text{enc}}$ is the time of the image encoder, which runs once (3,069 ms);
• $T_{\text{dec}}$ is the time of the prompt encoder and the mask decoder, which run once for each prompt (17.5 ms);
• $k$ is the number of prompts on the same picture.

Read the right side. The first part, $T_{\text{enc}} / k$, is the share of the encoder for one prompt. It becomes smaller when $k$ becomes larger. The second part, $T_{\text{dec}}$, does not change. So the time for each prompt comes near to 17.5 ms after many clicks, but it never goes below 17.5 ms.

With our numbers:

This is the meaning of "amortized real-time". The first click on a new picture is slow. All the next clicks are fast. In the annotation tool of the paper, the server computes the image embedding before the annotator starts. The annotator then sees only the decoder time. This is possible because 99.4% of the weights are in the encoder, and the 4.1-million-weight decoder is very small.

The data engine#

The three stages, in short (Part 4 gives the details):

1. Assisted-manual. A person clicks, SAM proposes a mask, and the person corrects it. 4.3 million masks on 120,000 pictures.
2. Semi-automatic. SAM first adds the masks that it is sure of. The person then labels the objects that SAM missed. 5.9 million more masks on 180,000 pictures.
3. Fully automatic. SAM gets a grid of 32 × 32 points on each picture and makes all the masks. 1.1 billion masks on 11 million pictures.

The dataset#

The paper's Figure 6 gives the size of the other datasets. We put them on one scale:

The ratios are simple divisions:

The paper rounds the first ratio down to "400×". It also says that SA-1B has "36× more masks per image" than Open Images. With the rounded numbers of the figure we get $103 / 2.7 \approx 38$. The small difference comes from the rounding of the dataset sizes.

Page 3 of the paper shows sample pictures of SA-1B with all their masks:

Responsible AI#

The experiments#

The five experiments go from low-level to high-level vision (Part 5):

The release#

Who's who: the papers that the introduction cites#

The introduction cites earlier work by number. This table puts the important references in one place. The full list is in the references at the end of this part.

Next, in Part 2: §2 of the paper, the task. Part 2 tells what a prompt can be and what "valid" means in exact terms. It shows why the task works as pre-training and how zero-shot transfer works with prompts. It also compares promptable segmentation with interactive and multi-task segmentation. With real runs, it shows SAM's three masks on more ambiguous clicks, and the effect of each new point.

Run it yourself

The script behind this part is code/papers/sam/sam_part1.py. It counts the weights of the three models and runs the picture through SAM ViT-H. It makes the masks for the two clicks and the box. It measures the time of the encoder and the decoder, and it saves every number in results/part1.json.

bashCopy

pip install torch transformers datasets pillow
python sam_part1.py        # downloads facebook/sam-vit-base, -large and -huge (about 3.8 GB in total)

The paper and the code#

Segment Anything (ICCV 2023) · code and weights · our scripts for this part