SAM, explained · Part 6 of 6 · Covers §8, Conclusion, and five later papers: SAM 2, MobileSAM, HQ-SAM, MedSAM, SAM 3

After SAM: Impact, Limits and Summary

Where SAM fails, measured with the released model on a bicycle, a drawn wheel and an edge with teeth. Then five later papers that fixed those limits, and the whole paper in one page.

Segment Anything. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick. ICCV 2023, 2023. arXiv:2304.02643

The last page of the paper is honest: it lists what SAM cannot do. We test each limit on the released model, then follow five later papers that took up those limits one by one.

Foundation models, once more

From the paper§8page 12

What it says

A foundation model is "trained on broad data at scale and are adaptable to a wide range of downstream tasks". SAM fits, but segmentation is "an inherently limited scope". Also, "the vast majority of its capabilities come from large-scale supervised training", not from self-supervised learning.

What it means

Part 1 defined a foundation model with the same words. Here the authors check SAM against the definition, and they add two careful notes.

The first note is modesty. A language model can write, translate and answer questions. SAM only cuts objects out of pictures. It is a foundation for one floor of the building, not for the whole building.

The second note is about how SAM learned. The usual story says that foundation models learn from raw data with no labels. SAM starts that way (the MAE pre-training of Part 3), but most of its skill comes from 1.1 billion masks with a correct answer. The data engine of Part 4 made those labels cheap, so labelled training became possible at scale again.

Composition: SAM as a part of a bigger system

From the paper§8page 12

What it says

The goal is to "make this kind of composition straightforward with SAM". A valid mask for any prompt creates "a reliable interface between SAM and other components". For example, SAM can be "prompted with gaze points detected by a wearable device", and such systems work "without need for additional training".

What it means

Think of a power socket. Any lamp, phone charger or kettle works with it, because the socket keeps one promise: the same voltage, the same shape. SAM's promise is "give me a point or a box, and I give you a valid mask". Any other program that can produce a point can use SAM.

The gaze example shows this well. Smart glasses track where the wearer looks. That point goes to SAM as a prompt, and out comes the object the person looks at. Nobody trained SAM for glasses. The prompt is the only connection. Part 2 explained why the task was chosen to make this possible.

The five later papers below are also compositions, in a way. Each one keeps most of SAM and changes one piece.

The limits

From the paper§8page 12

What it says

"While SAM performs well in general, it is not perfect. It can miss fine structures, hallucinates small disconnected components at times, and does not produce boundaries as crisply as more computationally intensive methods that 'zoom-in'."

What it means

Three faults in one sentence. "Fine structures" are thin parts: spokes, wires, legs of an insect, strands of hair. "Hallucinates" means the mask contains small islands of pixels where no object is. "Boundaries as crisply" is about the exact line of the edge.

The last phrase names the cure. A "zoom-in" method looks at the picture once and finds the rough edge. Then it cuts out a small window around the edge at high resolution and looks again. That costs more computation, and SAM does not do it. SAM looks once.

All three faults have one cause that we met in Part 3. The mask decoder predicts the mask on a grid of 256 × 256 cells, and the final mask is that grid, enlarged by interpolation. For a 1024 × 1024 picture, one cell covers 4 × 4 pixels. Anything thinner than a cell has no cell of its own.

We tested the three faults with the released ViT-H and ViT-B models.

Fault 1: thin structures

A bicycle is the classic test: a wheel is mostly empty space with thin spokes. We took a photo from the COCO dataset. SAM got two prompts: a box around the whole bicycle, and one point on the rear tyre.

plain text

ViT-H: box [195, 8, 497, 388] -> 1 mask, predicted IoU 0.972, 61,124 pixels
ViT-H: point (482, 270) on the rear tyre -> mask 2 (the wheel), predicted IoU 1.003, 34,976 pixels; holes inside it: 6 (77 pixels)

Two copies of a photo of a blue bicycle on a city sidewalk, both overlaid with a pink mask. Left: a box around the bicycle; the mask covers the frame and both wheels as solid discs, including the road visible between the spokes. Right: one green point on the rear tyre; the mask is the whole rear wheel as a solid disc — SAM ViT-H on a COCO photo. Both masks paint each wheel as a solid disc: the road seen between the spokes becomes "bicycle". The wheel mask has only 6 tiny holes (77 pixels in total).

The predicted IoU is 0.972 for the box and 1.003 for the wheel: SAM is very sure. But look at the wheels. The road between the spokes is pink. To SAM, the wheel is a disc.

A photo cannot tell us how wrong this is, because nobody drew the exact correct mask for every spoke. So we drew a wheel ourselves, where we know every pixel. The picture is 1024 × 1024 pixels with a dark rim, a hub and 24 spokes. We drew it eight times, with spokes 1 to 16 pixels wide, and gave SAM a box around the wheel each time.

Two numbers grade each mask. Spoke recall is the share of spoke pixels that the mask covers. Gap fill is the share of the empty pixels between the spokes that the mask covers. The correct mask has a spoke recall of 1 and a gap fill of 0.

plain text

model   spoke width  spoke px   score  spoke recall  gap fill    IoU  best of 3 IoU
ViT-H          1 px     7,907   0.917         0.000     0.000  0.835          0.839
ViT-H          2 px    16,188   0.733         0.208     0.014  0.745          0.771
ViT-H          3 px    26,062   0.756         0.620     0.534  0.263          0.709
ViT-H          4 px    34,234   0.893         0.988     0.991  0.193          0.656
ViT-H          6 px    52,289   0.989         1.000     1.000  0.222          0.569
ViT-H          8 px    69,115   0.994         1.000     1.000  0.249          0.586
ViT-B          3 px    26,062   0.871         0.506     0.046  0.681          0.708
ViT-B          4 px    34,234   0.915         0.888     0.402  0.358          0.681
ViT-H: widths with spoke recall > 0.9 and gap fill < 0.1 (the correct shape): none
ViT-B: widths with spoke recall > 0.9 and gap fill < 0.1 (the correct shape): none

Ten drawn wheels in two rows, ViT-H on top and ViT-B below, with spokes 1, 2, 3, 4 and 8 pixels wide. With 1 and 2 pixel spokes only the rim is pink. With 3 and 4 pixel spokes the inside is partly pink in a blotchy pattern. With 8 pixel spokes the whole disc is pink — The drawn wheel. Thin spokes vanish (only the rim is masked). Thick spokes turn the wheel into a solid disc. In between, the mask is a blotchy mix. No width gives spokes without gaps.

Spoke recall and gap fill against the spoke width, for ViT-H and ViT-B. All four lines stay near 0 for thin spokes and jump to near 1 between 3 and 8 pixels. The correct answer (recall 1, gap fill 0) never appears.

The result has two sides, and both are wrong. With spokes of 1 or 2 pixels, SAM does not see them: recall 0.000 and 0.208. With spokes of 6 pixels or more, SAM sees them and fills the whole disc: gap fill 1.000. The switch happens at 3 to 4 pixels, which is one cell of the 256 × 256 grid.

That is the grid at work. A gap between two spokes is wide near the rim and narrow near the hub. Near the hub, a cell holds spoke and gap together, and it can only say "in" or "out". The decoder learned from SA-1B that a wheel is usually one object, so it says "in" everywhere.

The IoU column hides all this. For 1-pixel spokes, the IoU is 0.835, a fine score, because the rim and the hub are most of the pixels. A good average can hide a total failure on the thin parts. The column "best of 3" gives the best of the three ambiguity masks; none of them is correct either.

ViT-B is closer at 3 pixels: recall 0.506 with a gap fill of only 0.046. That is not a general win for the small model. It only shows that the two models switch at slightly different widths.

What this test cannot show: a drawn wheel on a flat grey ground is not a photo. Real spokes have shadows, blur and a busy background. But the bicycle photo shows the same solid disc.

Fault 2: small islands

The second fault is "small disconnected components". A connected component is a group of mask pixels that touch each other (Part 4 defines it fully). A mask of one cat should be one component. Extra small islands are pixels that SAM marks as "object" far from the object.

We clicked a regular grid of 16 × 16 = 256 points on each of two photos, the bicycle and the two cats of Part 1. For each click we kept the mask with the highest predicted IoU and counted its components. An island under 100 pixels that is not the main piece is a "stray piece".

plain text

bike ViT-H: 256 prompts, 152 masks with more than one piece (59%), 145 with stray pieces under 100 pixels (57%), 1051 such pieces in total, median size 5 pixels
cats ViT-H: 256 prompts, 154 masks with more than one piece (60%), 87 with stray pieces under 100 pixels (34%), 1195 such pieces in total, median size 4 pixels
cats ViT-B: 256 prompts, 210 masks with more than one piece (82%), 209 with stray pieces under 100 pixels (82%), 9212 such pieces in total, median size 4 pixels

Two photos with pink masks and yellow rings. Top: the cats photo, a click on the blanket at the left; the blanket mask is pink, and a dense row of yellow rings marks hundreds of tiny islands on the red cloth at the top. Bottom: the bicycle photo, a click on the sidewalk; the sidewalk mask is pink, and many yellow rings mark small islands on the bricks around the tree — Stray pieces in confident ViT-H masks. Top: a click on the blanket gives a mask with 395 islands under 100 pixels on the red cloth (predicted IoU 0.959). Bottom: a click on the sidewalk gives 57 islands on the bricks (predicted IoU 0.924).

The pictures show where the islands come from. They sit on textures that look a little like the clicked surface: the red cloth beside the blanket, the bricks beside the sidewalk. The logit there is just above 0, and the threshold turns each lucky pixel into an island. The median island has 4 or 5 pixels.

The paper knew this. Appendix B estimates that "4% of masks include small, spurious components" in the automatic stage of the data engine. It removes islands under 100 pixels and fills holes under 100 pixels. It also keeps only masks with a predicted IoU above 0.88 and a high stability score (Part 4 explains both). We applied the two filters to our masks:

plain text

bike ViT-H: only masks with predicted IoU > 0.88: 231 masks, 125 with stray pieces under 100 pixels (54%)
   and also stable (IoU of the masks at logit +1 and -1 >= 0.95): 153 masks, 57 with stray pieces under 100 pixels (37%)
cats ViT-B: only masks with predicted IoU > 0.88: 224 masks, 182 with stray pieces under 100 pixels (81%)
   and also stable (IoU of the masks at logit +1 and -1 >= 0.95): 37 masks, 3 with stray pieces under 100 pixels (8%)

The share of masks with stray pieces under 100 pixels, for both photos and both models: all masks, then masks with predicted IoU above 0.88, then masks that are also stable. The filters lower the share, most for ViT-B, but they never remove it.

The stability filter helps most, because an island of 4 pixels with a logit near 0 is exactly what "unstable" means. For ViT-B on the cats it removes most masks: 37 of 224 survive. The predicted IoU alone barely helps. SAM gives a mask full of islands a score of 0.959.

Our 37% to 8% is far above the paper's 4%, and the two numbers are not comparable. The paper counted on the original photos, about 1500 pixels on the short side, with a special SAM trained for automatic generation. We counted on photos of 640 pixels, with raw single clicks and no crops. The lesson is the same: islands are common in raw masks, and a short clean-up step removes them.

Fault 3: edges that are not crisp

The third fault is the exact line of the edge. We zoomed into the tyre of the bicycle, 64 × 64 pixels, and drew four views of the same place:

Four zoomed views of the same 64 by 64 pixel patch at the edge of the bicycle tyre. First: the photo, dark tyre with knobbly tread on a grey road. Second: the 256 by 256 low-resolution mask as square blocks. Third: the final mask, almost the same shape with slightly smoother steps. Fourth: the probability map, white inside and black outside, with a thin grey edge — The tyre edge in four views. The final mask (third) is the 256 × 256 grid (second) with its steps smoothed; it adds no new detail. The probability (fourth) is uncertain only in a thin band at the edge.

plain text

ViT-H: low-resolution logits (256, 256); one cell covers 2.50 x 2.50 photo pixels
ViT-H: wheel mask edge 1,060 pixels long; pixels with probability between 0.1 and 0.9: 3,768; mean width of the soft band 3.55 pixels
ViT-H: the final mask and the 256 x 256 cell mask (blown up without interpolation) differ in 648 pixels, 1.9% of the mask area

The photo is 640 pixels wide, so one cell covers 2.5 × 2.5 photo pixels. The final mask and the blocky cell mask differ in only 1.9% of the mask's pixels. Interpolation rounds the corners of the blocks, but it cannot add a knob of the tread that the grid did not have.

The "soft band" is the strip where SAM is unsure: the probability is between 0.1 and 0.9. We divide its area by the length of the edge to get its average width. It is 3.55 pixels for ViT-H and 8.54 for ViT-B. Inside that band, the exact position of the line is a guess.

To measure crispness with a known answer, we drew a dark block with teeth on its top edge, 16 pixels high. The teeth and the gaps between them have the same width $t$ . We graded tooth recall and gap fill, as for the spokes:

plain text

model       t  tooth recall  gap fill   edge error (no teeth)
ViT-H    none                                         2.27 px
ViT-H     1 px         0.968     0.968
ViT-H     2 px         0.986     0.989
ViT-H     4 px         0.993     0.985
ViT-H     8 px         0.916     0.469
ViT-H    16 px         0.384     0.011
ViT-H    32 px         0.807     0.004

Four zoomed crops of a dark block with teeth on top, overlaid with the ViT-H mask in pink. Teeth of 2 and 4 pixels: the mask is a straight band that covers teeth and gaps alike. Teeth of 8 pixels: the mask follows the teeth with rounded corners and partly fills the gaps. Teeth of 16 pixels: the mask covers only a low rounded bump inside each tooth — The edge with teeth, ViT-H. Up to 4 pixels the mask draws a straight line over the tips. At 8 pixels it starts to follow the teeth, with round corners. At 16 pixels it cuts the teeth down to rounded bumps.

Tooth recall and gap fill against the tooth width. Up to 4 pixels both are near 1: the mask is a straight line over the tips. At 8 pixels the mask starts to follow the teeth. At 16 pixels it keeps only 38% of the tooth pixels (ViT-H).

Up to 4 pixels, the mask is a straight line over the tips: recall and gap fill both near 1. At 8 pixels, two cells per tooth, the mask starts to follow the teeth. Even at 32 pixels, the mask misses 19% of the tooth pixels, mostly at the sharp corners. Even a straight edge with no teeth is off by 2.27 pixels on average.

A zoom-in method would fix this. It would cut a window around the edge from the full photo and run a second, sharper pass on it. SAM chose one pass for speed.

Many clicks, and the heavy encoder

Our script counted the weights of the two released models:

plain text

ViT-H: 641,090,864 weights, of which image encoder 637,026,048, prompt encoder + mask decoder 4,064,816
ViT-B: 93,735,728 weights, of which image encoder 89,670,912, prompt encoder + mask decoder 4,064,816
ViT-H has 7.1 x the encoder weights of ViT-B

The decoder is the same size in both. Only the encoder changes. So does the small encoder give the same masks? We compared ViT-B with ViT-H on the same 256 clicks per photo, with ViT-H's top mask as the reference:

plain text

bike, ViT-B top mask: 256 prompts, mean IoU with the ViT-H top mask 0.753, median 0.915, 54% above 0.9, 18% below 0.5
bike, closest of 3 ViT-B masks: 256 prompts, mean IoU with the ViT-H top mask 0.884, median 0.944, 69% above 0.9, 5% below 0.5
cats, ViT-B top mask: 256 prompts, mean IoU with the ViT-H top mask 0.810, median 0.936, 58% above 0.9, 11% below 0.5

The median is high: the typical click gets nearly the same mask. The mean is lower, because 18% of the bicycle clicks get a very different mask. Most of those are ambiguity, not failure. When we let ViT-B pick the closest of its three masks, only 5% stay below 0.5. The small model often finds the same object, but ranks the part and the whole in another order.

Part 5 reads Figure 13 of the paper: ViT-H gains a lot over ViT-B, but little over ViT-L. Its caption adds that "smaller image encoders may be preferred in certain settings". Those settings are the phone and the video, and two later papers took them up.

Text, semantics and special domains

From the paper§8page 12

What it says

The text-to-mask work "is exploratory and not entirely robust". "It is unclear how to design simple prompts that implement semantic and panoptic segmentation." And there are "domain-specific tools", such as a tool for microscopy, "that we expect to outperform SAM in their respective domains".

What it means

Three more limits. The text prompt worked as a proof of concept in Part 5, and it was not released. Semantic segmentation needs a class name for every pixel, but SAM never names anything: a prompt asks for "the thing here", not "all the sky".

The last limit is the honest one for science. A tool trained for years on cell pictures knows things about cells that SA-1B never showed. SAM is a generalist, and a generalist loses to a specialist on the specialist's home ground.

The conclusion

Five limits of Section 8 and five later papers. SAM 2 adds a memory for video. MobileSAM trains a small encoder to copy ViT-H. HQ-SAM adds one output token for sharper masks. MedSAM fine-tunes SAM on medical images. SAM 3 finds all instances of a noun phrase.

After SAM: five papers

Video: SAM 2

From the paperRavi et al. (2024), SAM 2Abstractpage 1

What it says

SAM 2 is "a foundation model towards solving promptable visual segmentation in images and videos". It is "a simple transformer architecture with streaming memory for real-time video processing". On images, it is "more accurate and 6× faster than the Segment Anything Model (SAM)".

What it means

A video is a stack of pictures. SAM could cut an object out of each frame, but it would need a new click on every frame. It would also forget which object you meant. SAM 2 adds memory: you click once, and the model follows the object through the video.

Notice the same recipe as SAM: a task (prompts on any frame), a model, and a data engine that built a large video dataset. The SAM 2 paper says that a single image is treated as "a single-frame video". So SAM 2 also replaces SAM on photos.

From the paperRavi et al. (2024), SAM 2§4page 5

What it says

The memory bank keeps "a FIFO queue of memories of up to N recent frames" and a queue of "up to M prompted frames". It also stores "object pointers", small vectors that summarise the object.

What it means

Think of a guard who watches one person in a crowd. The guard remembers the last few seconds (the recent frames) and the photo on the poster (the frames where you clicked). On each new frame, the model looks at the new picture and asks the memory: "which of these regions looks like what I followed so far?"

SAM's decoder already used cross-attention to read the image embedding (Part 3). SAM 2 adds memory attention before the decoder, so the frame's embedding first reads the memory. The same paper adds a head that predicts whether the object is visible at all. In a video, the object can walk out of the picture.

A small encoder: MobileSAM

Sharper masks: HQ-SAM

From the paperKe et al. (2023), HQ-SAMAbstractpage 1

What it says

HQ-SAM adds "a learnable High-Quality Output Token" to SAM's mask decoder. It fuses the decoder's features "with early and final ViT features". It trains on "44K fine-grained masks", in "4 hours on 8 GPUs".

What it means

Part 3 showed that each mask comes from one output token. The token becomes 32 numbers, and a dot product with each pixel's 32 numbers gives the mask. HQ-SAM adds one more token of the same kind. SAM's weights stay frozen; only the new token and a few small layers learn.

The "early ViT features" are the important part for thin structures. Early blocks of the encoder still see edges and small details; late blocks see whole objects. HQ-SAM brings the early features back into the mask, at the same 256 × 256 grid as before. On the DIS dataset, the paper reports that the boundary quality rises from 52.8 to 70.4 mBIoU. That is the IoU measured only in a thin band along the edges.

The 44,000 training masks are tiny next to SA-1B's 1.1 billion. But the paper calls them "extremely accurate", taken from six existing datasets. A small set of very exact masks taught what a billion good masks did not.

Medicine: MedSAM

From the paperMa et al. (2024), MedSAMMethodspage 12

What it says

MedSAM starts from "the pre-trained SAM model with the ViT-Base model". The trainable parameters of the image encoder and mask decoder are "89,670,912 and 4,058,340". The box prompt gets "a random perturbation of 0-20 pixels".

What it means

Compare the first number with our weight count above: ViT-B's image encoder has 89,670,912 weights. It is the same network. MedSAM chose the small encoder, and it uses box prompts. Its paper argues that a box gives a less ambiguous target than a point.

The random shift of the box copies SAM's own training game from Part 3. There, boxes got noise of 10% of the side, at most 20 pixels. Fine-tuning means to continue the training of a trained model on new data: here, SAM's weights are the start, and medical images teach the rest.

Concepts: SAM 3

From the paperCarion et al. (2025), SAM 3Abstractpage 1

What it says

SAM 3 segments objects "based on concept prompts", which are "short noun phrases" such as "yellow school bus", image examples, or both. It returns masks for "all matching object instances". Its data engine produced "4M unique concept labels".

What it means

SAM answers "what is at this point?" SAM 3 answers "where are all the yellow school buses?" One click gives one object. One phrase gives every object that matches it, each with its own mask.

This is the answer to two limits of Section 8 at once. Text prompts are now a main feature, trained on millions of phrases, not a proof of concept. And "all instances of a concept" is close to the semantic and instance segmentation that SAM could not express with a click. The SAM 3 paper says that it predicts "instance and semantic masks".

The whole paper on one page

Six parts ago, we started with a photo, a click and three masks. Here is the paper again, in one picture and a few paragraphs.

The paper in one picture: the task (Part 2), the model (Part 3), the data engine and SA-1B (Part 4) and the zero-shot experiments (Part 5). The model makes the data, the data trains the model, and the prompt lets one model do many tasks.

The task (Part 2). Give the model any prompt, a point, a box, a rough mask or text. It must return a valid mask. If the prompt is ambiguous, any sensible object counts, so the model returns three masks: a part, a larger part and the whole. Language models learned from "predict the next word"; SAM learns from "predict the mask for this prompt".

The model (Part 3). A heavy ViT-H encoder reads the picture once and writes a 256 × 64 × 64 embedding. A light prompt encoder and mask decoder read that embedding for each click, fast enough for a person to click and see. The split into "once per picture" and "once per click" is the economic idea of the whole design.

The data engine and the dataset (Part 4). No dataset of the right size existed, so the team built one with the model in the loop. Annotators first clicked with SAM's help, then corrected SAM's automatic masks, and at the end SAM made masks alone. The result, SA-1B, has 11 million photos and 1.1 billion masks.

The experiments (Part 5). With no extra training, SAM was tested on 23 datasets and on other tasks: edges, object proposals, instance segmentation and text. It does not win every score, but people rate its masks highly, and one model covers all the tasks through prompts.

The limits (this part). SAM misses thin structures, makes small islands, draws edges at the resolution of a 256 × 256 grid and needs a heavy encoder. The later papers kept SAM's idea and fixed one limit each.

And the Background told where all this came from. Before SAM, there were two roads. Some models found and cut out a fixed list of things. Interactive tools cut out whatever you clicked. SAM joined the two roads, and it is still a central reference for the work that followed.

Key takeaways

SAM is a foundation model with a limited scope. It learned most of its skill from 1.1 billion labelled masks, not from self-supervision; the data engine made that possible.
Composition is the point of the prompt. Any system that can produce a point or a box can use SAM, with no training: glasses that track the eye, 3D tools, later models.
Thin structures fail both ways. On our drawn wheel, ViT-H misses spokes of 1 and 2 pixels (recall 0.000 and 0.208). For spokes of 6 pixels or more, it fills the whole disc (gap fill 1.000). No width gave the correct shape.
Islands are common in raw masks. 34% to 82% of our raw single-click masks had stray pieces under 100 pixels. The paper's filters and clean-up exist for this reason.
Edges follow a 256 × 256 grid. Teeth up to 4 pixels wide vanish under a straight line; teeth of 16 pixels keep only 38% of their pixels. The final mask differs from the blocky grid mask in 1.9% of its pixels.
The heavy encoder is the bottleneck. The decoder has 4.06 million weights in both models; the encoder has 637 million (ViT-H) or 89.7 million (ViT-B).
SAM 2 adds memory for video and is 6 times faster on images; MobileSAM copies ViT-H into a 5.78M encoder; HQ-SAM adds one output token for sharper masks; MedSAM fine-tunes ViT-B SAM on 1.57 million medical image-mask pairs; SAM 3 finds all instances of a noun phrase.
The paper's own test still holds: a foundation model is defined by its use. The later papers above are that use.

Run it yourself

The script code/papers/sam/sam_part6.py runs ViT-H and ViT-B on the bicycle photo, which it downloads from COCO. It draws the spoked wheel and the edge with teeth. It counts the islands in 512 masks per model, measures the soft band at the tyre edge and compares the two models. Every number of this part is in results/part6.json. We ran it on a laptop GPU (Apple MPS); no timing numbers are used, because other jobs ran at the same time.

bash

pip install torch transformers datasets pillow scipy
python sam_part6.py        # uses facebook/sam-vit-huge and facebook/sam-vit-base

Terminal output of sam_part6.py: the weights of the two models, the bicycle masks, the table of the drawn wheel with spoke recall and gap fill for each width, and the counts of stray pieces — The first part of the output of sam_part6.py, as it ran for this article.

The paper and the code

Segment Anything (ICCV 2023) · code and weights · our scripts for this part

After SAM: Impact, Limits and Summary

Foundation models, once more#

Composition: SAM as a part of a bigger system#

The limits#

Fault 1: thin structures#

Fault 2: small islands#

Fault 3: edges that are not crisp#

Many clicks, and the heavy encoder#

Text, semantics and special domains#

The conclusion#

After SAM: five papers#

Video: SAM 2#

A small encoder: MobileSAM#

Sharper masks: HQ-SAM#

Medicine: MedSAM#

Concepts: SAM 3#

The whole paper on one page#

The paper and the code#

Foundation models, once more

Composition: SAM as a part of a bigger system

The limits

Fault 1: thin structures

Fault 2: small islands

Fault 3: edges that are not crisp

Many clicks, and the heavy encoder

Text, semantics and special domains

The conclusion

After SAM: five papers

Video: SAM 2

A small encoder: MobileSAM

Sharper masks: HQ-SAM

Medicine: MedSAM

Concepts: SAM 3

The whole paper on one page

The paper and the code