The short answer
Removing duplicates is incomplete as an instruction until the fields defining equality are specified. The selected technical note and manual distinguish two operations that can legitimately return different row counts.
The records answer different questions
Keep the measures separate
The question changes when the object, unit or reporting period changes.
| Record or measure | What it establishes | What it does not establish |
|---|---|---|
| Duplicate key | Equality on selected ID fields. | Equality on every measurement. |
| Adjacent full-row comparison | Equality against the preceding output row. | An all-pairs duplicate search. |
| Distinct complete rows | Equality on every supplied field. | One row per ID regardless of other fields. |
Chronology
A timeline of the turning points
State the sequence
Ordering assumptions are part of the task.
State the equality rule
The retained rows depend on the chosen fields.
Equality depends on the selected fields
The Base SAS 9.3 manual says NODUPKEY removes observations sharing BY-variable values. EQUALS preserves relative input order for equal sort keys; NOEQUALS need not preserve that order.
A record with the same customer identifier but a different measurement is a duplicate key without necessarily being a duplicate full row. Keeping one record per key can therefore discard information that a full-row comparison would retain. The intended information unit must be decided before an option is chosen.
Sources for this section: [1]
A technical note corrects an intuitive model
Usage Note 1566 explains that NODUPRECS compares all variable values only with the previous observation written to output. It does not compare each observation with every other observation. The note recommends NODUPKEY with BY _ALL_ or SQL DISTINCT for removing all identical full observations.
Adjacency is the decisive detail. If two equal full rows are separated by a different row after sorting, a one-step comparison can leave both copies. This is a property of the specified procedure, not proof that one of the records is uniquely meaningful.
Sources for this section: [2]
Stable sorting needs an input order
The manual cautions that a stable sort cannot repair input whose order is not repeatable, such as some database retrievals.
The task below supplies a physical row sequence and assumes stable ordering within each key. Those assumptions let the reader reproduce the result. An unspecified retrieval order would make โkeep the first rowโ an incomplete policy. No claim about a particular production database follows from this small example.
Sources for this section: [1]
Original five-row trace
Invented input, in order: (A,10),(A,11),(A,10),(B,20),(B,20). Assume a stable Base-SAS-like sort with EQUALS and no host-specific exception. Trace three operations: NODUPKEY by ID; NODUPRECS after sorting only by ID; and NODUPKEY by every field. Which one answers โone record per IDโ, and which answers โdistinct complete rowsโ?
Reference answer: key-only deduplication leaves (A,10),(B,20): two rows. Adjacent full-row comparison leaves (A,10),(A,11),(A,10),(B,20): four rows, because the two A,10 copies are separated. Sorting by every field and retaining distinct complete rows leaves (A,10),(A,11),(B,20): three rows. The first answers the key question; the last answers full-row equality. These supplied algorithms were checked in Python, not executed in SAS.
Continue the chronology
Related history guides
Biography context
Explore Worldwide through connected lives
Connected biographies
The connected careers
Quick answers
Common questions
- Is one record per ID the same as distinct full rows?
- No. Different measurements under the same ID remain distinct full rows even when a key-only operation keeps only one.
Primary and institutional sources
How this history was checked
Compare the dated company records with the independent or regulatory context. Original task inputs are labelled separately from reported historical measures. This article was reviewed on 31 October 2026.