Data quality · build 2 October 2026
Every gate between a record and a number, with its count
A benchmark is only worth as much as the rule that decided what went into it. This page is the whole rule set: what came in, what each gate kept and excluded, what we kept but flagged, and what we cannot measure at all. Generated by the same build as the site, and served as JSON at /api/v1/quality with no key.
The gates
| Stage | The rule | Kept | Excluded | Changed |
|---|---|---|---|---|
| corpus | Every record the merged corpus and the watcher feed hold.Nothing is deleted anywhere in this pipeline; records are flagged and excluded from the counts they do not belong in. | 4,045 | 0 | 0 |
| developer names | One company spelled several ways is one developer.40 name groups merged, 92 records renamed. Before this, one builder could hold two pages and split its own price median. | 4,045 | 0 | 92 |
| duplicate projects | The same building listed twice under different spellings is one project.121 groups merged; 4 candidate groups left separate because their coordinates disagreed. | 4,045 | 0 | 121 |
| delivered buildings | A building that has handed over is not an off-plan project.Delivered buildings keep their record and their page and are counted separately. They are out of the headline count and out of every price benchmark. | 2,962 | 1,083 | 0 |
| unit records | A unit counts when it has a positive price and a plausible size for its bedroom count.36 kept records carry a size flag and are excluded from the benchmark set. | 59,806 | 9,069 | 0 |
| benchmark units | Only validated unit records with a price per sqft inside the plausible range enter a median.Every price per sqft figure on the site and in the licence is over this set. | 52,882 | 6,924 | 0 |
| payment plans | A plan is priced only when its steps sum to between 95 and 105 percent.A plan summing to 60 percent is a scrape failure, not a payment plan, and is dropped rather than rescaled. | 4,143 | 16 | 0 |
| published price | An off-plan project publishes a price when the listing states one.This is the count every page means by "priced". | 2,643 | 319 | 0 |
| payment plan priced | A priced off-plan project also gets a payment-plan-adjusted price when it has at least one qualifying plan.1888 of those carry a high or medium confidence grade and are the only ones published in a ranking. Delivered buildings are priced too but are out of every headline count, so they are not counted here. | 2,140 | 503 | 0 |
Excluded means excluded from the counts that gate governs, never deleted. A delivered building keeps its page and its record; it is out of the off-plan headline and out of every price benchmark.
What gets a page, and what gets into search
| Kind | Tracked | With a page | In the sitemap | Page, not indexed | No page |
|---|---|---|---|---|---|
| Developers | 681 | 392 | 285 | 107 | 289 |
| Communities | 231 | 168 | 145 | 23 | 63 |
A page exists when n_projects >= 2 or n_units >= 10 (developers: name not Unknown; communities: has a city). It enters the sitemap when n_projects >= 3 or n_units >= 25; below it a page exists with robots noindex, follow, and is not in the sitemap. A record below the first gate is still in the index, in every count and in the data licence; it has no page of its own because a page with three fields on it is not worth a reader’s click. A record between the gates has a page for anyone who reaches it and stays out of search, so a thin page never outranks a full one.
Records we kept and flagged
| Flag | Records |
|---|---|
| Unit size below the plausible floor for its bedroom count, so it is out of the benchmark setsize_below_floor | 24 |
| Unit size above the plausible ceiling for its bedroom count, so it is out of the benchmark setsize_above_ceiling | 12 |
| Flag | Records |
|---|---|
| The stated handover date has already passed, so there is no schedule left to discounthandover_past | 1,107 |
| The plan has a post-handover tail with no stated length, assumed at 24 months and graded mediumtail_assumed_24m | 113 |
| The listing advertises a discount whose terms are not published, so the discount is not modelleddiscount_plan_ignored | 56 |
| Both of the above on one projecthandover_past,discount_plan_ignored | 21 |
| No handover date is stated, so the schedule cannot be placed in time and the discount is not computedhandover_unknown | 17 |
What we cannot measure
- Prices are developer asking prices as published, never registered sale prices. The Dubai Land Department transaction file is not open data and we do not hold it.
- A project carries one stated handover date per read. A delivery rate cannot be computed from one read; where two dated reads exist, revisions between them are published separately with their sample sizes.
- Coverage follows the portals we read. A project no portal lists is not here.
- Communities are assigned one home emirate each; a community straddling a border is filed where most of its projects sit.
Freshness is three dates, never averaged into one: sources read 30 July 2026, corpus merged 18 August 2026, these tables built 2 October 2026, with 79 records read by the listings watcher to 2026-09-30. How every figure is derived.
Questions
Is anything deleted?
No. Nothing is deleted anywhere in the pipeline. A record that fails a gate keeps its row and usually keeps its page; it is excluded from the counts it does not belong in, and this page says which.
Why is the published project count lower than the record count?
4,045 records become 2,962 off-plan projects because 1,083 of them are buildings that have already handed over and are counted separately, and because duplicate listings of one building are merged into one project.
Can I check these numbers?
Every figure here is generated by the same build that generates the site and is served as JSON at /api/v1/quality with no key. Nothing on this page is typed by hand.
What is a flag?
A record we kept but marked, because one field is implausible or because a derived figure had to assume something. Flagged records stay in the index and are excluded from the specific number the flag concerns.
Every count here is produced by pipeline/quality.py from data/offplan.db and the build log, on the same build as the site. No figure on this page is typed by hand.