feat(taxa): per-taxon verification & agreement counts + verified filter (#1316)#1317
feat(taxa): per-taxon verification & agreement counts + verified filter (#1316)#1317mihow wants to merge 1 commit into
Conversation
…lter Adds to GET /api/v2/taxa/ (issue #1316): - verified_count and agreed_with_prediction_count annotations (always on), rolled up over descendant occurrences via a hierarchical parents_json match. - agreed_exact_count, gated behind with_agreement=true (and always on the detail view). - verified=true|false filter (EXISTS / strict complement), project-scoped and respecting apply_default_filters. - verified_count added to ordering_fields. The hierarchical descendant match uses a Postgres jsonb @> containment built from an OuterRef (literal __contains can't embed an OuterRef). Migration 0085 adds the supporting GIN index on Taxon.parents_json (jsonb_path_ops). Frontend: sortable "Verified" column + "Verification status" filter on the taxa list, and a Verification panel on the taxon detail page. Co-Authored-By: Claude <noreply@anthropic.com>
✅ Deploy Preview for antenna-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for antenna-ssec ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (11)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds per-taxon verification and model-agreement metrics to the taxa API and surfaces them in the taxa (“species”) UI, including a new verified filter and verified-count sorting/links.
Changes:
- Backend: annotate taxa with
verified_count+agreed_with_prediction_count, optionallyagreed_exact_count, and addverified=true|falsefiltering; add GIN index migration forparents_json. - Frontend: add “Verified” column, verified filter control, and taxon detail “Verification” panel; propagate
verifiedquery param via routing utils. - Tests/docs: add API tests covering counts, gating, filter complement behavior, ordering, and apply-defaults behavior; add planning doc.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| ui/src/utils/getAppRoute.ts | Adds verified to supported query filter keys for route generation. |
| ui/src/pages/species/species.tsx | Enables verified filter control and default column visibility. |
| ui/src/pages/species/species-columns.tsx | Adds sortable “Verified” column linking to verified occurrences for a taxon. |
| ui/src/pages/species-details/species-details.tsx | Displays verification/agreement counts on the taxon detail panel with a link to verified occurrences. |
| ui/src/data-services/models/species.ts | Exposes new server fields via getters (verified_count, agreement counts). |
| docs/claude/planning/2026-05-20-taxa-verification-guidance-ticket.md | Planning/spec documentation for the feature and performance considerations. |
| ami/main/tests.py | Adds TestTaxaVerification coverage for annotations, gating, filtering, ordering, and apply-defaults. |
| ami/main/models.py | Adds Taxon stub methods for new annotated fields to support serialization. |
| ami/main/migrations/0085_taxon_parents_json_gin_index.py | Adds concurrent GIN index on Taxon.parents_json for hierarchical containment performance. |
| ami/main/api/views.py | Implements hierarchical “under taxon” occurrence correlation, annotations, verified filter, and ordering field addition. |
| ami/main/api/serializers.py | Gates agreed_exact_count field on list responses unless with_agreement=true. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if not include_unobserved: | ||
| # Efficient EXISTS check that uses the composite index | ||
| qs = qs.filter(models.Exists(Occurrence.objects.filter(base_filter))) | ||
|
|
||
| qs = self.add_verification_data(qs, occurrence_filters, default_filters_q) | ||
|
|
| "created_at", | ||
| "updated_at", | ||
| "occurrences_count", | ||
| "verified_count", |
Implements #1316 — per-taxon verification / model-agreement data on the taxa list + detail, with matching UI.
Backend (
GET /api/v2/taxa/)verified_countandagreed_with_prediction_count— always-on annotations. Both roll up descendant occurrences (a Family/Order row aggregates its species), occurrence-weighted.agreed_exact_count— gated behindwith_agreement=true; always present on the detail view. Comparesoccurrence.determination_id(top human ID, already maintained) against the top machineClassification.taxon_id.verified=true|falsefilter —EXISTS/ strict complement, project-scoped, respectsapply_default_filters.verified_countadded toordering_fields./taxa/<id>/) returns all four counts unconditionally.The hierarchical descendant match reuses the
parents_jsoncontainment pattern from the occurrence-listtaxon=<id>filter, but the right-hand side is built from anOuterRefviajsonb_build_array(jsonb_build_object('id', <outer id>))— a literal__containslookup can't embed anOuterRef. Migration0085adds the supporting GIN index (parents_json jsonb_path_ops), createdCONCURRENTLY/IF NOT EXISTSso it co-exists with the same index if it lands separately via the #1307 follow-up.Frontend
verified=query param (reuses the existing occurrence-list filter component).Tests
ami.main.tests.TestTaxaVerification(9 tests): hierarchical rollup to ancestors, chosen-identification-onlyagreed_with_prediction, gatedagreed_exact_count(absent on list unlesswith_agreement),verifiedfilter + strict complement, ordering, andapply_defaultshandling. Existing taxon suite (47 tests) green, including the list query-count audit (the new annotations are inline correlated subqueries, so the statement count doesn't change).Performance — needs a decision before merge
Rough warm/cold timings for a 25-row page against a production DB copy, with the GIN index in place (numbers are measurements, the interpretation is mine):
verified=falseordering=verified_countwith_agreementSmall/mid meet the issue's targets. The largest project does not: the always-on count subqueries run ~8–9s cold, and
ordering=verified_countfalls off a cliff because it must compute the subquery for every taxon in the project before sorting, not just a page.This looks structural rather than a missing index. Directions worth discussing (ordered by effort):
occurrences_countpath.verified_countfor large projects, or back it with a denormalized column (the issue already lists "backfill counts onto a denormalized Taxon field" as a follow-up).agreed_exact_countdetail-only (already gated) and, if needed, add a/taxa/stats/verification/aggregate endpoint (the Endpoint for stats about verified occurrences #1307 fallback the issue calls out).I'd rather settle this than silently ship a list endpoint that times out on the biggest project. The correctness and UX are done; this is the open question.
Note on count semantics
occurrences_countstays direct-match (unchanged) whileverified_countrolls up descendants, per the ticket. For species rows (the common case) they're equivalent; for Family/Order rowsverified_countcan exceed the directoccurrences_count. Flagging in case we want them consistent.Test plan
🤖 Generated with Claude Code