Skip to content

Add FOCUS 1.4-preview ingestion (IngestionSetup_v1_4.kql)#2132

Draft
flanakin wants to merge 4 commits into
flanakin/focus14from
flanakin/focus14-phase4-ingestion-1.4
Draft

Add FOCUS 1.4-preview ingestion (IngestionSetup_v1_4.kql)#2132
flanakin wants to merge 4 commits into
flanakin/focus14from
flanakin/focus14-phase4-ingestion-1.4

Conversation

@flanakin
Copy link
Copy Markdown
Collaborator

@flanakin flanakin commented May 6, 2026

Summary

Phase 4 of FOCUS 1.4 work (parent: #2120). Adds preview-grade FOCUS 1.4 ingestion. FOCUS 1.4 is still in working_draft so this schema is labeled preview and may change before the spec ratifies.

Stacked on phase 3 (#2130).

Files

  • NEW: `IngestionSetup_v1_4.kql` (copied from v1_3 with renames + 1.4 deltas).
  • MODIFIED: `IngestionSetup_RawTables.kql` — `ContractCommitment_raw` gains the 14 new FOCUS 1.4 columns.
  • MODIFIED: `app.bicep` and `.build.config` — wires `IngestionSetup_v1_4.kql` into the deployment.

FOCUS 1.4 deltas applied

Cost and Usage

  • Removed `ProviderName` and `PublisherName` from `Costs_final_v1_4` table and projection (deprecated in 1.3, removed in 1.4).
  • Internal use of `ProviderName`/`PublisherName` retained inside the transform so `ServiceProviderName`/`HostProviderName` fall back to those values when source data still uses the old columns.

Contract Commitment (added 14 columns)

  • BenefitCategory, ContractCommitmentApplicability (JSON), Created, DiscountPercentage, DurationType, FulfillmentInterval, LastUpdated, LifecycleStatus, Model, OfferCategory, PaymentInterval, PaymentModel, PaymentUpfrontPercentage, PricingCurrencyContractCommitmentCost.
  • The transform uses `column_ifexists` for the new columns so partial 1.3-shaped source data still ingests cleanly.

Out of scope

  • HubSetup_v1_4.kql -> phase 5
  • New Invoice dataset (mentioned in 1.4 working_draft) -> deferred until spec stabilizes

Closes #2131

Test plan

  • Pester unit tests pass: 192/192 in HubsIngestionQueries.Tests.ps1
  • Bicep template compiles cleanly
  • Manual smoke test once Cost Management ships any FOCUS 1.4-preview export.

flanakin and others added 4 commits May 6, 2026 08:07
Adds IngestionSetup_v1_3.kql with Costs/Prices/CommitmentDiscountUsage/
Recommendations/Transactions transforms and final tables renamed for
FOCUS 1.3. Cost and Usage gains 8 new FOCUS 1.3 columns:
AllocatedMethodId/Details/ResourceId/ResourceName/Tags (data-generator
split cost allocation), ContractApplied (per-row contract commitment
application), ServiceProviderName + HostProviderName (replacing the
deprecated ProviderName/PublisherName, with empty-fallback for back
compat).

Costs_raw now carries the new columns so downstream v1_2 transforms
keep working and v1_3 transforms can read them directly. The v1_3
file is wired into the Bicep deployment alongside v1_0 and v1_2.

The unversioned Costs() function still aliases to v1_2 — phase 2
adds Costs_v1_3() and retargets Latest.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds HubSetup_v1_3.kql with Costs_v1_3, Prices_v1_3,
CommitmentDiscountUsage_v1_3, Recommendations_v1_3, and
Transactions_v1_3 that union the new Costs_final_v1_3 with the
existing Costs_final_v1_2 and Costs_final_v1_0 tables.

For v1_2 data unioned into the v1_3 view, the 8 new FOCUS 1.3 columns
default to null/empty and ServiceProviderName/HostProviderName are
populated from the deprecated ProviderName/PublisherName for back
compat. The same defaults apply to the v1_0 union arm on top of the
existing v1_0 -> v1_2 conversion.

HubSetup_Latest.kql aliases now point to *_v1_3 functions so the
unversioned Costs(), Prices(), etc. return the latest schema.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the FOCUS 1.3 Contract Commitment supplemental dataset:
- ContractCommitment_raw table in IngestionSetup_RawTables.kql with
  the 14 mandatory FOCUS 1.3 columns plus standard hubs source
  metadata.
- ContractCommitment_transform_v1_3 + ContractCommitment_final_v1_3
  in IngestionSetup_v1_3.kql, with update policy mapping raw to final.
- ContractCommitment_v1_3 hub function in HubSetup_v1_3.kql that
  reads the final table directly (no older versions to union).
- Unversioned ContractCommitment alias in HubSetup_Latest.kql.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds IngestionSetup_v1_4.kql as a preview schema for FOCUS 1.4
working_draft. Cost and Usage drops the deprecated ProviderName
and PublisherName columns (removed in 1.4 per spec). Contract
Commitment expands by 14 new columns: BenefitCategory,
ContractCommitmentApplicability (JSON), Created, DiscountPercentage,
DurationType, FulfillmentInterval, LastUpdated, LifecycleStatus,
Model, OfferCategory, PaymentInterval, PaymentModel,
PaymentUpfrontPercentage, PricingCurrencyContractCommitmentCost.

The internal ProviderName/PublisherName fix-up logic in the Costs
transform stays so HostProviderName / ServiceProviderName can fall
back to those values when source data does not yet supply them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@flanakin flanakin requested a review from MSBrett as a code owner May 6, 2026 15:39
Copilot AI review requested due to automatic review settings May 6, 2026 15:39
@microsoft-github-policy-service microsoft-github-policy-service Bot added the Needs: Review 👀 PR that is ready to be reviewed label May 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added the Tool: FinOps hubs Data pipeline solution label May 6, 2026
@flanakin flanakin changed the base branch from flanakin/focus14-phase3-contractcommitment to flanakin/focus14 May 6, 2026 23:00
@flanakin flanakin marked this pull request as draft May 6, 2026 23:16
@flanakin flanakin added this to the v15 milestone May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs: Review 👀 PR that is ready to be reviewed Tool: FinOps hubs Data pipeline solution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants