Data 360 · Data Model27 tasks
Configure Data 360 Batch Transforms
Build or rework Data 360 batch transforms so ingested data lands, reshaped, in the target data lake or data model objects, ready for identity resolution, data graphs and insights.
Method
How MeshMesh does it
- Researches the source data model object schemas before building.
- Builds or reworks the transform. In one case it joined many prediction outputs into one row per record. In another it added a field to data lake objects, then deleted and recreated the transforms, one with the new formula inlined and one passing it through.
- Fixes the target object so the transform can write to it. This meant creating missing fields through the Tooling API and repairing the primary key mapping.
- Runs the transform and checks the output records.
- When a transform shows data in preview but writes no records, it diagnoses the problem read-only. In one case the cause was escaping in formulas that the preview engine and the Spark SQL batch engine handle differently. MeshMesh then refined the formulas with the user and proposed rebuilding them with instr and concat.
Products
Products and features it works with
Bars show how many of this capability's tasks touched each product.
Features