Recreate Additional Referrals

Automatically create child referrals for referrals marked with "Multiple Individuals"

Overview

The Recreate Additional Referrals operation processes referrals that involve multiple individuals and creates separate referral records for each additional person. This ensures each individual has their own referral record while maintaining the relationship to the original (master) referral.

The operation:

Processes referrals with mag_additionalcontactids JSON data
Creates new referral records for each additional contact
Links child referrals to the master referral via mag_masterreferral
Copies all relevant referral data to child referrals
Clears exit data from child referrals (exits belong to master only)
Uses parallel processing for high performance

When to Use This Operation

Use This Operation When:

✅ After data migration where referrals have multiple individuals
✅ Referrals show "Multiple Individuals" but only one referral record exists
✅ You need to create individual referral records for each family member
✅ Reporting requires one referral per person (not one per household)
✅ Activities need to be linked to individual-specific referrals

Don't Use This Operation When:

❌ Referrals already have child referral records created
❌ mag_additionalcontactids field is empty/null
❌ You want to keep household-level referrals (not split them)
❌ Child referrals exist but are incorrect (use rebuild relationship operations instead)

Quick Start

Example 1: Preview Referral Recreation

Test to see how many child referrals will be created:

.\WhanauTahi.Xpm.Tooling.CLI.exe datamanipulation `
  --RecreateAdditionalReferrals `
  --EnvironmentUrl https://myorg.crm6.dynamics.com `
  --ClientId a1b2c3d4-e5f6-7890-1234-567890abcdef `
  --Secret YourClientSecretHere `
  --PreviewMode

Expected Output:

[2024-11-17 16:00:15.123] Starting additional referral recreation in PREVIEW MODE...
[2024-11-17 16:00:15.456] Counting referrals with additional contacts...
[2024-11-17 16:00:18.789] Found 1,240 parent referrals to process
[2024-11-17 16:00:19.012] Analyzing mag_additionalcontactids data...
[2024-11-17 16:00:22.345] PREVIEW: Would create 2,680 child referrals
[2024-11-17 16:00:22.678] PREVIEW: Average child referrals per parent: 2.16
[2024-11-17 16:00:22.901] Preview completed. No changes made.

Example 2: Create Additional Referrals

After verifying the preview, run for real:

.\WhanauTahi.Xpm.Tooling.CLI.exe datamanipulation `
  --RecreateAdditionalReferrals `
  --EnvironmentUrl https://myorg.crm6.dynamics.com `
  --ClientId a1b2c3d4-e5f6-7890-1234-567890abcdef `
  --Secret YourClientSecretHere

Expected Output:

[2024-11-17 16:05:22.123] Starting additional referral recreation...
[2024-11-17 16:05:22.456] Found 1,240 parent referrals to process
[2024-11-17 16:05:23.789] Creating pool of 50 service clients...
[2024-11-17 16:05:25.012] Processing referrals in parallel...
[2024-11-17 16:05:45.345] Progress: 500/1,240 (40.3%)
[2024-11-17 16:06:12.678] Progress: 1,000/1,240 (80.6%)
[2024-11-17 16:06:28.901] Progress: 1,240/1,240 (100.0%)
[2024-11-17 16:06:29.234] Additional referral recreation completed
[2024-11-17 16:06:29.567] Total parent referrals processed: 1,240
[2024-11-17 16:06:29.890] Total child referrals created: 2,680
[2024-11-17 16:06:29.123] Total time: 67.0 seconds
[2024-11-17 16:06:29.456] Average throughput: 40 child referrals/sec

How It Works

Processing Steps

Retrieve Parent Referrals: Queries mag_referral records where:
- mag_additionalcontactids is not null
- mag_masterreferral is null (not already a child referral)
Parse Additional Contacts: Extracts contact IDs from JSON in mag_additionalcontactids field
For Each Additional Contact:
- Retrieve contact details (name, family group, etc.)
- Create new referral record
- Copy all data from parent referral
- Set mag_contact to the additional contact
- Set mag_masterreferral to parent referral ID
- Clear exit data fields (mag_exitdate, mag_exitreason, mag_exitedby)
- Set state code and status code
Batch Processing: Uses ExecuteMultipleRequest for efficient creation
Parallel Execution: Processes multiple parent referrals concurrently (50 threads)

mag_additionalcontactids JSON Format

The operation expects this JSON structure:

[
  "12345678-1234-1234-1234-123456789012",
  "23456789-2345-2345-2345-234567890123",
  "34567890-3456-3456-3456-345678901234"
]

Each GUID represents a contact ID for an additional individual.

Field Mapping

Fields Copied from Parent to Child:

Service provider (mag_serviceprovider)
Referring organisation (mag_referringorganisation)
Referral date (mag_referraldate)
Source of referral (mag_sourceofreferral)
Priority (mag_priority)
Notes (mag_notes)
All custom fields

Fields Set Specifically for Child:

mag_contact → Additional contact ID
mag_familygroup → Additional contact's family group
mag_masterreferral → Parent referral ID
statecode → Active (0)
statuscode → Entered (809730008) or Active (809730002) based on parent

Fields Cleared for Child:

mag_exitdate → null
mag_exitreason → null
mag_exitedby → null

Note

Exit data belongs to the master referral only. Child referrals should be exited independently if needed.

Parameters

Required Parameters

`--RecreateAdditionalReferrals`

Flag to enable the operation.

Example:

--RecreateAdditionalReferrals

`--EnvironmentUrl` (-e)

The URL of your Dataverse environment.

Example:

--EnvironmentUrl https://myorg.crm6.dynamics.com

`--ClientId` (-c)

The Azure AD application (client) ID for authentication.

Example:

--ClientId a1b2c3d4-e5f6-7890-1234-567890abcdef

Optional Parameters

`--Secret` (-s)

The client secret for authentication. If omitted, uses the default secret.

Example:

--Secret YourClientSecretHere

`--PreviewMode`

Run in preview mode without creating referrals. Shows what would be done.

Example:

--PreviewMode

Tip

Always run with --PreviewMode first to verify the count of child referrals to be created.

Real-World Scenarios

Scenario 1: Post-Migration Referral Splitting

Situation: You migrated 5,000 referrals from a legacy system. In the old system, one referral could cover an entire household. You need to create individual referrals for each family member.

Solution:

# Step 1: Preview to verify count
.\WhanauTahi.Xpm.Tooling.CLI.exe datamanipulation `
  --RecreateAdditionalReferrals `
  -e https://prod.crm6.dynamics.com `
  -c a1b2c3d4-e5f6-7890-1234-567890abcdef `
  -s $env:DATAVERSE_SECRET `
  --PreviewMode

# Step 2: Review the log
$logFile = Get-ChildItem ".\Logs\ParallelProcessing_RecreateAdditionalReferrals_*" | Sort-Object LastWriteTime -Descending | Select-Object -First 1
Get-Content $logFile | Select-String "Would create"

# Step 3: If count looks correct, run for real
.\WhanauTahi.Xpm.Tooling.CLI.exe datamanipulation `
  --RecreateAdditionalReferrals `
  -e https://prod.crm6.dynamics.com `
  -c a1b2c3d4-e5f6-7890-1234-567890abcdef `
  -s $env:DATAVERSE_SECRET

# Step 4: Verify child referrals created
# Use Advanced Find in Power Apps:
# Entity: Referral
# Filter: Master Referral (Is Not Null)

Expected Outcome:

Each additional contact has their own referral record
Child referrals link back to parent via mag_masterreferral
Exit data remains only on parent referral
Activities can now be linked to individual-specific referrals

Scenario 2: Fix Incomplete Referral Creation

Situation: A previous run of this operation failed midway due to throttling. You have 800 parent referrals, but only 400 have child referrals created.

Solution:

The operation is idempotent - it checks if child referrals already exist before creating:

# Just run the operation again
.\WhanauTahi.Xpm.Tooling.CLI.exe datamanipulation `
  --RecreateAdditionalReferrals `
  -e https://prod.crm6.dynamics.com `
  -c a1b2c3d4-e5f6-7890-1234-567890abcdef `
  -s $env:DATAVERSE_SECRET

# The operation will:
# - Skip parent referrals that already have child referrals
# - Create child referrals for the remaining 400 parents

Expected Outcome:

All 800 parent referrals now have child referrals
No duplicate child referrals created
Operation completes successfully

Performance & Scalability

Processing Characteristics

Configuration	Value
Max Concurrent Threads	50
Records per Batch	10 parent referrals
Child Referrals per Batch	Variable (depends on mag_additionalcontactids)
Preview Mode Limit	100 parent referrals

Typical Performance

Parent Referrals	Avg Children/Parent	Total Children	Estimated Time	Throughput
100	2	200	15-30 seconds	30-60 children/sec
500	2	1,000	1-2 minutes	30-60 children/sec
1,000	2	2,000	2-4 minutes	30-60 children/sec
5,000	2	10,000	10-20 minutes	30-60 children/sec

Note

Performance depends on:

Number of additional contacts per parent referral
Network latency
Dataverse API throttling
Plugins/workflows triggered on referral creation

Troubleshooting

Issue: "No parent referrals found"

Cause: No referrals have mag_additionalcontactids populated.

Solution:

Verify referrals exist with additional contacts:

SELECT TOP 10 mag_referralid, mag_additionalcontactids 
FROM mag_referral 
WHERE mag_additionalcontactids IS NOT NULL 
AND mag_masterreferral IS NULL

Ensure data migration populated mag_additionalcontactids correctly

Issue: "Invalid JSON in mag_additionalcontactids"

Cause: Field contains malformed JSON.

Solution:

Check the JSON format:

SELECT mag_referralid, mag_additionalcontactids 
FROM mag_referral 
WHERE mag_additionalcontactids IS NOT NULL

Expected format: ["guid1","guid2","guid3"]
Fix malformed JSON manually or via script before running operation

Issue: "Contact not found"

Cause: Contact ID in mag_additionalcontactids doesn't exist.

Solution:

Verify contacts exist:

SELECT contactid FROM contact WHERE contactid = '<guid from JSON>'

If missing, create contacts before running operation
Or remove invalid GUIDs from mag_additionalcontactids

Issue: "Duplicate child referrals created"

Cause: Operation ran multiple times without checking for existing children.

Solution: This shouldn't happen as the operation checks for existing children. If duplicates exist:

Identify duplicate child referrals:

SELECT mag_masterreferral, mag_contact, COUNT(*) AS DuplicateCount
FROM mag_referral
WHERE mag_masterreferral IS NOT NULL
GROUP BY mag_masterreferral, mag_contact
HAVING COUNT(*) > 1

Delete duplicates manually (keep the newest):

-- Keep only the newest child referral per master+contact combination

Output & Logs

Log File Location

Logs\ParallelProcessing_RecreateAdditionalReferrals_2024-11-17_16-05-22-123.log

Log File Contents

Startup Phase:

[2024-11-17 16:05:22.123] Starting additional referral recreation...
[2024-11-17 16:05:22.456] Counting referrals with additional contacts...
[2024-11-17 16:05:23.789] Found 1,240 parent referrals to process
[2024-11-17 16:05:24.012] Creating pool of 50 service clients...
[2024-11-17 16:05:25.345] Service client pool created successfully

Processing Phase:

[2024-11-17 16:05:26.678] Processing parent referral: Family Smith Referral (12345678-1234-1234-1234-123456789012)
[2024-11-17 16:05:26.901] Found 3 additional contacts in mag_additionalcontactids
[2024-11-17 16:05:27.234] Creating child referral for contact: Jane Smith (23456789-2345-2345-2345-234567890123)
[2024-11-17 16:05:27.567] Creating child referral for contact: John Smith Jr (34567890-3456-3456-3456-345678901234)
[2024-11-17 16:05:27.890] Creating child referral for contact: Emily Smith (45678901-4567-4567-4567-456789012345)
[2024-11-17 16:05:28.123] Batch created 3 child referrals successfully
[2024-11-17 16:05:45.456] Progress: 500/1,240 (40.3%) - 35 children/sec
[2024-11-17 16:06:12.789] Progress: 1,000/1,240 (80.6%) - 38 children/sec

Completion Phase:

[2024-11-17 16:06:29.234] Additional referral recreation completed
[2024-11-17 16:06:29.567] Total parent referrals processed: 1,240
[2024-11-17 16:06:29.890] Total child referrals created: 2,680
[2024-11-17 16:06:29.123] Total time: 67.0 seconds
[2024-11-17 16:06:29.456] Average throughput: 40 child referrals/sec
[2024-11-17 16:06:29.789] Successful: 1,240 parents / 2,680 children
[2024-11-17 16:06:29.012] Failed: 0

Understanding the Results

Success Indicators:

✅ "Total child referrals created" matches expected count from preview
✅ "Failed: 0"
✅ No ERROR lines in log
✅ Child referrals visible in Dataverse with mag_masterreferral populated

Warning Signs:

⚠️ "Failed: X" where X > 0 - Review error messages
⚠️ "Invalid JSON" errors - Fix mag_additionalcontactids data
⚠️ "Contact not found" - Ensure contacts exist before running

Table of Contents

Recreate Additional Referrals

Overview

When to Use This Operation

Use This Operation When:

Don't Use This Operation When:

Quick Start

Example 1: Preview Referral Recreation

Example 2: Create Additional Referrals

How It Works

Processing Steps

mag_additionalcontactids JSON Format

Field Mapping

Note

Parameters

Required Parameters

`--RecreateAdditionalReferrals`

`--EnvironmentUrl` (-e)

`--ClientId` (-c)

Optional Parameters

`--Secret` (-s)

`--PreviewMode`

Tip

Real-World Scenarios

Scenario 1: Post-Migration Referral Splitting

Scenario 2: Fix Incomplete Referral Creation

Performance & Scalability

Processing Characteristics

Typical Performance

Note

Troubleshooting

Issue: "No parent referrals found"

Issue: "Invalid JSON in mag_additionalcontactids"

Issue: "Contact not found"

Issue: "Duplicate child referrals created"

Output & Logs

Log File Location

Log File Contents

Understanding the Results

See Also

Table of Contents

Recreate Additional Referrals

Overview

When to Use This Operation

Use This Operation When:

Don't Use This Operation When:

Quick Start

Example 1: Preview Referral Recreation

Example 2: Create Additional Referrals

How It Works

Processing Steps

mag_additionalcontactids JSON Format

Field Mapping

Note

Parameters

Required Parameters

--RecreateAdditionalReferrals

--EnvironmentUrl (-e)

--ClientId (-c)

Optional Parameters

--Secret (-s)

--PreviewMode

Tip

Real-World Scenarios

Scenario 1: Post-Migration Referral Splitting

Scenario 2: Fix Incomplete Referral Creation

Performance & Scalability

Processing Characteristics

Typical Performance

Note

Troubleshooting

Issue: "No parent referrals found"

Issue: "Invalid JSON in mag_additionalcontactids"

Issue: "Contact not found"

Issue: "Duplicate child referrals created"

Output & Logs

Log File Location

Log File Contents

Understanding the Results

See Also

`--RecreateAdditionalReferrals`

`--EnvironmentUrl` (-e)

`--ClientId` (-c)

`--Secret` (-s)

`--PreviewMode`