Recreate Family Groups

Automatically create family groups for individuals without existing groups

Overview

The Recreate Family Groups operation identifies individual contacts who don't belong to any family group (where mag_whanauid is null) and automatically creates new family groups for them. This ensures every individual has an associated family group, which is essential for proper data organization and reporting.

The operation:

Identifies contacts with mag_whanauid IS NULL
Creates a new mag_familygroup record for each individual
Creates mag_familygroupmembership linking the contact to the new group
Copies address data from contact to family group
Updates contact's mag_whanauid to point to the new family group
Processes in parallel (50 threads) for performance

When to Use This Operation

Use This Operation When:

✅ After data migration where family groups weren't imported
✅ New contacts created without family groups
✅ mag_whanauid is null for many contacts
✅ You need to ensure every individual has a family group
✅ After bulk contact imports that skipped family group creation
✅ Fixing data quality issues where groups are missing

Don't Use This Operation When:

❌ All contacts already have family groups (mag_whanauid is populated)
❌ Contacts should share existing family groups (use membership creation instead)
❌ You want to manually control family group creation
❌ Testing in production without preview mode first

Quick Start

Example 1: Preview What Will Be Created

.\WhanauTahi.Xpm.Tooling.CLI.exe datamanipulation `
  --RecreateFamilyGroup `
  --EnvironmentUrl https://myorg.crm6.dynamics.com `
  --ClientId a1b2c3d4-e5f6-7890-1234-567890abcdef `
  --Secret YourClientSecretHere `
  --PreviewMode

Expected Output:

[2024-11-17 18:00:15.123] Starting family group recreation (PREVIEW MODE)...
[2024-11-17 18:00:15.456] Retrieving contacts without family groups (mag_whanauid IS NULL)...
[2024-11-17 18:00:18.789] Found 1,234 contacts without family groups
[2024-11-17 18:00:19.012] PREVIEW: Would create 1,234 new family groups
[2024-11-17 18:00:19.345] Sample contacts to process:
[2024-11-17 18:00:19.678]   - John Smith (contactid: 12345678-1234-1234-1234-123456789012)
[2024-11-17 18:00:19.901]   - Jane Doe (contactid: 23456789-2345-2345-2345-234567890123)
[2024-11-17 18:00:19.234]   - Michael Johnson (contactid: 34567890-3456-3456-3456-345678901234)
... (first 10 shown)
[2024-11-17 18:00:19.567] No changes made (preview mode)

Example 2: Create Family Groups for Real

.\WhanauTahi.Xpm.Tooling.CLI.exe datamanipulation `
  --RecreateFamilyGroup `
  --EnvironmentUrl https://myorg.crm6.dynamics.com `
  --ClientId a1b2c3d4-e5f6-7890-1234-567890abcdef `
  --Secret YourClientSecretHere

Expected Output:

[2024-11-17 18:15:20.123] Starting family group recreation...
[2024-11-17 18:15:20.456] Retrieving contacts without family groups (mag_whanauid IS NULL)...
[2024-11-17 18:15:23.789] Found 1,234 contacts without family groups
[2024-11-17 18:15:24.012] Creating service client pool (50 threads)...
[2024-11-17 18:15:26.345] Processing contacts in parallel...
[2024-11-17 18:15:27.678] Progress: 100/1,234 (8.1%) - 45 contacts/sec
[2024-11-17 18:15:35.901] Progress: 500/1,234 (40.5%) - 48 contacts/sec
[2024-11-17 18:15:43.234] Progress: 1,000/1,234 (81.0%) - 52 contacts/sec
[2024-11-17 18:15:48.567] Family group recreation completed
[2024-11-17 18:15:48.890] Total contacts processed: 1,234
[2024-11-17 18:15:48.123] Total family groups created: 1,234
[2024-11-17 18:15:48.456] Total memberships created: 1,234
[2024-11-17 18:15:48.789] Total time: 28.7 seconds
[2024-11-17 18:15:48.012] Average throughput: 43 contacts/sec

How It Works

Processing Steps

Query Contacts: Retrieve all contacts where mag_whanauid IS NULL
Create Service Client Pool: Initialize 50 ServiceClient instances for parallel processing
For Each Contact (in parallel):
- Create new mag_familygroup record
- Set mag_name = contact's fullname
- Copy address data from contact to family group
- Create mag_familygroupmembership record linking contact to new group
- Update contact's mag_whanauid to new family group ID
Batch Processing: Groups operations into ExecuteMultipleRequest batches
Progress Reporting: Shows percentage complete and throughput

Address Data Copied to Family Group

The operation copies these fields from contact to the newly created family group:

Contact Field	Family Group Field	Description
address1_line1	mag_physicaladdress1	Physical address line 1
address1_line2	mag_physicaladdress2	Physical address line 2
address1_city	mag_physicalsuburb	Physical suburb/city
address1_stateorprovince	mag_physicalcity	Physical city/state
address1_postalcode	mag_physicalpostcode	Physical postal code
address2_line1	mag_postaladdress1	Postal address line 1
address2_line2	mag_postaladdress2	Postal address line 2
address2_city	mag_postalsuburb	Postal suburb/city
address2_stateorprovince	mag_postalcity	Postal city/state
address2_postalcode	mag_postalpostcode	Postal postal code
mag_meshblock	mag_meshblock	Geographic meshblock
mag_latitude	mag_latitude	Geographic latitude
mag_longitude	mag_longitude	Geographic longitude

Note

If address fields are null on the contact, they will be null on the family group.

Family Group Record Structure

Created Family Group:

mag_familygroup:
  mag_familygroupid: <new GUID>
  mag_name: "John Smith" (from contact.fullname)
  mag_physicaladdress1: "123 Main St" (from contact.address1_line1)
  mag_physicalsuburb: "Downtown" (from contact.address1_city)
  mag_meshblock: "0012345" (from contact.mag_meshblock)
  mag_latitude: -41.286461 (from contact.mag_latitude)
  mag_longitude: 174.776230 (from contact.mag_longitude)
  ... (all address fields copied)

Created Membership:

mag_familygroupmembership:
  mag_familygroupmembershipid: <new GUID>
  mag_familygroup: <new family group GUID>
  mag_contact: <original contact GUID>

Updated Contact:

contact:
  contactid: <original contact GUID>
  mag_whanauid: <new family group GUID> (updated)

Parameters

Required Parameters

`--RecreateFamilyGroup`

Flag to enable the operation.

Example:

--RecreateFamilyGroup

`--EnvironmentUrl` (-e)

The URL of your Dataverse environment.

Example:

--EnvironmentUrl https://myorg.crm6.dynamics.com

`--ClientId` (-c)

The Azure AD application (client) ID for authentication.

Example:

--ClientId a1b2c3d4-e5f6-7890-1234-567890abcdef

Optional Parameters

`--Secret` (-s)

The client secret for authentication. If omitted, uses the default secret.

Example:

--Secret YourClientSecretHere

Caution

Never commit secrets to source control.

`--PreviewMode`

Preview what will be created without making any changes. Limits to first 100 contacts.

Example:

--PreviewMode

Real-World Scenarios

Scenario 1: Post-Migration Family Group Creation

Situation: You migrated 15,000 contacts from a legacy system that didn't track family groups. All contacts have mag_whanauid IS NULL and need family groups created.

Solution:

# Step 1: Preview first
.\WhanauTahi.Xpm.Tooling.CLI.exe datamanipulation `
  --RecreateFamilyGroup `
  -e https://prod.crm6.dynamics.com `
  -c a1b2c3d4-e5f6-7890-1234-567890abcdef `
  -s $env:DATAVERSE_SECRET `
  --PreviewMode

# Review the preview output - confirm count looks correct

# Step 2: Create family groups for real
.\WhanauTahi.Xpm.Tooling.CLI.exe datamanipulation `
  --RecreateFamilyGroup `
  -e https://prod.crm6.dynamics.com `
  -c a1b2c3d4-e5f6-7890-1234-567890abcdef `
  -s $env:DATAVERSE_SECRET

# Step 3: Verify all contacts now have family groups
# Run in SQL/FetchXML:
# SELECT COUNT(*) FROM contact WHERE mag_whanauid IS NULL
# Expected: 0

Expected Outcome:

15,000 new family groups created
15,000 membership records created
All contacts have mag_whanauid populated
Family groups have addresses copied from contacts

Scenario 2: Fix Incomplete Bulk Import

Situation: Last night's automated contact import created 500 new contacts but failed to create family groups due to a script error. You need to create family groups for these orphaned contacts.

Solution:

# Identify affected contacts
$query = @"
SELECT contactid, fullname, createdon 
FROM contact 
WHERE mag_whanauid IS NULL 
  AND createdon >= '2024-11-16'
ORDER BY createdon DESC
"@

# Create family groups for these contacts
.\WhanauTahi.Xpm.Tooling.CLI.exe datamanipulation `
  --RecreateFamilyGroup `
  -e https://prod.crm6.dynamics.com `
  -c a1b2c3d4-e5f6-7890-1234-567890abcdef `
  -s $env:DATAVERSE_SECRET

# Verify fix
$verifyQuery = @"
SELECT COUNT(*) AS OrphanedContacts
FROM contact 
WHERE mag_whanauid IS NULL 
  AND createdon >= '2024-11-16'
"@
# Expected: 0

Expected Outcome:

500 family groups created for new contacts
Addresses and geocoding data copied to family groups
Automated import process can resume normally

Scenario 3: Scheduled Weekly Cleanup

Situation: Your organization occasionally creates contacts without family groups through various integrations. You want to automatically create family groups for orphaned contacts every Sunday night.

Solution:

# weekly-family-group-cleanup.ps1
$ErrorActionPreference = "Stop"

$env:CLIENT_SECRET = Get-Secret -Name "DataverseClientSecret"

# Check if there are any orphaned contacts
$checkQuery = "SELECT COUNT(*) FROM contact WHERE mag_whanauid IS NULL"
# (Assume you have a function to run this query)

$orphanCount = Get-OrphanedContactCount -EnvironmentUrl "https://prod.crm6.dynamics.com"

if ($orphanCount -gt 0) {
    Write-Host "Found $orphanCount orphaned contacts. Creating family groups..."
    
    & "C:\Tools\WhanauTahi.Xpm.Tooling.CLI.exe" datamanipulation `
      --RecreateFamilyGroup `
      -e https://prod.crm6.dynamics.com `
      -c a1b2c3d4-e5f6-7890-1234-567890abcdef `
      -s $env:CLIENT_SECRET
    
    # Archive log
    $logFile = Get-ChildItem ".\Logs\DataManipulation_*" | Sort-Object LastWriteTime -Descending | Select-Object -First 1
    Copy-Item $logFile "\\fileserver\logs\family-group-cleanup-$(Get-Date -Format 'yyyy-MM-dd').log"
    
    # Send notification
    Send-MailMessage -To "admin@example.com" `
                     -Subject "Family Group Cleanup: $orphanCount Created" `
                     -Body "Created $orphanCount family groups for orphaned contacts."
} else {
    Write-Host "No orphaned contacts found. Skipping cleanup."
}

Schedule via Windows Task Scheduler:

Trigger: Weekly on Sunday at 11:00 PM
Action: Run PowerShell script
Run whether user is logged on or not

Performance & Scalability

Processing Characteristics

Parallel Execution: 50 threads processing contacts simultaneously
Batch Processing: Groups operations into ExecuteMultipleRequest batches (10 records/batch)
Retry Logic: Exponential backoff for throttling errors (max 50 retries)

Typical Performance

Contacts Without Groups	Estimated Time	Throughput
100	5-10 seconds	10-20 contacts/sec
500	10-20 seconds	25-50 contacts/sec
1,000	20-40 seconds	25-50 contacts/sec
5,000	1.5-3 minutes	30-55 contacts/sec
10,000	3-6 minutes	30-55 contacts/sec
50,000	15-30 minutes	30-55 contacts/sec

Note

Performance depends on Dataverse API limits and network latency.

Optimization Tips

1. Run during off-peak hours to avoid Dataverse throttling:

# Schedule for evenings or weekends when user load is low

2. Monitor progress in real-time:

# In another PowerShell window
Get-Content ".\Logs\DataManipulation_*" -Wait -Tail 20

3. Process in chunks if dealing with very large datasets (100K+):

-- Create batches using modulo on contactid
-- Process each batch separately over multiple nights

Troubleshooting

Issue: "No contacts found without family groups"

Cause: All contacts already have mag_whanauid populated.

Solution:

Verify there are contacts without family groups:

SELECT COUNT(*) FROM contact WHERE mag_whanauid IS NULL

If count is 0, all contacts already have family groups (no action needed)

Issue: "Duplicate family groups created"

Cause: Operation was run multiple times on the same contacts.

Solution:

Identify contacts with multiple family groups:

SELECT c.contactid, c.fullname, c.mag_whanauid, COUNT(*) AS GroupCount
FROM contact c
INNER JOIN mag_familygroupmembership fgm ON c.contactid = fgm.mag_contact
GROUP BY c.contactid, c.fullname, c.mag_whanauid
HAVING COUNT(*) > 1

Manually consolidate or delete duplicate family groups
Use --PreviewMode to check before running again

Issue: "Operation slow or timing out"

Cause: Dataverse throttling due to API limits.

Solution:

The tool includes automatic retry with exponential backoff
Run during off-peak hours to reduce throttling
Check throttling in logs:

ERROR: Request throttled. Retrying in 30 seconds... (Attempt 5/50)

Issue: "Address data not copied to family group"

Cause: Source contact has null address fields.

Solution:

Verify contact has address data:

SELECT contactid, fullname, address1_line1, address1_city, mag_meshblock
FROM contact
WHERE contactid = '<specific-contact-id>'

If address fields are null, they can't be copied
Update contact addresses first, then recreate family groups if needed

Output & Logs

Log File Location

Logs\DataManipulation_2024-11-17_18-15-20-123.log

Log File Contents

Startup Phase:

[2024-11-17 18:15:20.123] Starting family group recreation...
[2024-11-17 18:15:20.456] Retrieving contacts without family groups (mag_whanauid IS NULL)...
[2024-11-17 18:15:23.789] Found 1,234 contacts without family groups
[2024-11-17 18:15:24.012] Creating service client pool (50 threads)...
[2024-11-17 18:15:26.345] Service client pool created successfully

Processing Phase:

[2024-11-17 18:15:27.678] Processing contact: John Smith (12345678-1234-1234-1234-123456789012)
[2024-11-17 18:15:27.901] Creating family group: "John Smith Family Group"
[2024-11-17 18:15:27.234] Copying address data: 123 Main St, Downtown, Auckland, 1010
[2024-11-17 18:15:27.567] Copying geocoding data: Meshblock 0012345, Lat -41.286461, Long 174.776230
[2024-11-17 18:15:27.890] Family group created: 98765432-9876-9876-9876-987654321098
[2024-11-17 18:15:27.123] Creating family group membership...
[2024-11-17 18:15:27.456] Membership created successfully
[2024-11-17 18:15:27.789] Updating contact mag_whanauid...
[2024-11-17 18:15:27.012] Contact updated successfully
[2024-11-17 18:15:28.345] Progress: 100/1,234 (8.1%) - 45 contacts/sec
... (continues)
[2024-11-17 18:15:35.678] Progress: 500/1,234 (40.5%) - 48 contacts/sec
[2024-11-17 18:15:43.901] Progress: 1,000/1,234 (81.0%) - 52 contacts/sec

Completion Phase:

[2024-11-17 18:15:48.567] Family group recreation completed
[2024-11-17 18:15:48.890] Total contacts processed: 1,234
[2024-11-17 18:15:48.123] Total family groups created: 1,234
[2024-11-17 18:15:48.456] Total memberships created: 1,234
[2024-11-17 18:15:48.789] Contacts with addresses copied: 1,180
[2024-11-17 18:15:48.012] Contacts without addresses: 54
[2024-11-17 18:15:48.345] Total time: 28.7 seconds
[2024-11-17 18:15:48.678] Average throughput: 43 contacts/sec
[2024-11-17 18:15:48.901] Success rate: 100%

Understanding the Results

Success Indicators:

✅ "Success rate: 100%"
✅ "Total contacts processed" matches expected count
✅ No ERROR lines in log
✅ All contacts now have mag_whanauid populated

Warning Signs:

⚠️ "Contacts without addresses" is very high - Address data may be missing
⚠️ Multiple throttling retries - Run during off-peak hours
⚠️ ERROR messages - Check permissions or data integrity
⚠️ Success rate < 100% - Review errors and retry failed contacts

Recreate Family Groups

Overview

When to Use This Operation

Use This Operation When:

Don't Use This Operation When:

Quick Start

Example 1: Preview What Will Be Created

Example 2: Create Family Groups for Real

How It Works

Processing Steps

Address Data Copied to Family Group

Note

Family Group Record Structure

Parameters

Required Parameters

--RecreateFamilyGroup

--EnvironmentUrl (-e)

--ClientId (-c)

Optional Parameters

--Secret (-s)

Caution

--PreviewMode

Real-World Scenarios

Scenario 1: Post-Migration Family Group Creation

Scenario 2: Fix Incomplete Bulk Import

Scenario 3: Scheduled Weekly Cleanup

Performance & Scalability

Processing Characteristics

Typical Performance

Note

Optimization Tips

Troubleshooting

Issue: "No contacts found without family groups"

Issue: "Duplicate family groups created"

Issue: "Operation slow or timing out"

Issue: "Address data not copied to family group"

Output & Logs

Log File Location

Log File Contents

Understanding the Results

See Also

`--RecreateFamilyGroup`

`--EnvironmentUrl` (-e)

`--ClientId` (-c)

`--Secret` (-s)

`--PreviewMode`