Background Process Execution

Last Updated: January 9, 2026

Learn how to run long-running bulk processes in the background without blocking the API request, and understand the implications for your BPM implementation.

The Problem

When processing large collections (e.g., 2000 items), the workflow can take several minutes to complete. If you start the process synchronously:

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Issues:

âŒ HTTP request timeout (30-120 seconds)
âŒ Browser shows loading spinner indefinitely
âŒ User can't navigate away or do other work
âŒ If connection drops, process state is unclear
âŒ Poor user experience

The Solution: Fire-and-Forget Pattern

Start the workflow in the background and return immediately with a tracking ID:

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Implementation Approach

Current BPM Implementation Status

Your current StartProcessCommand handler likely looks like this:

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

The Issue: _bpmnEngine.ExecuteProcess() runs synchronously through all tasks until completion or a wait state.

Solution 1: Immediate Return Pattern (Recommended)

Architecture Change

Modify the execution flow to:

Create the process instance (status = "Pending")
Return immediately with process instance ID
Queue the process for background execution
Background worker picks up and executes the process

Implementation

Step 1: Modify StartProcessCommand Handler

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Step 2: Add Background Job Queue Service

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Step 3: Add Background Worker Service

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Step 4: Register Services in Program.cs

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

API Controller Pattern

Controller Endpoint

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Frontend Integration

Starting the Process

async function startBulkProcess(items) {
    try {
        const response = await fetch('/api/bulk-process', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({
                items: items,
                batchSize: 100
            })
        });
        
        if (response.status === 202) {  // Accepted
            const result = await response.json();
            
            // Show success message
            showNotification('Bulk processing started', 'success');
            
            // Start polling for status
            pollProcessStatus(result.processInstanceId);
        }
    } catch (error) {
        showNotification('Failed to start process: ' + error.message, 'error');
    }
}

Polling for Status

function pollProcessStatus(processInstanceId) {
    const intervalId = setInterval(async () => {
        try {
            const response = await fetch(`/api/bulk-process/${processInstanceId}/status`);
            const status = await response.json();
            
            // Update UI
            updateProgressBar(status.percentComplete);
            updateStats({
                processed: status.processedItems,
                total: status.totalItems,
                success: status.successCount,
                failed: status.failureCount
            });
            
            // Check if complete
            if (status.isComplete) {
                clearInterval(intervalId);
                
                if (status.status === 'completed') {
                    showNotification('Bulk processing completed!', 'success');
                    loadResults(processInstanceId);
                } else if (status.status === 'failed') {
                    showNotification('Bulk processing failed', 'error');
                }
            }
        } catch (error) {
            console.error('Error polling status:', error);
        }
    }, 3000);  // Poll every 3 seconds
}

Progress UI Component

<div class="bulk-process-container">
    <div class="progress-section">
        <h3>Bulk Processing in Progress</h3>
        
        <!-- Progress Bar -->
        <div class="progress-bar">
            <div id="progress-fill" class="progress-fill" style="width: 0%">
                <span id="progress-text">0%</span>
            </div>
        </div>
        
        <!-- Stats -->
        <div class="stats-grid">
            <div class="stat-card">
                <span class="stat-label">Total Items</span>
                <span id="total-items" class="stat-value">-</span>
            </div>
            <div class="stat-card">
                <span class="stat-label">Processed</span>
                <span id="processed-items" class="stat-value">-</span>
            </div>
            <div class="stat-card success">
                <span class="stat-label">Success</span>
                <span id="success-count" class="stat-value">-</span>
            </div>
            <div class="stat-card failure">
                <span class="stat-label">Failed</span>
                <span id="failure-count" class="stat-value">-</span>
            </div>
        </div>
        
        <!-- Action Buttons -->
        <div class="actions">
            <button onclick="navigateAway()" class="btn btn-secondary">
                Continue Working
            </button>
            <button onclick="viewResults()" class="btn btn-primary" disabled id="view-results-btn">
                View Results
            </button>
        </div>
    </div>
</div>

<script>
function updateProgressBar(percent) {
    document.getElementById('progress-fill').style.width = percent + '%';
    document.getElementById('progress-text').innerText = percent.toFixed(1) + '%';
}

function updateStats(stats) {
    document.getElementById('total-items').innerText = stats.total;
    document.getElementById('processed-items').innerText = stats.processed;
    document.getElementById('success-count').innerText = stats.success;
    document.getElementById('failure-count').innerText = stats.failed;
    
    // Enable results button when complete
    if (stats.processed === stats.total) {
        document.getElementById('view-results-btn').disabled = false;
    }
}

function navigateAway() {
    // User can navigate away - process continues in background
    localStorage.setItem('activeProcessId', currentProcessInstanceId);
    window.location.href = '/dashboard';
}

// On page load, check for active processes
window.addEventListener('load', () => {
    const activeProcessId = localStorage.getItem('activeProcessId');
    if (activeProcessId) {
        // Resume polling for this process
        pollProcessStatus(activeProcessId);
    }
});
</script>

Impact on Current BPM Implementation

1. Process Instance State Management

Current State: Process instances likely don't track execution state granularly

Required Changes:

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

2. Database Schema Changes

-- Add new columns to ProcessInstance table
ALTER TABLE ProcessInstance 
    ADD StartedAt DATETIME2 NULL,
        CompletedAt DATETIME2 NULL,
        ErrorMessage NVARCHAR(MAX) NULL,
        CreatedBy NVARCHAR(255) NULL;

-- Add index for status queries
CREATE INDEX IX_ProcessInstance_Status_CreatedAt 
ON ProcessInstance(Status, CreatedAt DESC);

3. BPMN Engine Modifications

No Breaking Changes Required!

The BPMN execution engine (BpmnExecutionEngine) doesn't need changes. It continues to execute processes synchronously, but now:

âœ… Executes in background thread (not API request thread)
âœ… Can take minutes/hours without timing out
âœ… API returns immediately with process ID
âœ… Clients poll for status updates

4. Command Handler Pattern

Before (Synchronous):

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

After (Asynchronous):

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

5. Scalability Considerations

Current Limitations:

Single instance can handle ~10-20 concurrent background processes
Each process blocks a thread during execution

Future Enhancements (if needed):

Multiple Workers: Scale background worker instances
Distributed Queue: Use RabbitMQ/Redis instead of in-memory Channel
Process Pooling: Limit concurrent executions to prevent overload

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Alternative: SignalR for Real-Time Updates

Instead of polling, use SignalR for real-time push notifications:

SignalR Hub

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Update Progress from Worker

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Frontend SignalR Client

const connection = new signalR.HubConnectionBuilder()
    .withUrl("/hubs/process-progress")
    .build();

connection.on("ProgressUpdate", (update) => {
    // Update UI in real-time
    updateProgressBar(update.percentComplete);
    updateStats(update);
});

connection.start().then(() => {
    connection.invoke("SubscribeToProcess", processInstanceId);
});

Benefits:

âš¡ Real-time updates (no polling delay)
ðŸ”‹ Lower server load (no constant polling)
ðŸ“± Better mobile experience

Migration Strategy

Phase 1: Add Background Queue (Non-Breaking)

Add IBackgroundJobQueue interface and implementation
Add ProcessExecutionBackgroundService hosted service
Register services in DI container
Deploy - no impact on existing functionality

Phase 2: Update StartProcessCommand Handler

Modify handler to queue work instead of executing directly
Add status tracking columns to database
Deploy - existing synchronous callers still work (process just happens in background)

Phase 3: Update API Controllers

Change POST endpoints to return 202 Accepted instead of 200 OK
Add status polling endpoints
Update frontend to poll for status
Deploy with backward compatibility

Phase 4: Add Real-Time Updates (Optional)

Add SignalR hub
Update background worker to push updates
Update frontend to use SignalR
Deploy

Best Practices

âœ… DO

Return 202 Accepted for async operations (not 200 OK)
Provide status URL in response for tracking
Implement timeout limits for background processes
Log extensively in background workers
Handle cancellation properly (CancellationToken)
Limit queue size to prevent memory issues
Add retry logic for transient failures
Send notifications when long processes complete

âŒ DON'T

Return 200 OK for operations that aren't complete
Block HTTP requests for long-running operations
Forget to update process status on failure
Ignore CancellationToken in background work
Allow unlimited queue growth
Skip error handling in background workers

Testing Background Execution

Unit Test

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Integration Test

Code Removed

Implementation details removed for security.

Contact support for implementation guidance.

Conclusion

Impact Summary

Aspect	Current	After Background Execution
API Response Time	Minutes (timeout)	< 1 second âœ…
User Experience	Blocked/waiting	Can navigate away âœ…
Scalability	Limited	Better (controlled queue) âœ…
Error Handling	Unclear state	Tracked status âœ…
Progress Tracking	None	Polling/SignalR âœ…

Required Changes

âœ… Add background job queue - New service, no breaking changes
âœ… Update StartProcessCommand handler - Queue work instead of executing
âœ… Add status tracking - New DB columns, no schema breaking changes
âœ… Update API controllers - Return 202 Accepted with status URL
âœ… Add polling endpoints - New endpoints for status checking
âš¡ Optional: Add SignalR - Real-time updates instead of polling

Minimal Implementation (1-2 hours)

If you want the quickest solution:

Add IBackgroundJobQueue and BackgroundJobQueue classes
Add ProcessExecutionBackgroundService hosted service
Register in Program.cs
Modify StartProcessCommandHandler to queue work
Add /status endpoint to check process state

That's it! Your bulk processes now run in the background without blocking API requests. ðŸš€

The Problem​

The Solution: Fire-and-Forget Pattern​

Implementation Approach​

Current BPM Implementation Status​

Solution 1: Immediate Return Pattern (Recommended)​

Architecture Change​

Implementation​

Step 1: Modify StartProcessCommand Handler​

Step 2: Add Background Job Queue Service​

Step 3: Add Background Worker Service​

Step 4: Register Services in Program.cs​

API Controller Pattern​

Controller Endpoint​

Frontend Integration​

Starting the Process​

Polling for Status​

Progress UI Component​

Impact on Current BPM Implementation​

1. Process Instance State Management​

2. Database Schema Changes​

3. BPMN Engine Modifications​

4. Command Handler Pattern​

5. Scalability Considerations​

Alternative: SignalR for Real-Time Updates​

SignalR Hub​

Update Progress from Worker​

Frontend SignalR Client​

Migration Strategy​

Phase 1: Add Background Queue (Non-Breaking)​

Phase 2: Update StartProcessCommand Handler​

Phase 3: Update API Controllers​

Phase 4: Add Real-Time Updates (Optional)​

Best Practices​

âœ… DO​

âŒ DON'T​

Testing Background Execution​

Unit Test​

Integration Test​

Conclusion​

Impact Summary​

Required Changes​

Minimal Implementation (1-2 hours)​

The Problem

The Solution: Fire-and-Forget Pattern

Implementation Approach

Current BPM Implementation Status

Solution 1: Immediate Return Pattern (Recommended)

Architecture Change

Implementation

Step 1: Modify StartProcessCommand Handler

Step 2: Add Background Job Queue Service

Step 3: Add Background Worker Service

Step 4: Register Services in Program.cs

API Controller Pattern

Controller Endpoint

Frontend Integration

Starting the Process

Polling for Status

Progress UI Component

Impact on Current BPM Implementation

1. Process Instance State Management

2. Database Schema Changes

3. BPMN Engine Modifications

4. Command Handler Pattern

5. Scalability Considerations

Alternative: SignalR for Real-Time Updates

SignalR Hub

Update Progress from Worker

Frontend SignalR Client

Migration Strategy

Phase 1: Add Background Queue (Non-Breaking)

Phase 2: Update StartProcessCommand Handler

Phase 3: Update API Controllers

Phase 4: Add Real-Time Updates (Optional)

Best Practices

âœ… DO

âŒ DON'T

Testing Background Execution

Unit Test

Integration Test

Conclusion

Impact Summary

Required Changes

Minimal Implementation (1-2 hours)