Bulk Operations

The Conviso Go SDK is designed to be thread-safe and supports concurrent requests. While the API itself handles requests one by one, you can use Go's native concurrency features (goroutines) to perform efficient bulk operations.

Concurrency Limiting

The Client enforces a maximum number of concurrent requests (default: 10, max: 30) using an internal semaphore. This protects both your application and the API from being overwhelmed. You can safely launch more goroutines than this limit; the SDK will queue them automatically.

Bulk Identify Example

Here is how to efficiently Identify a large list of users using a worker pool pattern or simple wait groups.

package main

import (
	"context"
	"log"
	"sync"
	
	"github.com/codematic/opencdp-go/cdp"
)

type User struct {
	ID    string
	Email string
	Name  string
}

func BulkIdentify(client *cdp.Client, users []User) {
	var wg sync.WaitGroup
	ctx := context.Background()

	// Optionally use a semaphore here to limit client-side concurrency 
	// before it even hits the SDK lock, though the SDK handles it too.
	// For very large lists (e.g. 10k+), it's good practice to limit separate from the SDK.
	sem := make(chan struct{}, 50) 

	for _, user := range users {
		wg.Add(1)
		go func(u User) {
			defer wg.Done()
			
			// Acquire token
			sem <- struct{}{}
			defer func() { <-sem }()

			traits := map[string]interface{}{
				"email": u.Email,
				"name":  u.Name,
			}

			if err := client.Identify(ctx, u.ID, traits); err != nil {
				log.Printf("Failed to identify user %s: %v", u.ID, err)
			}
		}(user)
	}

	wg.Wait()
	log.Printf("Finished processing %d users", len(users))
}

Bulk Track Example

Similarly, you can track events in bulk.

func BulkTrack(client *cdp.Client, events []MyEvent) {
	var wg sync.WaitGroup
	ctx := context.Background()

	for _, event := range events {
		wg.Add(1)
		go func(e MyEvent) {
			defer wg.Done()
			
			if err := client.Track(ctx, e.UserID, e.Name, e.Properties); err != nil {
				log.Printf("Failed to track event for user %s: %v", e.UserID, err)
			}
		}(event)
	}

	wg.Wait()
}

Best Practices for Bulk Ops

1. Configure MaxConcurrentRequests: Set this in CDPConfig to an appropriate value (e.g., 20 or 30) for higher throughput.
2. Handle Errors: Since operations run concurrently, ensure you log individual errors.
3. Context Cancellation: Pass a cancellable context if you want the ability to stop the entire bulk operation.