Skip to main content

​
Firehose Multi-Service Configuration

​
Overview

Each flow is separately configured to use the firehose. This API allows firehose configuration to be tested before it is set on a flow. The Firehose feature supports multiple cloud storage services for event data export. This document describes the configuration options for both AWS S3 and Azure Blob Storage services.

​
Configuration Structure

​
Legacy Configuration (Backward Compatible)

For existing flows, the legacy configuration continues to work: 
{
  "firehose": {
    "enabled": true,
    "credential_id": "507f1f77bcf86cd799439011",
    "bucket": "my-s3-bucket",
    "prefix": "events/production"
  }
}

​
New Service-Based Configuration

The new service-based configuration allows multiple cloud storage providers: 
{
  "firehose": {
    "enabled": true,
    "services": {
      "aws": {
        "enabled": true,
        "credential_id": "507f1f77bcf86cd799439011",
        "bucket": "my-s3-bucket",
        "prefix": "events/aws"
      },
      "azure": {
        "enabled": true,
        "credential_id": "507f1f77bcf86cd799439012", 
        "bucket": "my-azure-container",
        "prefix": "events/azure"
      }
    }
  }
}

​
Credential Validation

​
AWS S3 Validation

The /firehose endpoint validates AWS credentials by:
  1. Creating a test file with unique name: leadconduit_verification_[flow_id_]YYYYMMDDHHMMSSMS.txt
  2. Uploading it to the specified bucket (with optional prefix) when verification_file=true (default)
  3. Alternatively, validating bucket access without creating file when verification_file=false
  4. Returning the S3 response with ETag or validation confirmation
Example Requests: Basic validation (creates file): 
curl -X GET "https://app.leadconduit.com/firehose?service=aws&access_key_id=AKIA...&secret_access_key=wJal...&bucket=my-bucket&prefix=test"
With flow ID (includes flow identifier in filename): 
curl -X GET "https://app.leadconduit.com/firehose?service=aws&access_key_id=AKIA...&secret_access_key=wJal...&bucket=my-bucket&flow_id=507f1f77bcf86cd799439011"
Validation only (no file created): 
curl -X GET "https://app.leadconduit.com/firehose?service=aws&access_key_id=AKIA...&secret_access_key=wJal...&bucket=my-bucket&verification_file=false"
Example Responses: File created: 
{
  "validated": true,
  "verification_file": true
}
Validation only: 
{
  "validated": true,
  "verification_file": false
}

​
Azure Blob Storage Validation

The /firehose endpoint validates Azure credentials by:
  1. Creating a BlobServiceClient from the connection string
  2. Checking if the specified container exists and is accessible
  3. Optionally creating verification file based on verification_file parameter
  4. Returning container validation information
Example Request: 
curl -X GET "https://app.leadconduit.com/firehose?service=azure&connection_string=DefaultEndpointsProtocol=https;AccountName=test;AccountKey=key;EndpointSuffix=core.windows.net&bucket=my-container"
Example Responses: With verification file: 
{
  "validated": true,
  "verification_file": true
}
Validation only (verification_file=false): 
{
  "validated": true,
  "verification_file": false
}

​
Event Processing

​
Single Service Configuration

When only one service is configured, events are sent to that service. If the service fails, events are spooled for retry.

​
Multi-Service Configuration

When multiple services are configured:
  1. Each enabled service receives the event independently
  2. If a service fails, it generates its own spool message for retry
  3. Each service can have different bucket/container and prefix configurations

​
Spooling and Retry

  • Each service failure generates a separate SQS message for retry
  • Spooled events contain service-specific metadata (bucket, prefix, credentials)
  • The unspooler processes each service’s failed events independently
  • File-based spooling creates separate paths for each service configuration

​
Migration Guide

​
From Legacy to Service-Based Configuration

  1. Keep existing configuration: Legacy configuration continues to work
  2. Add services object: Gradually migrate to service-based configuration
  3. Test thoroughly: Validate both configurations work as expected
Example migration: Before: 
{
  "firehose": {
    "enabled": true,
    "credential_id": "507f1f77bcf86cd799439011",
    "bucket": "my-bucket"
  }
}
After: 
{
  "firehose": {
    "enabled": true,
    "services": {
      "aws": {
        "enabled": true,
        "credential_id": "507f1f77bcf86cd799439011",
        "bucket": "my-bucket"
      }
    }
  }
}