> ## Documentation Index
> Fetch the complete documentation index at: https://developers.activeprospect.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Firehose

# Firehose Multi-Service Configuration

## Overview

Each flow is separately configured to use the firehose. This API allows firehose configuration to be tested before it
is set on a flow.

The Firehose feature supports multiple cloud storage services for event data export. This document describes the configuration options for both AWS S3 and Azure Blob Storage services.

## Configuration Structure

### Legacy Configuration (Backward Compatible)

For existing flows, the legacy configuration continues to work:

```json theme={null}
{
  "firehose": {
    "enabled": true,
    "credential_id": "507f1f77bcf86cd799439011",
    "bucket": "my-s3-bucket",
    "prefix": "events/production"
  }
}
```

### New Service-Based Configuration

The new service-based configuration allows multiple cloud storage providers:

```json theme={null}
{
  "firehose": {
    "enabled": true,
    "services": {
      "aws": {
        "enabled": true,
        "credential_id": "507f1f77bcf86cd799439011",
        "bucket": "my-s3-bucket",
        "prefix": "events/aws"
      },
      "azure": {
        "enabled": true,
        "credential_id": "507f1f77bcf86cd799439012", 
        "bucket": "my-azure-container",
        "prefix": "events/azure"
      }
    }
  }
}
```

## Credential Validation

### AWS S3 Validation

The `/firehose` endpoint validates AWS credentials by:

1. Creating a test file with unique name: `leadconduit_verification_[flow_id_]YYYYMMDDHHMMSSMS.txt`
2. Uploading it to the specified bucket (with optional prefix) when `verification_file=true` (default)
3. Alternatively, validating bucket access without creating file when `verification_file=false`
4. Returning the S3 response with ETag or validation confirmation

**Example Requests:**

*Basic validation (creates file):*

```bash theme={null}
curl -X GET "https://app.leadconduit.com/firehose?service=aws&access_key_id=AKIA...&secret_access_key=wJal...&bucket=my-bucket&prefix=test"
```

*With flow ID (includes flow identifier in filename):*

```bash theme={null}
curl -X GET "https://app.leadconduit.com/firehose?service=aws&access_key_id=AKIA...&secret_access_key=wJal...&bucket=my-bucket&flow_id=507f1f77bcf86cd799439011"
```

*Validation only (no file created):*

```bash theme={null}
curl -X GET "https://app.leadconduit.com/firehose?service=aws&access_key_id=AKIA...&secret_access_key=wJal...&bucket=my-bucket&verification_file=false"
```

**Example Responses:**

*File created:*

```json theme={null}
{
  "validated": true,
  "verification_file": true
}
```

*Validation only:*

```json theme={null}
{
  "validated": true,
  "verification_file": false
}
```

### Azure Blob Storage Validation

The `/firehose` endpoint validates Azure credentials by:

1. Creating a BlobServiceClient from the connection string
2. Checking if the specified container exists and is accessible
3. Optionally creating verification file based on `verification_file` parameter
4. Returning container validation information

**Example Request:**

```bash theme={null}
curl -X GET "https://app.leadconduit.com/firehose?service=azure&connection_string=DefaultEndpointsProtocol=https;AccountName=test;AccountKey=key;EndpointSuffix=core.windows.net&bucket=my-container"
```

**Example Responses:**

*With verification file:*

```json theme={null}
{
  "validated": true,
  "verification_file": true
}
```

*Validation only (verification\_file=false):*

```json theme={null}
{
  "validated": true,
  "verification_file": false
}
```

## Event Processing

### Single Service Configuration

When only one service is configured, events are sent to that service. If the service fails, events are spooled for retry.

### Multi-Service Configuration

When multiple services are configured:

1. Each enabled service receives the event independently
2. If a service fails, it generates its own spool message for retry
3. Each service can have different bucket/container and prefix configurations

### Spooling and Retry

* Each service failure generates a separate SQS message for retry
* Spooled events contain service-specific metadata (bucket, prefix, credentials)
* The unspooler processes each service's failed events independently
* File-based spooling creates separate paths for each service configuration

## Migration Guide

### From Legacy to Service-Based Configuration

1. **Keep existing configuration**: Legacy configuration continues to work
2. **Add services object**: Gradually migrate to service-based configuration
3. **Test thoroughly**: Validate both configurations work as expected

Example migration:

**Before:**

```json theme={null}
{
  "firehose": {
    "enabled": true,
    "credential_id": "507f1f77bcf86cd799439011",
    "bucket": "my-bucket"
  }
}
```

**After:**

```json theme={null}
{
  "firehose": {
    "enabled": true,
    "services": {
      "aws": {
        "enabled": true,
        "credential_id": "507f1f77bcf86cd799439011",
        "bucket": "my-bucket"
      }
    }
  }
}
```