Third-party Integration - 1/2

The $12 Million Integration Cascade That Destroyed Black Friday Launch

Picture this horror story: November 24th, 11:58 PM EST. One of the most anticipated e-commerce launches of the year is about to go live—a flash sale featuring limited edition products from major brands. The marketing campaign has generated 2.5 million pre-registered users. Everything has been tested. The servers are scaled. The databases are optimized.

Then 12:00 AM hits, and the digital equivalent of a house fire begins.

Within the first 10 minutes, users start experiencing a nightmare:

  • Credit card payments failing with cryptic “Service temporarily unavailable” messages
  • Users creating accounts but never receiving confirmation emails
  • Product images loading as broken placeholders from the CDN
  • Social media sharing buttons leading to 404 pages
  • Inventory counts wildly inaccurate because the ERP system integration froze
  • Customer support chatbot responding with “I don’t understand” to every query
  • Push notifications never sending, leaving users in the dark

The frontend was flawless. The backend could handle the load beautifully. But every single third-party integration—the invisible connective tissue that makes modern applications work—had failed spectacularly.

Here’s what went catastrophically wrong:

  • Payment processor rate limiting: No exponential backoff, causing payment failures to cascade
  • Email service overload: Sending 50,000 confirmation emails simultaneously without batching
  • CDN authentication expired: Image assets returning 403 errors at peak traffic
  • Social media APIs hitting quota limits: Share buttons breaking when users needed them most
  • ERP system timeout: Inventory updates taking 45 seconds, causing overselling
  • No fallback strategies: When one integration failed, it brought down related features

By 3 AM, they had:

  • $12 million in abandoned carts due to payment failures
  • 800,000 users who never received account confirmation emails
  • 1.2 million failed social media shares during peak virality window
  • Complete breakdown of inventory management causing legal issues with overselling
  • Emergency rollback to a version with 70% fewer features
  • Customer service overwhelmed with 50,000 support tickets

The brutal truth? They had built a technically perfect application that couldn’t communicate with the outside world when it mattered most.

The Uncomfortable Truth About Third-party Integration

Here’s what separates applications that gracefully orchestrate dozens of external services from those that collapse when a single API hiccups: Third-party integration isn’t just about making HTTP requests—it’s about building resilient, fault-tolerant communication layers that can handle the unpredictable nature of services you don’t control.

Most developers approach third-party integration like this:

  1. Find an API endpoint and start making requests
  2. Add basic error handling for 4xx/5xx responses
  3. Hope that external services will always be available
  4. Discover during traffic spikes that rate limits exist
  5. Panic when a third-party service goes down and takes your features with it

But systems that handle real-world integration complexity work differently:

  1. Design integration architecture with failure as the default assumption
  2. Implement intelligent retry strategies and circuit breakers from day one
  3. Build comprehensive monitoring for every external dependency
  4. Plan graceful degradation when third-party services fail
  5. Treat external API calls as distributed system challenges requiring sophisticated solutions

The difference isn’t just reliability—it’s the difference between applications that enhance user experience through seamless integrations and applications where third-party failures become user-facing disasters.

Ready to build integrations that work like Shopify’s payment processing instead of that API wrapper that breaks when someone sneezes at the data center? Let’s dive into the patterns that power bulletproof third-party integration.


REST API Consumption: Building Bulletproof Client Libraries

Production-Ready HTTP Client Architecture

// Robust REST API client with comprehensive error handling and resilience
import axios, { AxiosInstance, AxiosRequestConfig, AxiosResponse } from "axios";
import pRetry from "p-retry";

class RobustAPIClient {
  private client: AxiosInstance;
  private circuitBreaker: CircuitBreaker;
  private metrics: MetricsCollector;
  private cache: APICache;
  private rateLimiter: RateLimiter;

  constructor(config: APIClientConfig) {
    this.client = this.createAxiosInstance(config);
    this.circuitBreaker = new CircuitBreaker(config.circuitBreakerOptions);
    this.metrics = new MetricsCollector();
    this.cache = new APICache(config.cacheOptions);
    this.rateLimiter = new RateLimiter(config.rateLimitOptions);

    this.setupInterceptors();
    this.setupRequestTracing();
  }

  private createAxiosInstance(config: APIClientConfig): AxiosInstance {
    const instance = axios.create({
      baseURL: config.baseURL,
      timeout: config.timeout || 30000, // 30 second default
      headers: {
        "Content-Type": "application/json",
        "User-Agent": `${config.serviceName}/${config.version}`,
        ...config.defaultHeaders,
      },
      maxRedirects: 3,
      validateStatus: (status) => status < 500, // Don't throw on 4xx
    });

    return instance;
  }

  private setupInterceptors(): void {
    // Request interceptor
    this.client.interceptors.request.use(
      async (config) => {
        const startTime = Date.now();

        // Add correlation ID for tracing
        config.headers["X-Correlation-ID"] = this.generateCorrelationId();

        // Add request timestamp
        config.metadata = { startTime };

        // Apply rate limiting
        await this.rateLimiter.waitForToken();

        // Log request
        console.log(
          `API Request: ${config.method?.toUpperCase()} ${config.url}`
        );

        return config;
      },
      (error) => {
        console.error("Request interceptor error:", error);
        return Promise.reject(error);
      }
    );

    // Response interceptor
    this.client.interceptors.response.use(
      (response) => {
        const duration = Date.now() - response.config.metadata?.startTime;

        // Record metrics
        this.metrics.recordRequestSuccess(
          response.config.url || "",
          response.status,
          duration
        );

        // Log successful response
        console.log(
          `API Response: ${
            response.status
          } ${response.config.method?.toUpperCase()} ${
            response.config.url
          } (${duration}ms)`
        );

        return response;
      },
      async (error) => {
        const duration =
          Date.now() - (error.config?.metadata?.startTime || Date.now());

        // Record metrics
        this.metrics.recordRequestError(
          error.config?.url || "",
          error.response?.status || 0,
          duration,
          error.code
        );

        // Enhanced error logging
        console.error(
          `API Error: ${
            error.response?.status || "UNKNOWN"
          } ${error.config?.method?.toUpperCase()} ${
            error.config?.url
          } (${duration}ms)`,
          {
            status: error.response?.status,
            statusText: error.response?.statusText,
            data: error.response?.data,
            correlationId: error.config?.headers["X-Correlation-ID"],
          }
        );

        return Promise.reject(this.enhanceError(error));
      }
    );
  }

  // Resilient API request with retry logic and circuit breaker
  async request<T = any>(config: APIRequestConfig): Promise<APIResponse<T>> {
    const cacheKey = this.generateCacheKey(config);

    // Try cache first if enabled
    if (config.cache?.enabled) {
      const cachedResponse = await this.cache.get(cacheKey);
      if (cachedResponse) {
        console.log(`Cache hit: ${config.method} ${config.url}`);
        return cachedResponse;
      }
    }

    // Check circuit breaker
    if (this.circuitBreaker.isOpen()) {
      throw new APIError(
        "Circuit breaker is open",
        "CIRCUIT_BREAKER_OPEN",
        503
      );
    }

    try {
      const response = await this.executeWithRetry(config);

      // Cache successful responses
      if (config.cache?.enabled && response.status < 400) {
        await this.cache.set(
          cacheKey,
          response,
          config.cache.ttlSeconds || 300
        );
      }

      // Record circuit breaker success
      this.circuitBreaker.recordSuccess();

      return response;
    } catch (error) {
      // Record circuit breaker failure
      this.circuitBreaker.recordFailure();
      throw error;
    }
  }

  private async executeWithRetry<T>(
    config: APIRequestConfig
  ): Promise<APIResponse<T>> {
    const retryOptions = {
      retries: config.retry?.maxAttempts || 3,
      factor: config.retry?.backoffFactor || 2,
      minTimeout: config.retry?.initialDelayMs || 1000,
      maxTimeout: config.retry?.maxDelayMs || 30000,
      randomize: true, // Add jitter
      onRetry: (error: any, attempt: number) => {
        console.warn(
          `API retry attempt ${attempt} for ${config.method} ${config.url}:`,
          error.message
        );
        this.metrics.recordRetry(config.url || "", attempt);
      },
    };

    return pRetry(async () => {
      const axiosConfig: AxiosRequestConfig = {
        method: config.method,
        url: config.url,
        data: config.data,
        params: config.params,
        headers: {
          ...config.headers,
          // Add idempotency key for POST/PUT requests
          ...(config.method !== "GET" && {
            "Idempotency-Key": this.generateIdempotencyKey(config),
          }),
        },
        timeout: config.timeout,
      };

      const response = await this.client.request(axiosConfig);

      // Check if response indicates a retryable error
      if (this.isRetryableResponse(response)) {
        throw new APIError(
          "Retryable response received",
          "RETRYABLE_RESPONSE",
          response.status
        );
      }

      return this.transformResponse<T>(response);
    }, retryOptions);
  }

  private isRetryableResponse(response: AxiosResponse): boolean {
    // Retry on server errors and specific client errors
    const retryableStatuses = [
      429, // Rate limited
      500,
      502,
      503,
      504, // Server errors
      408, // Request timeout
    ];

    return retryableStatuses.includes(response.status);
  }

  private transformResponse<T>(response: AxiosResponse): APIResponse<T> {
    return {
      data: response.data,
      status: response.status,
      statusText: response.statusText,
      headers: response.headers,
      correlationId: response.config.headers?.["X-Correlation-ID"],
      requestDuration: Date.now() - (response.config.metadata?.startTime || 0),
    };
  }

  private enhanceError(error: any): APIError {
    const apiError = new APIError(
      error.message || "Unknown API error",
      this.getErrorCode(error),
      error.response?.status || 0,
      {
        originalError: error,
        correlationId: error.config?.headers?.["X-Correlation-ID"],
        url: error.config?.url,
        method: error.config?.method,
        requestData: error.config?.data,
        responseData: error.response?.data,
      }
    );

    return apiError;
  }

  private getErrorCode(error: any): string {
    if (error.code) return error.code;
    if (error.response?.status) {
      const statusCodes: Record<number, string> = {
        400: "BAD_REQUEST",
        401: "UNAUTHORIZED",
        403: "FORBIDDEN",
        404: "NOT_FOUND",
        422: "VALIDATION_ERROR",
        429: "RATE_LIMITED",
        500: "INTERNAL_SERVER_ERROR",
        502: "BAD_GATEWAY",
        503: "SERVICE_UNAVAILABLE",
        504: "GATEWAY_TIMEOUT",
      };
      return statusCodes[error.response.status] || "HTTP_ERROR";
    }
    return "UNKNOWN_ERROR";
  }

  // Convenience methods for common HTTP verbs
  async get<T = any>(
    url: string,
    config?: Omit<APIRequestConfig, "method" | "url">
  ): Promise<APIResponse<T>> {
    return this.request<T>({ ...config, method: "GET", url });
  }

  async post<T = any>(
    url: string,
    data?: any,
    config?: Omit<APIRequestConfig, "method" | "url" | "data">
  ): Promise<APIResponse<T>> {
    return this.request<T>({ ...config, method: "POST", url, data });
  }

  async put<T = any>(
    url: string,
    data?: any,
    config?: Omit<APIRequestConfig, "method" | "url" | "data">
  ): Promise<APIResponse<T>> {
    return this.request<T>({ ...config, method: "PUT", url, data });
  }

  async patch<T = any>(
    url: string,
    data?: any,
    config?: Omit<APIRequestConfig, "method" | "url" | "data">
  ): Promise<APIResponse<T>> {
    return this.request<T>({ ...config, method: "PATCH", url, data });
  }

  async delete<T = any>(
    url: string,
    config?: Omit<APIRequestConfig, "method" | "url">
  ): Promise<APIResponse<T>> {
    return this.request<T>({ ...config, method: "DELETE", url });
  }

  // Bulk operations with concurrency control
  async bulkRequest<T = any>(
    requests: APIRequestConfig[],
    concurrency: number = 5
  ): Promise<APIResponse<T>[]> {
    const results: APIResponse<T>[] = [];
    const executing: Promise<any>[] = [];

    for (const requestConfig of requests) {
      const promise = this.request<T>(requestConfig).then(
        (result) => {
          results.push(result);
          return result;
        },
        (error) => {
          // Store error as result for partial success handling
          results.push({
            data: null as any,
            status: error.status || 0,
            statusText: error.message,
            headers: {},
            error: error,
          } as APIResponse<T>);
          return error;
        }
      );

      executing.push(promise);

      if (executing.length >= concurrency) {
        await Promise.race(executing);
        executing.splice(
          executing.findIndex((p) => p === promise),
          1
        );
      }
    }

    await Promise.all(executing);
    return results;
  }

  // Health check for the API
  async healthCheck(): Promise<HealthCheckResult> {
    const startTime = Date.now();

    try {
      const response = await this.get("/health", {
        timeout: 5000,
        retry: { maxAttempts: 1 },
      });

      return {
        status: "healthy",
        responseTime: Date.now() - startTime,
        details: response.data,
      };
    } catch (error) {
      return {
        status: "unhealthy",
        responseTime: Date.now() - startTime,
        error: error.message,
      };
    }
  }

  // Get client statistics
  getStatistics(): ClientStatistics {
    return {
      circuitBreakerState: this.circuitBreaker.getState(),
      rateLimiterStats: this.rateLimiter.getStats(),
      cacheStats: this.cache.getStats(),
      requestMetrics: this.metrics.getMetrics(),
    };
  }

  private generateCorrelationId(): string {
    return `${Date.now()}-${Math.random().toString(36).substr(2, 9)}`;
  }

  private generateIdempotencyKey(config: APIRequestConfig): string {
    const payload = JSON.stringify({
      url: config.url,
      method: config.method,
      data: config.data,
      timestamp: Math.floor(Date.now() / 60000), // 1-minute window
    });
    return Buffer.from(payload).toString("base64");
  }

  private generateCacheKey(config: APIRequestConfig): string {
    const keyData = {
      method: config.method,
      url: config.url,
      params: config.params,
      // Don't include data for GET requests in cache key
      ...(config.method !== "GET" && { data: config.data }),
    };
    return Buffer.from(JSON.stringify(keyData)).toString("base64");
  }

  private setupRequestTracing(): void {
    // Add request/response logging and tracing
    if (process.env.NODE_ENV !== "production") {
      this.client.interceptors.request.use((config) => {
        console.debug(`→ ${config.method?.toUpperCase()} ${config.url}`, {
          headers: config.headers,
          data: config.data,
        });
        return config;
      });

      this.client.interceptors.response.use(
        (response) => {
          console.debug(`← ${response.status} ${response.config.url}`, {
            data: response.data,
          });
          return response;
        },
        (error) => {
          console.debug(`← ERROR ${error.config?.url}`, {
            status: error.response?.status,
            data: error.response?.data,
          });
          return Promise.reject(error);
        }
      );
    }
  }
}

// Supporting classes
class CircuitBreaker {
  private failures = 0;
  private lastFailureTime = 0;
  private state: "CLOSED" | "OPEN" | "HALF_OPEN" = "CLOSED";

  constructor(private options: CircuitBreakerOptions) {}

  isOpen(): boolean {
    if (this.state === "OPEN") {
      if (Date.now() - this.lastFailureTime > this.options.resetTimeoutMs) {
        this.state = "HALF_OPEN";
        return false;
      }
      return true;
    }
    return false;
  }

  recordSuccess(): void {
    this.failures = 0;
    this.state = "CLOSED";
  }

  recordFailure(): void {
    this.failures++;
    this.lastFailureTime = Date.now();

    if (this.failures >= this.options.failureThreshold) {
      this.state = "OPEN";
    }
  }

  getState(): string {
    return this.state;
  }
}

class RateLimiter {
  private tokens: number;
  private lastRefill: number;

  constructor(private options: RateLimitOptions) {
    this.tokens = options.maxTokens;
    this.lastRefill = Date.now();
  }

  async waitForToken(): Promise<void> {
    this.refillTokens();

    if (this.tokens <= 0) {
      const waitTime = this.calculateWaitTime();
      await new Promise((resolve) => setTimeout(resolve, waitTime));
      return this.waitForToken();
    }

    this.tokens--;
  }

  private refillTokens(): void {
    const now = Date.now();
    const timePassed = now - this.lastRefill;
    const tokensToAdd =
      Math.floor(timePassed / this.options.refillIntervalMs) *
      this.options.refillAmount;

    this.tokens = Math.min(this.options.maxTokens, this.tokens + tokensToAdd);
    this.lastRefill = now;
  }

  private calculateWaitTime(): number {
    return this.options.refillIntervalMs - (Date.now() - this.lastRefill);
  }

  getStats(): any {
    return {
      availableTokens: this.tokens,
      maxTokens: this.options.maxTokens,
    };
  }
}

class APICache {
  private cache = new Map<string, CachedResponse>();

  constructor(private options: CacheOptions) {}

  async get(key: string): Promise<any> {
    const cached = this.cache.get(key);
    if (!cached) return null;

    if (Date.now() > cached.expiresAt) {
      this.cache.delete(key);
      return null;
    }

    return cached.data;
  }

  async set(key: string, data: any, ttlSeconds: number): Promise<void> {
    this.cache.set(key, {
      data,
      expiresAt: Date.now() + ttlSeconds * 1000,
    });
  }

  getStats(): any {
    return {
      size: this.cache.size,
      maxSize: this.options.maxEntries || 1000,
    };
  }
}

class MetricsCollector {
  private metrics = {
    requestsTotal: 0,
    requestsSuccessful: 0,
    requestsFailed: 0,
    retriesTotal: 0,
    averageResponseTime: 0,
  };

  recordRequestSuccess(url: string, status: number, duration: number): void {
    this.metrics.requestsTotal++;
    this.metrics.requestsSuccessful++;
    this.updateAverageResponseTime(duration);
  }

  recordRequestError(
    url: string,
    status: number,
    duration: number,
    errorCode?: string
  ): void {
    this.metrics.requestsTotal++;
    this.metrics.requestsFailed++;
    this.updateAverageResponseTime(duration);
  }

  recordRetry(url: string, attempt: number): void {
    this.metrics.retriesTotal++;
  }

  private updateAverageResponseTime(duration: number): void {
    this.metrics.averageResponseTime =
      (this.metrics.averageResponseTime * (this.metrics.requestsTotal - 1) +
        duration) /
      this.metrics.requestsTotal;
  }

  getMetrics(): any {
    return { ...this.metrics };
  }
}

// Custom error class for API errors
class APIError extends Error {
  constructor(
    message: string,
    public code: string,
    public status: number,
    public metadata?: any
  ) {
    super(message);
    this.name = "APIError";
  }

  toJSON(): any {
    return {
      name: this.name,
      message: this.message,
      code: this.code,
      status: this.status,
      metadata: this.metadata,
    };
  }
}

// Interfaces
interface APIClientConfig {
  baseURL: string;
  serviceName: string;
  version: string;
  timeout?: number;
  defaultHeaders?: Record<string, string>;
  circuitBreakerOptions: CircuitBreakerOptions;
  rateLimitOptions: RateLimitOptions;
  cacheOptions: CacheOptions;
}

interface CircuitBreakerOptions {
  failureThreshold: number;
  resetTimeoutMs: number;
}

interface RateLimitOptions {
  maxTokens: number;
  refillIntervalMs: number;
  refillAmount: number;
}

interface CacheOptions {
  maxEntries?: number;
  defaultTTL?: number;
}

interface APIRequestConfig {
  method: "GET" | "POST" | "PUT" | "PATCH" | "DELETE";
  url: string;
  data?: any;
  params?: Record<string, any>;
  headers?: Record<string, string>;
  timeout?: number;
  retry?: {
    maxAttempts: number;
    initialDelayMs?: number;
    maxDelayMs?: number;
    backoffFactor?: number;
  };
  cache?: {
    enabled: boolean;
    ttlSeconds?: number;
  };
}

interface APIResponse<T = any> {
  data: T;
  status: number;
  statusText: string;
  headers: any;
  correlationId?: string;
  requestDuration?: number;
  error?: any;
}

interface CachedResponse {
  data: any;
  expiresAt: number;
}

interface HealthCheckResult {
  status: "healthy" | "unhealthy";
  responseTime: number;
  details?: any;
  error?: string;
}

interface ClientStatistics {
  circuitBreakerState: string;
  rateLimiterStats: any;
  cacheStats: any;
  requestMetrics: any;
}

Payment Gateway Integration: Handling Money Safely

Secure Payment Processing Architecture

// Comprehensive payment processing with multiple gateways and security
class PaymentGatewayManager {
  private gateways: Map<string, PaymentGateway> = new Map();
  private primaryGateway: string;
  private fallbackGateways: string[];
  private paymentLogger: PaymentLogger;
  private fraudDetector: FraudDetector;
  private webhookValidator: WebhookValidator;

  constructor(config: PaymentManagerConfig) {
    this.primaryGateway = config.primaryGateway;
    this.fallbackGateways = config.fallbackGateways || [];
    this.paymentLogger = new PaymentLogger();
    this.fraudDetector = new FraudDetector(config.fraudConfig);
    this.webhookValidator = new WebhookValidator();

    this.initializeGateways(config.gateways);
    this.setupHealthChecking();
  }

  private initializeGateways(gatewayConfigs: GatewayConfig[]): void {
    for (const config of gatewayConfigs) {
      let gateway: PaymentGateway;

      switch (config.provider) {
        case "stripe":
          gateway = new StripeGateway(config);
          break;
        case "paypal":
          gateway = new PayPalGateway(config);
          break;
        case "square":
          gateway = new SquareGateway(config);
          break;
        case "braintree":
          gateway = new BraintreeGateway(config);
          break;
        default:
          throw new Error(`Unsupported payment provider: ${config.provider}`);
      }

      this.gateways.set(config.name, gateway);
      console.log(`Initialized ${config.provider} gateway: ${config.name}`);
    }
  }

  // Process payment with automatic fallback
  async processPayment(request: PaymentRequest): Promise<PaymentResult> {
    const transactionId = this.generateTransactionId();

    try {
      // Log payment attempt
      await this.paymentLogger.logPaymentAttempt(transactionId, request);

      // Fraud detection
      const fraudCheck = await this.fraudDetector.analyzePayment(request);
      if (fraudCheck.risk === "HIGH") {
        throw new PaymentError(
          "Payment blocked due to fraud risk",
          "FRAUD_DETECTED",
          { fraudScore: fraudCheck.score }
        );
      }

      // Try primary gateway first
      const primaryGateway = this.gateways.get(this.primaryGateway);
      if (primaryGateway && (await primaryGateway.isHealthy())) {
        try {
          const result = await this.executePayment(
            primaryGateway,
            request,
            transactionId
          );

          await this.paymentLogger.logPaymentSuccess(transactionId, result);
          return result;
        } catch (error) {
          await this.paymentLogger.logPaymentError(
            transactionId,
            error,
            this.primaryGateway
          );

          // If primary fails, try fallbacks
          if (this.shouldFallback(error)) {
            console.warn(
              `Primary gateway ${this.primaryGateway} failed, trying fallbacks`
            );
            return await this.tryFallbackGateways(
              request,
              transactionId,
              error
            );
          }

          throw error;
        }
      }

      // Primary gateway unhealthy, go straight to fallbacks
      return await this.tryFallbackGateways(
        request,
        transactionId,
        new PaymentError("Primary gateway unavailable", "GATEWAY_UNAVAILABLE")
      );
    } catch (error) {
      await this.paymentLogger.logPaymentFailure(transactionId, error);
      throw error;
    }
  }

  private async executePayment(
    gateway: PaymentGateway,
    request: PaymentRequest,
    transactionId: string
  ): Promise<PaymentResult> {
    // Add transaction tracking
    const enhancedRequest = {
      ...request,
      transactionId,
      metadata: {
        ...request.metadata,
        gateway: gateway.getName(),
        timestamp: Date.now(),
        userAgent: request.userAgent,
        ipAddress: request.ipAddress,
      },
    };

    // Execute payment with timeout
    const timeoutPromise = new Promise<never>((_, reject) => {
      setTimeout(
        () => reject(new PaymentError("Payment timeout", "TIMEOUT")),
        30000 // 30 second timeout
      );
    });

    const paymentPromise = gateway.processPayment(enhancedRequest);
    const result = await Promise.race([paymentPromise, timeoutPromise]);

    // Validate result
    this.validatePaymentResult(result, request);

    return {
      ...result,
      transactionId,
      gateway: gateway.getName(),
      processedAt: Date.now(),
    };
  }

  private async tryFallbackGateways(
    request: PaymentRequest,
    transactionId: string,
    primaryError: Error
  ): Promise<PaymentResult> {
    for (const fallbackName of this.fallbackGateways) {
      const gateway = this.gateways.get(fallbackName);
      if (!gateway) continue;

      if (await gateway.isHealthy()) {
        try {
          console.log(`Trying fallback gateway: ${fallbackName}`);

          const result = await this.executePayment(
            gateway,
            request,
            transactionId
          );

          // Log successful fallback
          await this.paymentLogger.logFallbackSuccess(
            transactionId,
            fallbackName,
            primaryError
          );

          return result;
        } catch (fallbackError) {
          await this.paymentLogger.logPaymentError(
            transactionId,
            fallbackError,
            fallbackName
          );

          console.warn(
            `Fallback gateway ${fallbackName} failed:`,
            fallbackError.message
          );
        }
      }
    }

    // All gateways failed
    throw new PaymentError(
      "All payment gateways failed",
      "ALL_GATEWAYS_FAILED",
      { primaryError, gatewayCount: this.fallbackGateways.length + 1 }
    );
  }

  // Refund processing with gateway routing
  async processRefund(refundRequest: RefundRequest): Promise<RefundResult> {
    const originalPayment = await this.getPaymentDetails(
      refundRequest.originalTransactionId
    );

    if (!originalPayment) {
      throw new PaymentError("Original payment not found", "PAYMENT_NOT_FOUND");
    }

    const gateway = this.gateways.get(originalPayment.gateway);
    if (!gateway) {
      throw new PaymentError(
        "Original payment gateway not available",
        "GATEWAY_NOT_AVAILABLE"
      );
    }

    const refundId = this.generateRefundId();

    try {
      const result = await gateway.processRefund({
        ...refundRequest,
        refundId,
        originalPayment,
      });

      await this.paymentLogger.logRefundSuccess(refundId, result);
      return result;
    } catch (error) {
      await this.paymentLogger.logRefundError(refundId, error);
      throw error;
    }
  }

  // Webhook processing for payment updates
  async processWebhook(
    provider: string,
    signature: string,
    payload: any,
    headers: Record<string, string>
  ): Promise<WebhookResult> {
    const gateway = Array.from(this.gateways.values()).find(
      (g) => g.getProvider() === provider
    );

    if (!gateway) {
      throw new Error(`No gateway found for provider: ${provider}`);
    }

    // Validate webhook signature
    const isValid = await this.webhookValidator.validate(
      provider,
      signature,
      payload,
      headers
    );

    if (!isValid) {
      throw new PaymentError("Invalid webhook signature", "INVALID_SIGNATURE");
    }

    // Process webhook
    try {
      const result = await gateway.processWebhook(payload, headers);

      // Update payment status based on webhook
      if (result.transactionId) {
        await this.updatePaymentStatus(
          result.transactionId,
          result.status,
          result.metadata
        );
      }

      return result;
    } catch (error) {
      console.error(`Webhook processing error for ${provider}:`, error);
      throw error;
    }
  }

  // Payment method tokenization for recurring payments
  async tokenizePaymentMethod(
    tokenizationRequest: TokenizationRequest
  ): Promise<TokenizationResult> {
    const gateway = this.gateways.get(this.primaryGateway);
    if (!gateway) {
      throw new PaymentError(
        "Primary gateway not available",
        "GATEWAY_UNAVAILABLE"
      );
    }

    try {
      const result = await gateway.tokenizePaymentMethod(tokenizationRequest);

      // Store token securely (implementation depends on your security requirements)
      await this.storePaymentToken(result.token, {
        customerId: tokenizationRequest.customerId,
        gateway: gateway.getName(),
        paymentMethodType: result.paymentMethodType,
        lastFour: result.lastFour,
        expiryMonth: result.expiryMonth,
        expiryYear: result.expiryYear,
        createdAt: Date.now(),
      });

      return result;
    } catch (error) {
      console.error("Payment tokenization failed:", error);
      throw error;
    }
  }

  // Recurring payment processing
  async processRecurringPayment(
    recurringRequest: RecurringPaymentRequest
  ): Promise<PaymentResult> {
    const tokenData = await this.getPaymentToken(recurringRequest.tokenId);
    if (!tokenData) {
      throw new PaymentError("Payment token not found", "TOKEN_NOT_FOUND");
    }

    const gateway = this.gateways.get(tokenData.gateway);
    if (!gateway) {
      throw new PaymentError(
        "Gateway not available for token",
        "GATEWAY_UNAVAILABLE"
      );
    }

    const paymentRequest: PaymentRequest = {
      amount: recurringRequest.amount,
      currency: recurringRequest.currency,
      customerId: recurringRequest.customerId,
      paymentMethodToken: recurringRequest.tokenId,
      description: `Recurring payment: ${recurringRequest.description}`,
      metadata: {
        recurring: true,
        subscriptionId: recurringRequest.subscriptionId,
        ...recurringRequest.metadata,
      },
    };

    return await this.processPayment(paymentRequest);
  }

  // Payment gateway health monitoring
  private setupHealthChecking(): void {
    setInterval(async () => {
      for (const [name, gateway] of this.gateways) {
        try {
          const isHealthy = await gateway.isHealthy();
          if (!isHealthy) {
            console.warn(`Gateway ${name} is unhealthy`);
          }
        } catch (error) {
          console.error(`Health check failed for gateway ${name}:`, error);
        }
      }
    }, 60000); // Check every minute
  }

  // Utility methods
  private shouldFallback(error: any): boolean {
    // Determine if error warrants trying fallback gateways
    const fallbackErrors = [
      "GATEWAY_TIMEOUT",
      "GATEWAY_UNAVAILABLE",
      "RATE_LIMITED",
      "TEMPORARY_ERROR",
    ];

    return fallbackErrors.includes(error.code) || error.status >= 500;
  }

  private validatePaymentResult(
    result: PaymentResult,
    request: PaymentRequest
  ): void {
    if (!result.transactionId) {
      throw new PaymentError(
        "Missing transaction ID in result",
        "INVALID_RESULT"
      );
    }

    if (result.amount !== request.amount) {
      throw new PaymentError("Amount mismatch in result", "AMOUNT_MISMATCH");
    }

    if (result.currency !== request.currency) {
      throw new PaymentError(
        "Currency mismatch in result",
        "CURRENCY_MISMATCH"
      );
    }
  }

  private generateTransactionId(): string {
    return `txn_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
  }

  private generateRefundId(): string {
    return `ref_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
  }

  // Abstract methods that would be implemented based on your storage solution
  private async getPaymentDetails(transactionId: string): Promise<any> {
    // Implementation depends on your database
    return null;
  }

  private async updatePaymentStatus(
    transactionId: string,
    status: string,
    metadata?: any
  ): Promise<void> {
    // Implementation depends on your database
  }

  private async storePaymentToken(
    token: string,
    tokenData: any
  ): Promise<void> {
    // Implementation depends on your secure storage solution
  }

  private async getPaymentToken(tokenId: string): Promise<any> {
    // Implementation depends on your secure storage solution
    return null;
  }
}

// Example Stripe gateway implementation
class StripeGateway implements PaymentGateway {
  private stripe: any; // Stripe SDK instance
  private apiClient: RobustAPIClient;

  constructor(private config: GatewayConfig) {
    // Initialize Stripe SDK
    // this.stripe = require('stripe')(config.secretKey);

    this.apiClient = new RobustAPIClient({
      baseURL: "https://api.stripe.com",
      serviceName: "stripe-gateway",
      version: "1.0.0",
      defaultHeaders: {
        Authorization: `Bearer ${config.secretKey}`,
        "Stripe-Version": "2023-10-16",
      },
      circuitBreakerOptions: {
        failureThreshold: 5,
        resetTimeoutMs: 60000,
      },
      rateLimitOptions: {
        maxTokens: 100,
        refillIntervalMs: 1000,
        refillAmount: 10,
      },
      cacheOptions: {},
    });
  }

  async processPayment(request: PaymentRequest): Promise<PaymentResult> {
    try {
      const paymentIntentData = {
        amount: Math.round(request.amount * 100), // Convert to cents
        currency: request.currency.toLowerCase(),
        payment_method: request.paymentMethodId,
        customer: request.customerId,
        description: request.description,
        metadata: request.metadata,
        confirmation_method: "manual",
        confirm: true,
      };

      const response = await this.apiClient.post(
        "/v1/payment_intents",
        paymentIntentData
      );

      return this.transformStripeResult(response.data);
    } catch (error) {
      throw this.transformStripeError(error);
    }
  }

  async processRefund(
    request: RefundRequest & { originalPayment: any }
  ): Promise<RefundResult> {
    try {
      const refundData = {
        payment_intent: request.originalPayment.gatewayTransactionId,
        amount: request.amount ? Math.round(request.amount * 100) : undefined,
        reason: request.reason,
        metadata: request.metadata,
      };

      const response = await this.apiClient.post("/v1/refunds", refundData);

      return {
        refundId: response.data.id,
        amount: response.data.amount / 100,
        currency: response.data.currency.toUpperCase(),
        status: response.data.status,
        processedAt: Date.now(),
        gatewayRefundId: response.data.id,
      };
    } catch (error) {
      throw this.transformStripeError(error);
    }
  }

  async tokenizePaymentMethod(
    request: TokenizationRequest
  ): Promise<TokenizationResult> {
    // Stripe handles tokenization through payment methods
    try {
      const paymentMethodData = {
        type: "card",
        card: request.paymentMethodData,
        billing_details: request.billingDetails,
      };

      const response = await this.apiClient.post(
        "/v1/payment_methods",
        paymentMethodData
      );

      return {
        token: response.data.id,
        paymentMethodType: "card",
        lastFour: response.data.card.last4,
        expiryMonth: response.data.card.exp_month,
        expiryYear: response.data.card.exp_year,
        brand: response.data.card.brand,
      };
    } catch (error) {
      throw this.transformStripeError(error);
    }
  }

  async processWebhook(
    payload: any,
    headers: Record<string, string>
  ): Promise<WebhookResult> {
    const event = payload;

    switch (event.type) {
      case "payment_intent.succeeded":
        return {
          type: "payment_completed",
          transactionId: event.data.object.metadata?.transactionId,
          status: "completed",
          metadata: event.data.object,
        };

      case "payment_intent.payment_failed":
        return {
          type: "payment_failed",
          transactionId: event.data.object.metadata?.transactionId,
          status: "failed",
          metadata: event.data.object,
        };

      default:
        return {
          type: "unhandled",
          status: "ignored",
          metadata: event,
        };
    }
  }

  async isHealthy(): Promise<boolean> {
    try {
      const response = await this.apiClient.get("/v1/account", {
        timeout: 5000,
      });
      return response.status === 200;
    } catch (error) {
      return false;
    }
  }

  getName(): string {
    return this.config.name;
  }

  getProvider(): string {
    return "stripe";
  }

  private transformStripeResult(stripeResult: any): PaymentResult {
    return {
      transactionId: stripeResult.metadata?.transactionId || stripeResult.id,
      gatewayTransactionId: stripeResult.id,
      amount: stripeResult.amount / 100,
      currency: stripeResult.currency.toUpperCase(),
      status: stripeResult.status === "succeeded" ? "completed" : "pending",
      gatewayResponse: stripeResult,
    };
  }

  private transformStripeError(error: any): PaymentError {
    const stripeError = error.response?.data?.error || error;

    const errorCodeMap: Record<string, string> = {
      card_declined: "CARD_DECLINED",
      insufficient_funds: "INSUFFICIENT_FUNDS",
      invalid_cvc: "INVALID_CVC",
      expired_card: "EXPIRED_CARD",
      rate_limit: "RATE_LIMITED",
    };

    return new PaymentError(
      stripeError.message || error.message,
      errorCodeMap[stripeError.code] || "GATEWAY_ERROR",
      error.status,
      { stripeError, originalError: error }
    );
  }
}

// Supporting classes and interfaces
class PaymentLogger {
  async logPaymentAttempt(
    transactionId: string,
    request: PaymentRequest
  ): Promise<void> {
    console.log(`Payment attempt: ${transactionId}`, {
      amount: request.amount,
      currency: request.currency,
      customerId: request.customerId,
    });
  }

  async logPaymentSuccess(
    transactionId: string,
    result: PaymentResult
  ): Promise<void> {
    console.log(`Payment success: ${transactionId}`, result);
  }

  async logPaymentError(
    transactionId: string,
    error: any,
    gateway?: string
  ): Promise<void> {
    console.error(`Payment error: ${transactionId} (${gateway})`, error);
  }

  async logPaymentFailure(transactionId: string, error: any): Promise<void> {
    console.error(`Payment failure: ${transactionId}`, error);
  }

  async logFallbackSuccess(
    transactionId: string,
    fallbackGateway: string,
    primaryError: Error
  ): Promise<void> {
    console.log(`Fallback success: ${transactionId} via ${fallbackGateway}`, {
      primaryError: primaryError.message,
    });
  }

  async logRefundSuccess(
    refundId: string,
    result: RefundResult
  ): Promise<void> {
    console.log(`Refund success: ${refundId}`, result);
  }

  async logRefundError(refundId: string, error: any): Promise<void> {
    console.error(`Refund error: ${refundId}`, error);
  }
}

class FraudDetector {
  constructor(private config: FraudConfig) {}

  async analyzePayment(request: PaymentRequest): Promise<FraudAnalysis> {
    // Implement fraud detection logic
    // This is a simplified example
    let riskScore = 0;

    // Check for suspicious patterns
    if (request.amount > 1000) riskScore += 20;
    if (request.ipAddress && this.isHighRiskIP(request.ipAddress))
      riskScore += 30;
    if (request.userAgent && this.isSuspiciousUserAgent(request.userAgent))
      riskScore += 25;

    const risk = riskScore > 70 ? "HIGH" : riskScore > 40 ? "MEDIUM" : "LOW";

    return {
      score: riskScore,
      risk,
      reasons: [], // Would contain specific fraud indicators
    };
  }

  private isHighRiskIP(ip: string): boolean {
    // Implement IP risk checking
    return false;
  }

  private isSuspiciousUserAgent(userAgent: string): boolean {
    // Implement user agent analysis
    return false;
  }
}

class WebhookValidator {
  async validate(
    provider: string,
    signature: string,
    payload: any,
    headers: Record<string, string>
  ): Promise<boolean> {
    // Implement signature validation for different providers
    switch (provider) {
      case "stripe":
        return this.validateStripeSignature(signature, payload, headers);
      case "paypal":
        return this.validatePayPalSignature(signature, payload, headers);
      default:
        return false;
    }
  }

  private validateStripeSignature(
    signature: string,
    payload: any,
    headers: Record<string, string>
  ): boolean {
    // Implement Stripe signature validation
    return true; // Placeholder
  }

  private validatePayPalSignature(
    signature: string,
    payload: any,
    headers: Record<string, string>
  ): boolean {
    // Implement PayPal signature validation
    return true; // Placeholder
  }
}

class PaymentError extends Error {
  constructor(
    message: string,
    public code: string,
    public status?: number,
    public metadata?: any
  ) {
    super(message);
    this.name = "PaymentError";
  }
}

// Interfaces
interface PaymentManagerConfig {
  primaryGateway: string;
  fallbackGateways: string[];
  gateways: GatewayConfig[];
  fraudConfig: FraudConfig;
}

interface GatewayConfig {
  name: string;
  provider: string;
  secretKey: string;
  publicKey?: string;
  webhookSecret?: string;
  environment: "sandbox" | "production";
}

interface PaymentGateway {
  processPayment(request: PaymentRequest): Promise<PaymentResult>;
  processRefund(
    request: RefundRequest & { originalPayment: any }
  ): Promise<RefundResult>;
  tokenizePaymentMethod(
    request: TokenizationRequest
  ): Promise<TokenizationResult>;
  processWebhook(
    payload: any,
    headers: Record<string, string>
  ): Promise<WebhookResult>;
  isHealthy(): Promise<boolean>;
  getName(): string;
  getProvider(): string;
}

interface PaymentRequest {
  amount: number;
  currency: string;
  customerId?: string;
  paymentMethodId?: string;
  paymentMethodToken?: string;
  description?: string;
  metadata?: Record<string, any>;
  ipAddress?: string;
  userAgent?: string;
}

interface PaymentResult {
  transactionId?: string;
  gatewayTransactionId: string;
  amount: number;
  currency: string;
  status: "pending" | "completed" | "failed";
  gateway?: string;
  processedAt?: number;
  gatewayResponse?: any;
}

interface RefundRequest {
  originalTransactionId: string;
  amount?: number; // Partial refund if specified
  reason?: string;
  metadata?: Record<string, any>;
}

interface RefundResult {
  refundId: string;
  amount: number;
  currency: string;
  status: string;
  processedAt: number;
  gatewayRefundId: string;
}

interface TokenizationRequest {
  customerId: string;
  paymentMethodData: any;
  billingDetails?: any;
}

interface TokenizationResult {
  token: string;
  paymentMethodType: string;
  lastFour: string;
  expiryMonth: number;
  expiryYear: number;
  brand?: string;
}

interface RecurringPaymentRequest {
  tokenId: string;
  amount: number;
  currency: string;
  customerId: string;
  subscriptionId?: string;
  description: string;
  metadata?: Record<string, any>;
}

interface WebhookResult {
  type: string;
  transactionId?: string;
  status: string;
  metadata?: any;
}

interface FraudConfig {
  enabled: boolean;
  highRiskThreshold: number;
  mediumRiskThreshold: number;
}

interface FraudAnalysis {
  score: number;
  risk: "LOW" | "MEDIUM" | "HIGH";
  reasons: string[];
}

Email Service Integration: Reliable Communication

Multi-Provider Email System with Failover

// Comprehensive email service with multiple providers and advanced features
class EmailServiceManager {
  private providers: Map<string, EmailProvider> = new Map();
  private primaryProvider: string;
  private fallbackProviders: string[];
  private emailQueue: EmailQueue;
  private templateEngine: EmailTemplateEngine;
  private deliveryTracker: DeliveryTracker;
  private unsubscribeManager: UnsubscribeManager;

  constructor(config: EmailManagerConfig) {
    this.primaryProvider = config.primaryProvider;
    this.fallbackProviders = config.fallbackProviders || [];
    this.emailQueue = new EmailQueue(config.queueConfig);
    this.templateEngine = new EmailTemplateEngine(config.templateConfig);
    this.deliveryTracker = new DeliveryTracker();
    this.unsubscribeManager = new UnsubscribeManager();

    this.initializeProviders(config.providers);
    this.setupEmailProcessing();
  }

  private initializeProviders(providerConfigs: EmailProviderConfig[]): void {
    for (const config of providerConfigs) {
      let provider: EmailProvider;

      switch (config.service) {
        case "sendgrid":
          provider = new SendGridProvider(config);
          break;
        case "mailgun":
          provider = new MailgunProvider(config);
          break;
        case "ses":
          provider = new SESProvider(config);
          break;
        case "postmark":
          provider = new PostmarkProvider(config);
          break;
        default:
          throw new Error(`Unsupported email provider: ${config.service}`);
      }

      this.providers.set(config.name, provider);
      console.log(`Initialized ${config.service} provider: ${config.name}`);
    }
  }

  // Send email with automatic provider failover
  async sendEmail(emailRequest: EmailRequest): Promise<EmailResult> {
    const messageId = this.generateMessageId();

    try {
      // Validate email request
      this.validateEmailRequest(emailRequest);

      // Check unsubscribe status
      if (await this.unsubscribeManager.isUnsubscribed(emailRequest.to)) {
        throw new EmailError("Recipient has unsubscribed", "UNSUBSCRIBED", {
          recipient: emailRequest.to,
        });
      }

      // Process template if provided
      const processedEmail = await this.processEmailTemplate(emailRequest);

      // Add tracking and unsubscribe links
      const enhancedEmail = await this.enhanceEmailContent(
        processedEmail,
        messageId
      );

      // Try to send via primary provider
      const primaryProvider = this.providers.get(this.primaryProvider);
      if (primaryProvider && (await primaryProvider.isHealthy())) {
        try {
          const result = await this.sendViaProvider(
            primaryProvider,
            enhancedEmail,
            messageId
          );

          await this.deliveryTracker.trackSent(messageId, result);
          return result;
        } catch (error) {
          console.warn(
            `Primary provider ${this.primaryProvider} failed:`,
            error.message
          );

          // Try fallback providers if the error is retryable
          if (this.shouldTryFallback(error)) {
            return await this.tryFallbackProviders(
              enhancedEmail,
              messageId,
              error
            );
          }

          throw error;
        }
      }

      // Primary provider unhealthy, try fallbacks immediately
      return await this.tryFallbackProviders(
        enhancedEmail,
        messageId,
        new EmailError("Primary provider unavailable", "PROVIDER_UNAVAILABLE")
      );
    } catch (error) {
      await this.deliveryTracker.trackFailed(messageId, error);
      throw error;
    }
  }

  // Send bulk email with batching and rate limiting
  async sendBulkEmail(bulkRequest: BulkEmailRequest): Promise<BulkEmailResult> {
    const batchId = this.generateBatchId();
    const batchSize = bulkRequest.batchSize || 100;
    const delayMs = bulkRequest.delayBetweenBatches || 1000;

    try {
      const results: EmailResult[] = [];
      const errors: BulkEmailError[] = [];

      // Process template once for all recipients
      const baseEmail = await this.processEmailTemplate({
        template: bulkRequest.template,
        templateData: bulkRequest.commonData || {},
      } as EmailRequest);

      // Split recipients into batches
      const batches = this.chunkArray(bulkRequest.recipients, batchSize);

      for (let i = 0; i < batches.length; i++) {
        const batch = batches[i];
        console.log(
          `Processing batch ${i + 1}/${batches.length} (${
            batch.length
          } recipients)`
        );

        // Process batch concurrently
        const batchPromises = batch.map(async (recipient) => {
          try {
            // Personalize email for this recipient
            const personalizedEmail = await this.personalizeEmail(
              baseEmail,
              recipient
            );

            const result = await this.sendEmail(personalizedEmail);
            return { success: true, result, recipient: recipient.email };
          } catch (error) {
            return { success: false, error, recipient: recipient.email };
          }
        });

        const batchResults = await Promise.allSettled(batchPromises);

        // Process batch results
        for (const promiseResult of batchResults) {
          if (promiseResult.status === "fulfilled") {
            const { success, result, error, recipient } = promiseResult.value;
            if (success) {
              results.push(result);
            } else {
              errors.push({ recipient, error });
            }
          } else {
            errors.push({
              recipient: "unknown",
              error: new EmailError(
                promiseResult.reason.message,
                "BATCH_PROCESSING_ERROR"
              ),
            });
          }
        }

        // Delay between batches to respect rate limits
        if (i < batches.length - 1) {
          await new Promise((resolve) => setTimeout(resolve, delayMs));
        }
      }

      return {
        batchId,
        totalEmails: bulkRequest.recipients.length,
        successCount: results.length,
        failureCount: errors.length,
        results,
        errors,
        processedAt: Date.now(),
      };
    } catch (error) {
      throw new EmailError("Bulk email processing failed", "BULK_EMAIL_ERROR", {
        batchId,
        error,
      });
    }
  }

  // Queue email for later delivery
  async queueEmail(
    emailRequest: EmailRequest,
    options: EmailQueueOptions = {}
  ): Promise<string> {
    const messageId = this.generateMessageId();

    const queuedEmail: QueuedEmail = {
      messageId,
      emailRequest,
      queuedAt: Date.now(),
      priority: options.priority || "normal",
      sendAt: options.sendAt,
      maxAttempts: options.maxAttempts || 3,
      attemptCount: 0,
    };

    await this.emailQueue.enqueue(queuedEmail);

    console.log(`Email queued for delivery: ${messageId}`);
    return messageId;
  }

  // Process webhooks from email providers
  async processWebhook(
    provider: string,
    signature: string,
    payload: any,
    headers: Record<string, string>
  ): Promise<WebhookResult> {
    const emailProvider = Array.from(this.providers.values()).find(
      (p) => p.getProviderName() === provider
    );

    if (!emailProvider) {
      throw new Error(`No provider found for: ${provider}`);
    }

    // Validate webhook signature
    const isValid = await emailProvider.validateWebhook(
      signature,
      payload,
      headers
    );

    if (!isValid) {
      throw new EmailError("Invalid webhook signature", "INVALID_SIGNATURE");
    }

    // Process webhook events
    const events = await emailProvider.parseWebhookEvents(payload);

    for (const event of events) {
      await this.processWebhookEvent(event);
    }

    return {
      provider,
      eventsProcessed: events.length,
      processedAt: Date.now(),
    };
  }

  private async processWebhookEvent(event: EmailWebhookEvent): Promise<void> {
    switch (event.type) {
      case "delivered":
        await this.deliveryTracker.trackDelivered(
          event.messageId,
          event.timestamp,
          event.metadata
        );
        break;

      case "bounced":
        await this.deliveryTracker.trackBounced(
          event.messageId,
          event.timestamp,
          event.bounceType,
          event.reason
        );
        break;

      case "complained":
        await this.deliveryTracker.trackComplaint(
          event.messageId,
          event.timestamp,
          event.metadata
        );

        // Automatically unsubscribe spam complainers
        if (event.recipient) {
          await this.unsubscribeManager.addUnsubscribe(
            event.recipient,
            "spam_complaint",
            { messageId: event.messageId }
          );
        }
        break;

      case "clicked":
        await this.deliveryTracker.trackClicked(
          event.messageId,
          event.timestamp,
          event.url,
          event.metadata
        );
        break;

      case "opened":
        await this.deliveryTracker.trackOpened(
          event.messageId,
          event.timestamp,
          event.metadata
        );
        break;

      case "unsubscribed":
        if (event.recipient) {
          await this.unsubscribeManager.addUnsubscribe(
            event.recipient,
            "manual",
            { messageId: event.messageId }
          );
        }
        break;
    }
  }

  // Email analytics and reporting
  async getEmailAnalytics(
    dateRange: DateRange,
    filters?: EmailAnalyticsFilters
  ): Promise<EmailAnalytics> {
    return await this.deliveryTracker.getAnalytics(dateRange, filters);
  }

  private async tryFallbackProviders(
    email: ProcessedEmail,
    messageId: string,
    primaryError: Error
  ): Promise<EmailResult> {
    for (const fallbackName of this.fallbackProviders) {
      const provider = this.providers.get(fallbackName);
      if (!provider) continue;

      if (await provider.isHealthy()) {
        try {
          console.log(`Trying fallback provider: ${fallbackName}`);

          const result = await this.sendViaProvider(provider, email, messageId);

          // Log successful fallback
          console.log(
            `Email sent via fallback provider ${fallbackName}: ${messageId}`
          );

          await this.deliveryTracker.trackSent(messageId, result, {
            fallbackUsed: true,
            fallbackProvider: fallbackName,
            primaryError: primaryError.message,
          });

          return result;
        } catch (fallbackError) {
          console.warn(
            `Fallback provider ${fallbackName} failed:`,
            fallbackError.message
          );
        }
      }
    }

    // All providers failed
    throw new EmailError("All email providers failed", "ALL_PROVIDERS_FAILED", {
      primaryError,
      providerCount: this.fallbackProviders.length + 1,
      messageId,
    });
  }

  private async sendViaProvider(
    provider: EmailProvider,
    email: ProcessedEmail,
    messageId: string
  ): Promise<EmailResult> {
    const startTime = Date.now();

    try {
      const result = await provider.sendEmail({
        ...email,
        messageId,
        headers: {
          ...email.headers,
          "X-Message-ID": messageId,
        },
      });

      const duration = Date.now() - startTime;

      return {
        messageId,
        providerMessageId: result.providerMessageId,
        provider: provider.getProviderName(),
        status: "sent",
        sentAt: Date.now(),
        duration,
        recipient: email.to,
      };
    } catch (error) {
      const duration = Date.now() - startTime;

      throw new EmailError(
        error.message,
        error.code || "PROVIDER_ERROR",
        error.status,
        {
          provider: provider.getProviderName(),
          messageId,
          duration,
          originalError: error,
        }
      );
    }
  }

  private async processEmailTemplate(
    emailRequest: EmailRequest
  ): Promise<ProcessedEmail> {
    if (emailRequest.template) {
      return await this.templateEngine.render(
        emailRequest.template,
        emailRequest.templateData || {}
      );
    }

    return {
      to: emailRequest.to,
      from: emailRequest.from,
      subject: emailRequest.subject!,
      html: emailRequest.html,
      text: emailRequest.text,
      headers: emailRequest.headers || {},
      attachments: emailRequest.attachments || [],
    };
  }

  private async enhanceEmailContent(
    email: ProcessedEmail,
    messageId: string
  ): Promise<ProcessedEmail> {
    let enhancedHtml = email.html || "";
    let enhancedText = email.text || "";

    // Add tracking pixel for open tracking
    if (enhancedHtml) {
      const trackingPixel = `<img src="${process.env.TRACKING_URL}/open/${messageId}" width="1" height="1" alt="" style="display:none;" />`;
      enhancedHtml += trackingPixel;
    }

    // Add unsubscribe link
    const unsubscribeUrl = `${process.env.UNSUBSCRIBE_URL}/${messageId}`;
    const unsubscribeHtml = `<p><a href="${unsubscribeUrl}">Unsubscribe</a></p>`;
    const unsubscribeText = `\n\nUnsubscribe: ${unsubscribeUrl}`;

    if (enhancedHtml) {
      enhancedHtml += unsubscribeHtml;
    }
    if (enhancedText) {
      enhancedText += unsubscribeText;
    }

    return {
      ...email,
      html: enhancedHtml,
      text: enhancedText,
      headers: {
        ...email.headers,
        "List-Unsubscribe": `<${unsubscribeUrl}>`,
        "List-Unsubscribe-Post": "List-Unsubscribe=One-Click",
      },
    };
  }

  private async personalizeEmail(
    baseEmail: ProcessedEmail,
    recipient: EmailRecipient
  ): Promise<EmailRequest> {
    return {
      to: recipient.email,
      from: baseEmail.from,
      subject: this.personalizeString(baseEmail.subject, recipient.data),
      html: this.personalizeString(baseEmail.html || "", recipient.data),
      text: this.personalizeString(baseEmail.text || "", recipient.data),
      headers: baseEmail.headers,
      attachments: baseEmail.attachments,
    };
  }

  private personalizeString(
    template: string,
    data: Record<string, any>
  ): string {
    let result = template;

    // Simple template variable replacement
    for (const [key, value] of Object.entries(data)) {
      const pattern = new RegExp(`{{\\s*${key}\\s*}}`, "g");
      result = result.replace(pattern, String(value));
    }

    return result;
  }

  private setupEmailProcessing(): void {
    // Process queued emails
    setInterval(async () => {
      try {
        await this.processQueuedEmails();
      } catch (error) {
        console.error("Error processing queued emails:", error);
      }
    }, 10000); // Every 10 seconds

    // Health check providers
    setInterval(async () => {
      for (const [name, provider] of this.providers) {
        try {
          const isHealthy = await provider.isHealthy();
          if (!isHealthy) {
            console.warn(`Email provider ${name} is unhealthy`);
          }
        } catch (error) {
          console.error(`Health check failed for provider ${name}:`, error);
        }
      }
    }, 60000); // Every minute
  }

  private async processQueuedEmails(): Promise<void> {
    const emails = await this.emailQueue.dequeueReady();

    for (const queuedEmail of emails) {
      try {
        await this.sendEmail(queuedEmail.emailRequest);
        await this.emailQueue.markCompleted(queuedEmail.messageId);
      } catch (error) {
        queuedEmail.attemptCount++;

        if (queuedEmail.attemptCount >= queuedEmail.maxAttempts) {
          await this.emailQueue.markFailed(queuedEmail.messageId, error);
        } else {
          // Requeue with exponential backoff
          const delay = Math.pow(2, queuedEmail.attemptCount) * 60000; // Minutes
          await this.emailQueue.requeueWithDelay(queuedEmail.messageId, delay);
        }
      }
    }
  }

  // Utility methods
  private shouldTryFallback(error: any): boolean {
    const fallbackErrors = [
      "RATE_LIMITED",
      "PROVIDER_UNAVAILABLE",
      "TIMEOUT",
      "TEMPORARY_ERROR",
    ];

    return fallbackErrors.includes(error.code) || error.status >= 500;
  }

  private validateEmailRequest(request: EmailRequest): void {
    if (!request.to) {
      throw new EmailError("Missing recipient email", "MISSING_RECIPIENT");
    }

    if (!this.isValidEmail(request.to)) {
      throw new EmailError("Invalid recipient email", "INVALID_EMAIL");
    }

    if (!request.subject && !request.template) {
      throw new EmailError("Missing subject or template", "MISSING_SUBJECT");
    }

    if (!request.html && !request.text && !request.template) {
      throw new EmailError("Missing email content", "MISSING_CONTENT");
    }
  }

  private isValidEmail(email: string): boolean {
    const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
    return emailRegex.test(email);
  }

  private chunkArray<T>(array: T[], size: number): T[][] {
    const chunks = [];
    for (let i = 0; i < array.length; i += size) {
      chunks.push(array.slice(i, i + size));
    }
    return chunks;
  }

  private generateMessageId(): string {
    return `msg_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
  }

  private generateBatchId(): string {
    return `batch_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
  }
}

// Example SendGrid provider implementation
class SendGridProvider implements EmailProvider {
  private apiClient: RobustAPIClient;

  constructor(private config: EmailProviderConfig) {
    this.apiClient = new RobustAPIClient({
      baseURL: "https://api.sendgrid.com",
      serviceName: "sendgrid-email",
      version: "1.0.0",
      defaultHeaders: {
        Authorization: `Bearer ${config.apiKey}`,
        "Content-Type": "application/json",
      },
      circuitBreakerOptions: {
        failureThreshold: 5,
        resetTimeoutMs: 60000,
      },
      rateLimitOptions: {
        maxTokens: 100,
        refillIntervalMs: 1000,
        refillAmount: 10,
      },
      cacheOptions: {},
    });
  }

  async sendEmail(
    email: ProcessedEmail & { messageId: string }
  ): Promise<ProviderEmailResult> {
    try {
      const sendGridPayload = {
        personalizations: [
          {
            to: [{ email: email.to }],
            subject: email.subject,
          },
        ],
        from: { email: email.from },
        content: [
          ...(email.text ? [{ type: "text/plain", value: email.text }] : []),
          ...(email.html ? [{ type: "text/html", value: email.html }] : []),
        ],
        custom_args: {
          message_id: email.messageId,
        },
        tracking_settings: {
          click_tracking: { enable: true },
          open_tracking: { enable: true },
        },
      };

      const response = await this.apiClient.post(
        "/v3/mail/send",
        sendGridPayload
      );

      return {
        providerMessageId: response.headers["x-message-id"] || email.messageId,
        status: "accepted",
      };
    } catch (error) {
      throw this.transformSendGridError(error);
    }
  }

  async validateWebhook(
    signature: string,
    payload: any,
    headers: Record<string, string>
  ): Promise<boolean> {
    // Implement SendGrid webhook signature validation
    return true; // Placeholder
  }

  async parseWebhookEvents(payload: any): Promise<EmailWebhookEvent[]> {
    const events: EmailWebhookEvent[] = [];

    for (const event of payload) {
      events.push({
        type: this.mapSendGridEventType(event.event),
        messageId: event.sg_message_id,
        recipient: event.email,
        timestamp: event.timestamp,
        url: event.url,
        reason: event.reason,
        bounceType: event.type,
        metadata: event,
      });
    }

    return events;
  }

  async isHealthy(): Promise<boolean> {
    try {
      const response = await this.apiClient.get("/v3/user/account", {
        timeout: 5000,
      });
      return response.status === 200;
    } catch (error) {
      return false;
    }
  }

  getProviderName(): string {
    return "sendgrid";
  }

  private mapSendGridEventType(eventType: string): string {
    const eventMap: Record<string, string> = {
      delivered: "delivered",
      bounce: "bounced",
      dropped: "bounced",
      spamreport: "complained",
      click: "clicked",
      open: "opened",
      unsubscribe: "unsubscribed",
    };

    return eventMap[eventType] || eventType;
  }

  private transformSendGridError(error: any): EmailError {
    const sgError = error.response?.data?.errors?.[0] || {};

    return new EmailError(
      sgError.message || error.message,
      sgError.field === "from.email" ? "INVALID_FROM_EMAIL" : "SENDGRID_ERROR",
      error.status,
      { sendgridError: sgError, originalError: error }
    );
  }
}

// Supporting classes would be implemented similarly...

// Interfaces for email system
interface EmailManagerConfig {
  primaryProvider: string;
  fallbackProviders: string[];
  providers: EmailProviderConfig[];
  queueConfig: any;
  templateConfig: any;
}

interface EmailProviderConfig {
  name: string;
  service: string;
  apiKey: string;
  webhook_url?: string;
  sandbox?: boolean;
}

interface EmailProvider {
  sendEmail(
    email: ProcessedEmail & { messageId: string }
  ): Promise<ProviderEmailResult>;
  validateWebhook(
    signature: string,
    payload: any,
    headers: Record<string, string>
  ): Promise<boolean>;
  parseWebhookEvents(payload: any): Promise<EmailWebhookEvent[]>;
  isHealthy(): Promise<boolean>;
  getProviderName(): string;
}

interface EmailRequest {
  to: string;
  from: string;
  subject?: string;
  html?: string;
  text?: string;
  template?: string;
  templateData?: Record<string, any>;
  headers?: Record<string, string>;
  attachments?: EmailAttachment[];
}

interface ProcessedEmail {
  to: string;
  from: string;
  subject: string;
  html?: string;
  text?: string;
  headers: Record<string, string>;
  attachments: EmailAttachment[];
}

interface EmailResult {
  messageId: string;
  providerMessageId: string;
  provider: string;
  status: string;
  sentAt: number;
  duration: number;
  recipient: string;
}

interface BulkEmailRequest {
  template: string;
  commonData?: Record<string, any>;
  recipients: EmailRecipient[];
  batchSize?: number;
  delayBetweenBatches?: number;
}

interface EmailRecipient {
  email: string;
  data: Record<string, any>;
}

interface BulkEmailResult {
  batchId: string;
  totalEmails: number;
  successCount: number;
  failureCount: number;
  results: EmailResult[];
  errors: BulkEmailError[];
  processedAt: number;
}

interface BulkEmailError {
  recipient: string;
  error: Error;
}

interface EmailQueueOptions {
  priority?: "high" | "normal" | "low";
  sendAt?: number;
  maxAttempts?: number;
}

interface QueuedEmail {
  messageId: string;
  emailRequest: EmailRequest;
  queuedAt: number;
  priority: string;
  sendAt?: number;
  maxAttempts: number;
  attemptCount: number;
}

interface EmailWebhookEvent {
  type: string;
  messageId: string;
  recipient?: string;
  timestamp: number;
  url?: string;
  reason?: string;
  bounceType?: string;
  metadata?: any;
}

interface ProviderEmailResult {
  providerMessageId: string;
  status: string;
}

interface EmailAttachment {
  filename: string;
  content: string;
  type: string;
  disposition?: string;
}

interface DateRange {
  startDate: Date;
  endDate: Date;
}

interface EmailAnalyticsFilters {
  provider?: string;
  template?: string;
  recipient?: string;
}

interface EmailAnalytics {
  totalSent: number;
  delivered: number;
  bounced: number;
  opened: number;
  clicked: number;
  complained: number;
  unsubscribed: number;
  deliveryRate: number;
  openRate: number;
  clickRate: number;
  complaintRate: number;
}

interface WebhookResult {
  provider: string;
  eventsProcessed: number;
  processedAt: number;
}

class EmailError extends Error {
  constructor(
    message: string,
    public code: string,
    public status?: number,
    public metadata?: any
  ) {
    super(message);
    this.name = "EmailError";
  }
}

// Placeholder classes that would need full implementation
class EmailQueue {
  constructor(config: any) {}
  async enqueue(email: QueuedEmail): Promise<void> {}
  async dequeueReady(): Promise<QueuedEmail[]> {
    return [];
  }
  async markCompleted(messageId: string): Promise<void> {}
  async markFailed(messageId: string, error: Error): Promise<void> {}
  async requeueWithDelay(messageId: string, delayMs: number): Promise<void> {}
}

class EmailTemplateEngine {
  constructor(config: any) {}
  async render(
    templateName: string,
    data: Record<string, any>
  ): Promise<ProcessedEmail> {
    return {} as ProcessedEmail;
  }
}

class DeliveryTracker {
  async trackSent(
    messageId: string,
    result: EmailResult,
    metadata?: any
  ): Promise<void> {}
  async trackDelivered(
    messageId: string,
    timestamp: number,
    metadata?: any
  ): Promise<void> {}
  async trackBounced(
    messageId: string,
    timestamp: number,
    bounceType?: string,
    reason?: string
  ): Promise<void> {}
  async trackComplaint(
    messageId: string,
    timestamp: number,
    metadata?: any
  ): Promise<void> {}
  async trackClicked(
    messageId: string,
    timestamp: number,
    url?: string,
    metadata?: any
  ): Promise<void> {}
  async trackOpened(
    messageId: string,
    timestamp: number,
    metadata?: any
  ): Promise<void> {}
  async trackFailed(messageId: string, error: Error): Promise<void> {}
  async getAnalytics(
    dateRange: DateRange,
    filters?: EmailAnalyticsFilters
  ): Promise<EmailAnalytics> {
    return {} as EmailAnalytics;
  }
}

class UnsubscribeManager {
  async isUnsubscribed(email: string): Promise<boolean> {
    return false;
  }
  async addUnsubscribe(
    email: string,
    reason: string,
    metadata?: any
  ): Promise<void> {}
}

Key Takeaways

Third-party integration isn’t just about making HTTP requests—it’s about building resilient communication layers that can handle the unpredictable nature of services you don’t control.

Essential integration patterns:

  • Robust API clients with circuit breakers, retry logic, and comprehensive error handling
  • Payment processing with multi-gateway fallback and fraud detection
  • Email services with provider failover and delivery tracking
  • Webhook processing with signature validation and event handling
  • Health monitoring for all external dependencies

The production-ready integration framework:

  • Use comprehensive error handling that distinguishes between retryable and permanent failures
  • Implement intelligent fallback strategies when primary services fail
  • Build rate limiting and circuit breakers to protect both your system and external APIs
  • Design idempotent operations that can be safely retried
  • Plan graceful degradation when third-party services are unavailable

Integration best practices:

  • Always expect failures from external services and design accordingly
  • Monitor third-party service health as part of your system monitoring
  • Implement comprehensive logging for troubleshooting integration issues
  • Use correlation IDs to trace requests across service boundaries
  • Cache responses when appropriate to reduce external API calls

The architecture decision framework:

  • Use direct API integration for real-time, low-latency requirements
  • Use queue-based integration for high-volume, eventually consistent operations
  • Use webhook processing for event-driven updates from external services
  • Use batch processing for bulk operations and data synchronization
  • Use fallback services for critical functionality that must always work

What’s Next?

In the next blog, we’ll complete our third-party integration journey by diving into webhook handling and processing, API rate limiting and retry strategies, service reliability and fallbacks, integration testing strategies, and monitoring external dependencies.

We’ll explore the operational challenges of maintaining integrations at scale—from handling webhook floods during traffic spikes to debugging why a critical API integration started failing at 3 AM.

Because building integrations that work in development is straightforward. Making them bulletproof enough to handle the chaos of production traffic while maintaining your sanity—that’s where third-party integration becomes an art form.