Serverless & Cloud Functions

The $2.4 Million Serverless Bill That Killed a Startup

Picture this absolute engineering catastrophe: A promising food delivery startup with 500,000 users decides to go “fully serverless” to reduce infrastructure costs and complexity. Their CTO, fresh from a serverless conference, mandates that everything must run on AWS Lambda to achieve “infinite scalability” and “zero operational overhead.”

Six months later, their AWS bill arrived like a financial apocalypse:

The symptoms were financially devastating:

  • Monthly AWS costs exploded to $2.4 million: What started as a $50,000/month traditional server setup became a budget-destroying monster
  • Lambda invocations hit 847 million per month: Their inefficient function design was triggering cascading executions for simple user actions
  • Cold start delays averaged 8+ seconds: Users were abandoning orders faster than their functions could warm up
  • Memory usage averaged 2GB per function: Over-allocated memory was burning through compute costs unnecessarily
  • Data transfer costs reached $890,000: Functions were downloading the same static assets millions of times
  • CloudWatch logs storage hit $145,000: Excessive logging was generating terabytes of mostly useless data

Here’s what their expensive serverless audit revealed:

  • Monolithic functions: They migrated their entire Express.js app into single Lambda functions, negating all serverless benefits
  • Database connection chaos: Every function invocation was opening new database connections, exhausting connection pools
  • Excessive over-provisioning: Functions allocated 3GB of memory when they needed 128MB, paying 24x more than necessary
  • Anti-pattern event chains: Simple user actions triggered 15+ Lambda invocations in sequence
  • No cold start optimization: Functions were loading entire dependency trees on every cold start
  • Unoptimized bundling: Each function was 50MB+ because they bundled all dependencies instead of sharing common layers

The final damage:

  • $2.4 million monthly burn rate that consumed their entire Series A funding in 4 months
  • Company shutdown when they couldn’t raise emergency funding to cover infrastructure costs
  • Complete customer base exodus due to 8-second loading times that made the app unusable
  • Engineering team liquidation as developers fled a company that couldn’t afford basic infrastructure
  • Founder reputation destruction as the “serverless success story” became an industry cautionary tale

The brutal truth? Every single serverless anti-pattern they implemented could have been avoided with proper architecture design and understanding of when serverless actually makes sense versus when it becomes an expensive nightmare.

The Uncomfortable Truth About Serverless

Here’s what separates applications that benefit from serverless architecture from those that get bankrupted by it: Serverless isn’t about eliminating infrastructure complexity—it’s about shifting it to a different layer where you pay premium prices for convenience. Get the patterns wrong, and you’ll pay exponentially more while getting worse performance.

Most developers approach serverless like this:

  1. Assume “serverless” means “no infrastructure knowledge needed”
  2. Migrate existing monolithic applications directly to Lambda functions
  3. Use serverless for everything, regardless of workload characteristics
  4. Ignore cold start implications and over-provision memory “to be safe”
  5. Skip cost optimization until the monthly bill arrives

But developers who actually benefit from serverless think differently:

  1. Design for event-driven patterns where serverless provides genuine value over traditional architectures
  2. Right-size functions for specific workloads, not one-size-fits-all approaches
  3. Understand cost models and optimize for actual usage patterns, not theoretical maximums
  4. Architect for cold starts or choose alternatives when latency is critical
  5. Monitor and optimize continuously because serverless costs can spiral quickly with scale

The difference isn’t just monthly bills—it’s the difference between systems that scale cost-effectively and systems that become financially unsustainable as they grow.

Ready to build applications that leverage serverless benefits without the bankruptcy-inducing costs? Let’s dive into serverless patterns that actually work in production.


Serverless Architecture Fundamentals: Beyond the Marketing Hype

The Problem: Monolithic Functions That Defeat the Purpose

// The serverless nightmare that costs 10x more than necessary
exports.handler = async (event) => {
  // Loading entire application on every invocation - RED FLAG #1
  const express = require("express");
  const mongoose = require("mongoose");
  const jwt = require("jsonwebtoken");
  const bcrypt = require("bcrypt");
  const nodemailer = require("nodemailer");
  const aws = require("aws-sdk");
  const stripe = require("stripe");
  const redis = require("redis");

  // Creating new database connections on every invocation - RED FLAG #2
  await mongoose.connect(process.env.MONGODB_URL);

  const app = express();

  // Defining entire API inside Lambda function - RED FLAG #3
  app.get("/api/users", async (req, res) => {
    // No connection pooling or optimization - RED FLAG #4
    const users = await User.find({}).limit(1000);
    res.json(users);
  });

  app.post("/api/users", async (req, res) => {
    // Complex business logic in serverless function - RED FLAG #5
    const hashedPassword = await bcrypt.hash(req.body.password, 12);
    const user = new User({ ...req.body, password: hashedPassword });
    await user.save();

    // Synchronous email sending blocking response - RED FLAG #6
    await nodemailer.createTransport(emailConfig).sendMail({
      to: user.email,
      subject: "Welcome!",
      html: await renderComplexEmailTemplate(user),
    });

    res.json(user);
  });

  app.post("/api/orders", async (req, res) => {
    // Multiple external API calls in sequence - RED FLAG #7
    const user = await User.findById(req.body.userId);
    const inventory = await checkInventoryAPI(req.body.items);
    const pricing = await calculatePricingAPI(req.body.items);
    const tax = await calculateTaxAPI(req.body.total, user.address);
    const shipping = await calculateShippingAPI(req.body.items);

    // Payment processing in serverless function - RED FLAG #8
    const payment = await stripe.charges.create({
      amount: req.body.total,
      currency: "usd",
      source: req.body.paymentToken,
    });

    const order = new Order({ ...req.body, paymentId: payment.id });
    await order.save();

    res.json(order);
  });

  // Using serverless-http adapter - RED FLAG #9
  const serverlessHandler = require("serverless-http")(app);

  return await serverlessHandler(event, context);

  // Problems this creates:
  // - Every request loads 50MB+ of dependencies
  // - Cold starts take 8+ seconds
  // - Memory usage is 2GB+ per function
  // - Database connections are created/destroyed constantly
  // - No sharing of resources between invocations
  // - Costs scale linearly with requests instead of efficiently
  // - Single point of failure for entire API
};

The Solution: Event-Driven Microfunction Architecture

// Proper serverless architecture with focused functions
import { APIGatewayProxyEvent, APIGatewayProxyResult } from "aws-lambda";
import { DynamoDB } from "aws-sdk";
import { SQS } from "aws-sdk";
import { SNS } from "aws-sdk";

// Shared layer with common utilities (deployed as Lambda Layer)
import {
  createResponse,
  validateJWT,
  logMetrics,
  connectToDatabase,
} from "/opt/nodejs/shared-utils";

// Individual function: Get User Profile
export const getUserProfile = async (
  event: APIGatewayProxyEvent
): Promise<APIGatewayProxyResult> => {
  const startTime = Date.now();

  try {
    // JWT validation using shared utility
    const user = await validateJWT(event.headers.Authorization);

    // Optimized database query with connection reuse
    const db = await connectToDatabase();
    const userProfile = await db
      .collection("users")
      .findOne(
        { _id: user.id },
        { projection: { password: 0, internalNotes: 0 } }
      );

    if (!userProfile) {
      return createResponse(404, { error: "User not found" });
    }

    // Log metrics for monitoring
    await logMetrics("getUserProfile", {
      userId: user.id,
      duration: Date.now() - startTime,
      success: true,
    });

    return createResponse(200, userProfile);
  } catch (error) {
    await logMetrics("getUserProfile", {
      duration: Date.now() - startTime,
      success: false,
      error: error.message,
    });

    return createResponse(500, { error: "Failed to fetch user profile" });
  }
};

// Individual function: Create User (with proper async patterns)
export const createUser = async (
  event: APIGatewayProxyEvent
): Promise<APIGatewayProxyResult> => {
  const startTime = Date.now();

  try {
    const userData = JSON.parse(event.body || "{}");

    // Input validation
    if (!userData.email || !userData.password) {
      return createResponse(400, { error: "Email and password required" });
    }

    const db = await connectToDatabase();

    // Check if user exists
    const existingUser = await db.collection("users").findOne({
      email: userData.email,
    });

    if (existingUser) {
      return createResponse(409, { error: "User already exists" });
    }

    // Hash password (keeping CPU work in function)
    const bcrypt = await import("bcryptjs");
    const hashedPassword = await bcrypt.hash(userData.password, 12);

    // Create user
    const newUser = {
      ...userData,
      password: hashedPassword,
      createdAt: new Date(),
      verified: false,
    };

    const result = await db.collection("users").insertOne(newUser);

    // Publish event for async processing (email, analytics, etc.)
    const sns = new SNS();
    await sns
      .publish({
        TopicArn: process.env.USER_EVENTS_TOPIC,
        Message: JSON.stringify({
          eventType: "USER_CREATED",
          userId: result.insertedId,
          email: userData.email,
          timestamp: new Date().toISOString(),
        }),
      })
      .promise();

    // Return immediately without waiting for email/analytics
    const { password: _, ...userResponse } = newUser;

    await logMetrics("createUser", {
      duration: Date.now() - startTime,
      success: true,
    });

    return createResponse(201, { ...userResponse, id: result.insertedId });
  } catch (error) {
    await logMetrics("createUser", {
      duration: Date.now() - startTime,
      success: false,
      error: error.message,
    });

    return createResponse(500, { error: "Failed to create user" });
  }
};

// Background processing function: Handle User Events
export const processUserEvents = async (event: any): Promise<void> => {
  // Process SNS messages asynchronously
  for (const record of event.Records) {
    try {
      const message = JSON.parse(record.Sns.Message);

      switch (message.eventType) {
        case "USER_CREATED":
          await handleUserCreated(message);
          break;
        case "USER_UPDATED":
          await handleUserUpdated(message);
          break;
        default:
          console.log(`Unknown event type: ${message.eventType}`);
      }
    } catch (error) {
      console.error("Failed to process user event:", error);
      // Dead letter queue will handle failed messages
    }
  }
};

// Optimized order processing with proper async patterns
export const processOrder = async (
  event: APIGatewayProxyEvent
): Promise<APIGatewayProxyResult> => {
  const startTime = Date.now();

  try {
    const orderData = JSON.parse(event.body || "{}");
    const user = await validateJWT(event.headers.Authorization);

    // Validate order data
    if (!orderData.items || orderData.items.length === 0) {
      return createResponse(400, { error: "Order must contain items" });
    }

    // Step 1: Quick inventory check (synchronous)
    const inventoryCheck = await quickInventoryCheck(orderData.items);
    if (!inventoryCheck.available) {
      return createResponse(409, {
        error: "Some items are out of stock",
        unavailableItems: inventoryCheck.unavailableItems,
      });
    }

    // Step 2: Create pending order
    const db = await connectToDatabase();
    const pendingOrder = {
      ...orderData,
      userId: user.id,
      status: "pending",
      createdAt: new Date(),
    };

    const orderResult = await db.collection("orders").insertOne(pendingOrder);
    const orderId = orderResult.insertedId;

    // Step 3: Queue order for async processing
    const sqs = new SQS();
    await sqs
      .sendMessage({
        QueueUrl: process.env.ORDER_PROCESSING_QUEUE,
        MessageBody: JSON.stringify({
          orderId,
          userId: user.id,
          items: orderData.items,
          total: orderData.total,
          paymentMethod: orderData.paymentMethod,
        }),
        DelaySeconds: 0,
      })
      .promise();

    await logMetrics("processOrder", {
      orderId,
      userId: user.id,
      itemCount: orderData.items.length,
      total: orderData.total,
      duration: Date.now() - startTime,
      success: true,
    });

    // Return immediately with pending status
    return createResponse(202, {
      orderId,
      status: "pending",
      message: "Order is being processed",
    });
  } catch (error) {
    await logMetrics("processOrder", {
      duration: Date.now() - startTime,
      success: false,
      error: error.message,
    });

    return createResponse(500, { error: "Failed to process order" });
  }
};

// Dedicated function for heavy order processing
export const processOrderBackground = async (event: any): Promise<void> => {
  // Process SQS messages with proper error handling
  for (const record of event.Records) {
    let orderData;

    try {
      orderData = JSON.parse(record.body);

      // Parallel processing of order components
      const [inventory, pricing, tax, shipping] = await Promise.all([
        reserveInventory(orderData.items),
        calculatePricing(orderData.items),
        calculateTax(orderData.total, orderData.userId),
        calculateShipping(orderData.items, orderData.userId),
      ]);

      // Process payment
      const payment = await processPayment({
        amount: pricing.total,
        paymentMethod: orderData.paymentMethod,
        orderId: orderData.orderId,
      });

      // Update order status
      const db = await connectToDatabase();
      await db.collection("orders").updateOne(
        { _id: orderData.orderId },
        {
          $set: {
            status: "confirmed",
            paymentId: payment.id,
            inventoryReservation: inventory.reservationId,
            finalTotal: pricing.total,
            updatedAt: new Date(),
          },
        }
      );

      // Notify user of order confirmation
      await publishOrderConfirmation({
        orderId: orderData.orderId,
        userId: orderData.userId,
        total: pricing.total,
      });
    } catch (error) {
      console.error("Order processing failed:", error);

      // Update order with failure status
      if (orderData?.orderId) {
        const db = await connectToDatabase();
        await db.collection("orders").updateOne(
          { _id: orderData.orderId },
          {
            $set: {
              status: "failed",
              error: error.message,
              updatedAt: new Date(),
            },
          }
        );

        // Notify user of order failure
        await publishOrderFailure({
          orderId: orderData.orderId,
          userId: orderData.userId,
          error: error.message,
        });
      }

      // Re-throw to trigger DLQ if configured
      throw error;
    }
  }
};

// Utility functions for common operations
async function quickInventoryCheck(
  items: OrderItem[]
): Promise<InventoryCheckResult> {
  // Optimized inventory check using DynamoDB batch operations
  const dynamodb = new DynamoDB.DocumentClient();

  const keys = items.map((item) => ({ productId: item.productId }));

  const result = await dynamodb
    .batchGet({
      RequestItems: {
        [process.env.INVENTORY_TABLE!]: {
          Keys: keys,
          ProjectionExpression: "productId, availableQuantity",
        },
      },
    })
    .promise();

  const inventory = result.Responses?.[process.env.INVENTORY_TABLE!] || [];
  const unavailableItems: string[] = [];

  for (const item of items) {
    const inventoryItem = inventory.find(
      (inv) => inv.productId === item.productId
    );
    if (!inventoryItem || inventoryItem.availableQuantity < item.quantity) {
      unavailableItems.push(item.productId);
    }
  }

  return {
    available: unavailableItems.length === 0,
    unavailableItems,
  };
}

async function publishOrderConfirmation(orderInfo: OrderInfo): Promise<void> {
  const sns = new SNS();
  await sns
    .publish({
      TopicArn: process.env.ORDER_EVENTS_TOPIC,
      Message: JSON.stringify({
        eventType: "ORDER_CONFIRMED",
        ...orderInfo,
        timestamp: new Date().toISOString(),
      }),
    })
    .promise();
}

// Supporting interfaces
interface OrderItem {
  productId: string;
  quantity: number;
  price: number;
}

interface InventoryCheckResult {
  available: boolean;
  unavailableItems: string[];
}

interface OrderInfo {
  orderId: string;
  userId: string;
  total: number;
}

Advanced Serverless Patterns: Cold Start Optimization & Cost Management

The Problem: Cold Starts That Kill User Experience

// The cold start nightmare that makes users abandon your app
import * as AWS from "aws-sdk"; // 45MB dependency - RED FLAG #1
import mongoose from "mongoose"; // 12MB dependency - RED FLAG #2
import { v4 as uuidv4 } from "uuid";
import bcrypt from "bcrypt";
import jwt from "jsonwebtoken";
import nodemailer from "nodemailer";
import sharp from "sharp"; // 8MB image processing library - RED FLAG #3
import puppeteer from "puppeteer"; // 85MB Chrome installation - RED FLAG #4

// Global variables that get initialized on every cold start - RED FLAG #5
let dbConnection: any;
let emailTransporter: any;
let s3Client: any;
let redisClient: any;

exports.handler = async (event: any, context: any) => {
  const startTime = Date.now();

  // Expensive initialization on every cold start - RED FLAG #6
  if (!dbConnection) {
    console.log("Initializing database connection...");
    dbConnection = await mongoose.connect(process.env.MONGODB_URL, {
      useNewUrlParser: true,
      useUnifiedTopology: true,
      maxPoolSize: 1, // Only 1 connection per container - RED FLAG #7
    });
  }

  if (!emailTransporter) {
    console.log("Initializing email transporter...");
    emailTransporter = nodemailer.createTransporter({
      service: "gmail",
      auth: {
        user: process.env.EMAIL_USER,
        pass: process.env.EMAIL_PASS,
      },
    });
  }

  if (!s3Client) {
    console.log("Initializing S3 client...");
    s3Client = new AWS.S3({
      region: process.env.AWS_REGION,
    });
  }

  // Complex initialization logic - RED FLAG #8
  const browser = await puppeteer.launch({
    headless: true,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });

  try {
    // Heavy processing that should be optimized - RED FLAG #9
    const page = await browser.newPage();
    await page.goto("https://example.com");
    const screenshot = await page.screenshot();

    // Large memory allocation - RED FLAG #10
    const processedImage = await sharp(screenshot)
      .resize(1920, 1080)
      .jpeg({ quality: 90 })
      .toBuffer();

    const uploadResult = await s3Client
      .upload({
        Bucket: process.env.S3_BUCKET,
        Key: `screenshots/${uuidv4()}.jpg`,
        Body: processedImage,
      })
      .promise();

    console.log(`Total execution time: ${Date.now() - startTime}ms`);

    return {
      statusCode: 200,
      body: JSON.stringify({ imageUrl: uploadResult.Location }),
    };
  } finally {
    await browser.close();
  }

  // Problems this creates:
  // - 8-12 second cold start times
  // - 512MB-1GB memory allocation required
  // - Expensive package initialization on every cold start
  // - No connection reuse optimization
  // - Single function doing too many different things
};

The Solution: Optimized Serverless Architecture with Proper Resource Management

// High-performance serverless functions with cold start optimization
import {
  APIGatewayProxyEvent,
  APIGatewayProxyResult,
  Context,
} from "aws-lambda";

// Lightweight imports - only what's needed per function
import { DynamoDB } from "aws-sdk/clients/dynamodb";
import { S3 } from "aws-sdk/clients/s3";
import { SQS } from "aws-sdk/clients/sqs";

// Global variables for connection reuse (Lambda container reuse)
let dynamoClient: DynamoDB.DocumentClient;
let s3Client: S3;
let sqsClient: SQS;

// Connection initialization with proper error handling
const initializeClients = () => {
  if (!dynamoClient) {
    dynamoClient = new DynamoDB.DocumentClient({
      region: process.env.AWS_REGION,
      maxRetries: 3,
      retryDelayOptions: {
        customBackoff: (retryCount) => Math.pow(2, retryCount) * 100,
      },
    });
  }

  if (!s3Client) {
    s3Client = new S3({
      region: process.env.AWS_REGION,
      signatureVersion: "v4",
    });
  }

  if (!sqsClient) {
    sqsClient = new SQS({
      region: process.env.AWS_REGION,
    });
  }
};

// Lightweight function: Get User Data
export const getUserData = async (
  event: APIGatewayProxyEvent,
  context: Context
): Promise<APIGatewayProxyResult> => {
  // Initialize clients only when needed
  initializeClients();

  const startTime = Date.now();
  context.callbackWaitsForEmptyEventLoop = false; // Don't wait for event loop

  try {
    const userId = event.pathParameters?.userId;
    if (!userId) {
      return createResponse(400, { error: "User ID required" });
    }

    // Optimized DynamoDB query with projection
    const result = await dynamoClient
      .get({
        TableName: process.env.USERS_TABLE!,
        Key: { userId },
        ProjectionExpression: "userId, #name, email, createdAt, lastLoginAt",
        ExpressionAttributeNames: {
          "#name": "name", // Reserved keyword handling
        },
      })
      .promise();

    if (!result.Item) {
      return createResponse(404, { error: "User not found" });
    }

    // Add execution metrics
    const executionTime = Date.now() - startTime;
    const response = {
      ...result.Item,
      _metadata: {
        executionTime,
        coldStart:
          context.getRemainingTimeInMillis() === context.invokedFunctionArn
            ? true
            : false,
      },
    };

    return createResponse(200, response);
  } catch (error) {
    console.error("getUserData error:", error);
    return createResponse(500, {
      error: "Failed to fetch user data",
      requestId: context.awsRequestId,
    });
  }
};

// Optimized image processing function with proper memory management
export const processImage = async (
  event: any,
  context: Context
): Promise<void> => {
  initializeClients();
  context.callbackWaitsForEmptyEventLoop = false;

  // Process S3 events efficiently
  for (const record of event.Records) {
    try {
      const bucket = record.s3.bucket.name;
      const key = decodeURIComponent(record.s3.object.key.replace(/\+/g, " "));

      // Stream processing to minimize memory usage
      const originalImage = s3Client
        .getObject({
          Bucket: bucket,
          Key: key,
        })
        .createReadStream();

      // Use streaming transformation instead of loading entire image
      const { Transform } = await import("stream");
      const sharp = (await import("sharp")).default;

      const thumbnailTransform = sharp()
        .resize(200, 200, {
          fit: "cover",
          position: "center",
        })
        .jpeg({
          quality: 80,
          progressive: true,
          mozjpeg: true,
        });

      const mediumTransform = sharp()
        .resize(800, 600, {
          fit: "inside",
          withoutEnlargement: true,
        })
        .jpeg({
          quality: 85,
          progressive: true,
        });

      // Parallel processing with streams
      const [thumbnailUpload, mediumUpload] = await Promise.all([
        s3Client
          .upload({
            Bucket: bucket,
            Key: `thumbnails/${key}`,
            Body: originalImage.pipe(thumbnailTransform),
            ContentType: "image/jpeg",
            CacheControl: "public, max-age=31536000",
          })
          .promise(),

        s3Client
          .upload({
            Bucket: bucket,
            Key: `medium/${key}`,
            Body: originalImage.pipe(mediumTransform),
            ContentType: "image/jpeg",
            CacheControl: "public, max-age=31536000",
          })
          .promise(),
      ]);

      // Update database with processed image URLs
      await dynamoClient
        .update({
          TableName: process.env.IMAGES_TABLE!,
          Key: { imageId: key },
          UpdateExpression:
            "SET thumbnailUrl = :thumbnail, mediumUrl = :medium, processedAt = :processedAt",
          ExpressionAttributeValues: {
            ":thumbnail": thumbnailUpload.Location,
            ":medium": mediumUpload.Location,
            ":processedAt": new Date().toISOString(),
          },
        })
        .promise();

      console.log(`Successfully processed image: ${key}`);
    } catch (error) {
      console.error(`Failed to process image ${record.s3.object.key}:`, error);

      // Send to dead letter queue for manual review
      await sqsClient
        .sendMessage({
          QueueUrl: process.env.FAILED_PROCESSING_QUEUE!,
          MessageBody: JSON.stringify({
            error: error.message,
            s3Record: record,
            timestamp: new Date().toISOString(),
          }),
        })
        .promise();
    }
  }
};

// WebSocket connection handler with proper state management
export const handleWebSocketConnection = async (
  event: any,
  context: Context
): Promise<any> => {
  initializeClients();
  context.callbackWaitsForEmptyEventLoop = false;

  const { routeKey, connectionId } = event.requestContext;

  try {
    switch (routeKey) {
      case "$connect":
        return await handleConnect(connectionId, event);
      case "$disconnect":
        return await handleDisconnect(connectionId);
      case "sendMessage":
        return await handleSendMessage(connectionId, JSON.parse(event.body));
      default:
        return { statusCode: 400, body: "Unknown route" };
    }
  } catch (error) {
    console.error(`WebSocket ${routeKey} error:`, error);
    return { statusCode: 500, body: "Internal server error" };
  }
};

// Scheduled function: Clean up old data with batch processing
export const cleanupOldData = async (
  event: any,
  context: Context
): Promise<void> => {
  initializeClients();
  context.callbackWaitsForEmptyEventLoop = false;

  const thirtyDaysAgo = new Date(
    Date.now() - 30 * 24 * 60 * 60 * 1000
  ).toISOString();
  let processedItems = 0;

  try {
    // Batch process old items to avoid timeout
    let lastEvaluatedKey: any = undefined;

    do {
      const scanResult = await dynamoClient
        .scan({
          TableName: process.env.TEMP_DATA_TABLE!,
          FilterExpression: "createdAt < :thirtyDaysAgo",
          ExpressionAttributeValues: {
            ":thirtyDaysAgo": thirtyDaysAgo,
          },
          Limit: 25, // Process in small batches
          ExclusiveStartKey: lastEvaluatedKey,
        })
        .promise();

      if (scanResult.Items && scanResult.Items.length > 0) {
        // Batch delete items
        const deleteRequests = scanResult.Items.map((item) => ({
          DeleteRequest: {
            Key: { id: item.id },
          },
        }));

        // Process in chunks of 25 (DynamoDB batch limit)
        const chunks = chunkArray(deleteRequests, 25);

        for (const chunk of chunks) {
          await dynamoClient
            .batchWrite({
              RequestItems: {
                [process.env.TEMP_DATA_TABLE!]: chunk,
              },
            })
            .promise();
        }

        processedItems += scanResult.Items.length;
      }

      lastEvaluatedKey = scanResult.LastEvaluatedKey;

      // Check remaining execution time to avoid timeout
      if (context.getRemainingTimeInMillis() < 10000) {
        console.log("Approaching timeout, stopping cleanup");
        break;
      }
    } while (lastEvaluatedKey);

    console.log(`Cleaned up ${processedItems} old items`);
  } catch (error) {
    console.error("Cleanup failed:", error);
    throw error;
  }
};

// Advanced serverless monitoring and metrics
export class ServerlessMetrics {
  private static instance: ServerlessMetrics;

  static getInstance(): ServerlessMetrics {
    if (!ServerlessMetrics.instance) {
      ServerlessMetrics.instance = new ServerlessMetrics();
    }
    return ServerlessMetrics.instance;
  }

  async recordCustomMetric(
    metricName: string,
    value: number,
    unit: string = "Count",
    dimensions: Record<string, string> = {}
  ): Promise<void> {
    const cloudWatch = new (
      await import("aws-sdk/clients/cloudwatch")
    ).default();

    await cloudWatch
      .putMetricData({
        Namespace: "CustomApp/Serverless",
        MetricData: [
          {
            MetricName: metricName,
            Value: value,
            Unit: unit as any,
            Dimensions: Object.entries(dimensions).map(([Name, Value]) => ({
              Name,
              Value,
            })),
            Timestamp: new Date(),
          },
        ],
      })
      .promise();
  }

  async recordExecutionMetrics(
    functionName: string,
    executionTime: number,
    memoryUsed: number,
    coldStart: boolean
  ): Promise<void> {
    await Promise.all([
      this.recordCustomMetric("ExecutionTime", executionTime, "Milliseconds", {
        FunctionName: functionName,
      }),
      this.recordCustomMetric("MemoryUsed", memoryUsed, "Megabytes", {
        FunctionName: functionName,
      }),
      this.recordCustomMetric("ColdStart", coldStart ? 1 : 0, "Count", {
        FunctionName: functionName,
      }),
    ]);
  }
}

// Cost optimization utilities
export class ServerlessCostOptimizer {
  static calculateOptimalMemorySize(
    executionTimes: number[],
    memoryUsages: number[]
  ): number {
    // Analyze execution patterns to recommend optimal memory allocation
    const avgExecutionTime =
      executionTimes.reduce((a, b) => a + b, 0) / executionTimes.length;
    const maxMemoryUsage = Math.max(...memoryUsages);

    // Add 20% buffer to max memory usage, rounded to nearest 64MB increment
    const recommendedMemory = Math.ceil((maxMemoryUsage * 1.2) / 64) * 64;

    return Math.max(128, Math.min(recommendedMemory, 10240)); // AWS Lambda limits
  }

  static estimateMonthlyCost(
    invocationsPerMonth: number,
    avgExecutionTimeMs: number,
    memoryMB: number
  ): number {
    // AWS Lambda pricing (US East 1, as of 2024)
    const requestCost = (invocationsPerMonth / 1000000) * 0.2; // $0.20 per 1M requests
    const computeCost =
      ((invocationsPerMonth * (avgExecutionTimeMs / 1000) * (memoryMB / 1024)) /
        400000) *
      0.0000166667;

    return requestCost + computeCost;
  }
}

// Helper functions
function createResponse(statusCode: number, body: any): APIGatewayProxyResult {
  return {
    statusCode,
    headers: {
      "Content-Type": "application/json",
      "Access-Control-Allow-Origin": "*",
      "Access-Control-Allow-Credentials": true,
    },
    body: JSON.stringify(body),
  };
}

function chunkArray<T>(array: T[], chunkSize: number): T[][] {
  const chunks: T[][] = [];
  for (let i = 0; i < array.length; i += chunkSize) {
    chunks.push(array.slice(i, i + chunkSize));
  }
  return chunks;
}

// WebSocket helper functions
async function handleConnect(connectionId: string, event: any): Promise<any> {
  const userId = event.queryStringParameters?.userId;

  if (!userId) {
    return { statusCode: 401, body: "User ID required" };
  }

  // Store connection mapping
  await dynamoClient
    .put({
      TableName: process.env.CONNECTIONS_TABLE!,
      Item: {
        connectionId,
        userId,
        connectedAt: new Date().toISOString(),
      },
    })
    .promise();

  return { statusCode: 200, body: "Connected" };
}

async function handleDisconnect(connectionId: string): Promise<any> {
  // Clean up connection mapping
  await dynamoClient
    .delete({
      TableName: process.env.CONNECTIONS_TABLE!,
      Key: { connectionId },
    })
    .promise();

  return { statusCode: 200, body: "Disconnected" };
}

async function handleSendMessage(
  connectionId: string,
  message: any
): Promise<any> {
  // Implementation for sending messages through WebSocket
  // This would include message validation, routing, etc.
  return { statusCode: 200, body: "Message sent" };
}

Serverless vs Traditional Architecture: When to Choose What

The Problem: Using Serverless for Everything

# The serverless configuration nightmare that costs 10x traditional hosting
service: monolithic-serverless-disaster

provider:
  name: aws
  runtime: nodejs18.x
  memorySize: 3008 # Maximum memory for everything - RED FLAG #1
  timeout: 900 # 15 minutes timeout for all functions - RED FLAG #2

functions:
  # Everything is a Lambda function - RED FLAG #3
  webServer:
    handler: src/server.handler
    events:
      - http:
          path: /{proxy+}
          method: ANY
    reservedConcurrency: 1000 # Way too high - RED FLAG #4

  databaseMigrations:
    handler: src/migrations.handler
    timeout: 900
    memorySize: 3008 # Overpowered for simple DB updates - RED FLAG #5

  fileUpload:
    handler: src/upload.handler
    timeout: 300
    memorySize: 3008 # Waste of money for file operations - RED FLAG #6

  reportGeneration:
    handler: src/reports.handler
    timeout: 900
    memorySize: 3008 # Perfect use case, wrong configuration - RED FLAG #7

  cronJobs:
    handler: src/cron.handler
    events:
      - schedule: rate(1 minute) # Running every minute - RED FLAG #8

  realTimeChat:
    handler: src/chat.handler
    events:
      - websocket:
          route: $default
    reservedConcurrency: 10000 # Massive over-provisioning - RED FLAG #9

# Problems this creates:
# - Monthly costs of $50,000+ for what should cost $500
# - Cold start delays for user-facing functions
# - Resource waste on simple operations
# - Complexity in debugging and monitoring
# - Vendor lock-in without benefits

The Solution: Strategic Serverless Implementation

// Strategic serverless architecture with proper service selection

// ✅ PERFECT for Serverless: Event-driven processing
export const processImageUploads = {
  // Triggered by S3 events, scales to zero, perfect fit
  handler: "src/image-processor.handler",
  memorySize: 1024, // Right-sized for image processing
  timeout: 300, // 5 minutes max for image processing
  events: [
    {
      s3: {
        bucket: "user-uploads",
        event: "s3:ObjectCreated:*",
        rules: [{ prefix: "images/" }, { suffix: ".jpg" }],
      },
    },
  ],
  environment: {
    PROCESSED_BUCKET: "${self:custom.processedBucket}",
  },
};

// ✅ GOOD for Serverless: Scheduled tasks with variable workload
export const generateReports = {
  handler: "src/report-generator.handler",
  memorySize: 2048, // High memory for data processing
  timeout: 900, // 15 minutes for complex reports
  events: [
    {
      schedule: {
        rate: "cron(0 6 * * ? *)", // Daily at 6 AM
        input: {
          reportType: "daily",
        },
      },
    },
    {
      schedule: {
        rate: "cron(0 6 * * SUN *)", // Weekly on Sunday
        input: {
          reportType: "weekly",
        },
      },
    },
  ],
};

// ✅ EXCELLENT for Serverless: Webhook processors
export const processWebhooks = {
  handler: "src/webhook-processor.handler",
  memorySize: 512, // Lightweight processing
  timeout: 30, // Quick processing
  events: [
    {
      http: {
        path: "webhooks/{service}",
        method: "post",
        cors: false, // Security for webhooks
      },
    },
  ],
};

// ❌ POOR fit for Serverless: Persistent connections
// Use ECS/EKS instead for WebSocket servers
const realTimeChatService = `
# docker-compose.yml for persistent connection services
version: '3.8'
services:
  chat-server:
    image: node:18-alpine
    ports:
      - "3000:3000"
    environment:
      - REDIS_URL=redis://redis:6379
      - DATABASE_URL=postgresql://...
    deploy:
      replicas: 2
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M
`;

// ❌ POOR fit for Serverless: High-frequency operations
// Use traditional servers for frequent, predictable workloads
const frequentApiCalls = `
// Traditional Express.js server (ECS/EKS deployment)
// Better for constant traffic and predictable costs

import express from 'express';
import { createServer } from 'http';

const app = express();
const server = createServer(app);

// These endpoints get hit thousands of times per minute
// Serverless would be extremely expensive
app.get('/api/users/:id', getUserProfile);
app.post('/api/auth/login', handleLogin);
app.get('/api/products', listProducts);
app.post('/api/orders', createOrder);

// Health check for load balancer
app.get('/health', (req, res) => {
  res.json({ status: 'healthy', timestamp: new Date().toISOString() });
});

server.listen(3000, () => {
  console.log('API server running on port 3000');
});
`;

// Hybrid architecture decision framework
export class ServerlessDecisionMatrix {
  static evaluateWorkload(
    characteristics: WorkloadCharacteristics
  ): DeploymentRecommendation {
    const score = this.calculateServerlessScore(characteristics);

    if (score >= 80) {
      return {
        recommendation: "SERVERLESS",
        confidence: "HIGH",
        reasoning:
          "Perfect fit for serverless - event-driven, variable workload, scales to zero",
      };
    } else if (score >= 60) {
      return {
        recommendation: "SERVERLESS",
        confidence: "MEDIUM",
        reasoning: "Good fit for serverless with proper optimization",
      };
    } else if (score >= 40) {
      return {
        recommendation: "HYBRID",
        confidence: "MEDIUM",
        reasoning:
          "Consider hybrid approach - some components serverless, others traditional",
      };
    } else {
      return {
        recommendation: "TRADITIONAL",
        confidence: "HIGH",
        reasoning:
          "Traditional deployment (containers/VMs) will be more cost-effective",
      };
    }
  }

  private static calculateServerlessScore(
    chars: WorkloadCharacteristics
  ): number {
    let score = 0;

    // Traffic patterns (30 points)
    if (chars.trafficPattern === "sporadic") score += 30;
    else if (chars.trafficPattern === "scheduled") score += 25;
    else if (chars.trafficPattern === "burst") score += 20;
    else if (chars.trafficPattern === "steady") score += 5;

    // Execution duration (25 points)
    if (chars.avgExecutionTime < 30000) score += 25; // Under 30 seconds
    else if (chars.avgExecutionTime < 300000) score += 15; // Under 5 minutes
    else if (chars.avgExecutionTime < 900000) score += 5; // Under 15 minutes

    // State requirements (20 points)
    if (!chars.requiresPersistentState) score += 20;
    else if (chars.canUseExternalState) score += 10;

    // Scaling requirements (15 points)
    if (chars.needsAutoScaling) score += 15;
    else if (chars.predictableLoad) score += 5;

    // Integration complexity (10 points)
    if (chars.isEventDriven) score += 10;
    else if (chars.canBeEventDriven) score += 5;

    return score;
  }
}

// Real-world serverless cost calculator
export class ServerlessCostCalculator {
  static calculateMonthlyCost(params: CostParameters): CostBreakdown {
    // AWS Lambda pricing (as of 2024)
    const FREE_TIER_REQUESTS = 1000000; // 1M free requests per month
    const FREE_TIER_COMPUTE = 400000; // 400K GB-seconds per month

    const REQUEST_COST_PER_MILLION = 0.2; // $0.20 per 1M requests
    const COMPUTE_COST_PER_GB_SECOND = 0.0000166667; // $0.0000166667 per GB-second

    // Calculate billable requests
    const billableRequests = Math.max(
      0,
      params.monthlyInvocations - FREE_TIER_REQUESTS
    );
    const requestCosts =
      (billableRequests / 1000000) * REQUEST_COST_PER_MILLION;

    // Calculate compute costs
    const totalGBSeconds =
      params.monthlyInvocations *
      (params.avgExecutionTimeMs / 1000) *
      (params.memoryMB / 1024);
    const billableGBSeconds = Math.max(0, totalGBSeconds - FREE_TIER_COMPUTE);
    const computeCosts = billableGBSeconds * COMPUTE_COST_PER_GB_SECOND;

    // Additional AWS service costs
    const apiGatewayCosts = (params.httpRequests / 1000000) * 3.5; // $3.50 per million API calls
    const cloudWatchLogCosts = params.logDataGB * 0.5; // $0.50 per GB of log data

    const totalCost =
      requestCosts + computeCosts + apiGatewayCosts + cloudWatchLogCosts;

    return {
      totalMonthlyCost: totalCost,
      breakdown: {
        lambdaRequests: requestCosts,
        lambdaCompute: computeCosts,
        apiGateway: apiGatewayCosts,
        cloudWatchLogs: cloudWatchLogCosts,
      },
      freeTimerUsed: {
        requests: Math.min(params.monthlyInvocations, FREE_TIER_REQUESTS),
        computeSeconds: Math.min(totalGBSeconds, FREE_TIER_COMPUTE),
      },
    };
  }

  static compareWithTraditionalHosting(
    serverlessParams: CostParameters
  ): CostComparison {
    const serverlessCost = this.calculateMonthlyCost(serverlessParams);

    // Estimate traditional hosting costs (t3.medium instances)
    const instancesNeeded = Math.ceil(serverlessParams.avgConcurrency / 10); // 10 concurrent per instance
    const ec2Cost = instancesNeeded * 24.192; // t3.medium pricing per month
    const loadBalancerCost = instancesNeeded > 1 ? 22.5 : 0; // ALB pricing
    const traditionalCost = ec2Cost + loadBalancerCost;

    return {
      serverless: serverlessCost,
      traditional: {
        totalMonthlyCost: traditionalCost,
        breakdown: {
          ec2Instances: ec2Cost,
          loadBalancer: loadBalancerCost,
          monitoring: 10, // Basic CloudWatch
        },
      },
      recommendation:
        serverlessCost.totalMonthlyCost < traditionalCost
          ? "SERVERLESS"
          : "TRADITIONAL",
      savings: Math.abs(serverlessCost.totalMonthlyCost - traditionalCost),
    };
  }
}

// Supporting interfaces
interface WorkloadCharacteristics {
  trafficPattern: "steady" | "burst" | "sporadic" | "scheduled";
  avgExecutionTime: number; // milliseconds
  requiresPersistentState: boolean;
  canUseExternalState: boolean;
  needsAutoScaling: boolean;
  predictableLoad: boolean;
  isEventDriven: boolean;
  canBeEventDriven: boolean;
  avgConcurrency: number;
}

interface DeploymentRecommendation {
  recommendation: "SERVERLESS" | "TRADITIONAL" | "HYBRID";
  confidence: "HIGH" | "MEDIUM" | "LOW";
  reasoning: string;
}

interface CostParameters {
  monthlyInvocations: number;
  avgExecutionTimeMs: number;
  memoryMB: number;
  httpRequests: number;
  logDataGB: number;
  avgConcurrency: number;
}

interface CostBreakdown {
  totalMonthlyCost: number;
  breakdown: {
    lambdaRequests: number;
    lambdaCompute: number;
    apiGateway: number;
    cloudWatchLogs: number;
  };
  freeTimerUsed: {
    requests: number;
    computeSeconds: number;
  };
}

interface CostComparison {
  serverless: CostBreakdown;
  traditional: {
    totalMonthlyCost: number;
    breakdown: {
      ec2Instances: number;
      loadBalancer: number;
      monitoring: number;
    };
  };
  recommendation: "SERVERLESS" | "TRADITIONAL";
  savings: number;
}

This comprehensive serverless architecture guide gives you:

  1. Strategic serverless implementation that uses functions where they provide genuine value
  2. Cold start optimization techniques that minimize latency and improve user experience
  3. Cost optimization strategies that prevent budget explosions while maintaining performance
  4. Proper architectural patterns for event-driven, scalable applications
  5. Decision frameworks for choosing between serverless and traditional deployments

The difference between successful serverless implementations and expensive disasters isn’t just following best practices—it’s understanding when serverless solves real problems versus when it creates new ones.