Securing Web Applications with Custom WAF using OpenResty

A custom web application firewall (WAF) can help protect your web application from various threats like SQL injection, cross-site scripting (XSS), etc. If you ship software to customers or even if you host your services in the cloud but want more control, agility and/or simplicity, a custom WAF can help. In this article, I will show you how to build a custom WAF using OpenResty, a powerful web platform based on Nginx and Lua.

OpenResty, built on Nginx and LuaJIT, offers a powerful, scriptable platform to implement these capabilities in a lightweight, portable and cloud-native way.

Photo by Albert Stoynov on Unsplash

We will be building the following capabilities into our custom WAF:

Inspect request bodies for suspicious patterns.
Only runs on configured API routes (to ensure performance)
Logs malicious attempts
Authenticates requests using JWT
Supports blocking of specific APIs based on configuration or dynamic logic
Record all traffic with optional body logging
Functions as a lightweight API gateway for header manipulation

Setting Up OpenResty

FROM openresty/openresty:1.27.1.2-0-bookworm

# dependencies
RUN apt-get update && \
    apt-get install -y curl git luarocks && \
    luarocks install lua-resty-jwt && \
    luarocks install lua-cjson

COPY nginx.conf /usr/local/openresty/nginx/conf/nginx.conf
COPY waf.lua /usr/local/openresty/nginx/conf/waf.lua

EXPOSE 8080
CMD ["openresty", "-g", "daemon off;"]

This Dockerfile sets up OpenResty with a custom configuration and Lua script. The nginx.conf file will define the server and routing, while waf.lua will contain our WAF logic.

Nginx Configuration

worker_processes 1;

error_log /dev/stderr info;

events {
    worker_connections 1024;
}

http {
    lua_shared_dict waf_log 10m;

    server {
        listen 8080;

        location / {
            access_by_lua_file conf/waf.lua;
            proxy_pass http://backend:3333;
        }
    }
}

This configuration sets up a server that listens on port 8080 and proxies requests to a backend service running on port 3333. The access_by_lua_file directive calls our WAF logic before passing the request to the backend.

WAF Logic in Lua

local cjson = require "cjson"
local jwt = require "resty.jwt"

local protected_apis = {
  ["/api/v1/comments"] = true,
  ["/api/v1/submit"] = true,
}

local blocked_apis = {
  ["/api/v1/deprecated"] = true,
}

local suspicious_patterns = {
  "<script",
  "SELECT.+FROM",
  "UNION.+SELECT",
  "1=1",
  "DROP TABLE",
  ";--",
  "alert%(",
  "onerror=",
}

-- Block configured APIs
if blocked_apis[ngx.var.uri] then
  ngx.status = 403
  ngx.say("This API is currently unavailable.")
  return ngx.exit(403)
end

-- JWT Authentication
local auth_header = ngx.req.get_headers()["Authorization"]
if not auth_header or not auth_header:find("Bearer ") then
  ngx.status = 401
  ngx.say("Missing or invalid Authorization header")
  return ngx.exit(401)
end

local token = auth_header:sub(8)
local jwt_obj = jwt:verify("your_secret_key", token) -- read key from environment or config
if not jwt_obj.verified then
  ngx.status = 401
  ngx.say("Invalid token")
  return ngx.exit(401)
end

-- Malicious pattern detection
if protected_apis[ngx.var.uri] and ngx.req.get_method() == "POST" then
  ngx.req.read_body()
  local body = ngx.req.get_body_data() or ""

  for _, pattern in ipairs(suspicious_patterns) do
    if string.find(string.lower(body), string.lower(pattern)) then
      ngx.log(ngx.WARN, "WAF triggered: pattern '" .. pattern .. "' found in body")
      ngx.status = 403
      ngx.say("Request blocked by WAF")
      return ngx.exit(403)
    end
  end
end

-- General traffic logging (with optional body)
local log_obj = {
  method = ngx.req.get_method(),
  uri = ngx.var.request_uri,
  ip = ngx.var.remote_addr,
  status = ngx.status,
  timestamp = ngx.time()
}

if protected_apis[ngx.var.uri] and ngx.req.get_method() == "POST" then
  log_obj.body = ngx.req.get_body_data()
end

ngx.log(ngx.INFO, "Traffic Log: " .. cjson.encode(log_obj))

This Lua script implements the WAF logic:

API Protection: It checks if the requested API is protected and blocks access to configured APIs.
JWT Authentication: It verifies the JWT token in the Authorization header.
Malicious Pattern Detection: It inspects the request body for suspicious patterns and blocks requests that match any of them.
Traffic Logging: It logs all traffic, including request method, URI, IP address, status, and timestamp. For POST requests to protected APIs, it also logs the request body.

Dummy Backend Service

const http = require('http')
const url = require('url')

const server = http.createServer((req, res) => {
  const parsedUrl = url.parse(req.url, true)
  const method = req.method
  const pathname = parsedUrl.pathname

  res.setHeader('Access-Control-Allow-Origin', '*')
  res.setHeader('Access-Control-Allow-Methods', 'GET,POST,OPTIONS')
  res.setHeader('Access-Control-Allow-Headers', '*')

  if (method === 'OPTIONS') {
    res.writeHead(204)
    return res.end()
  }

  if (pathname === '/api/v1/submit' && method === 'POST') {
    handlePost(req, res, 'Submit API received your data.')
  } else if (pathname === '/api/v1/comments' && method === 'POST') {
    handlePost(req, res, 'Comment received and processed.')
  } else if (pathname === '/api/v1/deprecated' && method === 'GET') {
    res.writeHead(200, { 'Content-Type': 'application/json' })
    res.end(JSON.stringify({ message: 'Deprecated endpoint reached.' }))
  } else {
    res.writeHead(404, { 'Content-Type': 'application/json' })
    res.end(JSON.stringify({ error: 'Not found' }))
  }
})

function handlePost(req, res, message) {
  let body = ''
  req.on('data', (chunk) => {
    body += chunk.toString()
  })
  req.on('end', () => {
    console.log(`Received data: ${body}`)
    res.writeHead(200, { 'Content-Type': 'application/json' })
    res.end(JSON.stringify({ message, received: body }))
  })
}

server.listen(3333, () => {
  console.log('Dummy backend running on http://localhost:3333')
})

Docker compose for tying everything together

version: '3.8'
services:
  gateway:
    build: .
    ports:
      - '8080:8080'
    depends_on:
      - backend

  backend:
    image: node:22-slim
    working_dir: /app
    volumes:
      - .:/app
    command: ['node', 'backend.js']
    expose:
      - '3333'

Build and Run the Gateway

docker-compose up --build

Test

Replace <valid_token> with a valid JWT token generated using secret key in lua file ("your_secret_key").

Clean request

curl -X POST http://localhost:8080/api/v1/submit -d "hello=world" -H "Authorization: Bearer  <valid_token>"

Malicious request

curl -X POST http://localhost:8080/api/v1/submit -d "name=<script>alert(1)</script>" -H "Authorization: Bearer  <valid_token>"

Blocked API

curl -X GET http://localhost:8080/api/v1/deprecated

Unauthorized request

curl -X GET http://localhost:8080/api/v1/comments

Output from my local running WAF is shown below.

Output from my local running WAF

Detecting Anomalies in Server Traffic with Log Analysis

In my last post, I discussed setting up a local process to monitor server traffic and using a model like all-MiniLM-L6 detect anomalies in server traffic. You can also read my earlier post on OpenSearch and anomaly detection here.

There's a lot more we can do with this custom WAF implementation including implementing rate limiting, more sophisticated pattern matching, and integrating with a monitoring system to alert on suspicious activity.

Let me know your thoughts on this custom WAF implementation and how you might use it in your projects. Have you used any of the existing WAF solutions to dig deeper into traffic?