URL Encoding & Query Strings: Mastering RFC 3986, encodeURIComponent & URLSearchParams

Uniform Resource Identifiers (URIs) are constrained to a restricted character set. When parameters contain non-ASCII characters, spaces, punctuation, or reserved delimiters (such as &, =, ?, and /), they must be converted into standard percent-encoded octets (%XX) per RFC 3986.

Despite its ubiquity, URL encoding remains one of the most frequent sources of subtle production bugs: accidental double-encoding, mismatched plus sign (+) vs. percent-twenty (%20) decoding, broken Unicode surrogate pairs, and inconsistent nested array serialization across web frameworks.

This guide clarifies the mechanics of URL encoding specifications, dissects the differences between JavaScript's built-in encoding functions, and provides best practices for robust query string parsing and serialization.

---

1. The Anatomy of URI Characters (RFC 3986)

RFC 3986 divides ASCII characters into two distinct categories:

┌─────────────────────────────────────────────────────────────────────────────┐
│                       URI Character Classification                          │
├──────────────────────┬──────────────────────────────────────────────────────┤
│ Classification       │ Characters Included                                  │
├──────────────────────┼──────────────────────────────────────────────────────┤
│ Unreserved           │ A-Z  a-z  0-9  -  _  .  ~                            │
│ (Never Encoded)      │ (Safe in all URI components without ambiguity)       │
├──────────────────────┼──────────────────────────────────────────────────────┤
│ Reserved Delimiters  │ Gen-delims: : / ? # [ ] @                            │
│ (Encoded when used   │ Sub-delims: ! $ & ' ( ) * + , ; =                    │
│ as literal data)     │                                                      │
├──────────────────────┼──────────────────────────────────────────────────────┤
│ Disallowed / Special │ Spaces, Non-ASCII (UTF-8 bytes), Control Chars, < >  │
│ (Always Encoded)     │ " ` { } | \ ^                                        │
└──────────────────────┴──────────────────────────────────────────────────────┘

When a reserved delimiter (such as & or =) is part of a parameter's key or value rather than a structural delimiter, it must be percent-encoded:

• https://api.dev/search?query=rock%26roll ➔ Parameter query is "rock&roll".
• https://api.dev/search?query=rock&roll ➔ Parameters are query="rock" and boolean flag roll.

---

2. JavaScript Encoding Functions Compared

JavaScript provides three distinct built-in mechanisms for URL encoding, each designed for a specific level of the URI:

┌─────────────────────────┬───────────────────────────────┬──────────────────────────────┐
│ Function / API          │ Intended Scope                │ Characters Left Unencoded    │
├─────────────────────────┼───────────────────────────────┼──────────────────────────────┤
│ `encodeURI()`           │ Entire URI string             │ A-Z a-z 0-9 - _ . ! ~ * ' ( )│
│                         │ (Preserves URI structure)     │ ; , / ? : @ & = + $ #        │
├─────────────────────────┼───────────────────────────────┼──────────────────────────────┤
│ `encodeURIComponent()`  │ Single query param key/val    │ A-Z a-z 0-9 - _ . ! ~ * ' ( )│
│                         │ (Encodes delimiters like & =) │ (Leaves ! ~ * ' ( ) unencoded│
├─────────────────────────┼───────────────────────────────┼──────────────────────────────┤
│ `URLSearchParams`       │ Modern Query String Standard  │ Conforms to WHATWG URL spec  │
│                         │ (Encodes spaces as + or %20)  │ Automatically escapes all key│
└─────────────────────────┴───────────────────────────────┴──────────────────────────────┘

The RFC 3986 Strict encodeURIComponent Fix

Notice that standard encodeURIComponent() does not encode !, ', (, ), or *. While acceptable in older specs, RFC 3986 reserves these in certain contexts. For 100% strict compliance (e.g. AWS Signature V4 auth):

export function strictEncodeURIComponent(str: string): string {
  return encodeURIComponent(str).replace(
    /[!'()*]/g,
    (c) => `%${c.charCodeAt(0).toString(16).toUpperCase()}`
  );
}

---

3. The + vs %20 Space Mystery

One of the most confusing discrepancies is why spaces are encoded as + in query strings but %20 in path segments:

1. application/x-www-form-urlencoded (MIME standard for forms):
   • Historically specified that spaces (0x20) should be replaced by +.
   • URLSearchParams outputs + for spaces by default:

     const params = new URLSearchParams({ search: 'developer tools' });
     console.log(params.toString()); // search=developer+tools
     

2. RFC 3986 (URI Standard):
   • Standard percent-encoding strictly maps space to %20.
   • Path components (e.g. /files/my%20document.pdf) must use %20. Using + in a URL path literal represents an actual literal plus sign, not a space.

// Safe normalization helper
export function sanitizeQueryParam(val: string): string {
  // Replace + with space before decoding if decoding form-encoded data
  return decodeURIComponent(val.replace(/\+/g, ' '));
}

---

4. Serializing Complex Objects and Arrays

There is no formal RFC standard for serializing nested objects and arrays in URL query strings. Different backend ecosystems expect different conventions:

Format 1: Repeat Keys (FastAPI, Go, Django)
?filter=active&filter=pending

Format 2: Bracket Notation (PHP, Ruby on Rails, Express `qs`)
?filter[]=active&filter[]=pending

Format 3: Indexed Brackets (Strict PHP / C#)
?filter[0]=active&filter[1]=pending

Format 4: Comma-Separated Values (REST / OpenAPI default)
?filter=active,pending

Format 5: Deeply Nested Objects (Express / Axios)
?user[profile][theme]=dark&user[profile][notifications]=true

Universal Query Builder Implementation

export type QueryValue = string | number | boolean | null | undefined;
export type QueryParams = Record<string, QueryValue | QueryValue[] | Record<string, QueryValue>>;

export function buildQueryString(params: QueryParams, arrayFormat: 'repeat' | 'brackets' | 'comma' = 'brackets'): string {
  const parts: string[] = [];

  for (const [key, value] of Object.entries(params)) {
    if (value === null || value === undefined) continue;

    if (Array.isArray(value)) {
      if (arrayFormat === 'comma') {
        const encodedValues = value.map((v) => encodeURIComponent(String(v))).join(',');
        parts.push(`${encodeURIComponent(key)}=${encodedValues}`);
      } else {
        value.forEach((item, index) => {
          const paramKey = arrayFormat === 'brackets' ? `${key}[]` : key;
          parts.push(`${encodeURIComponent(paramKey)}=${encodeURIComponent(String(item))}`);
        });
      }
    } else if (typeof value === 'object') {
      // Nested object serialization (key[prop]=val)
      for (const [subKey, subVal] of Object.entries(value)) {
        if (subVal !== null && subVal !== undefined) {
          parts.push(`${encodeURIComponent(`${key}[${subKey}]`)}=${encodeURIComponent(String(subVal))}`);
        }
      }
    } else {
      parts.push(`${encodeURIComponent(key)}=${encodeURIComponent(String(value))}`);
    }
  }

  return parts.length > 0 ? `?${parts.join('&')}` : '';
}

---

5. Common Pitfalls & How to Prevent Them

1. Accidental Double Encoding

// BUG: Encoding an already encoded string
const term = "hello%20world";
const encoded = encodeURIComponent(term); // "hello%2520world" (Double encoded %)

// FIX: Always maintain clean, unencoded variables in application state,
// and encode only at the final boundary where the URL string is constructed.

2. Truncated Unicode Surrogate Pairs

JavaScript strings are UTF-16. Characters outside the Basic Multilingual Plane (such as emojis 🚀 or rare Asian glyphs) consist of two 16-bit surrogate code units. Splitting a string mid-surrogate and running encodeURIComponent() will throw URIError: URI malformed.

// Safe decoding wrapper
function safeDecodeURI(uri: string): string {
  try {
    return decodeURIComponent(uri);
  } catch (e) {
    console.error("Invalid UTF-8 sequence in URI:", uri);
    return uri; // Fallback
  }
}

---

6. Developer Tools & Verification

• Test & Convert: Use the URL Encoder/Decoder to inspect multi-byte UTF-8 percent sequences and compare standard vs component encoding.
• Inspect cURL Calls: Translate encoded cURL parameters into clean fetch or python code with the cURL Converter.
• Construct API Calls: Safely build query strings and headers using the API Request Builder.