Back to Blog
Tech

Deep Dive into URL Encoding

Why do URLs need encoding? How does percent-encoding work? Learn the rules, common pitfalls, and best practices for robust, interoperable URLs.

What Is URL Encoding

URL encoding converts characters into a transport-safe representation using percent-encoding. A byte is represented as % followed by two uppercase hexadecimal digits. This keeps URLs valid across browsers, servers, and protocols—preventing misinterpretation of spaces, non-ASCII characters, or reserved symbols that have structural meaning in a URL.

Under the hood, a string is first encoded to bytes (modern stacks use UTF-8). Each non-safe byte is then rendered as %HH, where HH is the uppercase hex of the byte. Safe bytes (unreserved characters) remain as-is.

URL Anatomy and Contexts

A URL comprises distinct components with different encoding rules:

  • Scheme: e.g., https. Alphabetic and a few symbols; generally no percent-encoding.
  • Authority: user:pass@host:port. User info may be encoded; host is subject to IDN/punycode; port is numeric.
  • Path: hierarchical segments separated by /. Segments may require encoding; / itself is a delimiter and must not be percent-encoded when used as a separator.
  • Query: key–value pairs after ?, separated by &, with = between key and value. Keys and values should be percent-encoded.
  • Fragment: after #; used for in-document references and client-side state. Encode textual content to avoid parser ambiguities.

Encoding decisions depend on which component you are working with: characters acceptable in path segments may not be acceptable unencoded in query values, and vice versa.

RFC 3986 Basics

  • Unreserved: A-Z a-z 0-9 - . _ ~ remain literal.
  • Reserved: ! * ' ( ) ; : @ & = + $ , / ? # [ ] carry structural meaning and may need encoding depending on context.
  • Everything else: spaces, quotes, control chars, and non-ASCII must be percent-encoded after UTF-8 conversion.

Path vs Query vs Fragment

  • Path segments: encode spaces and non-ASCII; leave / unencoded as a delimiter. Avoid encoding / unless it is part of a segment in a data URL or similar niche scenarios.
  • Query: encode keys and values. & separates pairs; = separates key and value. Percent-encode &, =, and other reserved characters when part of a key or value.
  • Fragment: though not sent to the server, fragments should be encoded to prevent client-side parsing issues and to ensure shareability.

UTF-8 and Byte-Level Encoding

Always normalize to UTF-8 before percent-encoding. Non-ASCII characters (e.g., é or Chinese characters) expand to multiple bytes in UTF-8. Each byte is percent-encoded individually. For example, “Café” becomes Caf%C3%A9: é is 0xC3 0xA9.

Mismatched encodings lead to mojibake—garbled text caused by decoding the wrong byte sequences. Standardize on UTF-8 throughout the stack to avoid subtle bugs.

Application/x-www-form-urlencoded Nuances

HTML forms historically use application/x-www-form-urlencoded, which differs from raw percent-encoding:

  • Spaces are serialized as + rather than %20.
  • Plus signs intended literally must be encoded as %2B.
  • Keys and values are percent-encoded; pairs separated by &, key–value by =.
  • Arrays and nested objects are convention-based (e.g., tags[]=a&tags[]=b).

When consuming such payloads, treat + as a space, then percent-decode. When producing, follow the same conventions to maintain interoperability.

encodeURI vs encodeURIComponent

encodeURI encodes a complete URL while preserving :/?#& separators. encodeURIComponent encodes a single component (e.g., a query value) and encodes most reserved characters. Use encodeURI when you already have a full URL; use encodeURIComponent for keys and values within queries or individual path segments.

const url = 'https://example.com/search?q=Café crème &sort=recent'; // Using encodeURI on the whole URL // https://example.com/search?q=Caf%C3%A9%20cr%C3%A8me%20&sort=recent // Using encodeURIComponent on the value only // q = encodeURIComponent('Café crème &sort=recent') // q = Caf%C3%A9%20cr%C3%A8me%20%26sort%3Drecent

WHATWG URL API and Query Handling

Modern browsers and Node.js ship the WHATWG URL interface, which standardizes parsing and serialization:

const u = new URL('https://example.com/search'); u.searchParams.set('q', 'Café crème &sort=recent'); u.searchParams.append('tags', 'c++'); // Serializes with proper percent-encoding: // https://example.com/search?q=Caf%C3%A9%20cr%C3%A8me%20%26sort%3Drecent&tags=c%2B%2B

Prefer URL and URLSearchParams over manual string concatenation to avoid subtle encoding bugs and parameter-order issues.

Internationalized Domain Names (IDN) and Punycode

Hosts may contain non-ASCII characters (e.g., Chinese, Arabic). These are encoded using punycode (IDNA). The browser renders a Unicode domain for display but uses an ASCII-compatible encoded form for DNS resolution, typically starting with xn--. Do not percent-encode host labels; rely on the URL implementation to handle IDN correctly.

Security Considerations

  • XSS vectors via URLs: injecting unencoded values into HTML attributes or scripts can create cross-site scripting risks. Always escape in the appropriate context (HTML attribute, text, JS string) in addition to URL encoding.
  • Open redirects: avoid blindly redirecting to user-provided URLs. Validate schemes and hosts; allow only whitelisted destinations.
  • Path traversal: percent-encoded ../ can bypass naive filters. Normalize paths server-side and restrict to allowed roots.
  • Double-decoding attacks: decoding twice can turn safe sequences into actionable payloads. Decode exactly once in a controlled pipeline.

Normalization vs Encoding

Normalization standardizes different textual representations before encoding. Examples include converting full-width characters to half-width, enforcing lowercase for percent-hex digits, and normalizing Unicode (NFC). Normalization reduces duplication (e.g., different representations of the same character) and improves comparability across systems.

Common Pitfalls

  • Spaces: percent-encoding uses %20; form submissions may use + in application/x-www-form-urlencoded.
  • Plus sign: + can mean a literal plus or a space depending on context; encode as %2B when literal.
  • Double-encoding: avoid encoding already encoded inputs (e.g., %2520 for %20).
  • Mojibake: use UTF-8 consistently; mismatches cause corrupted text.
  • Blind decoding: decode only components that were encoded; don’t decode entire URLs indiscriminately.
  • Incorrect separator handling: do not encode / in path separators; do encode & and = when inside values.

Good vs Bad Examples

Good

https://example.com/docs/url-encoding https://example.com/search?q=Caf%C3%A9%20cr%C3%A8me https://example.com/tags/C%2B%2B https://例子.xn--fiqs8s

Bad

https://example.com/search?q=Café crème https://example.com/tags/C++ https://example.com/a b/c d javascript:alert(1)  // unsafe scheme in user-controlled redirect

Testing and Validation

  • Use automated tests to assert round-trip correctness: encode then decode yields original.
  • Verify with multiple clients (browser, curl, Node) to ensure consistent behavior.
  • Lint and validate HTML output to confirm semantic correctness and proper attribute escaping.
  • Monitor logs for malformed URLs (spaces, unencoded non-ASCII) and fix upstream sources.

Implementation Patterns

  • Use URL and URLSearchParams instead of manual string concatenation.
  • Standardize on UTF-8 encoding in all layers (client, server, database).
  • Enforce canonicalization rules (lowercase hex in percent-encoding, consistent casing for hostnames).
  • Provide helper utilities for encoding/decoding to avoid duplicated, error-prone code paths.

Operational Checklist

  • Use UTF-8 for all text before encoding.
  • Encode query keys and values with encodeURIComponent or URLSearchParams.
  • Encode path segments; don’t encode / separators.
  • Handle application/x-www-form-urlencoded differences (+ vs %20).
  • Avoid double-encoding; validate and sanitize inputs.
  • Guard against unsafe schemes and redirect destinations.
  • Normalize Unicode and percent-hex casing for consistency.

FAQ

Do I need to percent-encode ASCII letters? No. Unreserved characters (A–Z a–z 0–9 - . _ ~) remain literal.

Should I encode slashes? No, / is a path separator. Encoding it alters structure and can break routing unless explicitly intended.

Why does my form use plus signs? Because application/x-www-form-urlencoded serializes spaces as +. Treat + as a space during decoding.

How do I handle international domains? Rely on the URL implementation to render Unicode for display and punycode for DNS. Don’t percent-encode hostname labels.

What if I see double-encoded sequences? Audit input sources and ensure decoding happens exactly once. Avoid encoding values that already contain percent-escapes.

Summary

URL encoding is foundational to robust, interoperable web systems. Follow RFC 3986, respect component-specific rules, normalize to UTF-8, understand application/x-www-form-urlencoded nuances, and prefer modern APIs like URL and URLSearchParams. With proper testing, validation, and security checks, your URLs will remain clear, portable, and resilient across clients and servers.