Text & Encoding guides

1 article · Browse the Text & Encoding tools →

Computers store text as numbers, and an encoding is the agreement about which number means which character. The oldest widely used agreement is ASCII, a 7-bit code with 128 values. Codes 0 to 31 and 127 are control characters such as tab, line feed and escape, and the rest are the printable letters, digits and punctuation on a US keyboard. The reference below lists all 128 in decimal, hexadecimal, octal and binary.

A few patterns make ASCII easy to read. The digit characters run in order starting at 48 (hex 30), so the character 7 is 55, and subtracting 48 recovers the number 7. Uppercase A is 65 (hex 41) and lowercase a is 97 (hex 61); the two differ by exactly 32, which is a single bit, so changing case is flipping one bit. Because the first 128 values of UTF-8 are identical to ASCII, a plain English file is valid in both.

Characters outside ASCII need more bytes in UTF-8, which uses one to four. The letter é (U+00E9) is two bytes, C3 A9; the euro sign (U+20AC) is three, E2 82 AC; and most emoji are four. When software reads UTF-8 bytes as an older single-byte encoding such as Windows-1252, you get the garbled text called mojibake: C3 A9 appears as “é”. The fix is to read the bytes with the encoding they were written in, not to edit the characters.

Encodings are different from the transformations built on top of them. Base64 turns every 3 bytes into 4 printable characters, so the three letters “Man” become TWFu and data grows by about one third. URL encoding replaces unsafe bytes with a percent sign and two hex digits, and HTML entities write characters such as the ampersand as &. All of these can be reversed. A hash cannot, which is why hashing is for checking integrity and not for hiding data.

Use the ASCII table when you need an exact code, such as a delimiter in a file format, a control character in a serial protocol, or the value behind a character in source code. For text outside ASCII, the Unicode Lab looks up any character and shows its code point and UTF-8 bytes.

Put it into practice

The Text & Encoding tools on Utilities Bunker run the ideas from these guides on your own input — free, in your browser, with nothing uploaded.

See all 24 Text & Encoding tools →