Why did ¥ and Ø vanish from my old DOS file?

Both live at byte 157, which Windows-1252 leaves empty. Code page 437 has ¥ there and code page 850 has Ø, so a US file loses its yen signs and a Danish or Norwegian one loses every capital Ø, turning Østerbro into sterbro. Whichever one went missing tells you which code page to decode with.

Why does my mojibake fix fail on Ý with '\x9d'?

The garbled form of Ý is Ã followed by an invisible control character, and Python's cp1252 codec refuses to encode that control, so the usual repair raises UnicodeEncodeError. For strings like this, encode with latin-1, which turns the control back into its byte. ftfy's fix_text copes with text that mixes both cases. If a tool already stripped the control, only Ã is left and the Ý can't be recovered.

ASCII 157

Empty on WindowsYen sign

ASCII 157 is empty on Windows and ¥ in old DOS.

Byte 157 (0x9D) is the last unassigned slot in Windows-1252, and it causes trouble when you're repairing mangled text. The closing curly quote ” is E2 80 9D in UTF-8, so its mangled form contains a 9D that browser-style decoders turn into the invisible control character U+009D. The one-line Python repair, s.encode('cp1252').decode('utf-8'), then fails on that character with UnicodeEncodeError, because Python's cp1252 has nothing at 0x9D. The ftfy library ships a sloppy-windows-1252 codec that passes these five bytes through, which is exactly what that repair needs.

¥ (Yen Sign) is byte 157 in code page 437, the original IBM PC character set, and maps to Unicode U+00A5. Code page 850 dropped ¥ from this byte and put Ø here instead, so a Danish Ø typed on a code page 850 machine shows up as ¥ under code page 437. Windows-1252 leaves byte 157 unassigned, so reading DOS text that way drops the character or turns it into an invisible control code.

UnassignedCurrency

Technical Details

Code Example

</>

/* Byte 157 (0x9D) has no character in Windows-1252.
   MultiByteToWideChar passes it through as the control code U+009D,
   so treat it as a sign that the text is not really Windows-1252. */
unsigned char b = 157;
char c = (char)157;   /* -99 on x86, where char is signed */

Code Example

</>

#include <stdio.h>

int main(void) {
    /* Byte 157 is ¥ only on a console using code page 437
       (chcp 437 on Windows, or DOSBox). On a UTF-8 terminal the
       lone byte 0x9D is invalid and prints as garbage, often �. */
    putchar(157);

    /* char is signed on x86, so a plain char holding 0x9D is -99.
       Use unsigned char when you compare or index by byte value. */
    char c = (char)157;
    unsigned char u = 157;
    printf("\n%d %d\n", c, u);   /* -99 157 */

    /* Portable: print the Unicode character as UTF-8 instead. */
    printf("\u00A5\n");          /* C2 A5 */
    return 0;
}

How to Type It

Windows: Hold Alt and type 157 on the numeric keypad, with no leading zero. US PCs insert ¥, but PCs set to code page 850, common in Western Europe, insert Ø.

¥ is also ASCII 165 on Windows, which has the full guide.

Frequently Asked Questions

The garbled form of Ý is Ã followed by an invisible control character, and Python's cp1252 codec refuses to encode that control, so the usual repair raises UnicodeEncodeError. For strings like this, encode with latin-1, which turns the control back into its byte. ftfy's fix_text copes with text that mixes both cases. If a tool already stripped the control, only Ã is left and the Ý can't be recovered.

Frequently Asked Questions

Both live at byte 157, which Windows-1252 leaves empty. Code page 437 has ¥ there and code page 850 has Ø, so a US file loses its yen signs and a Danish or Norwegian one loses every capital Ø, turning Østerbro into sterbro. Whichever one went missing tells you which code page to decode with.

ASCII 157

Empty on WindowsYen sign

ASCII 157 is empty on Windows and ¥ in old DOS.

Unassigned

Currency

/* Byte 157 (0x9D) has no character in Windows-1252.

MultiByteToWideChar passes it through as the control code U+009D,

so treat it as a sign that the text is not really Windows-1252. */

unsigned char b = 157;

char c = (char)157; /* -99 on x86, where char is signed */

#include <stdio.h>

int main(void) {

/* Byte 157 is ¥ only on a console using code page 437

(chcp 437 on Windows, or DOSBox). On a UTF-8 terminal the

lone byte 0x9D is invalid and prints as garbage, often �. */

putchar(157);

/* char is signed on x86, so a plain char holding 0x9D is -99.

Use unsigned char when you compare or index by byte value. */

char c = (char)157;

unsigned char u = 157;

printf("\n%d %d\n", c, u); /* -99 157 */

/* Portable: print the Unicode character as UTF-8 instead. */

printf("\u00A5\n"); /* C2 A5 */

return 0;

}