Why did Ålesund turn into lesund in my imported data?

The export came from a DOS program, where Å is byte 143, and it was read as Windows-1252, which has no character at 143, so the first letter of every Å name dropped out or became an invisible control code. Read it again with the Nordic DOS code page. In R, `readr::read_csv('old.csv', locale = readr::locale(encoding = 'CP865'))` does it.

UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f

The file is UTF-8 and was read as Windows-1252, which has no character at 0x8F. Russian text hits this constantly, since я is D1 8F in UTF-8, and Hebrew or Arabic text can carry it inside the invisible right-to-left mark, E2 80 8F. With the csv module, open the file as `open(path, newline='', encoding='utf-8')`.

ASCII 143

Empty on WindowsUppercase A with ring

ASCII 143 is empty on Windows and Å in old DOS.

Byte 143 (0x8F) has no character in Windows-1252, and ISO-8859-1 only has the control code SS3 (Single Shift Three) there. That control code still does real work in one place: EUC-JP, a Japanese encoding from the Unix world, uses 0x8F to introduce three-byte characters from the supplementary JIS X 0212 set. So if a decoder set to Windows-1252 fails on 0x8F and the rest of the file is mostly pairs of bytes between A1 and FE, EUC-JP is worth a try.

Å (Capital A with Ring) is byte 143 in code page 437, the original IBM PC character set, and maps to Unicode U+00C5. Windows-1252 leaves byte 143 unassigned, so Scandinavian DOS text read as Windows-1252 drops the letter or turns it into an invisible control code, and Ålesund comes out as lesund. Code page 850 kept Å at 143 too. Small å is 134 in DOS.

UnassignedLatin Letter

Technical Details

Code Example

</>

/* Byte 143 (0x8F) has no character in Windows-1252.
   MultiByteToWideChar passes it through as the control code U+008F,
   so treat it as a sign that the text is not really Windows-1252. */
unsigned char b = 143;
char c = (char)143;   /* -113 on x86, where char is signed */

Code Example

</>

#include <stdio.h>

int main(void) {
    /* Byte 143 is Å only on a console using code page 437
       (chcp 437 on Windows, or DOSBox). On a UTF-8 terminal the
       lone byte 0x8F is invalid and prints as garbage, often �. */
    putchar(143);

    /* char is signed on x86, so a plain char holding 0x8F is -113.
       Use unsigned char when you compare or index by byte value. */
    char c = (char)143;
    unsigned char u = 143;
    printf("\n%d %d\n", c, u);   /* -113 143 */

    /* Portable: print the Unicode character as UTF-8 instead. */
    printf("\u00C5\n");          /* C3 85 */
    return 0;
}

How to Type It

Windows: Hold Alt and type 143 on the numeric keypad, with no leading zero. Windows reads it as a DOS code and inserts Å.

Å is also ASCII 197 on Windows, which has the full guide.

Frequently Asked Questions

The file is UTF-8 and was read as Windows-1252, which has no character at 0x8F. Russian text hits this constantly, since я is D1 8F in UTF-8, and Hebrew or Arabic text can carry it inside the invisible right-to-left mark, E2 80 8F. With the csv module, open the file as `open(path, newline='', encoding='utf-8')`.

Frequently Asked Questions

The export came from a DOS program, where Å is byte 143, and it was read as Windows-1252, which has no character at 143, so the first letter of every Å name dropped out or became an invisible control code. Read it again with the Nordic DOS code page. In R, `readr::read_csv('old.csv', locale = readr::locale(encoding = 'CP865'))` does it.

ASCII 143

Empty on WindowsUppercase A with ring

ASCII 143 is empty on Windows and Å in old DOS.

Unassigned

Latin Letter

/* Byte 143 (0x8F) has no character in Windows-1252.

MultiByteToWideChar passes it through as the control code U+008F,

so treat it as a sign that the text is not really Windows-1252. */

unsigned char b = 143;

char c = (char)143; /* -113 on x86, where char is signed */

#include <stdio.h>

int main(void) {

/* Byte 143 is Å only on a console using code page 437

(chcp 437 on Windows, or DOSBox). On a UTF-8 terminal the

lone byte 0x8F is invalid and prints as garbage, often �. */

putchar(143);

/* char is signed on x86, so a plain char holding 0x8F is -113.

Use unsigned char when you compare or index by byte value. */

char c = (char)143;

unsigned char u = 143;

printf("\n%d %d\n", c, u); /* -113 143 */

/* Portable: print the Unicode character as UTF-8 instead. */

printf("\u00C5\n"); /* C3 85 */

return 0;

}