Dev Stack Source

Answer : So what's the problem, It's a ’ ( RIGHT SINGLE QUOTATION MARK - U+2019) character which is being decoded as CP-1252 instead of UTF-8. If you check the encodings table, then you see that this character is in UTF-8 composed of bytes 0xE2 , 0x80 and 0x99 . If you check the CP-1252 code page layout, then you'll see that each of those bytes stand for the individual characters â , € and ™ . and how can I fix it? Use UTF-8 instead of CP-1252 to read, write, store, and display the characters. I have the Content-Type set to UTF-8 in both my <head> tag and my HTTP headers: <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> This only instructs the client which encoding to use to interpret and display the characters. This doesn't instruct your own program which encoding to use to read, write, store, and display the characters in. The exact answer depends on the server side platform / database / progra...

Answer : It's interesting that though the question might seem reasonable, it's not that easy to figure out a practical reason why I would need to convert a StringIO into a BytesIO . Both are basically buffers and you usually need only one of them to make some additional manipulations either with the bytes or with the text. I may be wrong, but I think your question is actually how to use a BytesIO instance when some code to which you want to pass it expects a text file. In which case, it is a common question and the solution is codecs module. The two usual cases of using it are the following: Compose a File Object to Read In [16]: import codecs, io In [17]: bio = io.BytesIO(b'qwe\nasd\n') In [18]: StreamReader = codecs.getreader('utf-8') # here you pass the encoding In [19]: wrapper_file = StreamReader(bio) In [20]: print(repr(wrapper_file.readline())) 'qwe\n' In [21]: print(repr(wrapper_file.read())) 'asd\n' In [26]: bio.seek(0) Out[26]: 0 ...