How to print tokenized arabic text in python/nltk?

I'm doing the sentiment analysis for arabic language, I'm using python /nltk and the dream pie shell ,this problem occurs when I apply the function of tokenization , how to display these words ?

>>> import nltk
>>> sentence = "مصادمات عنيفه في"
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['\xd9\x85\xd8\xb5\xd8\xa7\xd8\xaf\xd9\x85\xd8\xa7\xd8\xaa', '\xd8\xb9\xd9\x86\xd9\x8a\xd9\x81\xd9\x87', '\xd9\x81\xd9\x8a']


ANSWERS:


By printing tokens, you're printing the list and the \x... is the bytecode representation. If you want to print out the arabic form, just loop through the list and print the tokens one by one.

>>> import nltk
>>> sentence = "مصادمات عنيفه في"
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['\xd9\x85\xd8\xb5\xd8\xa7\xd8\xaf\xd9\x85\xd8\xa7\xd8\xaa', '\xd8\xb9\xd9\x86\xd9\x8a\xd9\x81\xd9\x87', '\xd9\x81\xd9\x8a']
>>> for i in tokens:
...     print i
... 
مصادمات
عنيفه
في


 MORE:


 ? Python NLTK Collocations for tagged text
 ? NLTK tokenize with collocations
 ? NLTK import Error Python
 ? NLTK import Error Python
 ? NLTK import Error Python
 ? Python import nltk error
 ? Import Error, When I import nltk.corpus.framenet in NLTK Python
 ? python import nltk error on Mac
 ? Proxy error while downloading NLTK
 ? Not able to Import in NLTK - Python