ezoic

Monday, December 4, 2017

Python encoding and decoding

1. Unicode and UTF-8

Unicode is character set. UTF-8 is a way of decoding for unicode. And it includes UTF-16, UTF-32.we can find the unicode of a character by searching the standard. The UTF-8 decode can be calculated by unicode.

2. str and unicode in python 2

str and unicode are two types.

str stores the bytes after encoding. When outputting, every byte is presented by 16 digits, starting with \x. Every chinese character has three  bytes.

str can be converted to unicode by decode() method.

example:

a.decode('utf-8')

unicode can be converted to str by encode() method.

example:

b.encode('utf-8')



 ENCODE METHODS

1. head of a script

#-*- coding: utf-8 -*-

or

# coding=utf-8

if we do not set up, the default one is ascii. And you will get error.

2. sys.stdin.encoding and sys.stdout.encoding

3. sys.getdefaultencoding()

CHARACTER SET CONCATENATION

when concatenating str and unicode, the output is unicode. first decode str to unicode by decode(), then concatenating. Sometimes we will have errors.

READ IN FILE AND JSON

read in a file and we get str type, presented as 16-digits starting by "\x".

f=open('t.txt')
a=f.read()

a:

'{"hello":"\xe5\x92\xa9"}\n'

can be decoded to unicode by json

json.loads(a)

{u'hello':u'\u54a9'}

OUTPUT

str can be outputed to files. unicode needs to be encoded to str by encode()

COMPUTE md5

md5 computation requires unicode to be encoded to str first.

hashlib.md5(a).hexdigest()

OUTPUT TO STDOUT

when outputing to stdout , default encoding is sys.stdout.encoding, it depends on the default setting of the system.

import sys
sys.stdout.encoding

'UTF-8'

in the environment of zh_CN.GB2312, default is not UTF-8, we can not output normally.

COMMAND PARAMETER  READIN

the parameters gotten by sys.argv and argparse are all str type, presented as 16-digits starting with \x. We can get encoding type by sys.stdin.encoding , and then convert to unicode.

#! /usr/bin/evn python

# coding =utf-8

import sys

print repr(sys.argv[1])

print sys.stdin.encoding

print repr(sys.argv[1].decode(sys.stdin.encoding))

python hello.py "哇嘿嘿”

'\xe5\x93\x87\xe5\x98\xbf\xe5\x98\xbf'
UTF-8
u'\u54c7\u563f\u563f'


CHARACTER START WITH \u CONVERTED  TO UNICODE

b='\u54a9'

b

'\\u54a9'


convert b to chinese

1. unicode-escape

unicode(b,'unicode-escape')

u'\u54a9'

or

b.decode('unicode-escape')

u'\u54a9'

2. eval concatenation

eval('u"'+b.replace('"',r'\"')+'"'

u'\u54a9'










Python package structures and how to build a package automatically

A python package usually has the following structure:

funniest/
    funniest/
        __init__.py
        command_line.py
        tests/
            __init__.py
            test_joke.py
            test_command_line.py
    MANIFEST.in
    README.rst
    setup.py
    .gitignore

http://python-packaging.readthedocs.io/en/latest/everything.html


To build one package automatically, one can use cookiecutter

https://www.pydanny.com/cookie-project-templates-made-easy.html
https://github.com.audreyr/cookiecutter-pypackage.git




Thursday, October 12, 2017

Some sourcetree, stash tutorial

Atlassian Stash tutorial:



https://www.youtube.com/watch?v=QWS2yXehCNk


Sourcetree tutorial:


https://www.youtube.com/watch?v=PkVMgh1q33Q



More stash:

https://www.youtube.com/watch?v=y9rTRk-5uSQ



If your web.py read in some garbage character, try this method

I once used the python web.py to set up a web service. But when reading in non English words, it will show garbage characters.

Here is a way to fix that problem.

add the two lines at the top of the script:

#!/usr/bin/env python# -*- coding:utf-8 -*-

Add the lines to the beginning import of the script:

import sys
reload(sys)
sys.setdefaultencoding('utf-8')


Add the header in the get function like below. 




It will solve the problem. 

Python's zip, map and lambda


Python's zip, map and lambda:

https://bradmontgomery.net/blog/pythons-zip-map-and-lambda/


R is not a simple programming language, and it does better on reading excel files than python

R is not a simple programming language, and it does better on reading excel files than python . tried to read excel files to python and R. i...