How to install MySQL on linux

How to install MySQL on linux:

First, you need to install linux.

I installed linux on windows using this method:

https://builtvisible.com/the-ubuntu-installation-guide/#Install-from-USB

Linux is one of the best operation system that I have used. It is better than windows etc.

And then, you can install MySQL on linux:

https://www.linode.com/docs/databases/mysql/how-to-install-mysql-on-ubuntu-14-04

Then you can write some  script for mysql  and run it on linux.

data science/analysis website

https://www.kaggle.com/

http://blog.yhat.com/

www.r-bloggers.com 

A tricky program

I have a list of files in python, each of having a bunch of elements. I want to count the total number elements from all of the files. There are two ways to do it:
1. concatenate all the sub files into one list, and count all the elements in the bigger list. But when I try to concatenate all the sub files into one list, it exceeded the memory quota.
2. read in one file into one list each time, and count the number of elements in the list, and add the number of the elements of each list together, each time initialize one new list when reading in a new file.

import glob 

a=0

 for file in glob.glob("/home/adam/*.txt"):
      list=[]
      file=open(file,'rb') as b:
           for row in b:
             list.append(b)
      a=a+len(list)
print a

then , we will see the total number of elements from printing a. 

Have a good habit on programming

There are some tips on writing program:

1. Try to give the objects some meaningful names, like when I create a list containing a list of fruit. Then we cab type: fruit=['kiwi','orange','pear'], instead b=['kiwi','orange','pear'].

2. Give the files some handles before reading them in, like:

file1="aaa.txt"

file2="bbb.txt"

So when you change the file names to be read in, just change the handles.

3. When a piece of code is repeatedly used, we can code them into functions, or classes etc.

One tricky thing about excel

This week, I did some data manipulation to some excel files.

I found one tricky thing about excel.

I used python module called, xlrd to read the excel file into python.

The following is the python code from xlrd:

import xlrd

wb=xlrd.open_workbook("xl1.xlsx")
sh=wb.sheet_by_index(0)

And I found that sometimes when you count the number of the tab, if the tab is on the third
one by counting, but when you try sh=wb.sheet_by_index(3), it may fail. But if you try some other number for that tab, it may work.
And for the same file, if you save is as different formats, like excel 1997-2003 and excel 2010, when you read the file, the number of the tab, i.e. sheet_by_index which will work may be different for
the same tab. You sometimes need to try different numbers to find the right one.

One method to avoid this problem is to use sheet_by_name not sheet_by index, like the following.

But it will need more coding characters.

Save file in .csv file or tab delimited file

One day, I tried to finish a project in which I used some flat file. First, I save the file to .txt tab delimited file. And when I read in the file, there was always an error, like the rows are not of the same length. Then I converted the file to .csv file. There was no more error. Later on, I found, since when it was tab delimited file, some columns were not actually delimited by tab, i.e. two or more columns are collapsed together, and became one column. So for different rows the number of columns are not equal. But when I used the csv file, no such issue, since the commas are always there, and separated the columns.

So maybe the csv files are safer to used than tab delimited file.

Python's Levenshtein method is much faster than difflib.SquenceMatcher method

I did one project last week. The project is mainly about text mining, comparing the similarity between two strings.

And I used Python's difflib.SequenceMatcher to do the similarity comparison. It took around 120 mins to finish a 400M times computations.

And after I switched to Levenshtein method on Python, it only took 30 mins to finish the same amount of computation.

I was told Levenshtein used C to do the computation and difflib.SequenceMatcher used Python to do the computation. So Levenshtein is much faster.