One way to install linux on your pc without a disk

The official way to install linux on PC is to use an installing disk, like a CD.

I found a way to install it without a disk, but a USB:

https://builtvisible.com/the-ubuntu-installation-guide/#Install-from-USB

It can help you to install the ubuntu on your computer properly.

One way to deal with the csv file " comma inside quotes" problem.

Sometimes, we got some annoying csv files, they are not properly delimited. When we open it, we saw the "," showing in the file together with the text. And when we try to open it through excel, it won't work either.

One way to deal with this kind of csv file, is:

1. download the csv file, but do not touch it, like opening it or something.
2. change the file name ".csv" to ".txt" without opening it.
3. open an excel, and use "open file" inside the excel to open the file, step by step, use delimiter as "comma". and use the Text_Qualifier field in your Flat file connection manager to as ". This should wrap your data in quotes and only separate by commas which are outside the quotes.

The data looks like below:

 The step to open the new txt file:

How to install MySQL on linux

How to install MySQL on linux:

First, you need to install linux.

I installed linux on windows using this method:

https://builtvisible.com/the-ubuntu-installation-guide/#Install-from-USB

Linux is one of the best operation system that I have used. It is better than windows etc.

And then, you can install MySQL on linux:

https://www.linode.com/docs/databases/mysql/how-to-install-mysql-on-ubuntu-14-04

Then you can write some  script for mysql  and run it on linux.

data science/analysis website

https://www.kaggle.com/

http://blog.yhat.com/

www.r-bloggers.com 

A tricky program

I have a list of files in python, each of having a bunch of elements. I want to count the total number elements from all of the files. There are two ways to do it:
1. concatenate all the sub files into one list, and count all the elements in the bigger list. But when I try to concatenate all the sub files into one list, it exceeded the memory quota.
2. read in one file into one list each time, and count the number of elements in the list, and add the number of the elements of each list together, each time initialize one new list when reading in a new file.

import glob 

a=0

 for file in glob.glob("/home/adam/*.txt"):
      list=[]
      file=open(file,'rb') as b:
           for row in b:
             list.append(b)
      a=a+len(list)
print a

then , we will see the total number of elements from printing a. 

Have a good habit on programming

There are some tips on writing program:

1. Try to give the objects some meaningful names, like when I create a list containing a list of fruit. Then we cab type: fruit=['kiwi','orange','pear'], instead b=['kiwi','orange','pear'].

2. Give the files some handles before reading them in, like:

file1="aaa.txt"

file2="bbb.txt"

So when you change the file names to be read in, just change the handles.

3. When a piece of code is repeatedly used, we can code them into functions, or classes etc.

One tricky thing about excel

This week, I did some data manipulation to some excel files.

I found one tricky thing about excel.

I used python module called, xlrd to read the excel file into python.

The following is the python code from xlrd:

import xlrd

wb=xlrd.open_workbook("xl1.xlsx")
sh=wb.sheet_by_index(0)

And I found that sometimes when you count the number of the tab, if the tab is on the third
one by counting, but when you try sh=wb.sheet_by_index(3), it may fail. But if you try some other number for that tab, it may work.
And for the same file, if you save is as different formats, like excel 1997-2003 and excel 2010, when you read the file, the number of the tab, i.e. sheet_by_index which will work may be different for
the same tab. You sometimes need to try different numbers to find the right one.

One method to avoid this problem is to use sheet_by_name not sheet_by index, like the following.

But it will need more coding characters.