overfitting and underfitting

 

https://towardsdatascience.com/what-are-overfitting-and-underfitting-in-machine-learning-a96b30864690

accuracy, precision and recall

 

https://towardsdatascience.com/accuracy-precision-recall-or-f1-331fb37c5cb9

random forest, boosting and bagging

 

https://sebastianraschka.com/faq/docs/bagging-boosting-rf.html#:~:text=tl%3Bdr%3A%20Bagging%20and%20random,that%20underfit%20the%20training%20data.

feature selection techniques , linear regession

 

https://towardsdatascience.com/feature-selection-techniques-in-regression-model-26878fe0e24e

partition by and group by in sql

 

https://www.sqltutorial.org/sql-window-functions/sql-partition-by/#:~:text=SQL%20PARTITION%20BY%20clause%20overview&text=The%20PARTITION%20BY%20clause%20divides,and%20recalculate%20for%20each%20partition.&text=You%20can%20specify%20one%20or,to%20partition%20the%20result%20set.

NLP , AI and machine learning

 

https://monkeylearn.com/blog/nlp-ai/

ROC curve

 

https://medium.com/swlh/roc-curve-and-auc-detailed-understanding-and-r-proc-package-86d1430a3191

Anova type I/II/III

 

http://md.psych.bio.uni-goettingen.de/mv/unit/lm_cat/lm_cat_unbal_ss_explained.html

F-test

 

https://people.richland.edu/james/lecture/m170/ch13-f.html

collinear testing

 

https://www.stat.tamu.edu/~hart/652/collinear.pdf

Create a new project under cookiecutter

 

https://cookiecutter.readthedocs.io/en/1.7.2/first_steps.html

https://www.youtube.com/watch?v=5VjuG5lliYU

https://openfolder.sh/django-how-to-scaffold-cookiecutter

cookiecutter building using template

 

https://nsls-ii.github.io/scientific-python-cookiecutter/preliminaries.html

https://cookiecutter.readthedocs.io/en/1.7.2/first_steps.html

install centos and install hugo using snap

 

https://snapcraft.io/install/hugo/centos

https://www.youtube.com/watch?v=31Xfju-yCHw

cookiecutter make template

 https://raphael.codes/blog/create-your-own-cookiecutter-template/

top 10 static site generators

 

https://www.youtube.com/watch?v=RsxTAuP_EQ8

build a serverless blog with hugo and netlify

 

https://www.youtube.com/watch?v=gSiAcyTWU3c

build a website using hugo

 

https://www.sitepoint.com/premium/books/a-beginner-s-guide-to-creating-a-static-website-with-hugo/read/1

https://www.chuxinhuang.com/blog/noobs-guide-to-hugo/

easy ways to make money

 

https://elnacain.com/blog/easy-ways-to-make-money/

Linux desktop version

 

https://www.youtube.com/watch?v=SlmgwHAXgB4

How to generate a static website using hugo

 

https://thenewstack.io/tutorial-use-hugo-to-generate-a-static-website/

How to build a website using hugo and post on it

 

https://www.freecodecamp.org/news/your-first-hugo-blog-a-practical-guide/

hugo installing a theme

 https://www.mikedane.com/static-site-generators/hugo/installing-using-themes/

assign a domain name to localhost

 

windows 

https://www.tutorialsteacher.com/articles/set-custom-domain-name-for-localhost-website-iis

linux 

https://www.geeksforgeeks.org/creating-custom-domain-name-instead-of-localhost-in-ubuntu/

Hugo themes and what you can do with it

 

https://themes.gohugo.io/simplicity/

The 50 best blogging platforms

 https://sharethis.com/thought-leadership/2019/12/best-blogging-platforms/

How to code a website (html , css, python)

 

https://websitesetup.org/website-coding-html-css/

https://www.probytes.net/blog/use-python-make-websites/

https://code.tutsplus.com/articles/python-from-scratch-create-a-dynamic-website--net-22787

https://pythonhow.com/building-a-website-with-python-flask/

clustering

 k-means clustering 

https://towardsdatascience.com/understanding-k-means-clustering-in-machine-learning-6a6e67336aa1

hierarchical clustering 

https://towardsdatascience.com/understanding-the-concept-of-hierarchical-clustering-technique-c6e8243758ec

https://uc-r.github.io/hc_clustering

natural language processing

 https://towardsdatascience.com/your-guide-to-natural-language-processing-nlp-48ea2511f6e1

recommendation systems

user-based 

 https://medium.com/sfu-cspmp/recommendation-systems-user-based-collaborative-filtering-using-n-nearest-neighbors-bf7361dc24e0

https://towardsdatascience.com/build-a-user-based-collaborative-filtering-recommendation-engine-for-anime-92d35921f304

https://www.r-bloggers.com/2014/05/basic-recommendation-engine-using-r/

item based

https://towardsdatascience.com/comprehensive-guide-on-item-based-recommendation-systems-d67e40e2b75d

user based and item based 

https://medium.com/@cfpinela/recommender-systems-user-based-and-item-based-collaborative-filtering-5d5f375a127f

how to create a website (in chinese)

 

https://yihui.org/cn/2009/06/how-to-build-a-website-as-a-dummy/

how to add adsense to your website or hugo website

 

https://www.quicksprout.com/how-to-add-adsense-to-your-website/

https://bash-prompt.net/guides/custom-html-jugo/#:~:text=Copy%20your%20AdSense%20code%20into,%22google%2Ffile%22%20.%20%7D%7D

https://www.godo.dev/tutorials/hugo-in-article-ad/

https://www.quicksprout.com/how-to-add-adsense-to-your-website/

dual boot debian and windows

 https://www.linuxtechi.com/dual-boot-windows-10-debian-10/

https://www.youtube.com/watch?v=HlD6E0Vp8is

dual boot fedora and windows

 https://www.fosslinux.com/842/how-to-install-fedora-in-a-dual-boot-setup-with-windows.htm

install debian using rufus

 https://linuxhint.com/install_debian9_stretch_usb/#:~:text=Downloading%20Rufus%20and%20Making%20a,should%20see%20the%20following%20window.

self attention

 https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a

Wild and Deep Learning

 https://ai.googleblog.com/2016/06/wide-deep-learning-better-together-with.html

python efficiently

 https://www.cnblogs.com/xueweihan/p/13624201.html

Build a website using hugo

 Build a website using hugo

https://www.youtube.com/watch?v=c7vpcqA6SEQ

https://gohugo.io/getting-started/installing/

How to create a new hugo blog post.

https://www.youtube.com/watch?v=cunYNe7qLjE

some other methods:

bo-blog

wordpress

jekyll

some other related articles 

https://jdhao.github.io/2019/09/22/google_adsense_hugo_config/

https://sspai.com/post/59504

https://www.qikqiak.com/post/add-adsense-in-hugo-article/

Git push and pull on git bash windows

 https://www.datacamp.com/community/tutorials/git-push-pull

regularization bias and variance

https://towardsdatascience.com/regularization-in-machine-learning-76441ddcf99a#:~:text=Regularization,linear%20regression%20looks%20like%20this.

How to fix the USB when it lost space

Sometimes, there is not much things in your usb. And it shows that there is no much space left in your usb.  Your USB lost space. But it can be fixed.

First move the things in your USB to another place. The steps here will make things in USB lost.

In a windows environment, follow the steps

1. In the start menu, search for and launch diskpart.
2. Type list disk to see all current disk volumes on your system,
3. Plug in the USB and type list disk. Note the newly listed volume, that's your USB's.
4. Type select disk#, where # is your USB's volume number.
5. Type Clean to wipe the volume of alll partitions.
6. Type create partition primary to make a new partition with all unallocated space.
7. type exit to finish.

When I tried to fix my USB, I tried steps 4,5 twice. It did not work at the first time.

When I tried to use my USB, it showed that I need to format the USB. And I formatted the USB. It got back to normal.

Using ubuntu, and sometimes it freezes and crashes, how I fixed it

One my operation systems is ubuntu.
sometimes it freezes and crashes.
And I found that most of the time, the browsers freeze. I used firefox and chrome.
And I look for some other browsers. And I found opera and chromium.
I tried to install opera.
I downloaded opera deb file. And tried to install deb.
I tried the command first:

sudo apt install ...deb

not working

Then I tried sudo dpkg -i xxx.deb

works.



finally installed opera. now it seems no crashes or freezes

Opera seems stable. But it does not support some videos.

To install chromium, I used
sudo apt install -y chromium-browser

And I installed it. chromium seems not so stable as opera.

how to fix R's error Error: variables "xxx" were specified with different types from the fit

In R's predicting modeling, sometimes there is an error:

 Error: variables ‘xxx"  were specified with different types from the fit

How to fix it?

Mainly the reason for it is that in the data where we did the predicting  the data type for some variables are different from the data type for some variables from the modeling.

You can check the data types in the data where we will do the predicting.

Once I found that , there were some quotes in the data which made the data into character. Then I unquoted the data and made the data types into numerical. problem solved.

Sample script:

newData=noquote(newData)

newData$in_dream1<-as.numeric(newData$in_dream1)
newData$selected_by1<-as.numeric(newData$selected_by1)
newData$form1<-as.numeric(newData$form1)
newData$points_game1<-as.numeric(newData$points_game1)
newData$element_type1<-as.factor(newData$element_type1)

inc.com is an interesting webpage

inc.com is an interesting webpage.

there is a 5000 companies list every year:

https://www.inc.com/inc5000/2019/top-private-companies-2019-inc5000.html 

10 linux simple free games

Linux offers some free games. Here are 10 free linux games:

1. Card game, kpatience

2. Brutal Chess

3. SuperTuxKart

4. SDL-Ball

5. KBlocks

6. Secret Maryo Chronicles

7. Gweled

8 Zaz

9. Frozen Bubble

10. Torcs

How to use git and github on linux

https://product.hubspot.com/blog/git-and-github-tutorial-for-beginners


Awesome one!

Not able to install kernlab package in R, how to resolve the problem

Tried to install kernlab package in R, tried install.packages("kernlab")  but failed.

Tried to install it by install.packages("kernlab", type="source"), but failed.

And download from

https://cran.r-project.org/web/packages/kernlab/index.html

And install on a terminal

R CMD INSTALL kernlab_0.9-19.tar

And library(kernlab)

model selection caret in R

caret is a package in R.

Caret is the short for Classification And REgression Training.

How to do the predictive modeling in caret/R, steps

https://towardsdatascience.com/create-predictive-models-in-r-with-caret-12baf9941236


How to do the classification modeling in caret/R, steps:

https://rpubs.com/ezgi/classification

Working with json file using R

https://blog.exploratory.io/working-with-json-data-in-very-simple-way-ad7ebcc0bb89


read in a json file from webpage using the package in R rjson:
newElements<-fromJSON(file("https://fantasy.premierleague.com/api/bootstrap-static/"))

the data we needed is an element in json file , how to read it and make it a data frame

newElements <- as.data.frame(newElements$elements)

R rds and rda models

https://www.mydatahack.com/how-to-save-machine-learning-models-in-r/


commands:

save rda model

save(model_nnet, file = "/tmp/model_nnet.rda")

load rda model

load(file = "/tmp/model_nnet.rda")

save rds 

saveRDS(model_nnet, file = "/tmp/model_nnet2.rda")

load rds 

model2 <- readRDS("/tmp/model_nnet2.rda")

Top 10 programming languages used in web development 

Java

Python

C

SQL

Go

JavaScript

CSS / HTML

C++

PHP

Ruby

statistics' application on medical area

1. Clinical trial sample size and power analysis



2. cox proportional hazards model

Things about classification AUC, ROC, recall, precision, sensitivity, specificity and F1

Classification methods predict the probability of the values of  a categorical  responsible variable  given some predictors.

There are a lot of classification methods. logistic, support vector machine, random forest

In R, there is a package called caret , classification and regression training . There are a lot of built-in classification methods :

https://rdrr.io/cran/caret/man/models.html

To measure the effectiveness of a classification method, there are some metrics :

AUC, ROC, recall, precision, sensitivity, specificity and F1

1.  ROC and AUC

A receiver operating characteristic curve or ROC curve, is a graphical plot that illustrates the diagnostic ability of binary classifier system as its discrimination threshold is varied.

The ROC curve is created by plotting the true positive rate against the false positive rate at various threshold settings. True positive rate is also known as sensitivity, recall or probability of detection. The false positive rate is also known as probability of false alarm, and can be calculated as (1-specificity).  It can also be thought of as a plot of the power as a function of the Type I Error of the decision rule.

AUC means area under the ROC curve. It is a value between 0 and 1.  The closer AUC is to 1, the more accurate the classification prediction is.


2. Recall, precision , sensitivity , specificity , F1

The values are from confusion matrix

precision=true positive/(true positive + false positive)

recall ( sensitivity) =true positive /(true positive + false negative)

specificity =true negative /(true negative + false positive)

F1 Score = 2*(Recall * Precision) / (Recall + Precision)

High recall and high precision show that the classifier is returning accurate  results ( high precision ) , as well as returning a majority of all positive results ( high recall).

specificity is the true negative rate. high specificity means that the classifier is retuning a majority of all negative results.

recommendation system

https://towardsdatascience.com/how-to-build-a-simple-recommender-system-in-python-375093c3fb7d

recommendation systems

recommendation systems

item based

https://en.wikipedia.org/wiki/Item-item_collaborative_filtering

item based and user based recommendation systems

https://medium.com/@cfpinela/recommender-systems-user-based-and-item-based-collaborative-filtering-5d5f375a127f

data pipeline

https://towardsdatascience.com/data-science-for-startups-data-pipelines-786f6746a59a

predictive modeling in python

https://www.analyticsvidhya.com/blog/2017/06/a-comprehensive-guide-for-linear-ridge-and-lasso-regression/

https://github.com/agrawal-priyank/machine-learning-regression

https://towardsdatascience.com/ridge-and-lasso-regression-a-complete-guide-with-python-scikit-learn-e20e34bcbf0b

two good articles on statistics and data analysis

https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229


https://towardsdatascience.com/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8

10 useful python data visilization tools

https://mode.com/blog/python-data-visualization-libraries

I used matplotlib and ggplot

ggplot2 is for R

predictive modeling python with missing values in data

https://towardsdatascience.com/end-to-end-python-framework-for-predictive-modeling-b8052bb96a78

build a website in 11 mins python django

https://www.youtube.com/watch?v=PqeAvFf_HDI