I once worked on big data projects. I analyzed 5,000,000,000 rows of data each day. I used hadoop/hive. To analyze the data with some scripts took a long time. Sometimes when there were some errors with the scripts, the program would break, and I needed to start over. And it cost time. So sometimes it took relatively long time to get projects done.
So, when you have the problem, start with small samples of the data. Then the programs run faster. you will get the jobs done sooner. time saving.
Subscribe to:
Post Comments (Atom)
-
I tried to commit script to bitbucket using sourcetree. I first cloned from bitbucket using SSH, and I got an error, "authentication ... -
https://github.com/boto/boto3/issues/134 import boto3 import botocore client = boto3.client('s3') result = client.list_obje...
-
Previously, I wanted to install "script" on Atom to run PHP. And there was some problem, like the firewall. So I tried atom-runner...
No comments:
Post a Comment