1. Random Sample

1.1. Situation:

• I want to random sample a 0.3 billion rows data

1.2. Solution:

• Some preparing work

  import random
  import sys
  random.seed( 10 )
  print "Random number with seed 10 : ", random.random()
  sample_num = int(sys.argv[1])
  

• Read Data into Memory

  f = open('target_file.txt')
  lines = f.readlines()
  print 'Read Data Complete'
  original_data = list()
  count = 0
  for line in lines:
      count = count + 1
      # indicator
      if count % 1000000 == 0:
          print count
      line = line.strip()
      target, num = line.split('\t')
      original_data.append(target)
  f.close()
  print('Save to List Complete!')
  

• Random Sampling

  sampled_data = random.sample(original_data, sample_num)
  print('Sampling Data Complete')
  

• Write into another file

  f1 = open('sampled_data_'+str(sample_num)+'.txt','w')
  for sd in sampled_data:
      print >> f1, sd
  f1.close()
  print('Output Sample Data Complete!')
  

• Do other stuff…