We keep telling you how web scraping can make your life a little better place. In part 1 you could learn what are the legal issues regarding it are nothing but the discussions of how ethic is behavior of some people in the Internet. The technique of web scraping can do a lot of good. And today we will tell you how to use it with the help of Ruby and Nokogirl.
Getting started
Ruby and Nokogirl work great, when you need to web scrape HTML + CSS pages. It would save you tons of time for migration and exporting data. As soon as you learn this technique, you will be ready for any data transition that world has in the sleeve for you. So let’s get down to some Ruby stuff! You will need:
- Ruby 1.9.3 or higher
- RubyGems
- Nokogiri, CSV, Sanitize, and Find gems
- And some really basic Ruby skills even a beginner has.
If you don’t have any of those, use ‘gem install’ in the Terminal.
Prepare for scraping
The technique of web scraping is basically telling the system to read all the pages and save them locally. If you have a cheap hosting, this might lead to some load issues. Another problem is that not all URLs might be available for you. But let’s imagine we have a pretty accessible and stable website.
Create a new file and name it something like srape.rb. Declare required gems in the beginning.
|
1 2 3 4 5 6 7 8 9 10 11 |
# include required gems require 'find' require 'rubygems' require 'nokogiri' require 'sanitize' require 'csv' |
It is a good idea to get rid of all bad symbols right away, so that you won’t have to bother later. Here’s how we can remove apostrophes and quotes Microsoft uses:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# generic function to replace MS word smart quotes and apostrophes def strip_bad_chars(text) text.gsub!(/"/, "'"); text.gsub!(/\u2018/, "'"); text.gsub!(/[”“]/, '"'); text.gsub!(/’/, "'"); return text end |
And now let’s clean HTML tags to keep text well-formatted:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
def clean_body(text) text.gsub!(/(\r)?\n/, ""); text.gsub!(/\s+/, ' '); # extra muscle, clean up crappy HTML tags and specify what attributes are allowed text = Sanitize.clean(text, :elements => ['h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'p', 'a', 'b', 'strong', 'em', 'img', 'iframe'], :attributes => { 'a' => ['href', 'title', 'name'], 'img' => ['src', 'title', 'alt'], 'iframe' => ['src', 'url', 'class', 'id', 'width', 'height', 'name'], }, :protocols => { 'a' => { 'href' => ['http', 'https', 'mailto'] }, 'iframe' => { 'src' => ['http', 'https'] } }) # clean start and end whitespace text = text.strip; return text end |
Nokogiri time
Now we are ready for our first web scraping. Nokogiri gem will basically do all the hard work for us. Let’s see how it scrapes data from /blog:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 |
# this is the main logic that recursively searches from the current directory down, and parses the HTML files. def parse_html_files Find.find(Dir.getwd) do |file| if !File.directory? file and File.extname(file) == '.html' # exclude and skip if in a bad directory # we may be on an html file, but some we just do not want current = File.new(file).path # stick to just the blog folder if not current.match(/(blog)/) next end # however, skip these folders entirely if current.match(/(old|draft|archive)/) next end # open file, pluck content out by its element(s) page = Nokogiri::HTML(open(file)); # grab title title = page.css('title').text.to_s; title = strip_bad_chars(title) # for page title, destroy any pipes and MS pipes and return the first match title.gsub!(/[│,|],{0,}(.*)+/, '') # grab the body content body = page.css('section article').to_html body = clean_body(body) # clean the file path path = File.new(file).path path.gsub! $base_path, "/" # if we have content, add this as a page to our page array if (body.length > 0) $count += 1 puts "Processing " + title # insert into array data = { 'path' => path, 'title' => title, 'body' => body, } $posts.push data end end end write_csv($posts) report($count) end |
What happens here is that Ruby is checking files in the blog folder one after another looking for HTML ones. It makes sure old, draft and archive posts are not included.
Using .css one can create content structure. Text and to_html methods will let do some formatting.
Rolling things up
Now when Nokogiri did some scrpaing magic, let’s save it to CSV file. Guess what would help us with that? CSV gem!
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# This creates a CSV file from the posts array created above def write_csv(posts) CSV.open('posts.csv', 'w' ) do |writer| writer << ["path", "title", "body"] $posts.each do |c| writer << [c['path'], c['title'], c['body']] end end end |
Don’t forget to echo the job done:
|
1 2 3 4 5 6 7 |
# echo to the console how many posts were written to the CSV file. def report(count) puts "#{$count} html posts were processed to #{Dir.getwd}/posts.csv" end |
And trigger the script
|
1 2 3 |
# trigger everything parse_html_files |
This is it. Now you know how to web scrape HTML + CSS pages. But you should remember that this method won’t work for js files. What shall I do then? We will tell you the answer in the next article. Stay tuned!
