Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Darrell Abrau - Gutenberg #79

Open
wants to merge 2 commits into
base: gutenberg
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions gutenberg/Gemfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
source 'https://rubygems.org'

gem 'pry-byebug'
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion data/stopwords.txt → gutenberg/data/stopwords.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
a,able,about,across,after,all,almost,also,am,among,an,and,any,are,as,at,be,because,been,but,by,can,cannot,could,dear,did,do,does,either,else,ever,every,for,from,get,got,had,has,have,he,her,hers,him,his,how,however,i,if,in,into,is,it,its,just,least,let,like,likely,may,me,might,most,must,my,neither,no,nor,not,of,off,often,on,only,or,other,our,own,rather,said,say,says,she,should,since,so,some,than,that,the,their,them,then,there,these,they,this,tis,to,too,twas,us,wants,was,we,were,what,when,where,which,while,who,whom,why,will,with,would,yet,you,your,one,out,more,now,first,two,very,such,same,shall,upon,before,therefore,great,made,even,same,work,make,being,through,here,way,true,see,time,those,place,much,without,body,whole,another,thus,set,new,given,both,above,well,part,between,end,order,each,form,gutenberg
a,able,about,across,after,all,almost,also,am,among,an,and,any,are,as,at,be,because,been,but,by,can,cannot,could,dear,did,do,does,either,else,ever,every,for,from,get,got,had,has,have,he,her,hers,him,his,how,however,i,if,in,into,is,it,its,just,least,let,like,likely,may,me,might,most,must,my,neither,no,nor,not,of,off,often,on,only,or,other,our,own,rather,said,say,says,she,should,since,so,some,than,that,the,their,them,then,there,these,they,this,tis,to,too,twas,us,wants,was,we,were,what,when,where,which,while,who,whom,why,will,with,would,yet,you,your,one,out,more,now,first,two,very,such,same,shall,upon,before,therefore,great,made,even,same,work,make,being,through,here,way,true,see,time,those,place,much,without,body,whole,another,thus,set,new,given,both,above,well,part,between,end,order,each,form,gutenberg,
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion gutenberg.rb → gutenberg/gutenberg.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
require_relative 'lib/simple_predictor'
require_relative 'lib/complex_predictor'
require 'pry-byebug'

def run!(predictor_klass, opts={})
puts "+----------------------------------------------------+"
Expand All @@ -15,7 +16,6 @@ def run!(predictor_klass, opts={})
start_time = Time.now
predictor.train!
puts "Training took #{Time.now - start_time} seconds."

puts "Predicting..."
start_time = Time.now
accuracy = predictor.predict_test_set(opts)
Expand Down
80 changes: 80 additions & 0 deletions gutenberg/lib/complex_predictor.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
require_relative 'predictor'

class ComplexPredictor < Predictor
# Public: Trains the predictor on books in our dataset. This method is called
# before the predict() method is called.
#
# Returns nothing.
def train!
@data = Hash.new()

get_categories(@data, @all_books)
compile_words_by_category(@data, @all_books)
build_top_words_hash_for_each_category(@data, 50)
end

def build_top_words_hash_for_each_category(compiled_words_by_category_hash, number_of_pop_words)
@data.each do |category, books|
histogram = build_histogram(books)
top_words = find_top_words(histogram, number_of_pop_words)
@data[category] = top_words
end
end

def get_categories(empty_hash, given_books)
given_books.keys.each {|key| empty_hash[key] = []}
end

def compile_words_by_category(categories_hash, given_books)
given_books.each do |category, books|
books.each do |book|
book[1].each do |word|
categories_hash[category] << word
end
end
end
end

def build_histogram(array)
histogram = Hash.new(0)

array.each do |word|
if good_token?(word)
histogram[word] += 1
end
end

histogram
end

def find_top_words(histogram, number_of_words)
sorted = histogram.sort_by {|k, v| v}
Hash[sorted[-number_of_words..-1]]
end


# Public: Predicts category.
#
# tokens - A list of tokens (words).
#
# Returns a category.
def predict(tokens)
book_histogram = build_histogram(tokens)
top_words_for_book = find_top_words(book_histogram, 300)
decide_category(top_words_for_book, @data)
end

def decide_category(top_words, training_data)
count_hash = Hash.new(0)
training_data.each do |category, pop_words|
top_words.each_key do |word|
if pop_words.has_key?(word)
count_hash[category] = count_hash[category] + 1
end
end
end
count_hash.max_by{|k,v| v}.first
end
end


File renamed without changes.
File renamed without changes.
File renamed without changes.
22 changes: 0 additions & 22 deletions lib/complex_predictor.rb

This file was deleted.