Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Server search #5

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Server search #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
871abc7
search for subscriptions in web server
bleonard Jul 9, 2019
bf13e2c
Always do JSON for server event search.
bleonard Jul 9, 2019
9c1d498
try various ways to fix up the search query
bleonard Jul 9, 2019
29f2ea1
Clean up a view things in event search
bleonard Jul 9, 2019
c13f907
support hash.inspect logging from event publish
bleonard Jul 9, 2019
2f3149f
Sort query for display
bleonard Jul 9, 2019
efdafa9
handle parsing of enqueued array
bleonard Jul 9, 2019
e7bc606
add the subscription key to the table.
bleonard Jul 9, 2019
cddbb5c
button to set the text area to the pretty version.
bleonard Jul 9, 2019
5591dec
Don’t flush it all!
bleonard Jul 22, 2019
58d214e
adds some specs
bleonard Jul 22, 2019
9df1f8e
rescue StandardError
bleonard Jul 22, 2019
295634e
Clearer comment
bleonard Jul 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions lib/resque_bus/server.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,125 @@ def self.included(base)

}
end

class Helpers
class << self
def parse_query(query_string)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this move to QueueBus so that it can be shared between sidekiq-bus and resque-bus?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds like a plan to me.

query_string = query_string.to_s.strip
has_open_brace = query_string.include?("{")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's generally a flat JSON, but perhaps we could use a regex here to anchor the opening and closing brace to the start and end? I think we could also strip the string and then look at the first and last character.

has_close_brace = query_string.include?("}")
has_multiple_lines = query_string.include?("\n")
has_colon = query_string.include?(":")
has_comma = query_string.include?(",")
has_quote = query_string.include?("\"")

exception = nil

# first let's see if it parses
begin
query_attributes = JSON.parse(query_string)
raise "Not a JSON Object" unless query_attributes.is_a?(Hash)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could also just set the exception variable here. That would avoid the raise and immediate rescue.

rescue StandardError => e
exception = e
end
return query_attributes unless exception

if query_attributes
# it parsed but it's something else
if query_attributes.is_a?(Array) && query_attributes.length == 1
# maybe it's pasted from the inputs in the web UI like queues/bus_incoming
# this is an array (of job arguments) and the first one is a JSON string
json_string = query_attributes.first
fixed = JSON.parse(json_string) rescue nil
return fixed if fixed
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we validate that this is also a Hash as we do earlier?

end

# something else?
raise exception
end

if !has_open_brace && !has_close_brace
# maybe they just forgot the braces
fixed = JSON.parse("{ #{query_string} }") rescue nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't we do this on line 38 before we parse? If we're concerned about the possibility that it's an array, maybe we can also look at the opening/closing brackets to see if they are square braces.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I tried to do was not mess with the input if it would parse. That's why I parsed first. Seems like a safe practice - if they put something that works use it before messing around with it.

return fixed if fixed
end

if !has_open_brace
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be able to reduce the complexity. of these conditions:

query_string = "{ #{query_string}" unless has_open_brace
query_string = "#{query_string} }" unless has_close_brace
fixed = JSON.parse(query_string) rescue nil
return fixed if fixed

# maybe they just forgot the braces
fixed = JSON.parse("{ #{query_string}") rescue nil
return fixed if fixed
end

if !has_close_brace
# maybe they just forgot the braces
fixed = JSON.parse("#{query_string} }") rescue nil
return fixed if fixed
end

if !has_multiple_lines && !has_colon && !has_open_brace && !has_close_brace
# we say they just put a bus_event type here, so help them out
return {"bus_event_type" => query_string, "more_here" => true}
end

if has_colon && !has_quote
# maybe it's some search syntax like this: field: value other: true, etc
# maybe use something like this later: https://github.com/dxwcyber/search-query-parser

# quote all the strings, (simply) tries to avoid integers
test_query = query_string.gsub(/([a-zA-z]\w*)/,'"\0"')
if !has_comma
test_query.gsub!("\n", ",\n")
end
if !has_open_brace && !has_close_brace
test_query = "{ #{test_query} }"
end

fixed = JSON.parse(test_query) rescue nil
return fixed if fixed
end

if has_open_brace && has_close_brace
# maybe the whole thing is a hash output from a hash.inspect log
ruby_hash_text = query_string.clone
# https://stackoverflow.com/questions/1667630/how-do-i-convert-a-string-object-into-a-hash-object
# Transform object string symbols to quoted strings
ruby_hash_text.gsub!(/([{,]\s*):([^>\s]+)\s*=>/, '\1"\2"=>')
# Transform object string numbers to quoted strings
ruby_hash_text.gsub!(/([{,]\s*)([0-9]+\.?[0-9]*)\s*=>/, '\1"\2"=>')
# Transform object value symbols to quotes strings
ruby_hash_text.gsub!(/([{,]\s*)(".+?"|[0-9]+\.?[0-9]*)\s*=>\s*:([^,}\s]+\s*)/, '\1\2=>"\3"')
# Transform array value symbols to quotes strings
ruby_hash_text.gsub!(/([\[,]\s*):([^,\]\s]+)/, '\1"\2"')
# fix up nil situation
ruby_hash_text.gsub!(/=>nil/, '=>null')
# Transform object string object value delimiter to colon delimiter
ruby_hash_text.gsub!(/([{,]\s*)(".+?"|[0-9]+\.?[0-9]*)\s*=>/, '\1\2:')
fixed = JSON.parse(ruby_hash_text) rescue nil
return fixed if fixed
end

raise exception
end

def sort_query(query_attributes)
query_attributes.each do |key, value|
if value.is_a?(Hash)
query_attributes[key] = sort_query(value)
end
end
query_attributes.sort_by { |key| key }.to_h
end

def query_subscriptions(app, query_attributes)
# TODO: all of this can move to method in queue-bus to replace event_display_tuples
if query_attributes
subscriptions = app.subscription_matches(query_attributes)
else
subscriptions = app.send(:subscriptions).all
end
end
end
end
end
end

Expand Down
81 changes: 71 additions & 10 deletions lib/resque_bus/server/views/bus.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,34 +9,64 @@ if (agree)
else
return false ;
}

function setSample() {
var text = document.getElementById("query_attributes").textContent;
var textArea = document.getElementById('querytext');
textArea.value = text;
return false;
}
// -->
</script>

<%
app_hash = {}
class_hash = {}
event_hash = {}
query_string = params[:query].to_s.strip

query_attributes = nil
query_error = nil
if query_string.length > 0
begin
query_attributes = ::ResqueBus::Server::Helpers.parse_query(query_string)
raise "Not a JSON Object" unless query_attributes.is_a?(Hash)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make the view lighter if we put this logic into the parser. This is already repeated with some of the logic in the parse_query fuction.

rescue Exception => e
query_attributes = nil
query_error = e.message
end

if query_attributes
# sort keys for display
query_attributes = ::ResqueBus::Server::Helpers.sort_query(query_attributes)
end
end

# collect each differently
::QueueBus::Application.all.each do |app|
app_key = app.app_key

app_hash[app_key] ||= []
app.event_display_tuples.each do |tuple|
class_name, queue, filters = tuple
subscriptions = ::ResqueBus::Server::Helpers.query_subscriptions(app, query_attributes)
subscriptions.each do |sub|
class_name = sub.class_name
queue = sub.queue_name
filters = sub.matcher.filters
sub_key = sub.key

if filters["bus_event_type"]
event = filters["bus_event_type"]
else
event = "see filter"
end

app_hash[app_key] << [event, class_name, queue, filters]
app_hash[app_key] ||= []
app_hash[app_key] << [sub_key, event, class_name, queue, filters]

class_hash[class_name] ||= []
class_hash[class_name] << [app_key, event, queue, filters]
class_hash[class_name] << [app_key, sub_key, event, queue, filters]

event_hash[event] ||= []
event_hash[event] << [app_key, class_name, queue, filters]
event_hash[event] << [app_key, sub_key, class_name, queue, filters]
end
end

Expand All @@ -53,14 +83,14 @@ else
form = ""
if button
text, url = button
form = "<form method='POST' action='#{u url}' style='float:left; padding:0 5px 0 0;margin:0;'><input type='submit' name='' value='#{text}' style='padding:0;margin:0;' onclick=\"return confirmSubmit();\"/><input type='hidden' name='name' value='#{h(name)}' /></form>"
form = "<form method='POST' action='#{u url}' style='float:left; padding:0 5px 0 0;margin:0;'><input type='submit' name='' value='#{h(text)}' style='padding:0;margin:0;' onclick=\"return confirmSubmit();\"/><input type='hidden' name='name' value='#{h(name)}' /></form>"
end

if !val
out = "<td>&nbsp;</td><td>&nbsp;</td>"
else
one, two, queue, filters = val
out = "<td>#{h(one)}</td><td>#{h(two)}</td><td><a href='#{u("queues/#{queue}")}'>#{h(queue)}</a></td>"
one, two, three, queue, filters = val
out = "<td>#{h(one)}</td><td>#{h(two)}</td><td>#{h(three)}</td><td><a href='#{u("queues/#{queue}")}'>#{h(queue)}</a></td>"
out << "<td>#{h(::QueueBus::Util.encode(filters).gsub(/\"bus_special_value_(\w+)\"/){ "(#{$1})" }).gsub(" ", "&nbsp;").gsub('&quot;,&quot;', '&quot;, &quot;')}</td>"
end

Expand All @@ -85,19 +115,48 @@ else
end
%>

<h1 class='wi'>Sample Event</h1>
<p class='intro'>Enter JSON of an event to see applicable subscriptions.</p>
<div style="text-align: center;width:700px;">
<form method="GET" action="<%= u "bus" %>" style="float:none;padding:0;margin:0;">
<textarea id="querytext" name="query" style="padding: 10px;height:150px;width:700px;font-size:14px;font-family:monospace"><%=
h(query_string)
%></textarea>
<br/>
<button onclick="window.location.href = '<%= u "bus" %>'; return false;">Clear</button>
<input type="submit" name="" value="Filter results to this event"/>
</form>
</div>

<% if query_error %>
<blockquote><pre style="text-align:left;font-family:monospace;margin:5px 0 5px 0;padding:10px;background:#f2dede;color:#a94442;"><code><%=
h(query_error.strip)
%></code></pre></blockquote>
<% end %>
<% if query_attributes %>
<blockquote><pre style="text-align:left;font-family:monospace;margin:5px 0 5px 0;padding:10px;background:#dff0d8;color:#3c763d;"><code id="query_attributes"><%=
h(JSON.pretty_generate(query_attributes).strip)
%></code></pre></blockquote>
<div style="text-align:right;">
<button onclick="return setSample();">Set Sample</button>
</div>
<% end %>

<hr/>

<h1 class='wi'>Applications</h1>
<p class='intro'>The apps below have registered the given classes and queues.</p>
<table class='queues'>
<tr>
<th>App Key</th>
<th>Subscription Key</th>
<th>Event Type</th>
<th>Class Name</th>
<th>Queue</th>
<th>Filters</th>
</tr>
<%= output_hash(app_hash, ["Unsubscribe", "bus/unsubscribe"]) %>

<%= output_hash(app_hash, query_attributes ? false : ["Unsubscribe", "bus/unsubscribe"]) %>
</table>

<p>&nbsp;</p>
Expand All @@ -108,6 +167,7 @@ else
<tr>
<th>Event Type</th>
<th>App Key</th>
<th>Subscription Key</th>
<th>Class Name</th>
<th>Queue</th>
<th>Filters</th>
Expand All @@ -125,6 +185,7 @@ else
<tr>
<th>Class Name</th>
<th>App Key</th>
<th>Subscription Key</th>
<th>Event Type</th>
<th>Queue</th>
<th>Filters</th>
Expand Down
114 changes: 114 additions & 0 deletions spec/server_helper_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
require 'spec_helper'
require_relative '../lib/resque_bus/server'

describe "Web Server Helper" do
describe ".parse_query" do
it "should pass through valid json" do
input = %Q{
{ "name": "here", "number": 1, "bool": true }
}
output = {"name"=>"here", "number"=>1, "bool"=>true}
check = ::ResqueBus::Server::Helpers.parse_query(input)
check.should == output
end

it "should handle multi-line json" do
input = %Q{
{
"name": "here",
"number": 1,
"bool": true
}
}
output = {"name"=>"here", "number"=>1, "bool"=>true}
check = ::ResqueBus::Server::Helpers.parse_query(input)
check.should == output
end

it "should interpret simple string as bus_event_type" do
input = %Q{ user_created }
output = {"bus_event_type" => "user_created", "more_here" => true}
check = ::ResqueBus::Server::Helpers.parse_query(input)
check.should == output
end

it "should raise error on valid json that's not an Object" do
input = '[{ "name": "here" }]'
lambda {
::ResqueBus::Server::Helpers.parse_query(input)
}.should raise_error("Not a JSON Object")
end

it "should allow array from resque server panel with encoded json string arg array" do
input = '["{\"name\":\"here\"}"]'
output = {"name" => "here"}
check = ::ResqueBus::Server::Helpers.parse_query(input)
check.should == output
end

it "should take in a arg list and make it json" do
input = %Q{
bus_event_type: my_event
user_updated: true
}
output = {"bus_event_type" => "my_event", "user_updated" => "true"}
check = ::ResqueBus::Server::Helpers.parse_query(input)
check.should == output
end

it "should take in an arg list with quoted json and commas" do
input = %Q{
"bus_event_type": "my_event",
"user_updated": true
}
output = {"bus_event_type" => "my_event", "user_updated" => true}
check = ::ResqueBus::Server::Helpers.parse_query(input)
check.should == output
end

it "should parse logged output from event.inspect" do
input = %Q{
{"bus_published_at"=>1563793250, "bus_event_type"=>"user_created", :user_id=>42, :name=>"Brian" }
}
output = {
"bus_published_at" => 1563793250,
"bus_event_type" => "user_created",
"user_id" => 42,
"name" => "Brian"
}
check = ::ResqueBus::Server::Helpers.parse_query(input)
check.should == output
end

it "should throw json parse error when it can't be handled" do
input = '{ "name": "here" q }'
lambda {
::ResqueBus::Server::Helpers.parse_query(input)
}.should raise_error(/unexpected token/)
end
end

describe ".sort_query" do
it "should alphabetize a query hash" do
input = {"cat" => true, "apple" => true, "dog" => true, "bear" => true}
output = {"apple" => true, "bear" => true, "cat" => true, "dog" => true }
check = ::ResqueBus::Server::Helpers.sort_query(input)
check.should == output
end

it "should alphabetize a query sub-hashes but not arrays" do
input = {"cat" => true, "apple" => [
"jackal", "kangaroo", "iguana"
], "dog" => {
"frog" => 11, "elephant" => 12, "hare" => 16, "goat" => 14
}, "bear" => true}
output = {"apple" => [
"jackal", "kangaroo", "iguana"
], "bear" => true, "cat" => true, "dog" => {
"elephant" => 12, "frog" => 11, "goat" => 14, "hare" => 16
}}
check = ::ResqueBus::Server::Helpers.sort_query(input)
check.should == output
end
end
end
Loading