License issue #10
Comments
|
it worked finally, a similar case with solution as follows: |
|
Hi. This is because the output is not registered within logstash. I'll have to search how I did it in the past when I was building this output... |
|
@valentinul , we did install it successfully as follows: how come the last step fails ? logstash -e 'output {cassandra {}}' |
|
Ok....that looks good. what logstash version are you using ? |
|
@valentinul , it's logstash_2.2.2-1_all.deb |
|
Ok... I remember having this kind of issue .... If you're installing a local gem file, put the path to the file in GEM_PATH. Edit /opt/logstash/Gemfile to include the line:gem "logstash-output-cassandra", "0.1.1" Installenv GEM_HOME=/opt/logstash/vendor/bundle/jruby/1.9 /opt/logstash/vendor/jruby/bin/jruby /opt/logstash/vendor/jruby/bin/gem install logstash-output-cassandra -v 0.1.1 Try it |
|
env GEM_HOME=/opt/logstash/vendor/bundle/jruby/1.9 /opt/logstash/vendor/jruby/bin/jruby /opt/logstash/vendor/jruby/bin/gem install logstash-output-cassandra -v 0.1.1 but the last step still failed: |
|
Hmmm, It seems that the file is missing. Is the cassandra.rb file in there ? |
|
yes, it's there : |
|
Do a env GEM_HOME=/opt/logstash/vendor/bundle/jruby/1.9 /opt/logstash/vendor/jruby/bin/jruby /opt/logstash/vendor/jruby/bin/gem list Do you see the cassandra output in there ? |
|
I think it's there: *** LOCAL GEMS *** addressable (2.3.8) |
|
Seems to be in there. I think it's a path issue.... Add -vvvv and see if anything is fishy... |
|
sorry , I don't get it , could your provide command line ? Thanks , or any chance for a teamviewer ? |
|
Start logstash using /etc/init.d/logstash start and get/grep the full command line used to start it. Use that full command and try to start it with it to see the actual PATH used to start it.... Can't help you with TV. |
|
not quite understand what you told me, is the following ok? |
|
Run as root. /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -Xmx1g -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log --debug |
|
Also change /etc/logstash/conf.d to a different path if you have the config file in some other place. |
|
the error message: Error: No config files found: /etc/logstash/conf.d/* Could you tell me the config file name ? |
|
@valentinul , there's no file beneath /etc/logstash/conf.d , Could you tell me the name of config file? |
|
@vkjuju the file has to be called logstash.conf |
|
it's weird I don't have logstash.conf in my ubuntu ? |
|
Depends how you installed logstash.
In general it should be there as that is the default location ...
…On Thu, Oct 5, 2017 at 9:36 AM, vkjuju ***@***.***> wrote:
it's weird I don't have logstash.conf in my ubuntu ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlsVtm3bpsm2yyvz_jzlHv4s3peNGFHks5spIbugaJpZM4Pqa7F>
.
|
|
there's a file: etc/default/logstash is it this one ? there's no logstash.conf , I searched it from / |
|
If there is none, just make that folder in etc or wherever and create an
logstash.conf file and put in it the input/filter/output section you need...
…On Thu, Oct 5, 2017 at 9:49 AM, vkjuju ***@***.***> wrote:
there's a file:
vi /var/lib/dpkg/info/logstash.conffiles
etc/default/logstash
/etc/init.d/logstash
/etc/logrotate.d/logstash
is it this one ? there's no logstash.conf , I searched it from /
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlsVmMjnsC3HAEnxY7xvOMfX2941hy1ks5spIoTgaJpZM4Pqa7F>
.
|
|
I created an empty logstash.conf on /etc/logstash/conf.d and run debug: |
|
Yea...it expects some input/filter/output section...
…On Thu, Oct 5, 2017 at 10:06 AM, vkjuju ***@***.***> wrote:
I created an empty logstash.conf on /etc/logstash/conf.d and run debug:
Error: Expected one of #, input, filter, output at line 2, column 1 (byte
2) after
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlsVu38gvt-_swn7Nkyu2a37ggjGMr1ks5spI3ogaJpZM4Pqa7F>
.
|
|
sorry , I don't get it on this "put in it the input/filter/output section you need" |
|
Means you have to write the config you want/need into that file.
example:
output { cassandra {} }
…On Thu, Oct 5, 2017 at 10:17 AM, vkjuju ***@***.***> wrote:
sorry , I don't get it on this "put in it the input/filter/output section
you need"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlsVkK9_nzi6V_6mlaivibtFnbB-AUXks5spJCSgaJpZM4Pqa7F>
.
|
|
sorry, I got stuck on logstash.conf, I created a empty one underneath /etc/logstash/conf.d and I don't know how to do next ... |
|
rm -rf /etc/logstash/conf.d/logstash.conf; echo "output { cassandra {} }" >
/etc/logstash/conf.d/logstash.conf;
then start logstash with the command like I said earlier..
…On Thu, Oct 5, 2017 at 10:45 AM, vkjuju ***@***.***> wrote:
sorry, I got stuck on logstash.conf, I created a empty one underneath
/etc/logstash/conf.d and I don't know how to do next ...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlsVqQG3MutbSzj332pxhBjr6GXv9Qxks5spJclgaJpZM4Pqa7F>
.
|
|
@valentinul , any more advice would be appreciated ^^" |
|
Take a look more at gem. The issue is that there are multiple gem/jruby installations on the system and logstash has it's own. So try to find which is the correct path to install the cassandra output. You have to use that gem install in the proper jruby path. So in conclusion, find all the "gem" binary location and do gem install logstash-output-cassandra -v 0.1.1 until it gets registered into the logstash system/path. |
|
Sorry, just came back from national holidays: root@199mysqlmove:/opt/logstash/vendor/jruby/bin# ./gem build logstash-output-cassandra.gemspec @valentinul , any advice would be appreciated... |
|
Hi,
You have to export the jruby path..search in this issue I already pasted
it.
…On Wed, Oct 11, 2017 at 4:27 AM, vkjuju ***@***.***> wrote:
***@***.***:/# find . -name gem
./home/mysqlmove/logstash-5.5.1/vendor/jruby/bin/gem
./opt/logstash/vendor/jruby/bin/gem
./usr/bin/gem
***@***.***:/opt/logstash/vendor/jruby/bin# ./gem build
logstash-output-cassandra.gemspec
/usr/bin/env: jruby: no such file or directory
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlsVvRhUp9As1L_bZHHQtlMXJhFfOFMks5srCeUgaJpZM4Pqa7F>
.
|
|
@valentinul , sorry , I don't get it , is it export JRUTY_PATH= ? |
|
env GEM_HOME=/opt/logstash/vendor/jruby |
|
the same error after executing env GEM_HOME=/opt/logstash/vendor/jruby: |
|
@valentinul , is it ok if you have a chance to ssh our ubuntu server? we have been getting stuck on this issue for quite some time... |
|
Hi,
Send me some info at valentin at servergeek . at and I'll see if I can help
you.
…On Wed, Oct 11, 2017 at 11:03 AM, vkjuju ***@***.***> wrote:
@valentinul <https://github.com/valentinul> , is it ok if you have a
chance to ssh our ubuntu server? we have been getting stuck on this issue
for quite a long time....
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlsVvQFEjd-GCdhSsn9VpfDscg9KqVxks5srIR3gaJpZM4Pqa7F>
.
|
|
Ok , I have sent some info to you , btw: my skype: joesonga at hotmail.com, Thanks |
|
@valentinul , there's still some errors as follows: output { output { output { output { output { |
|
Yes. You have to fill/use all the mandatory settings for the cassandra output |
|
sorry , I don't know how to fix the above error , could you write down your solution once you get a chance? |
|
Thanks, Could you write down how to fix it? btw, I saw some errors as follows, how to fix that ?
@valentinul , there's still some errors as follows:
root@199mysqlmove:/opt/logstash/bin# ./logstash -e 'output {cassandra {}}'
plugin is using the 'milestone' method to declare the version of the plugin this method is deprecated in favor of declaring the version inside the gemspec. {:level=>:warn}
Missing a required setting for the cassandra output plugin:output {
cassandra {
hosts => # SETTING MISSING
...
}
} {:level=>:error}
Missing a required setting for the cassandra output plugin:output {
cassandra {
keyspace => # SETTING MISSING
...
}
} {:level=>:error}
Missing a required setting for the cassandra output plugin:output {
cassandra {
table => # SETTING MISSING
...
}
} {:level=>:error}
Missing a required setting for the cassandra output plugin:output {
cassandra {
username => # SETTING MISSING
...
}
} {:level=>:error}
Missing a required setting for the cassandra output plugin:output {
cassandra {
password => # SETTING MISSING
...
}
} {:level=>:error}
Error: Something is wrong with your configuration.
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system.
Valentin Fischer <[email protected]> 於 2017/10/12 (週四) 4:22 PM 寫道﹕
Yes. You have to fill/use all the mandatory settings for the cassandra output—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Nop, you have to dig the problem out. You have to make the actual configuration for the cassandra output. An example is the following. |
|
so we don't have to take care errors from this command ? logstash -e 'output {cassandra {}}'
Valentin Fischer <[email protected]> 於 2017/10/12 (週四) 4:31 PM 寫道﹕
Nop, you have to dig the problem out. You have to make the actual configuration for the cassandra output.An example is the following.output {
cassandra {
# List of Cassandra hostname(s) or IP-address(es)
hosts => [ "cass-01", "cass-02" ]
# The port cassandra is listening to
port => 9042
# The protocol version to use with cassandra
protocol_version => 4
# Cassandra consistency level.
# Options: "any", "one", "two", "three", "quorum", "all", "local_quorum", "each_quorum", "serial", "local_serial", "local_one"
# Default: "one"
consistency => 'any'
# The keyspace to use
keyspace => "a_ks"
# The table to use (event level processing (e.g. %{[key]}) is supported)
table => "%{[@metadata][cassandra_table]}"
# Username
username => "cassandra"
# Password
password => "cassandra"
# An optional hints hash which will be used in case filter_transform or filter_transform_event_key are not in use
# It is used to trigger a forced type casting to the cassandra driver types in
# the form of a hash from column name to type name in the following manner:
hints => {
id => "int"
at => "timestamp"
resellerId => "int"
errno => "int"
duration => "float"
ip => "inet"
}
# The retry policy to use (the default is the default retry policy)
# the hash requires the name of the policy and the params it requires
# The available policy names are:
# * default => retry once if needed / possible
# * downgrading_consistency => retry once with a best guess lowered consistency
# * failthrough => fail immediately (i.e. no retries)
# * backoff => a version of the default retry policy but with configurable backoff retries
# The backoff options are as follows:
# * backoff_type => either * or ** for linear and exponential backoffs respectively
# * backoff_size => the left operand for the backoff type in seconds
# * retry_limit => the maximum amount of retries to allow per query
# example:
# using { "type" => "backoff" "backoff_type" => "**" "backoff_size" => 2 "retry_limit" => 10 } will perform 10 retries with the following wait times: 1, 2, 4, 8, 16, ... 1024
# NOTE: there is an underlying assumption that the insert query is idempotent !!!
# NOTE: when the backoff retry policy is used, it will also be used to handle pure client timeouts and not just ones coming from the coordinator
retry_policy => { "type" => "default" }
# The command execution timeout
request_timeout => 1
# Ignore bad values
ignore_bad_values => false
# In Logstashes >= 2.2 this setting defines the maximum sized bulk request Logstash will make
# You you may want to increase this to be in line with your pipeline's batch size.
# If you specify a number larger than the batch size of your pipeline it will have no effect,
# save for the case where a filter increases the size of an inflight batch by outputting
# events.
#
# In Logstashes <= 2.1 this plugin uses its own internal buffer of events.
# This config option sets that size. In these older logstashes this size may
# have a significant impact on heap usage, whereas in 2.2+ it will never increase it.
# To make efficient bulk API calls, we will buffer a certain number of
# events before flushing that out to Cassandra. This setting
# controls how many events will be buffered before sending a batch
# of events. Increasing the `flush_size` has an effect on Logstash's heap size.
# Remember to also increase the heap size using `LS_HEAP_SIZE` if you are sending big commands
# or have increased the `flush_size` to a higher value.
flush_size => 500
# The amount of time since last flush before a flush is forced.
#
# This setting helps ensure slow event rates don't get stuck in Logstash.
# For example, if your `flush_size` is 100, and you have received 10 events,
# and it has been more than `idle_flush_time` seconds since the last flush,
# Logstash will flush those 10 events automatically.
#
# This helps keep both fast and slow log streams moving along in
# near-real-time.
idle_flush_time => 1
}
}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
It seems that you don't understand the actual issue. You don't have a valid logstash configuration yet. When you run logstash -e, thats equal with running logstash with a configuration taken from from the command line. So, in other words, YOU have to write a valid logstash.conf |
|
Ok, will look into it , Thanks a lot for your patience,
Valentin Fischer <[email protected]> 於 2017/10/12 (週四) 4:43 PM 寫道﹕
It seems that you don't understand the actual issue. You don't have a valid logstash configuration yet.When you run logstash -e, thats equal with running logstash with a configuration taken from from the command line.So, in other words, YOU have to write a valid logstash.conf—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
@valentinul , we used to collect iis log and send output to elasticsearch as follows, we'd like to know if we just replace the output to your example? thanks } filter { if [message] =~ "^#" { grok { date { if [bytesSent] { if [bytesReceived] { mutate { } } |
|
Hi, No..it's not that simple...you have to create the tables in cassandra first and make sure you have the correct structure so it's matching your message structure and so on ... You also have to clean/strip the message from extra stuff that it matches EXACTLY your structure from your cassandra table. Unfortunately you have to work on the config and debug it until you are able to make it work. |
|
yes, we are creating table and columns on cassandra based on grok: |
|
Ok.
Then make the correct structure and make sure the logstash message is
having the exact same structure and you should be ok.
…On Mon, Oct 16, 2017 at 12:17 PM, vkjuju ***@***.***> wrote:
yes, we are creating table and columns on cassandra based on grok:
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp}
%{WORD:serviceName} %{WORD:serverName} %{IP:serverIP} %{WORD:method}
%{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} %{NOTSPACE:username}
%{IPORHOST:clientIP} %{NOTSPACE:protocolVersion} %{NOTSPACE:userAgent}
%{NOTSPACE:cookie} %{NOTSPACE:referer} %{NOTSPACE:requestHost}
%{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:win32response}
%{NUMBER:bytesSent} %{NUMBER:bytesReceived} %{NUMBER:timetaken}"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlsVgkZnTVedheYkSbmwzDi0syXMa6Zks5ssy05gaJpZM4Pqa7F>
.
|
|
@valentinul , we hit some errors as follows, any advice would be appreciated, welcome to connect to our server once you get a chance... |
|
seems like you are trying to set things/options that are not supported by the cassandra output. |
|
@valentinul , we think something wrong with "hints" ? we're not sure how to define those "hints" ? any advice would be appreciated... logstash config file: } } } |
|
Hi,
The hints are the actual fields / table structure for your Cassandra database.
This helps the Cassandra output to match the message to the Cassandra tables.
From: vkjuju <[email protected]>
Reply-To: otokarev/logstash-output-cassandra <[email protected]>
Date: Wednesday, 18. October 2017 at 04:55
To: otokarev/logstash-output-cassandra <[email protected]>
Cc: Valentin Fischer <[email protected]>, Mention <[email protected]>
Subject: Re: [otokarev/logstash-output-cassandra] License issue (#10)
@valentinul , we think something wrong with your "hints" ? we not not sure how to define those "hints" ?
our iis log file looks like this:
2017-09-01 01:32:04 W3SVC1 155sqlsrv01 192.168.112.155 GET / - 80 - 192.168.25.80 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:52.0)+Gecko/20100101+Firefox/52.0 - 304 0 0 262
2017-09-01 01:32:04 W3SVC1 155sqlsrv01 192.168.112.155 GET /iis-85.png - 80 - 192.168.25.80 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:52.0)+Gecko/20100101+Firefox/52.0 http://192.168.112.155/ 304 0 0 135
2017-09-01 01:32:05 W3SVC1 155sqlsrv01 192.168.112.155 GET /favicon.ico - 80 - 192.168.25.80 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:52.0)+Gecko/20100101+Firefox/52.0 - 404 0 2 205
2017-09-01 01:32:05 W3SVC1 155sqlsrv01 192.168.112.155 GET /favicon.ico - 80 - 192.168.25.80 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:52.0)+Gecko/20100101+Firefox/52.0 - 404 0 2 248
2017-09-01 01:32:14 W3SVC1 155sqlsrv01 192.168.112.155 GET / - 80 - 192.168.25.80 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:52.0)+Gecko/20100101+Firefox/52.0 - 304 0 0 32
logstash config file:
input {
file {
type => "iis-w3c"
path => "/home/mysqlmove/logstash-5.5.1/iis_logs/*"
}
}
filter {
if [message] =~ "^#" {
drop {}
}
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:serviceName} %{WORD:serverName} %{IP:serverIP} %{WORD:method} %{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientIP} %{NOTSPACE:protocolVersion} %{NOTSPACE:userAgent} %{NOTSPACE:cookie} %{NOTSPACE:referer} %{NOTSPACE:requestHost} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:win32response} %{NUMBER:bytesSent} %{NUMBER:bytesReceived} %{NUMBER:timetaken}"]
}
date {
match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
timezone => "Etc/UTC"
}
if [bytesSent] {
ruby {
code => "event['kilobytesSent'] = event['bytesSent'].to_i / 1024.0"
}
}
if [bytesReceived] {
ruby {
code => "event['kilobytesReceived'] = event['bytesReceived'].to_i / 1024.0"
}
}
mutate {
convert => ["bytesSent", "integer"]
convert => ["bytesReceived", "integer"]
convert => ["timetaken", "integer"]
add_field => { "clientHostname" => "%{clientIP}" }
remove_field => [ "log_timestamp"]
}
dns {
action => "replace"
reverse => ["clientHostname"]
}
useragent {
source=> "useragent"
prefix=> "browser"
}
}
output {
cassandra {
username => "cassandra"
password => "cassandra"
hosts => ["192.168.112.171"]
keyspace => "mykeyspace"
table => "query_log"
consistency => "all"
source => "payload"
hints => {
id => "int"
at => "timestamp"
resellerId => "int"
errno => "int"
duration => "float"
ip => "inet"}
ignore_bad_messages => true
ignore_bad_values => true
batch_size => 100
batch_processor_thread_period => 1
max_retries => 3
retry_delay => 3
}
}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
@valentinul , logstash -f ../iis.conf works, but when we add some iis logs, we hit the following error message , any advice would be appreciated...root@199mysqlmove:/opt/logstash/bin# ./logstash -f ../iis.conf
|
|
Hi,
You have stuff in your iis logs that is not mapped in your cassandra
structure. In other words your cassandra structure is not matching the
message which is coming from logstash.
You have to make sure that your logstash message being sent to cassandra is
always 1:1 with your table structure.
…On Tue, Oct 24, 2017 at 7:59 AM, vkjuju ***@***.***> wrote:
@valentinul <https://github.com/valentinul> , logstash -f ../iis.conf
works, but when we add some iis logs, we hit the following error message ,
any advice would be appreciated... ***@***.***:/opt/logstash/bin#
./logstash -f ../iis.conf
***@***.***:/opt/logstash/bin# ./logstash -f ../iis.conf
plugin is using the 'milestone' method to declare the version of the
plugin this method is deprecated in favor of declaring the version inside
the gemspec. {:level=>:warn}
Settings: Default pipeline workers: 2
Logstash startup completed
Failed to prepare query {:action=>{"table"=>"query_log",
"data"=>{"message"=>"2017-09-05 09:45:31 W3SVC1 155sqlsrv01
192.168.112.155 GET / - 80 - 192.168.25.80 curl/7.55.1 - 200 0 0 59\r",
"path"=>"/opt/logstash/iis_logs/u_ex170905xxx01.log",
"host"=>"199mysqlmove", "type"=>" iis-w3c", "tags"=>["_grokparsefailure"],
"clientHostname"=>"%{clientIP}"}}, :exception=>#<Cassandra::Errors::InvalidError:
Undefined colum n name message>, :backtrace=>[], :level=>:error}
Failed to prepare query {:action=>{"table"=>"query_log",
"data"=>{"message"=>"2017-09-05 09:46:30 W3SVC1 155sqlsrv01
192.168.112.155 GET / - 80 - 192.168.25.80 curl/7.55.1 - 200 0 0 5\r",
"path"=>"/opt/logstash/iis_logs/u_ex170905xxx01.log",
"host"=>"199mysqlmove", "type"=>"i is-w3c", "tags"=>["_grokparsefailure"],
"clientHostname"=>"%{clientIP}"}}, :exception=>#<Cassandra::Errors::InvalidError:
Undefined column name message>, :backtrace=>[], :level=>:error}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlsVsNbL524oLzs13IoVAPOkhdcQfxSks5svXzYgaJpZM4Pqa7F>
.
|
|
Thanks @valentinul , logstash.conf : we are not sure how to create table based on the above iis log, logstash.conf and your output(hints), any chance to connect to our server to take a look? it's the last one mile from the begining ... Thanks |
|
Hi,
Unfortunately not. You have to make the cassandra table and make the
logstash config. I cannot create them for you.
You have to invest time and effort in order the make them match.
1. Strip/delete fields using logstash when the iss message comes.
2. Make sure the cassandra table is created correctly and you can insert
into it.
…On Thu, Oct 26, 2017 at 3:47 PM, vkjuju ***@***.***> wrote:
Thanks @valentinul <https://github.com/valentinul> ,
we got stuck at how to match them 1:1, could you verify it for us?
iis log:
2017-09-07 09:44:02 W3SVC1 155sqlsrv01 192.168.112.155 GET /iis-85.png -
80 - 192.168.20.32 Mozilla/5.0+(Windows+NT+10.0;+
Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/60.0.3112.113+Safari/537.36
http://192.168.112.155/ 200 0 0 599
logstash.conf :
grok {
## Very helpful site for building these statements:
# http://grokdebug.herokuapp.com/
#
# This is configured to parse out every field of IIS's W3C format when
# every field is included in the logs
#
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp}
%{WORD:serviceName} %{WORD:serverName} %{IP:serverIP} %{WORD:method}
%{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} %{NOTSPACE:username}
%{IPORHOST:clientIP} %{NOTSPACE:protocolVersion} %{NOTSPACE:userAgent}
%{NOTSPACE:cookie} %{NOTSPACE:referer} %{NOTSPACE:requestHost}
%{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:win32response}
%{NUMBER:bytesSent} %{NUMBER:bytesReceived} %{NUMBER:timetaken}"]
we are not sure how to create table based on the above iis log,
logstash.conf and your output(hints), any change to connect to our server
to take a look, it's the last mile from the begining ...
Thanks
Joe
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlsVtgxoG9_ZEE9IXiwd99z3JfrvQw9ks5swI1vgaJpZM4Pqa7F>
.
|
Hi, When we ran gem build logstash-output-cassandra.gemspec, there's a license issue as follows, any advice would be appreciated.
root@199mysqlmove:/home/mysqlmove/download/logstash-output-cassandra-master# gem build logstash-output-cassandra.gemspec
fatal: Not a git repository (or any of the parent directories): .git
WARNING: WARNING: license value 'Apache License (2.0)' is invalid. Use a license identifier from
http://spdx.org/licenses or 'Nonstandard' for a nonstandard license.
WARNING: open-ended dependency on cassandra-driver (>= 0) is not recommended
if cassandra-driver is semantically versioned, use:
add_runtime_dependency 'cassandra-driver', '
> 0'> 0'WARNING: open-ended dependency on logstash-devutils (>= 0, development) is not recommended
if logstash-devutils is semantically versioned, use:
add_development_dependency 'logstash-devutils', '
WARNING: See http://guides.rubygems.org/specification-reference/ for help
Successfully built RubyGem
Name: logstash-output-cassandra
Version: 0.1.1
File: logstash-output-cassandra-0.1.1.gem
The text was updated successfully, but these errors were encountered: