Skip to content

Commit

Permalink
Make the received endpoint the default for logshare-cli (#10)
Browse files Browse the repository at this point in the history
  • Loading branch information
@elithrar
elithrar authored Nov 20, 2017
1 parent a0fddcc commit 8c69caf
Show file tree
Hide file tree
Showing 2 changed files with 58 additions and 42 deletions.
81 changes: 45 additions & 36 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,14 +63,17 @@ GLOBAL OPTIONS:
--zone-id value The zone ID of the zone you are requesting logs for
--zone-name value The name of the zone you are requesting logs for. logshare will automatically fetch the ID of this zone from the Cloudflare API
--ray-id value The ray ID to request logs from (instead of a timestamp)
--start-time value The timestamp (in Unix seconds) to request logs from. Defaults to 30 minutes behind the current time (default: 1504137645)
--end-time value The timestamp (in Unix seconds) to request logs to. Defaults to 20 minutes behind the current time (default: 1504138245)
--start-time value The timestamp (in Unix seconds) to request logs from. Defaults to 30 minutes behind the current time (default: 1511219860)
--end-time value The timestamp (in Unix seconds) to request logs to. Defaults to 20 minutes behind the current time (default: 1511220460)
--count value The number (count) of logs to retrieve. Pass '-1' to retrieve all logs for the given time period (default: 1)
--by-received Retrieve logs by the processing time on Cloudflare. This mode allows you to fetch all available logs vs. based on the log timestamps themselves.
--by-received (default behaviour) Retrieve logs by the processing time on Cloudflare. This mode allows you to fetch all available logs vs. based on the log timestamps themselves.
--legacy-endpoint (deprecated) Retrieve logs using the 'legacy' endpoint, where results are returned by log timestamp.
--fields value Select specific fields to retrieve in the log response. Pass a comma-separated list to fields to specify multiple fields.
--list-fields List the available log fields for use with the --fields flag
--google-storage-bucket value Full URI to a Google Cloud Storage Bucket to upload logs to
--google-project-id value Project ID of the Google Cloud Storage Bucket to upload logs to
--help, -h show help
--version, -v print the version
```

Typically you will need the zone ID from the Cloudflare API to retrieve logs from the ELS REST API.
Expand All @@ -85,10 +88,10 @@ Although `logshare-cli` can be used in multiple ways, and for ingesting logs int
common use-case is ad-hoc analysis of logs when troubleshooting or analyzing traffic. Here are a few examples that
leverage [`jq`](https://stedolan.github.io/jq/) to parse log output.

#### Distribution of Origin Response Status Codes
#### Distribution of Edge (client-facing) Response Status Codes

```
$ logshare-cli --api-key=<snip> --api-email=<snip> --zone-name=example.com --start-time=1453307871 --count=20000 | jq '.[] | .originResponse.status // empty' | sort -rn | uniq -c | sort -rn
$ logshare-cli --api-key=<snip> --api-email=<snip> --zone-name=example.com --start-time=1453307871 --count=20000 | jq '.[] | .EdgeResponseStatus empty' | sort -rn | uniq -c | sort -rn
```
```
35954 200
Expand All @@ -106,42 +109,48 @@ $ logshare-cli --api-key=<snip> --api-email=<snip> --zone-name=example.com --sta
1 405
```

#### Top 10 Visitor Countries
#### List Available Log Fields

```
logshare-cli --api-key=<snip> --api-email=<snip> --zone-name=example.com --start-time=1453307871
--count=20000 | jq '. | .client.country' | uniq -c | sort -rn | head -n 10
$ logshare-cli --api-key=<snip> --api-email=<snip> --zone-name=example.com --list-fields | jq
```
```
39384 "us"
1276 "de"
933 "ie"
743 "gb"
597 "ca"
587 "in"
528 "id"
476 "au"
464 "jp"
437 "fr"
```

#### Client IPs and endpoints that were impacted by rate-limiting rules

```
logshare-cli --zone-name example.com --api-key $CF_API_KEY --api-email $CF_API_EMAIL --start-time 1453307871 --count 20000| jq 'select(.edge.rateLimit.processedRules | length > 0)| {ts: .timestamp, ruleID: .edge.rateLimit.processedRules[], url: .clientRequest.uri, srcIP: .client.ip}'
```

```
{
"ts": 1503002722080000000,
"ruleID": {
"ruleId": 96612,
"ruleSrc": "user",
"status": "allow",
"ruleType": "ban"
},
"url": "/download",
"srcIP": "45.56.123.254"
"CacheCacheStatus": "unknown | miss | expired | updating | stale | hit | ignored | bypass | revalidated",
"CacheResponseBytes": "Number of bytes returned by the cache",
"CacheResponseStatus": "HTTP status code returned by the cache to the edge: all requests (including non-cacheable ones) go through the cache: also see CacheStatus field",
"ClientASN": "Client AS number",
"ClientCountry": "Country of the client IP address",
"ClientDeviceType": "Client device type",
"ClientIP": "IP address of the client",
"ClientIPClass": "Client IP class",
"ClientRequestBytes": "Number of bytes in the client request",
"ClientRequestHost": "Host requested by the client",
"ClientRequestMethod": "HTTP method of client request",
"ClientRequestProtocol": "HTTP protocol of client request",
"ClientRequestReferer": "HTTP request referrer",
"ClientRequestURI": "URI requested by the client",
"ClientRequestUserAgent": "User agent reported by the client",
"ClientSSLCipher": "Client SSL cipher",
"ClientSSLProtocol": "Client SSL protocol",
"ClientSrcPort": "Client source port",
"EdgeColoID": "Cloudflare edge colo id",
"EdgeEndTimestamp": "Unix nanosecond timestamp the edge finished sending response to the client",
"EdgePathingStatus": "Edge pathing status",
"EdgeResponseBytes": "Number of bytes returned by the edge to the client",
"EdgeResponseCompressionRatio": "Edge response compression ratio",
"EdgeResponseStatus": "HTTP status code returned by Cloudflare to the client",
"EdgeStartTimestamp": "Unix nanosecond timestamp the edge received request from the client",
"OriginIP": "IP of the origin server",
"OriginResponseBytes": "Number of bytes returned by the origin server",
"OriginResponseHTTPExpires": "Value of the origin 'expires' header in RFC1123 format",
"OriginResponseHTTPLastModified": "Value of the origin 'last-modified' header in RFC1123 format",
"OriginResponseStatus": "Status returned by the origin server",
"OriginResponseTime": "Number of nanoseconds it took the origin to return the response to edge",
"RayID": "Ray ID of the request",
"WAFAction": "Action taken by the WAF, if triggered",
"WAFRuleID": "ID of the applied WAF rule",
"ZoneID": "Internal zone ID"
}
```

Expand Down
19 changes: 13 additions & 6 deletions cmd/logshare-cli/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,8 @@ func run(conf *config) func(c *cli.Context) error {
conf.apiKey,
conf.apiEmail,
&logshare.Options{
ByReceived: conf.byReceived,
// Pass the inverse of the legacy flag to invoke the old behaviour.
ByReceived: !conf.legacy,
Fields: conf.fields,
Dest: outputWriter,
})
Expand Down Expand Up @@ -143,6 +144,7 @@ func parseFlags(conf *config, c *cli.Context) error {
conf.endTime = c.Int64("end-time")
conf.count = c.Int("count")
conf.byReceived = c.Bool("by-received")
conf.legacy = c.Bool("legacy-endpoint")
conf.fields = c.StringSlice("fields")
conf.listFields = c.Bool("list-fields")
conf.googleStorageBucket = c.String("google-storage-bucket")
Expand All @@ -161,6 +163,7 @@ type config struct {
endTime int64
count int
byReceived bool
legacy bool
fields []string
listFields bool
googleStorageBucket string
Expand All @@ -177,14 +180,14 @@ func (conf *config) Validate() error {
return errors.New("zone-name OR zone-id must be set")
}

if conf.legacy && conf.byReceived {
return errors.New("you must specify either --legacy-endpoint or --by-received (the default), not both. The default mode is --by-received")
}

if len(conf.fields) > 0 && !conf.byReceived {
return errors.New("specifying --fields is only supported when using the --by-received endpoint")
}

// if conf.count -1 || conf.count > 0 {
// return errors.New("count must be > 0, or set to -1 (no limit)")
// }

if (conf.googleStorageBucket == "") != (conf.googleProjectId == "") {
return errors.New("Both google-storage-bucket and google-project-id must be provided to upload to Google Storage")
}
Expand Down Expand Up @@ -230,7 +233,11 @@ var flags = []cli.Flag{
},
cli.BoolFlag{
Name: "by-received",
Usage: "Retrieve logs by the processing time on Cloudflare. This mode allows you to fetch all available logs vs. based on the log timestamps themselves.",
Usage: "(default behaviour) Retrieve logs by the processing time on Cloudflare. This mode allows you to fetch all available logs vs. based on the log timestamps themselves.",
},
cli.BoolFlag{
Name: "legacy-endpoint",
Usage: "(deprecated) Retrieve logs using the 'legacy' endpoint, where results are returned by log timestamp.",
},
cli.StringSliceFlag{
Name: "fields",
Expand Down

0 comments on commit 8c69caf

Please sign in to comment.