Update for temporary parameter support (#547)

See rapid7/recog-ruby#12

Gemfile

@@ -2,7 +2,7 @@ source 'https://rubygems.org'
 
 gemspec name: 'recog-content'
 
-gem 'recog', '~>3.0'
+gem 'recog', '~>3.1'
 
 group :test do
   gem 'rake'

README.md

@@ -127,8 +127,6 @@ At least one `example` element should be present, however multiple `example` ele
 
 tests that `RomSShell_4.62` matches the provided regular expression and that the value of `service.version` is 4.62.
 
-The `param` elements contain a `pos` attribute, which indicates what capture field from the `pattern` should be extracted, or `0` for a static string. The `name` attribute is the key that will be reported in the case of a successful match and the `value` will either be a static string for `pos` values of `0` or missing and taken from the captured field.
-
 The `example` string can be base64 encoded to permit the use of unprintable characters.  To signal this to Recog an `_encoding` attribute with the value of `base64` is added to the `example` element.  Based64 encoded text that is longer than 80 characters may be wrapped with newlines as shown below to aid in readability.
 
 ```xml
@@ -155,6 +153,51 @@ They can then be loaded using the `_filename` attribute:
 
 This is useful for long examples.
 
+The `param` elements contain a `pos` attribute, which indicates what capture field
+from the `pattern` should be extracted, or `0` for a static string. The `name` attribute
+is the key that will be reported in the case of a successful match and the `value`
+will either be a static string for `pos` values of `0` or missing and taken from the
+captured field.
+
+The `value` attribute supports interpolation of data from other fields. This is
+often useful when capturing the value for `hw.product` via regex and re-using this
+value in `os.product`.
+
+Here is an example from`http_servers.xml` where `hw.product` is captured and reused.
+
+```xml
+  <fingerprint pattern="^Eltex (TAU-\d+[A-Z]*(?:\.IP)?)$">
+    <description>Eltex TAU model VoIP gateway</description>
+    <example hw.product="TAU-72">Eltex TAU-72</example>
+    <example hw.product="TAU-1.IP">Eltex TAU-1.IP</example>
+    <param pos="0" name="os.vendor" value="Eltex"/>
+    <param pos="0" name="os.product" value="{hw.product} Firmware"/>
+    <param pos="0" name="os.device" value="VoIP Gateway"/>
+    <param pos="0" name="hw.vendor" value="Eltex"/>
+    <param pos="1" name="hw.product"/>
+    <param pos="0" name="hw.device" value="VoIP Gateway"/>
+  </fingerprint>
+```
+
+There is special handling for temporary attributes that have a name starting with
+`_tmp.`. These attributes can be used for interpolation but are not emitted in the
+output. This is useful when a particular product name is inconsistent in various
+banners, vendor marketing, or with NIST values when trying to generate CPEs. In
+these cases the useful parts of the banner can be extracted and a new value
+crafted without cluttering the data emitted by a match.
+
+```xml
+<fingerprint pattern="^foo baz switchThing-(\d{4})$">
+  <description>NetCorp NX series switches</description>
+  <example hw.product="NX8200">foo baz switchThing-8200</example>
+  <param pos="0" name="hw.vendor" value="NetCorp"/>
+  <param pos="0" name="hw.product" value="NX{_tmp.001}"/>
+  <param pos="2" name="_tmp.001"/>
+</fingerprint>
+```
+
+These temporary attributes are not tracked in the `identifiers/fields.txt`.
+
 [^back to top](#recog-a-recognition-framework)
 
 ## Contributing

bin/recog_standardize

@@ -59,7 +59,7 @@ end
 
 # @param current [Hash] Indentifiers extracted from fingerprints
 # @param original [Hash] Indentifiers loaded from the existing identifiers file
-# param msg [String] Context to include in messaging to user
+# @param msg [String] Context to include in messaging to user
 # @param ident_type [String] Key used to get the identifier file path
 # @param write [Boolean] Indicate if changes should be written to disk
 def handle_changes(current, original, msg, ident_type, write)
@@ -151,6 +151,8 @@ ARGV.each do |arg|
     ndb.fingerprints.each do |f|
       f.params.each do |k, v|
 
+        # Don't track temporary attributes.
+        next if k.start_with?("_tmp.")
         curr_fields[k] = true
 
         param_index, val = v

features/data/successful_tests.xml

@@ -15,4 +15,11 @@
      <param pos="1" name="os.version" />
      <param pos="0" name="os.name" value="Bar" />
    </fingerprint>
+   <fingerprint pattern="^foo sb\-([\d.]+)$">
+     <description>test of temp params</description>
+     <example os.version="Super Beta 1.0">foo sb-1.0</example>
+     <param pos="1" name="_tmp.001" />
+     <param pos="0" name="os.version" value="Super Beta {_tmp.001}" />
+     <param pos="0" name="os.name" value="Bar" />
+   </fingerprint>
 </fingerprints>

features/verify.feature

@@ -12,7 +12,7 @@ Feature: Verify
     When I run `recog_verify successful_tests.xml`
     Then it should pass with exactly:
       """
-      successful_tests.xml: SUMMARY: Test completed with 4 successful, 0 warnings, and 0 failures
+      successful_tests.xml: SUMMARY: Test completed with 5 successful, 0 warnings, and 0 failures
       """
 
   @no-clobber