Fix R900 Decoder and New Release Tool

Fix R900 Decoder and New Release Tool

This minor release is a fix for the R900 decoder.

I've also moved to a new release tool that greatly simplifies making binary releases for new versions. As a result, I'm now only providing a signature for the checksums file generated by the tool, which should be sufficient to verify that I built the binaries uploaded for each release.

Changes:

• 4805445 Fix version flags for use with goreleaser.
• cec57c9 Store decoder packet config in r900 parser. Fixes #102

Concurrent Multi-Protocol Decoding!

This release contains primarily code and documentation cleanup. There are a few optimizations and some functionality has been both added and removed.

There has also been an unusual amount of time/work between releases, so it's time to provide new pre-compiled binaries. All of the binary releases were compiled using go version go1.11 linux/amd64.

Big Changes:

• Removed -logfile, see 3fef211 for console redirection equivalent functionality.
• Removed -decimation due to performance improvements, no longer necessary.
• Added new -msgtype value r900bcd for those funky R900 meters that transmit binary-coded digits for consumption values.
• Added new -msgtype value netidm for type 8 net meters that track both consumption and production simultaneously.
• For ease of use with docker and systemd, flags can now be set with environment variables, see 9e258fd.
• Add netidm for type 8 power meters. Same preamble and CRC as standard IDM, different packet structure includes aggregate consumption and generation.
• Add preliminary multi-protocol decoding support.

All Changes:

• 6dcad79 Fix travis-ci configuration.
• fb5c246 Update travis-ci for new go versions.
• 44e1520 Remove generator test.
• 0d41d2f Simplify input buffering.
• 7f30323 Move decode and parse to protocol.
• 6bfcafd Add multi-protocol decoding support.
• d4b9855 Correct filter benchmark. Remove symbol length test.
• 4be127c Rename wiki page Command Line Flags to Configuration.
• 7d7707c Move usage from README.md to wiki. Expand message type section.
• 1d3a52b Merge branch 'netidm'.
• 301eb82 Switch intervals back to unsigned integers.
• 8f43170 Move content of help.md to wiki.
• c6540fe Add multiple of 8 requirement to symbollength.
• a0452b4 Move quantization into filter.
• 74bd955 Change feedback.
• b2f2254 Delete docs folder for conversion from github pages to wiki.
• 596595b Add scm_polynomial.svg
• 4540f7d Add images and equations for migration to wiki.
• cbd4df4 Round packed signal length up to nearest byte.
• 19cf823 Replace transposition and byteFinder preamble search.
• dfc901f Add LastGeneration field.
• bc732cb Assume intervals are signed 14-bit integers.
• 42bfbac Invert quantize output to match previous behavior.
• f56d86e Apply a few new optimizations.
• 26da5a3 Remove decimation from tests and DecCfg from Decoder.
• bac8031 Remove decimation functionality.
• 50292b1 Add link to rtlamr-collect.
• 26f7dba Add rtltcp as git subtree.
• 48b7cad Remove rtltcp submodule.
• ce8f749 Change FROM to allow arm compatibility.
• 83208e4 Fix table in help.md
• 19e3236 Change format description, update help.md Fixes #81
• 31f4d92 Update Dockerfile and vendor bemasher/rtltcp.
• 6dbef3d Add description for Transmit Time Offset.
• b84344e Don't use channel as buffer, rtl_tcp handles that sufficiently.
• f8b5ccb Update readme and travis-ci with new version requirements.
• f25fa42 Fix race condition introduced in 8a0e86e
• 152b416 Update Dockerfile (#79)
• 8a0e86e Handle blocks asynchronously.
• 5910c8f Send "Time Limit Reached" to stderr. Fixes #76
• bf6b099 Don't fail hard on invalid environment variable values.
• 914fe8d Fix front-matter.
• 3fef211 Remove -logfile flag.
• b2dba49 Revert change to Quantize invocation.
• 57926fc Refactor symbolsPerBlock, should have no effect though. Add decoder benchmarks.
• 1da5fcd Minor optimizations in Transpose and Quantize.
• 9e258fd Allow overriding flag defaults with environment variables.
• a0ec0e8 Revert change to invocation of Quantize.
• 9f1f0de Remove reference to alpha-max beta-min magnitude calculation.
• fcc41fd Remove remnants of MathJax and duplicate katex loads.
• 89d2a9a Wrapping with divs works, but need unescaped $$'s for KaTeX.
• 4e31f28 Yet another attempt at getting kramdown to behave with latex.
• 0788ed9 Output html entities as is.
• a23d0b5 Kramdown, again. Using nomarkdown tags.
• 8a9045a Convert <p>'s to <span>'s...
• 830cd37 Wrap KaTeX blocks in <p>, kramdown is still mangling these.
• 3e0b042 Escape $$'s, kramdown mangles these with their own latex handling.
• 2ace562 Add KaTeX fonts.
• ce569ed Add auto-render source file.
• a318039 Fix typo in KaTeX's auto-render invocation.
• 8040242 Replace pygments with rouge for code highlighting.
• 823ed3a Switch to KaTeX for equation typesetting.
• 911713f Move github pages content into /docs
• afde396 Add Decoder benchmark.
• e7e0e2d Rename SymbolLength to ChipLength and SymbolLength2 to SymbolLength.
• 2ae5485 Minor fix and optimizations.
• 49906ad Update comment about break point in SCM+ decoder.
• 8e98509 Add ProtocolID check to SCM+ decoder.
• 7de771c Correct random frequency selection for SCM generation test.

v0.8.0 - Performance Improvements and R900 BCD

Cut away a lot of the fat in various buffers and implemented a protocol wrapper for R900 meters that transmit consumption as a hexadecimal coded digits. This also marks the first version which performs well enough to receive at full sample rate on an RPi2 without dropping any samples.

Changes:

• 2b386b0 Add R900 BCD Protocol (#58)
• 17b0016 Remove IQ buffer from decoder, implement buffer for sample dumping in main program.
• cedc2f8 Rename SCM test and update meters.csv
• 082d452 Maintain internal signal buffer since we trimmed the decoder-specific one.
• b567710 Resize Filter and csum buffers. Rename Pack to Transpose.
• 1c5bbb4 Simulation (#51)
• 2bee3a8 Rename recv.go to main.go
• 1742cc5 Fix length of bits to pack.
• 508c42c Remove old calculation from MagLUT.
• 1378c68 Remove Sqrt in MagLUT, sensitivity is unaffected. Use appropriate scaling in LUT calculation.
• faba109 Trim Signal and Filtered buffers to only necessary lengths.

v0.7.1 - Decimation Fix

Decimation was broken in the previous release when the new preamble search was implemented. This release fixes that bug.

Changes:

• 31bbf3f Use decimated packet configuration in preamble search. Fixes #47.

v0.7

This release includes a lot of changes most important of which is SCM+ support!

Changes:

• e8ea9d9 Remove cpuprofile flag, add version flag.
• dad7ae0 Check lowercase version of msgType against registered parsers.
• 99be3e7 Parsers register themselves like in database/sql.
• 206c0dc Refactor packet config to allow modification. Closes #43
• 64a0ded Merge pull request #42 from bemasher/SCM+
• a6286ba Add prototype SCM+ parser.
• 99f5b70 Merge pull request #41 from bemasher/search
  cb511be Use the go standard library implementation of Boyer-Moore string search from the strings * package for preamble searching.
• 40470d4 Wait for a message from each meter given in -filterid when combined with -single. Closes #38
• 1e03a51 Clarify Pack description.
• c45fdef Remove old travis-ci badge.
• 1364a85 Remove flag debug print.
• 5555d22 Fix travis-ci badge.
• d453c86 Remove filter chain debug print.
• 42b1b3d Merge branch 'msghandler'
• 6b376a3 Write plain text encoder.
• f8797da Update travis-ci config to use new infrastructure.
• 938607e Oops, Filter is the method, MessageFilter is the interface.
• 7314e7a Simplify filter flag handling. Rename Filter interface.
• 548dac1 Refactor message filtering.
• 767fea7 Remove -fastmag from usage.
• 42ceb2c Remove FastMag
• 30ee13e Remove unnecessary benchmark.
• de00b1d Update usage and unique flag description.
• c4340c3 Fix symbol index for checksum copy.
• 34b9eb6 Refactor to check against checksum for uniqueness.
• dbdbcae minor fixes
• b9240f9 add -unique

v0.6.2

This release is sort of half-way between major and minor:

• Decimation is now available through the -decimation flag, reduces cpu resource cost at the expense of sensitivity, factors between 2 and 8 still seem to provide reasonably good sensitivity at full bandwidth. Will require simulation to determine effective sensitivity (coming soon).
• Some minor refactoring in SCM and IDM packet handlers.
• Whole blocks of samples are now read and buffered from rtl_tcp.

Changes:

• ae0364d Merge branch 'decimation'
• 07607af Refactor message unpacking in SCM and IDM.
• 7f05214 Add decimation factor checking. Clarify warning.
• 8f3cb52 Apply gofmt to everything.
• 04d73e4 Update logging to include decimated configuration.
• de4bffe Fix csum length.
• 953f371 Move R900 message statement.
• 22906a5 Update link in README to point to gh-pages documentation.
• 3bd0a4e Update flags in README.
• efea189 Implement decimation in AlphaMaxBetaMinLUT.
• 3a29e9b Implement decimation in R900 decoder.
• 586359c Move decimation to demodulator, only MagLUT for now.
• 1373ceb Fix block read, reads might span writes to the pipe.
• 588ef52 Read whole blocks from server concurrently with io.Pipe().
• f782879 Remove config debugging statements.
• c2855e7 Implement decimation, except in R900 decoder.
• f3d51f2 Update compatible meter table location.
• 3249c73 Move compatible meter table to csv format.

v0.6.1

This is a minor release:

• The r900 decoder now handles a few more of the unknown fields.
• Documentation added for a few new supported meters submitted by users.
• Optimizations in the magnitude execute loop and the preamble search.

Changes:

• e1d9b04 Merge pull request #27 from sandeen/master
• 2336082 r900: decode leak, fraud, and backflow status flags
• 9be534c Add use cases.
• d4afba8 Add 100GDLAN, update commodity type list.
• 15e1d1d Optimize execute loop for magnitude demodulators.
• 169c530 Switch to bytes.Equal in preamble search. Simplify calculations for slice allocation.
• 0f08d69 Add user-submitted meters.

v0.6

Now with experimental Neptune R900 support!

Changes:

• eb2b055 Fix galois field package path.
• a88bbf1 Simplify symbol decoding process.
• 57e1be9 Add gf package license.
• 6f3d973 Integrate R900 decoder.
• 8a57030 Add R900 decoder.
• 19dadbb Move CenterFreq to PacketConfig for per protocol defaults.
• 8865d2e Update travis-ci configuration.
• 979aa93 Fix rounding problem in parser.
• 0c47867 Move s to correct scope.
• c9c1d99 Simplify duplicate packet check.
• 3c66161 Separate demodulated signal from filtered signal.
• dbb4227 Set actual field lengths.
• 86a2abf Refactor for signal access from parser.
• 11d5802 Rename for later protocols.
• 03369d1 Simplify packed byte array length calculation.
• 3bcb52b Fix packed byte array length calculation.
• b326ea9 Limit checksum to valid length.
• 2d12445 Update usage.
• 76ee4cb Update badges.
• ea81369 Add new error case for SCM parsing.
• 2b055fc Convert filterid and filtertype flags to comma-separated lists.
• ed47c5c Fix NewDataFromBits
• 9994aaa Add calculator for valid symbol lengths.

v0.5.1

This is a minor release mostly dealing with documentation and some refactoring. The refactoring will make future work on FHSS tools less painful.

This is also the first release to include detached gpg signatures with each archive:

# Extract a binary archive.
tar -xzvf rtlamr_linux_amd64.tar.gz

# Retrieve public key used to sign the binary.
gpg --keyserver pgp.mit.edu --recv-keys DD1EC7EE

# Verify the signature.
gpg --verify rtlamr.asc rtlamr

See my keybase profile for proof of ownership: https://keybase.io/bemasher

Changes:

• f955a45 Fix magnitude benchmarks.
• 5f508e5 Refactor parsing into packages for reuse.
• 0092dd1 Fix FastMag.
• 659a8b3 Fix for package refactor.
• b650170 Refactor decoding to package for reuse.
• d0f46b3 Add comments on decoding methods.
• df217be Add format message structures.
• bdd008f Update sensitivity.
• 16e4b4a Added description for each flag.
• 69d955c Create help.md
• 1f4e9dd Update purpose for clarity.

v0.5.0

First major release without any external dependencies! FFTW is no longer required for builds which allows cross-compilation.

Changes:

• README.md updated to reflect new build process along with other information new to the project.
• Flags which override values determined by rtlamr are now handled properly (gain, center frequency, sample rate).
• Matched filter optimized, ~2x performance increase.

Future:

• Changes to log format in plain-text may change for IDM packets, they're still huge.
• FHSS tools may need to wait. Testing shows rtl-sdr dongles don't have good enough rejection at the moment to reasonably determine that a decoded message occured in the band we expected them to due to aliasing and the like. These will take a lot more effort in signal processing to accurately break the FHSS hopping pattern.
• Writing raw samples to file still need to be non-blocking.