ipc_playground

This code is used to test IPC protocols communicating messages between 2 processes running on the same pc.

In addition to a interprocess communication protocol, this code also includes a message protocol, random data generator, a data file storing system, and unit tests that can be used to evalute the system.

Table of Contents

1. File-Description
2. Set Up
3. How to Run
4. Solution and Rationale
   • Transfer Protocol Selections and Design Trade-offs
   • Modular Interfaces
   • Other Design Considerations

---

File-Description:

• core.messageprotocol.py

  • Discerns the message protocol from Variables.json file.

• core.generator.py

  • Defines the random data generator using inputs from from Variables.json file.

• core.stenographer.py

  • Defines the data logging methods.
  • Each session is saved with an index file, and increasing numbered files.
  • Amount of data saved per numbered file is set in this submodule

• core.ipcprotocols.py

  • Lays down the common functionality to be used by transfer protocols.

• transfer_protocols

  • Transmission methods for sending and receiving files.
  • Currently supporting ftp and named_pipes.
  • template.py is provided to give core functionality and structure for newly added transfer protocols.
  • Note: all files here inherit from core.ipcprotocols.py, and I've attempted to standardize their usage. Thus, there is some reptition the code within these files. I felt that this was acceptable given the inconsistencies in the transfer protocols main loops -- particularly with respect to the "try" commands.

• test_ipc.py

  • Unit tests for ipc transmission.
  • test_standard: standard use transmission (60 Hz as defined by code challenge guidelines). Also checks data storage and retrieval.
  • test_latency: tracks time it takes to pack, send, receive, unpack, and generate a new timestamp
    • Note: Some transfer protocols implementations may need to use a different measure of latency (e.g. if they don't need to pack the data)
  • test_throughput: send data at maximum speed. Records total messages transmitted.

• Variables.json

  • Defines the message variable names, format, and functions used to generate randomized data.
  • json format was selected for readability, and ease of modification.
  • See ipc.py for formatting guidelines

• MessageRegister.py

  • Provides a space for providing definitions to message variables
  • MessageCommands: indicates transmission commands. Currently only "continue" and "stop" are used
  • MessageSource: indicates sender signature. Currently only TEST_PROCESS is being used.

• main.py

  • Enables parsing of command line options. I felt this was much easier to use to demonstrate functionality.

---

Set Up

Easily done with following command from terminal in base directory:

sudo python setup.py install

How to Run

To quickly demonstrate functionality, code can be run in one of two ways:

• Run standard sender and standard receiver in separate terminal windows (in ipc_playground directory)
  1. python ipc_playground -r receiver -t named_pipes
  2. python ipc_playground -r sender -t named_pipes
  • Note: Always run recv_ftp() first, as the sender will not wait for the receiver to join connection.
• Run standard case unit test file in a terminal window (in ipc_playground directory)
  1. python ipc_playground
  • Note: This uses parallel processing to run both from the same terminal simultaneously. This option does not work in a Windows machine due to a documented issue in parallel processing.

Solution and Rationale

Unit tests on my pc clock named_pipes with the lowest latencies (lat < 1 ms). Furthermore it scales well and transmit at nearly 15000 Hz, much higher than the standard use of 60. Furthermore, it operates on a FIFO buffer, meaning that data should not be lost under normal circumstances.

Transfer Protocol Selections and Design Trade-offs

Named_pipes was selected as the solution due to its reputation for fast and efficient streaming. One downside is its lack of cross-compatability, although there are potential workarounds.

FTP was selected as an alternative method due to my own familiarity using it as well as its strong documentation. Additionally, the FTP module was tested and shown to run on a windows pc. Tried and true, its been around for a long time and I wouldn't expect it to be deprecated any time soon.

While Named_pipes was selected as the solution, both methods appear to be plenty fast enough for the design challenge -- even on my dinosaur of a laptop. Furthermore, throughput was tested to demonstrate one aspect of scalability. Another aspect is data storage, and both methods generate files that are about 58 bytes per data point, or 12Mb/hr at a sample frequency of 60Hz.

Modular Interfaces

File organization was broken down to demonstrate modular interfacing. Each file, save for the test file, primarily only describes one functionality. Test file was left as the sum of multiple tests in order to improve the printed output.

The Variables.json file was used to enable easy, and readable modifications to the message protocol.

Multiple transfer protocols were used to demonstrate modularity in the transfer protocols also. Additional protocols can be easily implemented after development.

Other Design Considerations

Sender process identity has been implemented. The current MessageProtocol specifies a mssgsource that is associated with the MessageRegister. Additional processes need only register themselves as a unique enumeration in the MessageRegister file.

A quick querying of the data occurs at the end of the unit test for standard use. This is done to ensure that the timestamps are incrementing, as the rest of the data in my messages is randomized and can't really be interpretted. Data is stored in csv files due to its simple format, and can easily retreived into a dataframe matrix using pandas, as shown in the unit test. An index file can be used to reinforce the understanding of which file data will be in, as it records the first data iteration added to each file.

If we were to implement a wireless eye tracker was used instead of component A, we would no longer need to populate messages with the Generator. In its place, we could do one of two things: 1) implement an interpreter that would read the messages sent by the eyetracker and translate them to match the format of our messages and to be sent through the current transfer protocols, or 2) scrap the Sender along with the Generator attempt to get the Receiver/Writer to listen for the messages on the wireless receiver directly.