net.slide

Network, Net package
16 May 2017

Ivan Kutuzov
ikut@softserveinc.com
https://discuss.7insyde.com
https://golang.org.ua
@arbrix

* License and Materials

This presentation is licensed under the [[https://creativecommons.org/licenses/by-sa/4.0/][Creative Commons Attribution-ShareAlike 4.0 International]] license.

You are encouraged to remix, transform, or build upon the material, providing you give appropriate credit and distribute your contributions under the same license.

If you have suggestions or corrections to this presentation, please raise [[https://github.com/GolangUA/go-training/issues][an issue on the GitHub project]].

* Agenda

- Recap
- Networking tasks
- Net package
- Net/http package
- Other net/... packages
- Testing Network

* Recap

- Goals of programming

* Recap

- Goals of programming
  - Reduce costs
  - Increase revenue

* Recap

- Goals of programming
  - Reduce costs
  - Increase revenue

How to achieve this goals?

* How to achieve programming goals

- Understanding the problem (problem defenition)

* How to achieve programming goals

- Understanding the problem (problem defenition)
- Understanding the domain (all related technologies and existing solutions)

* How to achieve programming goals

- Understanding the problem (problem defenition)
- Understanding the domain (all related technologies and existing solutions)
- Understanding general theory (data structures and algorithms / patterns)

* How to achieve programming goals

- Understanding the problem (problem defenition)
- Understanding the domain (all related technologies and existing solutions)
- Understanding general theory (data structures and algorithms / patterns)
- Understanding toolset (programmin language)

* Network

* Concepts in web principles

- Request: request data from users, including POST, GET, Cookie and URL.+
- Response: response data from server to clients.
- Conn: connections between clients and servers.
- Handler: Request handling logic and response generation.

* Standard http workflow

.image ./img/how-web-works.png _ 800

* Two general tasks

- Client

  resp, err := http.Get("http://example.com/")
  ...
  resp, err := http.Post("http://example.com/upload", "image/jpeg", &buf)
  ...
  resp, err := http.PostForm("http://example.com/form",
  	url.Values{"key": {"Value"}, "id": {"123"}})

- Server

  http.Handle("/foo", fooHandler)
  
  http.HandleFunc("/bar", func(w http.ResponseWriter, r *http.Request) {
  	fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
  })
  
  log.Fatal(http.ListenAndServe(":8080", nil))

* Net package

Package net provides a portable interface for network I/O, including TCP/IP, UDP, domain name resolution, and Unix domain sockets.

Let's go deeply.

* Net general

Although the package provides access to low-level networking primitives, most clients will need only the basic interface provided by the Dial, Listen, and Accept functions and the associated Conn and Listener interfaces. The crypto/tls package uses the same interfaces and similar Dial and Listen functions.

* Net Dial

The Dial function connects to a server:

 conn, err := net.Dial("tcp", "golang.org:80")
 if err != nil {
 	// handle error
 }
 fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n")
 status, err := bufio.NewReader(conn).ReadString('\n')
 // ...

* Net Listen

The Listen function creates servers:

 ln, err := net.Listen("tcp", ":8080")
 if err != nil {
 	// handle error
 }
 for {
 	conn, err := ln.Accept()
 	if err != nil {
 		// handle error
 	}
 	go handleConnection(conn)
 }

* Net IP

An IP is a single IP address, a slice of bytes. Functions in this package accept either 4-byte (IPv4) or 16-byte (IPv6) slices as input.

 type IP []byte
 func IPv4(a, b, c, d byte) IP

* net/url

Package url parses URLs and implements query escaping. [[https://golang.org/pkg/net/url/][link]]

 u, err := url.Parse("http://bing.com/search?q=dotnet")
 if err != nil {
 	log.Fatal(err)
 }
 u.Scheme = "https"
 u.Host = "google.com"
 q := u.Query()
 q.Set("q", "golang")
 u.RawQuery = q.Encode()
 fmt.Println(u)

* net/http

Package http provides HTTP client and server implementations. [[https://golang.org/pkg/net/http/][link]]

Get, Head, Post, and PostForm make HTTP (or HTTPS) requests:

 resp, err := http.Get("http://example.com/")
 ...
 resp, err := http.Post("http://example.com/upload", "image/jpeg", &buf)
 ...
 resp, err := http.PostForm("http://example.com/form",
 	url.Values{"key": {"Value"}, "id": {"123"}})

The client must close the response body when finished with it:

 resp, err := http.Get("http://example.com/")
 if err != nil {
 	// handle error
 }
 defer resp.Body.Close()
 body, err := ioutil.ReadAll(resp.Body)
 // ...

* net/http (continue)

For control over HTTP client headers, redirect policy, and other settings, create a Client:

 client := &http.Client{
 	CheckRedirect: redirectPolicyFunc,
 }
 
 resp, err := client.Get("http://example.com")
 // ...
 
 req, err := http.NewRequest("GET", "http://example.com", nil)
 // ...
 req.Header.Add("If-None-Match", `W/"wyzzy"`)
 resp, err := client.Do(req)
 // ...

* net/http (continue 2)

For control over proxies, TLS configuration, keep-alives, compression, and other settings, create a Transport:

 tr := &http.Transport{
 	TLSClientConfig:    &tls.Config{RootCAs: pool},
 	DisableCompression: true,
 }
 client := &http.Client{Transport: tr}
 resp, err := client.Get("https://example.com")
 // ...

*Important* Clients and Transports are safe for concurrent use by multiple goroutines and for efficiency should only be created once and re-used.

* net/http (continue 3)

ListenAndServe starts an HTTP server with a given address and handler. The handler is usually nil, which means to use DefaultServeMux. Handle and HandleFunc add handlers to DefaultServeMux:

 http.Handle("/foo", fooHandler)
 
 http.HandleFunc("/bar", func(w http.ResponseWriter, r *http.Request) {
 	fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
 })
 
 log.Fatal(http.ListenAndServe(":8080", nil))

More control over the server's behavior is available by creating a custom Server:

 s := &http.Server{
 	Addr:           ":8080",
 	Handler:        myHandler,
 	ReadTimeout:    10 * time.Second,
 	WriteTimeout:   10 * time.Second,
 	MaxHeaderBytes: 1 << 20,
 }
 log.Fatal(s.ListenAndServe())

* net/http (handler)

  type Handler interface {
      ServeHTTP(ResponseWriter, *Request)
  }

A Handler responds to an HTTP request.

ServeHTTP should write reply headers and data to the ResponseWriter and then return. Returning signals that the request is finished; it is not valid to use the ResponseWriter or read from the Request.Body after or concurrently with the completion of the ServeHTTP call.

* net/http (handler continue)

 func FileServer(root FileSystem) Handler

FileServer returns a handler that serves HTTP requests with the contents of the file system rooted at root.

To use the operating system's file system implementation, use http.Dir

 func NotFoundHandler() Handler

NotFoundHandler returns a simple request handler that replies to each request with a “404 page not found” reply.

 func RedirectHandler(url string, code int) Handler

RedirectHandler returns a request handler that redirects each request it receives to the given url using the given status code.

* net/http (handler continue2)

 func StripPrefix(prefix string, h Handler) Handler

StripPrefix returns a handler that serves HTTP requests by removing the given prefix from the request URL's Path and invoking the handler h.

 func TimeoutHandler(h Handler, dt time.Duration, msg string) Handler

TimeoutHandler returns a Handler that runs h with the given time limit.

* net/http type ServeMux

ServeMux is an HTTP request multiplexer. It matches the URL of each incoming request against a list of registered patterns and calls the handler for the pattern that most closely matches the URL.

Patterns name fixed, rooted paths, like "/favicon.ico", or rooted subtrees, like "/images/" (note the trailing slash). Longer patterns take precedence over shorter ones, so that if there are handlers registered for both "/images/" and "/images/thumbnails/", the latter handler will be called for paths beginning "/images/thumbnails/" and the former will receive requests for any other paths in the "/images/" subtree.

Note that since a pattern ending in a slash names a rooted subtree, the pattern "/" matches all paths not matched by other registered patterns, not just the URL with Path == "/".

Third party packages
- [[http://github.com/gorill/mux][gorilla mux]]
- [[https://github.com/julienschmidt/httprouter][HttpRouter]]

* A web server

[[https://golang.org/doc/effective_go.html#web_server][Effective Go]]

 var addr = flag.String("addr", ":1718", "http service address") // Q=17, R=18
 
 var templ = template.Must(template.New("qr").Parse(templateStr))
 
 func main() {
     flag.Parse()
     http.Handle("/", http.HandlerFunc(QR))
     err := http.ListenAndServe(*addr, nil)
     if err != nil {
         log.Fatal("ListenAndServe:", err)
     }
 }
 
 func QR(w http.ResponseWriter, req *http.Request) {
     templ.Execute(w, req.FormValue("s"))
 }
 
 const templateStr = `...

* Variety of techniques

- TCP / UDP
- HTTP / HTTP/2
- Socket
- RPC
- REST

* WebSockets

- Only one TCP connection for a single web client.
- WebSocket servers can push data to web clients.
- Lightweight header to reduce data transmission overhead.

.link https://astaxie.gitbooks.io/build-web-application-with-golang/en/08.2.html Build web application with Golang

* REST

REpresentational State Transfer

- Every URI represents a resource.
- There is a representation layer for transferring resources between clients and servers.
- Clients use four HTTP methods to implement "Presentation Layer State Transfer", allowing them to operate on remote resources.

* RPC

RPC is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types. On the server side, the server implements this interface and runs a RPC server to handle client calls. On the client side, the client has a stub (referred to as just a client in some languages) that provides the same methods as the server.

- Functions are exported (capitalized).
- Functions must have two arguments with exported types.
- The first argument is for receiving from the client, and the second one has to be a pointer and is for replying to the client.
- Functions must have a return value of error type.

  func (t *T) MethodName(argType T1, replyType *T2) error

* Not only existing protocol

We are able to create a new ones.

* net/textproto

Package textproto implements generic support for text-based request/response protocols in the style of HTTP, NNTP, and SMTP.

The package provides:
- Error, which represents a numeric error response from a server.
- Pipeline, to manage pipelined requests and responses in a client.
- Reader, to read numeric response code lines, key: value headers, lines wrapped with leading spaces on continuation lines, and whole text blocks ending with a dot on a line by itself.
- Writer, to write dot-encoded text blocks.
- Conn, a convenient packaging of Reader, Writer, and Pipeline for use with a single network connection.

* net/textproto Conn

A Conn represents a textual network protocol connection. It consists of a Reader and Writer to manage I/O and a Pipeline to sequence concurrent requests on the connection. These embedded types carry methods with them; see the documentation of those types for details.

 type Conn struct {
     Reader
     Writer
     Pipeline
     // contains filtered or unexported fields
 }

* net/textproto Cmd

Cmd is a convenience method that sends a command after waiting its turn in the pipeline. The command text is the result of formatting format with args and appending \r\n. Cmd returns the id of the command, for use with StartResponse and EndResponse.

For example, a client might run a HELP command that returns a dot-body by using:

 id, err := c.Cmd("HELP")
 if err != nil {
 	return nil, err
 }
 
 c.StartResponse(id)
 defer c.EndResponse(id)
 
 if _, _, err = c.ReadCodeLine(110); err != nil {
 	return nil, err
 }
 text, err := c.ReadDotBytes()
 if err != nil {
 	return nil, err
 }
 return c.ReadCodeLine(250)

* net/textproto Pipeline

A Pipeline manages a pipelined in-order request/response sequence.

To use a Pipeline p to manage multiple clients on a connection, each client should run:

 id := p.Next()	// take a number
 
 p.StartRequest(id)	// wait for turn to send request
 «send request»
 p.EndRequest(id)	// notify Pipeline that request is sent
 
 p.StartResponse(id)	// wait for turn to read response
 «read response»
 p.EndResponse(id)	// notify Pipeline that response is read

A pipelined server can use the same calls to ensure that responses computed in parallel are written in the correct order.

* Network Summary

- Standard packages in Go provide all necessaries for working with network
 - as client
 - as server
 - using TCP or UDP
 - using WebSockets
 - using REST
 - using RPC
 - using HTTP or HTTP/2
- possability of control on each level
 - transport
 - client
 - listener
 - connection
 - server



* Testing & Profiling

* net/http/httptest

Package httptest provides utilities for HTTP testing. [[https://golang.org/pkg/net/http/httptest][link]]

 Constants
 func NewRequest(method, target string, body io.Reader) *http.Request
 type ResponseRecorder
     func NewRecorder() *ResponseRecorder
     func (rw *ResponseRecorder) Flush()
     func (rw *ResponseRecorder) Header() http.Header
     func (rw *ResponseRecorder) Result() *http.Response
     func (rw *ResponseRecorder) Write(buf []byte) (int, error)
     func (rw *ResponseRecorder) WriteHeader(code int)
     func (rw *ResponseRecorder) WriteString(str string) (int, error)
 type Server
     func NewServer(handler http.Handler) *Server
     func NewTLSServer(handler http.Handler) *Server
     func NewUnstartedServer(handler http.Handler) *Server
     func (s *Server) Close()
     func (s *Server) CloseClientConnections()
     func (s *Server) Start()
     func (s *Server) StartTLS()

* net/http/httptest ResponseRecorder

 handler := func(w http.ResponseWriter, r *http.Request) {
 	http.Error(w, "something failed", http.StatusInternalServerError)
 }
 
 req := httptest.NewRequest("GET", "http://example.com/foo", nil)
 w := httptest.NewRecorder()
 handler(w, req)
 
 fmt.Printf("%d - %s", w.Code, w.Body.String())

* net/http/httptest Server

 ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 	fmt.Fprintln(w, "Hello, client")
 }))
 defer ts.Close()
 
 res, err := http.Get(ts.URL)
 if err != nil {
 	log.Fatal(err)
 }
 greeting, err := ioutil.ReadAll(res.Body)
 res.Body.Close()
 if err != nil {
 	log.Fatal(err)
 }
 
 fmt.Printf("%s", greeting

* net/http/pprof

Package pprof serves via its HTTP server runtime profiling data in the format expected by the pprof visualization tool.

The package is typically only imported for the side effect of registering its HTTP handlers. The handled paths all begin with /debug/pprof/.

To use pprof, link this package into your program:

 import _ "net/http/pprof"

If your application is not already running an http server, you need to start one. Add "net/http" and "log" to your imports and the following code to your main function:

 go func() {
 	log.Println(http.ListenAndServe("localhost:6060", nil))
 }()

* net/http/pprof how to use

Then use the pprof tool to look at the heap profile:

 go tool pprof http://localhost:6060/debug/pprof/heap

Or to look at a 30-second CPU profile:

 go tool pprof http://localhost:6060/debug/pprof/profile

Or to look at the goroutine blocking profile, after calling runtime.SetBlockProfileRate in your program:

 go tool pprof http://localhost:6060/debug/pprof/block

Or to collect a 5-second execution trace:

 wget http://localhost:6060/debug/pprof/trace?seconds=5

To view all available profiles, open http://localhost:6060/debug/pprof/ in your browser.

* Tasks

1. Create TCP Proxy server
2. Create HTTP Proxy server
3. Create simple CRUD server (with saving data in DB (PostgreSQL, MySQL, MongoDB))
4. Create simple chat server on TCP base

* Resources

- [[http://thenewstack.io/understanding-golang-packages/][Understanding Golang Packages]]
- [[https://golang.org/pkg/][Golang std pkg]]
- [[https://golang.org/src][Golang std src]]
- Alan A. A. Donovan and Brian W. Kernighan 'The Go Programming Language'
- [[http://networkstatic.net/golang-network-ops/][Golang for Network Ops]]
- [[http://www.golang-book.com/books/intro][An Introduction to Programming in Go]]
- [[http://blog.scottlowe.org/2015/01/26/using-git-with-github/][Using Git with Github]]
- [[http://www.minaandrawos.com/2016/05/14/udp-vs-tcp-in-golang/][UDP vs TCP in Go]]
- [[https://astaxie.gitbooks.io/build-web-application-with-golang/en/03.1.html][Web App with Golang]]
- [[http://soryy.com/blog/2014/not-another-go-net-http-tutorial/][Not Another Go/Golang net/http Tutorial]]


* Questions?