What is the best way to reduce my image file size?
At minimum: use ImageOptim. It can significantly reduce the size of images while preserving visual quality. Windows and Linux alternatives are also available.
More specifically: run your JPEGs through MozJPEG (q=80 or lower is fine for web content) and consider Progressive JPEG support, PNGs through pngquant and SVGs through SVGO. Explicitly strip out metadata (--strip for pngquant) to avoid bloat. Instead of crazy huge animated GIFs, deliver H.264 videos (or WebM for Chrome, Firefox and Opera)! If you can’t at least use Giflossy. If you can spare the extra CPU cycles, need higher-than-web-average quality and are okay with slow encode times: try Guetzli.
From the excellent images.guide.
What is the best format for performant images? PNG? JPG? SVG?
It's between SVG and JPG. It really depends on how much detail and color variation the image has. If it's a simple vector graphic (Something like a logo or icon) you should safely be able to go with SVG. This can be under 1k - 5k in size. If the image is more like a photo with a considerable amount of detail and colors I would go with JPG. You should only feel compelled to reach for PNG if the file has transparency.