We present a new version of YOLO with better performance and extended with instance segmentationcalled Poly-YOLO. Poly-YOLO builds on the original ideas of YOLOv3 and removes two of itsweaknesses: a large amount of rewritten labels and inefficient distribution of anchors. Poly-YOLOreduces the issues by aggregating features from a light SE-Darknet-53 backbone with a hypercolumntechnique, using stairstep upsampling, and produces a single scale output with high resolution. Incomparison with YOLOv3, Poly-YOLO has only 60% of its trainable parameters but improvesmAP by a relative 40%. We also present Poly-YOLO lite with fewer parameters and a loweroutput resolution. It has the same precision as YOLOv3, but it is three times smaller and twiceas fast, thus suitable for embedded devices. Finally, Poly-YOLO performs instance segmentationusing bounding polygons. The network is trained to detect size-independent polygons defined ona polar grid. Vertices of each polygon are being predicted with their confidence, and thereforePoly-YOLO produces polygons with a varying number of vertices. Source code is available at https://gitlab.com/irafm-ai/poly-yolo.