Among popular techniques in remote sensing image(RSI) segmentation, Deep Neural Networks (DNNs) have gainedincreasing interest but often require high computation complex-ity, which largely limit their applicability in on-board space plat-forms. Therefore, various dedicated hardware designs on FPGAshave been developed to accelerate DNNs. However, it imposesdifficulty on the design of efficient accelerator for DNN-basedsegmentation algorithms, since they need to perform both convo-lution and deconvolution which are two fundamentally differenttypes of operations. This paper proposes a uniform architectureto efficiently implement both convolution and deconvolution inone vector multiplication module. This architecture is furtheroptimized through exploiting different levels of parallelism andlayer fusion to achieve low latency for RSI segmentation tasks.Moreover, an optimized DNN model is developed for real-timeRSI segmentation, which shows preferable accuracy comparedto other methods. The proposed hardware accelerator efficientlyimplements the DNN model on Intel’s Arria 10 device, demon-strating 1578 GOPS of throughput and 17.4 ms of latency, i.e.,57 images per second.