Forecasting nationwide passenger flows at city-level is an important but challenging task for passenger flow management and effective allocation of national transportation resources, as it can be affected by multiple complex factors. This study develops a forecasting framework to simultaneously forecast inbound and outbound passenger flows in irregular-shaped region. First, region segmentation and approximation strategies are adopted to segment the irregular-shaped regions to a set of regular grids and generate the passenger flow heatmap, enabling convolution operation to capture spatial dependencies among irregular-shaped regions. We consider intra-city dependency between inflow and outflow by treating the input heatmap as the two-channel image. Then, three temporal features and external factors are trained via the residual network units and standard Fully-Connected layers, respectively. The developed framework is validated by the inter-city travel counts data in China. The results show that the approach can well capture the spatiotemporal dependencies between passenger flows, and outperforms eight competitive methods in terms of forecasting accuracy.