Federated Learning (FL) is a promising framework for enabling edge intelligence while preserving privacy and enhancing communication efficiency. In real-world scenarios, limited and heterogeneous hardware resources among users threaten the overall training efficiency in FL. To address this issue, several model heterogeneity frameworks have emerged in recent years. However, existing studies primarily focus on mitigating idle times on powerful devices, overlooking the fact that strong devices may not always have a positive impact and can potentially become a burden instead. Furthermore, as the resource gap among clients widens or the number of weak devices increases, the model may be hindered by the influence of strong devices. Directly applying existing model heterogeneity frameworks leads to model performance degradation due to the neglect of contributions from weak devices. To make a performance breakthrough, we proposed a novel FL framework called Progressive Width Mixing (PWM), which extracts diverse features and balances the contribution of strong and weak devices and mitigates straggler effects under scenarios with various device distributions. To our best knowledge, we are the first model heterogeneity work that considered straggler effects from the weak devices' point of view. The experiments show that PWM maintains stability over different datasets and outperforms all baselines across various device distributions and non-IID scenarios. PWM makes a leap forward and surpasses all baselines by 7%, and mitigates the negative impact from the strong devices.