Excellent article over at Search Engine Journal:
Regardless if responsive design remains popular or disappears from the online landscape, there is one fact we’re certain of: speed will always be important. And that should be a concern for sites using responsive design. If you’re site is bloated with CSS files, for example, it’s going to take longer to download. If you have a well-developed responsive site, than there should be no worries.
What we’re getting at is that it doesn’t matter if you’re using a desktop, smart phone or tablet, speed is a deciding factor in if customer’s will stick around. It also will determine if they come back again, which can improve sales. And, the speed of your site also matters to Google.
No surprise that you can have your cake and eat it, too. It all comes down to how well-developed your site is. Extraneous stylesheets and excessive image downloads on mobile devices aren’t good for anyone. The best approach is to start with the mobile experience and then add features and functionality as screen sizes increase.