People are five times more likely to leave your site if it is not mobile friendly.
From the Google Webmaster Central Blog:
“Getting good, relevant answers when you search shouldn’t depend on what device you’re using. You should get the best answer possible, whether you’re on a phone, desktop or tablet. Last year, we started using mobile-friendliness as a ranking signal on mobile searches. Today we’re announcing that beginning in May (2016), we’ll start rolling out an update to mobile search results that increases the effect of the ranking signal to help our users find even more pages that are relevant and mobile-friendly.”
If your site is not mobile friendly, you will suffer in Google rankings.