SEO

Fixing Robots.txt and Other Common Mobile Site Issues

Busy day? If this blog post is TL;DR (Too Long; Didn’t Read), click through the slideshow below. We summarize the whole post in 3 slides.

The mobile algorithm may have already come, but it’s certainly not gone. Being mobile friendly is the new normal. It’s a process, not an event. twitter-icon

At Oracle, where I worked with numerous enterprise websites on their technical SEO, I began to notice a few mobile site issues – Robots.txt, faulty redirects, and mobile only 404s — that popped up over and over again.

The mobile site issues I’ve compiled here are not the only issues you could run into, but they have two things in common: they happen all the time, and they’re easy to fix.

If you want more information, you can check out my presentation Protect Your Mobile Web Presence: Tips for the Procrastinator from Conductor’s Road to C3 conference in DC.

Mobile Site Issue #1: Robots.txt blocking JS, CSS, and Image Files

Google has become more efficient in crawling websites over the past couple years, and they can crawl through JavaScript, CSS, and image elements without any issues. However, if you don’t show search engines the webpage like you would be showing it to a user, then crawlers will have problems with your website.

Robots.txt was an issue for almost all of the websites that we worked on at Oracle. This is common because developers or IT teams often add disallow statements to the robots.txt, not realizing they are the blocking elements you need to get crawled by a search engine.

In optimizing our clients websites for mobile friendliness, we saw a direct correlation between disallow statements and a “not mobile friendly” designation.

mobile-friendly-test-2

Fortunately, it’s an easy fix. All you need to do is simplify your robots.txt, remove unnecessary disallow statements, and create separate disallow statement for user-agent: googlebot. Send a .txt file over to your developer, and they’ll upload it for you. twitter-icon

Problem: Robots.txt blocking JS, CSS, and image files.
 
Fix: Simplify your robots.txt, remove unnecessary disallow statements, and create separate disallow statements for user-agent: googlebot.

Mobile Site Issue #2: Faulty Redirects

We probably all have personal experience being unhappy users who run into faulty mobile redirects.

I was recently trying to look up the class schedule for this yoga studio that I was planning on visiting on my way home from work. But the class schedule kept redirecting to the mobile homepage… so I finally just quit and went running instead.

Mobile redirects happen when a user clicks on one page on a mobile site, expecting to go to the specified destination. However, the link redirects you to a different page, or back to the home page.

mobile-redirects

Faulty mobile redirects create a negative user experience. Since usability is an important ranking factor, it’s a no brainer that it will hurt your SEO.

What’s the fix? First, monitor faulty redirects in Google webmaster tools. Then, fix any broken links and implement 301 redirects. twitter-icon

If a page doesn’t have a smartphone equivalent, keep users on the desktop page, rather than redirecting them to the smartphone site’s homepage.

Problem: Faulty redirects.
 
Fix: Monitor faulty redirects in Google webmaster tools. Then, fix any broken links and implement 301 redirects.

Mobile Site Issue #3: Mobile-Only 404s

Mobile-only 404s are something that webmasters should look out for. This happens when a site serves content to desktop users who are accessing a url, but show an error page to mobile users.

It’s another big usability issue. Say I send you a link that looks fine on my computer, but when you open it on your phone and it goes right to a 404. This happens a lot on dedicated mobile sites where the example.com/page might be fine, but the m.example.com webpage that is the equivalent is broken.

404-mobile-url-img desktop-mobile-redirect

To fix this, use Google Webmaster tools to find mobile-only 404s. Ensure you’ve got proper 301 redirects implemented, and make sure that the mobile page itself is not an error page. twitter-icon

Problem: Mobile-Only 404s.
 
Fix: Ensure you’ve got proper 301 redirects implemented, and make sure that the mobile page itself is not an error page.

A Quick Case Study — Think of Your Mobile Site as an Opportunity

It’s easy to get bogged down in issues and fixes when it comes to optimizing your site for multiple devices. It’s worth mentioning that there’s a massive opportunity, too.

When I went to C3 last year, I heard Sushma Tayur from FedEx give an incredible talk about local SEO. She outlined how FedEx creates multiple pages for local stores, instead of creating a single store locator page. It really made me think differently about structuring a site around local visibility.

On top of that, I read an interesting article in Think With Google around mobile-specific keywords. That inspired me to start thinking about the different ways people would search on a mobile device.

top-near-me-searches

Combining these ideas lead to a successful new strategy for our team at Oracle. We started aggregating lists of our clients’ geo-modified keywords, including phrases like “near me” and “closest.” We used Searchlight’s Visibility Explorer to identify these, selecting them based on high search volumes and whether they ranked for with the highest search volume.

We built out and optimized new pages based on these insights. This led to more traffic for our clients, and more business for our company. twitter-icon Wins all around!

All this to say: fix your mobile site, but don’t miss the forest for the trees. Look for the opportunities, too.

Are you focusing on local search as well? Read this piece on The Power of Local Search: More Conversions for Less Traffic.

mobile-friendly-checklist-featured-4

This Mobile Friendly Checklist Could Save You from Mobilegeddon

Read More