Insights to Ignite Your Mind

Director of SEO Innovation

Mobile Apps

Mobile SEO: Google Won’t Be Crawling Your Site, It Will Crawl Your API

I started my first company, Blue Gecko Web Design, in the spring of 2002. It was me and a buddy from college. He would go around to local store owners trying to sell them a new website whereas I’d be back at the apartment building the sites.

I distinctly remember going on a sales round with him when we visited a local Chinese buffet that we frequently ate. We asked to speak to the manager and the conversation went like this:

Us: Hey, I noticed you don’t have a website.
Manager: We don’t need a website. All our customers see our sign and come in.
Us: Ok. Thanks for your time.

I admit, we were really bad salesmen, but sure enough today they have a website up. No surprise there. Every company has a website. In 2002, not every company thought it needed one! In 2012, not every company thought it needed a Facebook page. And in 2012, not every company thinks it needs a mobile app, but they all do, and they will all create one…eventually.

The Shift

By now, businesses should realize that the world is going mobile. As people move from PCs to mobile devices, mobile SEO is becoming increasingly important.

Apple sold more tablets than any of the top PC makers sold computers last year, and this was even before the iPad mini.

Responsive websites are great, but as Facebook has learned, they are no replacement for native apps. Native apps are simply the best way to consume content on a mobile device. Like it or not, every company that has a website needs a native app.

How does one power a native app?

Some apps, like games, have most of the content stored right within the app itself. Other apps, like Twitter’s or Facebook’s, pull content directly from the Internet through APIs.

APIs are kind of like a special way of requesting information from a database. Instead of writing a complicated piece of code to look up special values in the database, you simply tell the API, “Give me this user’s profile information” and the API converts that request into a complicated code that looks for special values in the database.

These APIs make it easy for external folks, like outside developers, to use your content and data. That’s why Facebook and Twitter have so many apps built on top of their platforms. Their APIs are open to the public and easy to use. Letting developers easily access Facebook and Twitter data in the same way Facebook’s and Twitter’s own mobile applications do will make accessing your native app’s content easier  for external developers, as well as search engines.

Search Engines

Search engines are great at crawling web pages, but they’re really bad at crawling apps. Even if Coca-Cola decides to announce the reveal its secret formula on its iPhone app, Google would never be able to see that information.

We know that users prefer to consume content on native mobile apps. Because of this, companies will need to ensure that their content is available through their native apps. However, we also know that search engines are not able to crawl content on those native apps. What’s a search engine to do?

Simple. Request the content through an API.

If a company wants search engines to continue  crawling their site content, they’ll need to open its native app’s API to search engines. Believe me, it won’t be long until search engines come up with a standardized way to do so, just like they did with

Get Ready

The key to beating the search engines is always to skate to where the puck is going. In this case, it’s APIs. Here is a recap of the steps you need to take to get ready.

  1. Create a native mobile app for both Android and iOS.
  2. Deliver content to your app through a private API.
  3. Be ready to open parts of your API to search engines when they are ready.

Sure, you can wait until everyone else does it first, but do you really want to be like the Chinese Buffet in our example, waiting until the competition forces your company to get a website? If you know what the future holds, why not be the first to market?

Need More of the Future?

If you thought this post was insightful, and you don’t want to miss any future posts from Catalyst, you can join our tribe.

Being part of our tribe you’ll get three great perks:

  1. Never miss a Catalyst post. Whenever there’s a new post you’ll get it delivered to your email inbox.
  2. Keep your followers happy. You can manually or automatically share Catalyst posts with your social networks.
  3. Meet the team. You’ll be able to interact with dozens of SEO/SEM experts who work for Catalyst.

We’re looking forward to chatting with you in the tribe. Join us.

Share this:

6 Responses to Mobile SEO: Google Won’t Be Crawling Your Site, It Will Crawl Your API

  1. Clayburn says:

    I disagree. I think we only “prefer” native apps to web apps because there aren’t good web app options yet. We’re still in the early stages of mobile where apps are required. Eventually, just like desktops, the browser will be the thing in which all apps live, and they’ll be web-based. If you asked the question in 2004, desktop apps would have been “preferred”. But now I can check my email, create documents and play games all inside of Chrome. It’s far easier and more efficient than installing and running a new program for everything I want to do.

    Still, APIs are important. I just don’t think native mobile apps are the future. I think they’re the present and will soon be the past.

    • Dan Cristo says:

      I think of the mobile device itself as a browser, with apps you download to your homescreen as “favorites” or “bookmarked” applications. It’s similar to how with the Chrome OS, everything take place inside your browser.

      When you have the mobile device itself as your browser, and downloading apps becomes as fast as opening a web page, why would we need an application like a mobile web browser anymore. The mobile device itself is more capable of accessing what we need from the Internet through apps.

  2. Matthew Proctor says:

    I agree with Clayburn. I think that web apps will work out just as well as native ones. I also don’t think that it will ever be an either or case. While it may be true that Google and other search engines will eventually have the capability to crawl your API, they won’t suddenly stop crawling the web.

    More likely, both crawls will run simultaneously and index together. What the search engines prefer to use will become the best practice and eventually companies will get on board with whatever that may be but we can’t know which one the engines will choose yet.

    Either way, its essential that companies start thinking in terms of mobile apps (be they native or web based) now so that when the time comes they’ll be in a much better position to adapt. So, cheers to that sentiment Dan!

    • I also agree with both Clayburn & Matthew Proctor…if this is skating to where the puck “is going” then its one that is not on the rink.

      Having an API is great, but most companies won’t see value in that, as such search engines aren’t going to be giving additional “credit” to those that actually have one and would be difficult for search engines to understand what the various APIs do.

      I think the author is getting lost in apps ruling the world just as software did in the 80s/90s until computing power & Internet speeds got strong enough to remove the need for Native tools and simplify for all platforms under the browser…same for mobile devices, as connection speeds and processing improves there wont be a need to create and then UPDATE an app each time for Android, iOs, Windows, RIM,..never mind versioning for tablets and then future device types coming out…solidification of HTML5/CSS3 removes that need with the exception of high end games and tools (which is really the only software you still buy outside the OS for PCs)

      I post data and facts behind my reasoning with articles at Search Engine Land and Search Engine Watch, which this gives me a good idea for my next post :)

    • Dan Cristo says:

      I think you’re right in the sense that both native apps and web sites will be crawled at the same time. I think we’ll go through that transition for several years. It really comes down to whether or not native mobile apps will completely replace web pages or not. I think they will eventually, and I think human wearable computers will put the nail in the coffin of the legacy web, so to speak.

  3. As a note, I posted about this article and cited it in today’s Search Engine Land article at

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>