Why isn’t my site crawling since 1.5 months ago?

robot spider crawling on tower building

How to optimise a website for its crawl budget and get your most important pages indexed »

Answer by Jason Darrell:

Hi, Maulik Solanki. Thanks for the invitation to answer.

One of the lesser known elements of SEO is a site’s ‘crawl budget‘. Every website has a threshold that bots will crawl. If you’ve reached that limit, your site may seem not to have been indexed for a while.

Assuming that your robots.txt files are allowing indexers to crawl your site and haven’t been accidentally switched to noindex, it could be that your website has reached that saturation point. That’s the best way I can think of describing what I mean.

Imagine you’re targeting specific search terms across your site. If the only new content you’re producing is similar to that which already exists, why would Google or other search engines reward the new content with QDF/displaying it in SERPs?

The way Google is headed is that it’s feeding spiders on awesome, quality content. The more you feed it, the more its capacity for lunch. If you’re not telling the web anything new, your site will starve.

So, if you’re adding no new value to your website, the indexers will look at the overall power of your website, deem they’ve crawled all they want to and show no signs of having crawled the deeper pages.

How to optimise your website to make best use of its crawl budget

There is a way you can point the indexers in the direction of the pages that you want crawling. That’s whether you have fresh content or not.

But if you have got new content that you want indexing, it’s imperative that you know how to send the bots there. A share on Google+ is often a quick fix for a specific page, but doesn’t serve your long term purpose for your entire website.

Here’s what Matt Cutts has to say about freshness as a signal, in the context of what I’m trying to get across:

Now, hands up, the technical aspect of implementing 304s is beyond my remit – strictly an on-page semantic copy guy.

But my CTO at SEO Workers, John S. Britsios, has  written an in-depth article about the topic on algohunters, a subsidiary site:
Building a Solid Index Presence by Optimizing your Crawl Budget

The above video is one of three in the article that help underline why crawl budget is so important to understand and to optimise for. That goes for any webmaster who wants organic traffic.

As Matt himself would say, #HopeThatHelps ☺
image credit: algohunters

Visit the question on Quora, here: Why isn’t my site crawling since 1.5 months ago?

About these ads

How can I improve SEO for a very big dictionary website?

Screaming Frog SEO Spider Tool & Crawler Download Links

Answer by Jason Darrell, via Quora; for full context, visit the original Quora thread.

My question would be: “if my site truly "is already the biggest and official dictionary in that language", why isn't it ranking first, anyway?”

You can approach the ranking aspect from two directions:

  1. content;
  2. usability.

As Jaime J Candau suggests, try offering pseudonyms and antonyms (as per Thesaurus) and other language tools on top of your explanation of the word itself.

Also, you may want to consider:

  • is your description totally unique to the Internet?
  • does the way you describe the meaning of a word add more potential value to the user than other online dictionaries?
  • is the word offered in all tenses and formats, with examples of it being used in a sentence?

Then, as Mohit Maheshwari alludes, do you offer a way for your users to share their grammatical discovery right from your website?

Do you have the time/budget to run a word-a-day promo on social?

Are you engaging with people who want to develop or learn from scratch your language?

Of course, check all the SEO basics for every page, if you've not done so.

This META SEO Inspector extension for Chrome will help you check page-by-page. It lives in the browser as an extension and you can see what's right or wrong with a page's SEO (or mark up) at a glance.

But I guess with 2.5M words you'll want to check in bulk. In that case, Screaming Frog's SEO Spider will let you download your content in bulk.

nb – if memory serves, 500 URLs are free to extract, but you'd need a pro membership for as many pages as you need to SEO.

Your SEO priority has to be the UX.

But more than anything, what does each page contain in the way of ads, flash and widgets?

From my experience of Collins and other online dictionaries, they generate income by plastering ads everywhere. In fact, there's sometimes so much advertising it's difficult to pick out what it is you've visited the page for.

Is the information the user is looking for expressly clear and visible as soon as they land on page? Try to make the user's experience as painless as possible.

More page load speed time, the less likely it is your customer will hang around. Less return visitors for you, less authority in Google's eyes.

And Google did confirm (not that we didn't know anyway) that load speed is now a definite ranking factor at State of Search Dallas not so long ago.

And, of course, optimise your website for mobile with a responsive design.

In short, aim to provide the best service possible, add the most value and your customers will begin to bookmark your dictionary and return time and again. Word-of-mouth will earn links to your site, thus may have additional benefit because of a heightened link profile.

SEO is not an overnight process and the results – as well as the implementation – will take time. But if you do it the right way, the results will be greater and last a lot longer.

image credit: screenshot of Screaming Frog free download links

What are your predictions for digital and internet marketing for 2015?

Forecast for Search and Social for 2015 | via Quora

Answer by Jason Darrell:

Saw this article, thought of you, Frank Tsaur : 2015 Digital Marketing Predictions. Well, specifically, this question.

The linked post contains summaries from industry experts about their 2015 forecasts for:

  • SEO
  • AdWords
  • SocMed advertising
  • Consumer traits
  • More advertising and social media predictions
  • you get the picture.

Article originally found on Google+: Online Marketing Predictions for 2015

Changing face of SEO for 2015

For myself, I agree with Rick Eliason's forecast for SEO, to some extent.

Keywords will still play a part, just their role has changed. Marrying keywords' relationships with context and concepts around them will be the winning factor for ranking in search.

But this means a change in the way we write for Web 3.0. Brands will (hopefully) realise that Google does not exist purely as a channel along which brands can expect website traffic. «« Tweet This!

A company's entire digital strategy has to merge and beat with the same heart rate. Yes, even if SEO remains somewhat symbiotic to traditional advertising methods.

But that's no bad thing. Your search campaign should mirror your offline promotions, products and services. That's what Google wants.

Speak to your clients (not at them) through your social channels and your blog and website content. Discover what it is they like about your brand. Or what they don't.

Only by understanding your evangelists' motivation and your naysayers' chagrin will your digital marketing be successful.

I want to see more brands become more tangible, letting customers get touchy-feely with them.

If they can highlight the benefits of their products, they'll both attract a more relevant audience in the first instance and garner word of mouth traction, which is advertising money just can't buy.

How do I write SEO text?

related content positions for semantic copywriting

Found on Quora; my answer to: *How Do I Write SEO text?*

Answer by Jason Darrell:

I’d just like to add to this conversation, because some of the information contained herein is misleading or misguided. Unintentionally so, but nevertheless, could cause problems if you ‘ignore’ keywords altogether.

First and foremast, a brand new post on Moz – Topic Targeting: An Illustrated Guide for Advanced On-page SEO – suggests that keyword placement and their relationships with other key phrases on the page carry weight.

This is especially true in the semantic landscape, which we’ve been following, preaching and pre-empting for some years.


I’d also like to temper the “keyword density is irrelevant” comment

What the commentators mean when they say that ‘keywords don’t matter’ is that there is no mythical optimum keyword density to aim for.

Don’t think that you can include the keyword over and over again because density no longer matters. You can over-cook the amount of times you use a keyword, which will lead to an awful UX and may even see your page (and site) penalised.

The relationship your main keyword has with supporting keywords is what’s critical to Google. The search indexers should be able to relate ‘entities’ to concepts it has confirmed from various databases about said people/products/places/services on its Knowledge Graph.

The article on Moz (linked above) shows keyword placement optimisation. If it helps, think about your main keyword and its supporting phrases like a pyramid.

Choose the main keyword then, as the guys have already suggested in these comments, use Google Keyword Planner or ‘related articles’ at the bottom of SERPs to discover on-topic (longer-tail) keywords.

Spearhead your paragraph with a keyword from the top layers, then support those main keywords with distanced concepts. This will also help you link out naturally with a variety of different anchor texts that will appear natural.

And that really is what Google is looking for: content that flows naturally with related topics/keywords and links that support your page’s main theme. The Moz article goes on to describe this, too. But…

Your copy CAN transcend the need for the main keyword, but your copy has to be awesome

…if your content is truly awesome, you can teach an algorithm to extract the main concept of your article without actually including the main keyword.

This takes skill, patience, editing and a knowledge of the topic about which you’re writing.

Here’s a SlideShare I did for a client that demonstrates what I mean: Entity Disambiguation – the Semantic XRay.

In fact, it’s a bit crap on SlideShare; here’s the original in my OneDrive folder: Semantic X-Ray.pptx – feel free to download it for reference.

The presentation takes you step-by-step through my process of taking a client’s copy and re-writing it so that it’s better optimised for relevant search terms.

Slide 19 shows that, although I didn’t include the main keyword in the rewrite of the original content, AlchemyLanguage data extraction tool rated the main keyword as 0.94 relevant to the content (where 1.0 is optimum).

Organic = Natural – it’s not rocket science

But above all else, the key thing to remember is that you’re writing for a reader, not the search engine.

Write naturally, as if you were talking to your client, not preaching to an audience you’ve never met.

Then go back in and edit. Use NLP* to ensure that your copy is crystal clear. Use data extraction tools to test if your copy is telling the search engine and reader what you set out to tell the world.

Stay on that track and you won’t go far wrong.

For further reading about semantic copywriting, here are a couple of resources:

Screen shot: Moz, from the Illustrated On-Page Topic Targeting Guide, linked above.

p.s. - you don’t have to use schema, as is suggested in the Moz article. We use RDFa Lite mark up and replicate pretty much everything you can achieve with schema.

How do I get known or promote my online profile?

Examples of Hootsuite Stream Twitter Search

Jason Darrell’s answer (on Quora) to: How do I get known or promote my online profile?

Answer by Jason Darrell:

First, you want to get known for the right reasons.

Start with social listening – follow conversations where your expertise can help people out. Don’t try and sell anything at first – just be a hero to those in need.

If your knowledge is sound and advice clear, people will start to follow you for the right reasons.

Be where your customer is
If you simply gatecrash a conversation and paste a link to an answer you have existing on your own web real estate, (social media or website/blog), people will see you as a spammer, even if you do it with good intentions!

The psychology adopted by today’s digital-savvy consumer makes them very protective of their little bit of the internet.

As marketers, we have to reach people where they are – if they’re using facebook, don’t try to send them to Google+ or Twitter.

In order to encapsulate the whole of this market, you need to be wherever you think potential customers will be.

How to begin Social Listening
The temptation is to push, push, push your own content. Then push it some more. You cannot do that until you’ve developed trust and displayed your authority by sharing relevant knowledge with your potential audience.

To begin, you can follow hashtags, perform your own searches on the platforms themselves and follow thought leaders in your field to see how they’re responding to people who you believe will benefit from your help.

Hootsuite Social Media Management offers ways to follow “twitter searches”, as per this image, depicting a couple of terms I’ve got a vested interest in:

To accomplish following hashtags using Hootsuite, simply:

  • create a stream in Hootsuite (after you’ve signed up, if you’ve not done so already),
  • choose the Twitter account via which you want to follow conversations (must be linked via Hootsuite),
  • choose the ‘search’ option for the new stream (it will have ‘mentions‘, ‘inbox‘, ‘home stream‘, etc.)
  • and finally enter the search terms for which you want Hootsuite to render tweets in your dashboard (as above).

NOD3x Social Media Analytics and Data Discovery does a similar thing for Google+.

Both have free versions, with options for greater social network coverage (both in volume and number of networks you can follow/post to) for upgraded licenses.

Also, make sure you’re following Twitter chats around your sector and are involved in Facebook, LinkedIn and Google+ groups/communities of the same.

The new term for this ‘phenomenon’ is “Relationship Marketing“. It’s no different from the 1-2-1 help pro digital marketers have been offering for years.

If you’re looking for an overall social strategy, here’s a 12-step guide I put together in a similar answer right here on Quora: How should I stop my self-promoting on social media?

I think that should cover all your bases; the very best of luck.

I am trying to get more website traffic and social networking traffic. Any help?

Hummingbird in a Nutshell

…question found on @Quora

Answer by Jason Darrell:

Some great answers here already. But there are three main routes via which relevant traffic can reach your website:

  • organic (SERPs)
  • social (networking/engagement)
  • paid (sponsored/adwords)

First, make sure you have unique quality content that adds value to your readership/following when they get there. Both in the service pages and on your blog.

Then, make sure they can find their way around your website.

According to #StateofSearch this week in Dallas, high quality content and navigability are the two key factors of the UX that will help your website in SERPs for organic traffic.

More info in this SlideShare, here:

Hummingbird and Semantic Search – State of Search Dallas from Eric Enge

Then, you need to be the face of your brand on social. Begin with social listening, pick out the conversations that point to your product/service being of benefit, then offer to help.

Don’t go in with a hard sell – just help out and your authority will lead to traffic via social in time.

I outline out a social engagement strategy based on my personal experience (if you don’t have one), here: Jason Darrell’s answer to How should I stop my self-promoting on social media?

Then, there’s paid traffic. Whether you use adwords and/or sponsored/boosted posts on social will depend upon the nature of your product.

Your ‘social listening’ efforts will help you to determine which platforms’ users have most in common with you, your brand and your product.

SEM, unfortunately, is beyond my remit, therefore take on board what others have said about driving paid traffic, if you will.