Have you found that no matter what you do, how much content you publish or how many links to attract, your website is never found in Google? Wondering why, no matter how hard you try, you can never get in the top ten?

Well, it’s probably due to a few really simple mistakes you’ve made, so let’s have a look at the common ones and see how we can fix them once and for all.

Is your robots.txt file misbehaving?

If you’re totally new to this web lark then you might have no idea what I mean by robots.txt, well it’s a small text file that sits on your server, and it gives instructions to search engines.

It’s usually used to tell the search engines which pages to view and which ones to ignore. In the case of Google, you can, for example, tell it to ignore all of your administration pages.

However, a mistake made by many is to use it to stop search engines viewing any of the site. Why would you want this?

When creating a new website, the web developers will often create a test site for you. This will have the ‘disallow all’ operator in the robots.txt file to stop any issues with duplicate content.

The problem comes when the site gets moved to live. Sometimes the developer forgets to change the operator, and suddenly, your site disappears.

How to test it

The file is usually in the root of your website, for example, here’s the one on the BMW website: http://www.bmw.co.uk/robots.txt

And here’s the one for the BBC: http://www.bbc.co.uk/robots.txt

Yours doesn’t need to be as complex as either of those, and you can tell if it’s causing you problems by replacing the domain from one of the ones above with your own.

If you see something like this:

User-agent: *
Disallow: /

Then you know you’ve got a problem. Those two lines mean that for all search engines, disallow everything. If this is on your site, it’ll never be found.

Talk to your web developer about changing that.

Does your site have a penalty?

If your site has been penalised by Google, then you need to discover why and how. It could be that the links to your site have been causing you problems, or it could be that your content is at fault.

To find out, head across to:

Pop your web address in, and take a look at what is displayed. Here’s how BMW fared:

bmw penalty


Now, I don’t want to go into the real big details of what each of the penalties are, how they happened and how to get out of them, but what you’re looking for is a dip in traffic at around the time of one of the coloured bars.

As you can see, BMW seems to have a nice steady rise, with the odd drop, although those drops don’t seem to coincide with any Google changes, so they’re probably OK.

However, take a look at Interflora:


What happened here was that they were found to be flaunting Google’s rules and so they received a penalty. Tsk!

If you find that you have a dip that coincides with one of these lines on Google, then you might be suffering a penalty, so you need to take action to find out what’s gone on.

Keyword Stuffing

Take a look at this paragraph of text:

We are the best car valeting company in Birmingham! We offer top price cheap Birmingham car valeting at the best rates so that if you were looking for car valeting, you can be sure that we offer the best car valeting in town. To get a car valeting quote, give us a call on 0555 553343 now and someone from our Birmingham car valeting office will be in touch.

As you can probably tell, they really want to be found for “car valeting”. Preferably in Birmingham.

Thing is, this looks terrible and Google knows when you’re doing it.

There’s no right answer to the question “what keyword density should I be looking to achieve”, as it’s different for every single subject. However, if it looks odd to you, then it will look odd to others and Google will frown upon it.

How to fix the problem

Your keyword (in this case, “car valeting”) should be in the title, the header and somewhere in the text. That’s enough. If it needs to, then it can appear a couple of times in your text, but it shouldn’t be repeated for the sake of it.

Check all of the pages of your site and see if any of them are culprits. If they are, re-write the text.

Incorrect use of Titles

The titles of your page are one of the most important aspects of your on-site SEO, but even though they’re easy to get right, so many people get them wrong.

They are NOT there for you to stuff in lots of keywords, they are there to be a very short description of what’s on the page.

An example of a wrong title: “car valeting, car valeting Birmingham, Birmingham car valeting, cheap car valeting”

That’s a list of keywords, it’s pointless and won’t help (and could hinder) your SEO efforts.

This is an example of a good title: “Car valeting service based in Birmingham”

Job done.

Your titles should be static, too. I’ve seen an instance where the titles changed every time someone visited the site, this is not a good idea and could lead to a penalty – it’s just confusing.

And finally, your titles should be unique across your site. If you’ve got the same titles on multiple pages, get in there and change them.

Duplicate Content

Now, I’m going to say this for the gazillionth time: Duplicate content is not a penalty as such. But it could be the reason why your site isn’t being ranked well.

If you have a lot of content on your site, then the vast majority of it should be unique. If it’s a common subject, then the articles you write should be your take on that subject. For example, a local accounting practice might write an article about how to handle tax at the end of the year. You can write something about it, too, but it needs to be unique.

If you simply take an article from someone else’s site and copy it word-for-word on to yours, then you could be storing up problems.

First of all, there’s copyright.

The original writer of the content might get a bit upset and ask you to take your version down. You should.

If you refuse, you might end up in court.

Secondly, you might end up with yours not appearing. This is where the almost mythical duplicate content penalty kicks in.

You see, Google will display the content that was there first, has the most shares, is the most popular. Yours will not appear because, let’s face it, you copied it.

If you do this a lot, then you’ll find that your content is never found. Google’s getting better at knowing which is original and which is copied, and they’ll reward the good stuff and leave the copies out.

Is this the same as curation?

Nope. Curation is something we’ll come on to another time, and it’s a very powerful way of publishing content, but it has to be done in a certain way.

If you have a store that sells other people’s stuff, then there’s bound to be some copying going on.

Product descriptions do tend to be duplicated across sites, but notice how the good sites make it work for them. They add their own spin on the products, include videos and other content and make the pages, on the whole, unique.

When is it time to give up?

Sometimes a site will never rank. It’s usually because of Google penalties and they’re just so bad, it’ll be consigned to the nether-reaches of the Internet forever.

So, it might be time to give up.

It could be that nothing can be done. So you have to make the decision.

Do you start again with a new domain?

Many SEO companies would discount this as a crazy idea, but I would like to offer the opposite view.

Sometimes, it can be a lot easier to just rip up a site, and start again.

If you feel there’s no way forward and you just can’t make any inroads, consider it as the last option.