15 replies to this topic

[Michael Martinez]

In December Jill said that Google will indeed crawl iFramed pages and follow the links. I had never seen an example of this behavior before, as in my previous tests that was not happening. In fact, for some existing sites I'm pretty sure the links are not passing value but that doesn't mean Google isn't crawling the iframed content and evaluating the links.

So I designed a simple two-stage test in December to see what would happen. The test is such that if the link passes anchor text in the first stage there is no need to try the second stage.

I embedded an iframed page on one of my older sites. It is well-indexed, has never been penalized, and has sufficient inbound links that I was confident it would be recrawled and re-indexed within a short period of time. It is also a low traffic site (it used to have more traffic a few years ago) so I felt it was probably a safe place to run a link test without attracting too much attention.

The iframed page only contains one in-body HTML element, a link pointing to HighRankings.com with the anchor text of "jill whalen says search engines crawl iframes". This was, until I ran my test, an expression that did not appear in Google's search results (when you wrap quotes around it).

I let the test be through the holidays and checked it today. The expression now appears in Google's search results and it shows only one listing for it:

SEO Consulting: High Rankings Expert Search Engine Optimization ...
www.highrankings.com/ - Report spam

by Jill Whalen · in 4,445 Google+ circles · More by Jill Whalen
Over the past 16 years, SEO Consultant, Jill Whalen has worked with hundreds of SEO clients across more than 40 industries to enhance their online presence ...

I don't know how long the link's value will remain in the index, as I will take it down from my site soon. But for now you can use this query to see that Jill's site is indeed ranking for the expression above.

After I publish this discussion, this query will no longer return no results but for now it tells me:

No results found for intext:"jill whalen says search engines crawl iframes".

There are rogue crawlers out there which DO crawl iFramed content and republish it on suspicious Websites. That was another reason why I chose the low-traffic site I used for my test. Had I found this anchor text (or the link) on other sites I would have been forced to go to stage two of the test plan to see if I could filter out any influence from rogue crawlers.

I'll say that I am .01% uncertain about any possible contamination of the test from unforeseen factors.

I did, by the way, embed a robots meta directive on my iframed page with "noindex,nofollow,noarchive". This appears to be the only safe way to ensure that iFramed content does not appear in search results. I don't know when that changed but Jill is correct in saying you can no longer rely upon iFramed pages NOT being crawled and indexed (it was always sketchy anyway, as the iFramed pages could have been found through other links).

[Jill]

Thanks, Michael. I knew it to be true since my RatOutYourCompetitor.com site is simply an iframe to a Wufoo form and only the form on the Wufoo subdomain gets indexed now. (Previously the actual domain had been indexed.) I had only recently noticed this as well, so it maybe somewhat newish behavior on Google's part.

[BostonSEO]

It would seem more likely that the search engines found the page through a sitemap. I have several pages with iframes running, I use them to manipulate my link flow throughout my web site. I have not seen any caching of my iframes within the page they are installed on, although they are cached. So either they found it in a sitemap or followed the link from the iframe itself. Either way it is not affecting the link flow on the page the iframe is on. I am going to continue to monitor my link flow and see if anything changes.

[Jill]

It would seem more likely that the search engines found the page through a sitemap.

Nope. No site map. They follow the link from the iframe.

[Michael Martinez]

It would have defeated the purpose of the test to link to the test page with anything other than an iframe reference. I assure you, there was only ever 1 link pointing to that page (and I just took it down this morning).

It is not necessary to inspect my Website to analyze the test results. I explained how I performed the test so anyone should be able to replicate it. If my explanation seems unclear, I'll be glad to post more details about the methodology.

[Jill]

I use them to manipulate my link flow throughout my web site.

How would an iframe even manipulate link flow (whatever that is) anyway?

[Michael Martinez]

Yeah, I'm starting to get the impression that people are confusing the link on the iFramed page with the link in the iFrame HTML element.

The test was designed to show whether or not Google would follow the IFRAME ELEMENT's link to the iFramed page. We have always known that if you link to an iFramed page normally that a search engine can find the page and index it.

So the fact that the iFramed page in this test could ONLY be found through an iFrame is significant in that the iFramed link passed anchor text to its destination. One can infer that the IFRAME ELEMENT link may be passing PageRank to the iFramed page but the test wasn't designed to try to determine that (there is no way to track and measure the flow of PageRank).

I am sure that people will now try all sorts of experiments but it's hard to say if there is really any value to trying to leverage iFrames for SEO.

What the test clarifies is whether using IFRAME elements to partially block pages from search engines has any benefit at all.

So far as I can determine, Bing did NOT follow the IFRAME ELEMENT link, but Google did. Hence, if you don't want the content on the iFramed page indexed, you need to embed robots meta directives on the page or set up a robots.txt DISALLOW.

The embedded meta directives are the only way to ensure that the iFrame page doesn't appear in the search index at all, as blocking with robots.txt will only tell a search engine not to crawl the page; a search engine may still show an uncrawled URL in its results.

And rogue crawlers may still create external links to iFramed pages. In my test I could not find any trace of rogue crawling activity, so I am pretty confident that the test confirms what Jill has been saying, that Google follows the IFRAME ELEMENT's link.

[Michael Martinez]

I did, by the way, embed a robots meta directive on my iframed page with "noindex,nofollow,noarchive"....

So this was brought to my attention.

I did, actually, specify "FOLLOW" in the robots meta directive. Sorry about the typo.

[SEOmofo]

As far as I can tell, this experiment/thread can be explained with just a couple of basic concepts.

1. Google processes a web page (i.e., reads the HTML source code returned in the body of a given URL's HTTP response); if it finds a string that looks like a URL, it adds the string to the Crawler queue. In other words, Google will try to crawl anything and everything. The src attribute of iframe elements is easily "discoverable" in this regard.

2. Once a URL makes it into Google's Crawler queue, Google will request it--without making any assumptions about the content it expects to find. It's just a list of URLs; where or how they were discovered doesn't matter.

3. When Google requests a URL that it discovered in an iframe element, most servers simply respond with the content of the requested URL. That is, most servers return the same HTTP response whether the request was made directly or via an iframe element. Therefore, Google requests AND receives an iframe src URL just like any other URL. It's simply another web page, as far as Google's concerned.



Considering these concepts, nothing mentioned in this thread is surprising, is it?



intext: operator => no results - Only 1 URL (i.e., the iframe src attribute value) would have satisfied this query, but it was noindexed via robots meta tag. No surprise there.

inanchor: operator => Jill's home page - Google knew of one page (i.e., the iframe src attribute value) that links to Jill's home page with that anchor text. Google wasn't Disallowed from requesting the page (i.e., the iframe src attribute value), and Google wasn't prevented from following links on the page (i.e., the iframe src attribute value). No surprise here either.



I'm not trying to be Mr. Knowitall here...I'm just confused as to what this experiment has to do with "passing value." As far as I can tell, "Google passed value" really means "Google crawled a URL it found in an iframe element's src attribute."



Wouldn't the following experiments essentially "confirm" the same thing?

1. iframe src attribute (i.e., the experiment mentioned in this thread)
- You embed an iframe on your web page. The iframe src="URL".
- No web pages contain links pointing to URL.
- Google requests URL and receives page content.
- Page content contains a link to Jill's home page.


2. Direct submission
- You submit URL via Google's URL submission form.
- No web pages contain links pointing to URL.
- Google requests URL and receives page content.
- Page content contains a link to Jill's home page.


3. sitemap.xml
- You include URL in your XML Sitemap.
- No web pages contain links pointing to URL.
- Google requests URL and receives page content.
- Page content contains a link to Jill's home page.



The basic idea here is that Google will treat damn near ANYTHING like a URL and try to request it. Here are a few more examples (none of which I've actually confirmed, but strongly suspect to be true):

1. Google pulls URL-like strings out of inline JavaScript (I've actually confirmed this one. See http://www.seomofo.c...avascript.html)

2. Google indexes URLs mentioned in robots.txt directives (stupid idea, I know, but it's the only explanation I've seen in some cases).

3. Google tries requesting parent directories, e.g., if www.seomofo.com/foo/bar.html is valid, Google will try www.seomofo.com/foo/ ...even if www.seomofo.com/foo/ has never existed, never been mentioned or submitted, and never been linked to.

[Jill]

I agree, SEOmofo. Not sure why this is a big deal either.

[Michael Martinez]

Crawl is a form of value that a link may pass (a concept I documented on SEO Theory as far back as 2007). Whether Google specifically looks for iframe SRC references or just grabs uncommented URLs for discovery or parses all URLs from a page for discovery is trivial with respect to this discussion. The point of the test was to determine if what Jill had observed was verifiable through experimentation: that Google would find and index a Web page solely through the link embedded in an IFRAME element on a page.

The test is not designed to determine if PageRank flows through the link. However, I felt that using a link on the iFramed page to pass a unique anchor expression to another page (in this case HighRankings) would show definitively that Google could only have associated that unique expression with the destination by finding and following the iFramed link.

The results I described are no longer confirmable because discussing the test on the open Web, where the various links and discussions are indexed by Google, populates queries with clutter.

In retrospect, I see that I should not have written "passed value" in the thread title because clearly many people misunderstood what I was referring to (the value of CRAWL) and assumed I was talking about PageRank.

[SEOmofo]

Crawl is a form of value that a link may pass (a concept I documented on SEO Theory as far back as 2007).

I'm not familiar with this concept. Can you cite your sources of this info?

Whether Google specifically looks for iframe SRC references or just grabs uncommented URLs for discovery or parses all URLs from a page for discovery is trivial with respect to this discussion. The point of the test was to determine if what Jill had observed was verifiable through experimentation: that Google would find and index a Web page solely through the link embedded in an IFRAME element on a page.

What you're saying just doesn't make sense.



For clarity, let's assume the following:

- Page1 is a page on your site.
- You create a new page, Page2.
- You add an "iframe" element to the code of Page1, and the iframe's src attribute is the URL of Page2.
- You add a link (i.e., an "a" element) to the code of Page2, and the link's href attribute is the URL of Jill's home page.



Here's a visual representation:


Page1 code
________________________
| . . . . . . . . . . . . . . . . . . . . . . . |
| . . . . . . . . . . . . . . . . . . . . . . . |
| . <iframe src="Page2" /> . . . |
| . . . . . . . . . . . . . . . . . . . . . . . |
| . . . . . . . . . . . . . . . . . . . . . . . |
| _______________________|



Page2 code
___________________________________________
| . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
| . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
| . <a href="HighRankings.com">. . . . . . . . . . . . . . . . . |
| . . . jill whalen says search engines crawl iframes . . . |
| . </a> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .|
| . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
| . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
| __________________________________________|


If all you wanted to do was prove that Google finds and indexes URLs found in iframe src attributes, then the content of Page2 doesn't matter--you could simply wait for Page2 to appear in Google's search results. In this case, the link to Jill's site and the inanchor: query results would be irrelevant and unnecessary.

If you're trying to prove that Page1 has passed any sort of value to Page2, then you've failed, because you haven't shown that Google *only* indexes pages that have received some kind of value. Nor have you shown that Google *only* follows links (in this case, from Page2 to HR.com) if Page2 has value above that of any other orphaned page.

The way this experiment was presented, most people are going to walk away with the assumption that Google treats <iframe src="..."> just like <a href="...">, but that's not what's been shown here. We still have no reason to believe that Page2 has received any value or benefit from Page1--beyond the basic discovery of Page2's URL.

Another way of putting it is: we have no reason to believe that Google has acknowledged any kind of link or relationship between Page1 and Page2.

[JohnORegan]

Thanks, Michael. I knew it to be true since my RatOutYourCompetitor.com site is simply an iframe to a Wufoo form and only the form on the Wufoo subdomain gets indexed now. (Previously the actual domain had been indexed.) I had only recently noticed this as well, so it maybe somewhat newish behavior on Google's part.

Hi Jill, Michael and all reading !

I am just starting in business again as a web designer / online marketer for small business's having spent a number of years doing other things and have found myself referring to topics in your forum on a few occasions so said I might as well join. I'm really just learning the whole SEO business and am being cautious with it. The sites I produce are all for legitimate business's and sole traders and it is my intention to use some third party complimentary advertising on most sites in order to provide further content for site users and further revenue sources for myself and individual site owners. Also I am based in West Cork, Ireland if that's any help to your members !

My query concerning links in iframes is somewhat dealt with in this thread but some specifics are missing.

In my case I would like to . . . .

1. place a file on my own site which contains links to all the sites I produce.

2. place an iframe containing this file in all the sites I produce, so if I have 10 sites they now all linkshare with each other.

so my Query is will the search engines identify the links on each site as external links from that site or will they only register those links as coming from the site on which they are hosted ?

any advise is appreciated and I hope to be of some help to someone on here someday !

Kind Regards,

John O'Regan

[Jill]

ill the search engines identify the links on each site as external links from that site or will they only register those links as coming from the site on which they are hosted ?

They would be links from whatever the URL is that's referenced in the iFrame since that's where the links come from.

[SEOmofo]

1. place a file on my own site which contains links to all the sites I produce.
2. place an iframe containing this file in all the sites I produce, so if I have 10 sites they now all linkshare with each other.

This would definitely not work.

To accomplish what you're trying to do, you would need to handle this on the server-side, for example:

- Store the file of links on your server as a block of HTML code.

- Configure each website (server) to include/import/fetch the contents of that file when each web page is being built. There are several ways you can do this, depending on your server configuration, e.g., Apache includes, PHP cURL, etc.

- The link file's HTML block must be fetched and inserted directly into the HTML code of each web page before being sent to the client (i.e., browser, search engine crawler, etc.).

- Even if you successfully implement this idea, be aware that Google is constantly making progress towards (1) devaluing boilerplate content (e.g., global footers), (2) devaluing inconspicuous links (e.g., small footer links), and (3) detecting and devaluing nepotistic links (e.g., interlinking all your sites, even if they're topically unrelated). So yeah...probably just a waste of time.

...will the search engines identify the links on each site as external links from that site or will they only register those links as coming from the site on which they are hosted ?

Jill answered this already, but I'd like to reiterate, because people seem to have a hard time understanding what an iframe really is.

Search engines do not "look through" iframes and see the content you've referenced. Instead, they treat the iframe file as its own separate page and crawl it like any other URL. Therefore, in the example you're asking about, Google would not look at it like many different pages all linking to each other. Rather, Google would look at it like one page (i.e., the iframe URL) linking to several websites.

And to make matters worse, we don't know if Google thinks the iframe URL is even important. In other words, an iframe tag is not the same as a link tag, so Google might look at your iframe file like it has 0 inbound links. Furthermore, the iframe URL presumably has no other content besides the links.



Bottom line:

Your attempt to "linkshare" would result in Google finding 1 new web page...

...with 0 inbound links pointing to it...

...and no content on it...

...except for a list of links that point to your websites.




P.S. Jill, Michael, et all: by all means, if you have evidence that suggests that Google *does* in fact treat iframe references like links (beyond basic discovery), then please correct me.