Is your website doing well for SEO? You probably know that a search-engine-optimized website is important for your online business. In this Screaming Frog SEO Spider review, I am going to show you how to use Screaming Frog to optimize your SEO for better search results.
Users from the search are more likely to convert to a paid customer compared to other channels because they have an intent. If you worry about your site may have SEO issues, then you should try Screaming Frog, one of my all-time favorite SEO tools.
Is Screaming Frog worth the money? I’m going to show you ten ways you can use it to improve your SEO and increase your traffic to get more deals and sales.
What is Screaming Frog?
Screaming frog is website crawler, that crawls websites’ URLs and collects data, so you can analyze the onsite SEO. It basically works like Google and other search engines use to collect data from the web.
How does it work?
You can start your project with a seed URL or a list of URLs. For analyzing your website or your client’s site, you simply input the domain and start the crawl. If needed, you can also change crawl settings in Configuration.
When the crawl starts, first it collects all of the information on that first page and then it finds all of the links on that page to and it follows every single link and collects all the information from every one of those pages.
When it finishes, the program will display all the information including links, status code, protocol, meta title, description, H tags, etc. you can go all some of this information to make improvements on your website.
Screaming Frog Pricing Plan
Screaming Frog provides both a free and a paid version for Mac, Windows or Linux. For the free version, you can use to crawl up to 500 URLs. 500 URLs does not mean 500 pages. It includes all the images, CSS JavaScript, and any other files were linked on your website.
If you have a small site, the free version is good enough. If you have a large website with thousands of pages, you need to upgrade to the paid version to crawl pages in one go. The paid version is 149 pounds per year, which is almost 200 in dollars.
Compared to other tools that do a similar thing, the price is good enough. Deepcrawl online web crawler starts at $79/month.
Find non-secured pages
For non-secure pages, the Chrome web browser will not show a green secure bar at the address bar, which may affect credibility.
If your site is migrated from HTTP to HTTPS, your site probably has some non-secured pages. Screaming Frog allows you to export a sheet with all the details about all the non-secured pages. Just Go to Reports and select Insecure content, also known as mixed content.
Non-secure pages are very easy to fix. It is mainly because of mixed content. Just make sure all links are HTTPS instead of HTTP.
Make or check a sitemap
If you have a rather large website, a sitemap is necessary to let search engine bots better crawl your site. Screaming Frog can help you create a sitemap effortlessly.
At the top menu, choose Sitemap and Create XML sitemap, and you can just leave default settings and just click Next to create an XML sitemap. When it is done, you can upload or update this to your site.
If you already have a sitemap, it’s always a good practice to make sure that there are no errors. Your sitemap may have broken URLs or redirected URLs.
So to make your sitemap clean, in the top menu, change the Mode from Spider mode to List and upload your sitemap to crawl all the links in the sitemap.
Most of the other sitemap generator apps cannot crawl websites that are largely written in JavaScript. Screaming Frog can help you. You just need to go to Screaming Frog > Configuration > Spider > Rendering. From there, select JavaScript instead of Text Only.
Check broken links on your site
You can simply put your own website’s URL to start crawling. When the crawl finishes, you click on the Response Code is sort the list at your preference. In this case, we want to show all the URLs with a 404 response code at first.
You should fix URLs with response code 4XX or 5XX. 200 means okay. If you export all these problem URLs to an Excel file. Just click the Bulk Export > Response Code > Client Error (4XX) inlinks.
After finding the broken links, these are the two things you need to do. You can check the bad link itself and the pages that link to it. You can click on the bad link and click the Inlink on the bottom menu to show pages that link to this bad link.
Another great way of using Screaming Frog is to use it for broken link building. You also upload a list of links to check if there are any broken external links or internal links.
Optimize your page titles
Google Search Console will show pages with missing title tags, duplicate title tags, long/short title tags, and non-informative title tags in Search Appearance > HTML Improvement if Google bots detect them.
Use Screaming Frog, you can also identify these problems after you’ve crawled any site. It gives you all of the information here, including the length of the title, and not only the length in characters, but also the length in pixels, which Is really the thing you want to go by when you’re looking at optimizing your Google search results.
At some point, if your title gets too long, it’s going to get cut off in Google search results.
You should really optimize your titles because that’s the main factor if someone decides to click on your page.
Identify pages with thin content
Many case studies have proved that long-form content tends to rank better from a search engine. Therefore, you need to find pages with thin content and optimize them.
For example, let’s find pages with less than 300 words. Just scroll to the right to find the Word Count column and click on it to sort words from small to a large number.
Find links that are being redirected
Redirects aren’t as bad as broken links but redirect chain is an issue. Screaming Frog has a special report for redirect chain. You can simply click Reports and select Redirect Chains.
In addition, It’s still a good idea to avoid using redirects, because too many redirects may slow down your site performance and make it hard for bots to find the right content.
To find redirects, you just go to bulk export and go to response codes and then choose redirection 300 in links. This will give you a spreadsheet with all the redirected URLs and the original URLs.
Find slow loading pages
Site load speed is certainly a ranking factor and it will affect your conversion rate. Screaming Frog cannot tell you how long a page actually takes to load, but it can tell you if there is a problem with your server.
To check that, you can click the Response time column. if you get a slow server, sometimes it could take a few seconds to respond to a request, which is a really long time.
Check internal links on your site
Better onsite linking can make a big difference in your SEO effort. First, you should check the Crawl depth Inlinks column. Level: zero, means your page where the crawl began. The level goes up as each link it has to follow. So each page your homepage linking to is level one.
For pages with crawl depth higher than 3 (at least 4), so should consider making more pages link to them. The best practice is to make every page less than 3 clicks away.
Second, you should check the Inlinks column. For really important pages, you may want to add more links to them.
Other useful features
Check how your website is going to look in Google’s search results. To do that, click on one of the results and click the SERP snippet at the bottom menu. This will show you how your pages are displayed in the search results.
To see more data, Screaming Frog allows you to connect multiple APIs including Google Analytics, Google Search Console, PageSpeed Insights, Moz, Ahrefs, and Majestic.
Other reports are also helpful. Colonial errors, Hreflang error report, which is helpful for multiple language websites, Rel next prev errors for pagination SEO, and Orphan pages report for optimizing internal linking.
Conclusion
For regular users, the free version of Screaming Frog is enough. If you meet the crawl limit, you can run it multiple times and crawl it once in a while to monitor if there are any problems with your site.
Even if you own a very big site with thousands of pages, I don’t recommend you buy the Pro version of Screaming Frog, because you can use other tools that have more features or is cheaper.
For site owners, I recommend you Ahrefs, SEMrush, or Link Assistant Site Auditor. These tools give a more clear clue about what you should fix but cost you more.
If you do not have the budget, then I recommend you use Xunx, a free open source software for Windows, to find broken links. Then you can use Google Sheets to scrape content from your site.
For SEO specialists/agents or those who need to manage multiple websites, I highly recommend you use the paid version of Screaming Frog.
Hello I am so thrilled I found your blog page, I really found you by mistake, while I was researching on Bing for something else, Anyways I am here now and would just like to say thanks for a fantastic post and a all round entertaining blog (I also love the theme/design), I don’t have time to read it all at the minute but I have book-marked it and also added your RSS feeds, so when I have time I will be back to read more, Please do keep up the awesome work.
Good day! This is my 1st comment here so I just wanted to give a quick shout out and say I really enjoy reading your blog posts. Can you recommend any other blogs/websites/forums that cover the same subjects? Thanks a lot!