{"id":4099,"date":"2023-10-17T12:00:36","date_gmt":"2023-10-17T17:00:36","guid":{"rendered":"https:\/\/www.innovaxisinc.com\/?p=4099"},"modified":"2023-12-04T12:20:13","modified_gmt":"2023-12-04T18:20:13","slug":"demystify-yoasts-advanced-crawl-optimization","status":"publish","type":"post","link":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization","title":{"rendered":"Demystify Yoast\u2019s Advanced Crawl Optimization"},"content":{"rendered":"<p class=\"c-image\"><img loading=\"lazy\" decoding=\"async\" title=\"Crawl optimization\" src=\"\/wp-content\/uploads\/2023\/10\/crawl-optimization.png\" alt=\"Crawl optimization graphic\" width=\"300\" height=\"225\" \/><\/p>\n<p>You might be familiar with Yoast, a popular and widely used WordPress plugin to fine-tune your SEO needs, but have you ever explored their Advanced Crawl Optimization settings?<\/p>\n<p>These settings help prevent URLs and links with no SEO value from being crawled, reducing the electricity usage and carbon footprint of your website. WordPress automatically adds these URLs and links to the head section of your site pages. You may not be actively using this metadata and content for your website so Yoast gives you the option to remove them and improve your website\u2019s crawl budget.<\/p>\n<h2>What is Crawl Budget?<\/h2>\n<p>For websites with many URLs and pages, crawl budget can become an issue.<\/p>\n<p>Crawl budget is determined by a <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget#:~:text=Crawl%20capacity%20limit,-Googlebot%20wants%20to&amp;text=This%20is%20calculated%20to%20provide,can%20be%20used%20to%20crawl.\" target=\"_blank\" rel=\"noopener\">multitude of factors<\/a> such as the:<\/p>\n<ul class=\"main\">\n<li>Size of your website<\/li>\n<li>&#8220;Health\u201d of your website (does the crawler encounter any errors?)<\/li>\n<li>Popularity of your website (number of site visits &amp; links pointing to you)<\/li>\n<\/ul>\n<p>The pages on a site with low crawl budget will take longer to be crawled by Google. However, this is only a problem for large websites with at least 10,000 unique pages that are being updated often. If this is the case, these settings help crawlers ignore the unnecessary pages and focus on crawling more important content.<\/p>\n<p>To help you decide which settings will work best for your site, here is an overview:<\/p>\n<h3>Remove unwanted metadata<\/h3>\n<p>These settings add links and content to the site\u2019s &lt;head&gt; and HTTP headers.<\/p>\n<h3>Remove shortlinks<\/h3>\n<p>Remove links to WordPress&#8217; internal &#8216;shortlink&#8217; URLs for your posts.<\/p>\n<h3>Remove REST API links<\/h3>\n<p>Remove links to the location of your site\u2019s REST API endpoints.<\/p>\n<ul class=\"main\">\n<li>The REST API is a developer-oriented feature that allows applications to retrieve data from or modify WordPress\u2019 database. These applications could be plugins, so turning off this feature can break website functionality.<\/li>\n<\/ul>\n<h3>Remove RSD \/ WLW links<\/h3>\n<p>Remove links used by external systems for publishing content to your blog.<\/p>\n<ul class=\"main\">\n<li>We recommend turning this setting on. Really Simple Discovery (RSD) allows external applications to communicate and modify WordPress (ex: Posting content through the WordPress mobile app). However, this functionality has been replaced by the REST API and can be a security risk. Windows Live Writer (WLW) is also a discontinued application.<\/li>\n<\/ul>\n<h3>Remove oEmbed links<\/h3>\n<p>Remove links used for embedding your content on other sites.<\/p>\n<h3>Remove generator tag<\/h3>\n<p>Remove information about the plugins and software used by your site.<\/p>\n<p>Keeping this tag visible can be a security threat.<\/p>\n<h3>Pingback HTTP header<\/h3>\n<p>Remove links which allow others sites to \u2018ping\u2019 yours when they link to you.<\/p>\n<ul class=\"main\">\n<li>Trackbacks are used to notify other blogs when you have linked to them. Pingbacks are an automatic version of trackbacks. If a link to an external site is made, the external site can choose to publish the trackback\/pingback in their blog comment section. This setting is mainly useful for notifications when others have linked to you.<\/li>\n<\/ul>\n<h3>Remove powered by HTTP header<\/h3>\n<p>Remove information about the plugins and software used by your site.<\/p>\n<h3>Remove unused resources<\/h3>\n<p>WordPress loads lots of resources, some of which your site might not need. If you\u2019re not using these, removing them can speed up your pages and save resources.<\/p>\n<h3>Remove emoji scripts<\/h3>\n<p>Remove JavaScript used for converting emoji characters in older browsers.<\/p>\n<h3>Remove WP-JSON API<\/h3>\n<p>Add a \u2018disallow\u2019 rule to your robots.txt file to prevent crawling of WordPress&#8217; JSON API endpoints.<\/p>\n<ul class=\"main\">\n<li>The REST API is a developer-oriented feature that allows applications to retrieve data from or modify WordPress\u2019 database. These applications could be plugins, so Google needs to crawl the endpoints to properly render and index pages where plugins are used.<\/li>\n<\/ul>\n<p>Now you might be wondering, can smaller websites still benefit from utilizing these settings? Since unused JavaScript links and other content are being removed from the &lt;head&gt; section, we hypothesized that a website may load faster, as minifying and removing unnecessary code is a proven method in improving performance. However, after testing these settings on our site and checking site speed through Google PageSpeed Insights, there was unfortunately no difference in scores.<\/p>\n<p>Beside these settings, Yoast can also prevent RSS feed pages from being crawled. RSS feed pages provide metadata for website updates, such as new blogs or site comments. RSS or Really Simple Syndication allows for publishers to distribute content to subscribers who use feed readers to aggregate their news and information. Feeds are XML or Atom files with URLs that are meant for machines to read, providing information about the content including titles, summaries, and links.<\/p>\n<p>There are <a href=\"https:\/\/www.hostinger.com\/tutorials\/wordpress-rss-feed\" target=\"_blank\" rel=\"noopener\">benefits to using RSS feeds<\/a> for website owners, but if you aren\u2019t using these, Yoast can remove them as well and help improve crawl budget.<\/p>\n<h2>More Strategies for Improving the Performance of Your B2B Website<\/h2>\n<p>At Innovaxis, our digital marketing experts can help you with strategies for maximizing the speed and performance of <a href=\"\/de\/marketing-strategy-b2b\/website-content-strategy\/\">your B2B website<\/a> as part of a comprehensive marketing program.<\/p>\n<p>Reach out to one of our team members today to get started.<\/p>\n<div style=\"text-align: center;\"><a class=\"btn btn-primary\" href=\"\/de\/contact-innovaxis-b2b-marketing-consulting\/\"><strong>Learn more about B2B website best practices<\/strong><\/a><\/div>","protected":false},"excerpt":{"rendered":"<p class=\"c-image\"><img loading=\"lazy\" decoding=\"async\" title=\"Crawl optimization\" src=\"\/wp-content\/uploads\/2023\/10\/crawl-optimization.png\" alt=\"Crawl optimization graphic\" width=\"300\" height=\"225\" \/><\/p>\n<p>You might be familiar with Yoast, a popular and widely used WordPress plugin to fine-tune your SEO needs, but have you ever explored their Advanced Crawl Optimization settings?<\/p>\n<p>These settings help prevent URLs and links with no SEO value from being crawled, reducing the electricity usage and carbon footprint of your website. WordPress automatically adds these URLs and links to the head section of your site pages. You may not be actively using this metadata and content for your website so Yoast gives you the option to remove them and improve your website\u2019s crawl budget.<\/p>\n<h2>What is Crawl Budget?<\/h2>\n<p>For websites with many URLs and pages, crawl budget can become an issue.<\/p>&#8230; <a class=\"view-article\" href=\"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization\">Continue Reading >><\/a>","protected":false},"author":12,"featured_media":4127,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[220,218,217,219],"coauthors":[],"class_list":["post-4099","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-advanced-crawl-optimization","tag-b2b-best-practices","tag-crawl-budget","tag-yoast-seo"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.6 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Demystifying Yoast Crawl Optimization Options | Innovaxis Marketing Consulting<\/title>\n<meta name=\"description\" content=\"Learn about Yoast\u2019s Advanced Crawl Optimization settings and how they can benefit crawl budget for larger websites\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Demystifying Yoast Crawl Optimization Options | Innovaxis Marketing Consulting\" \/>\n<meta property=\"og:description\" content=\"Learn about Yoast\u2019s Advanced Crawl Optimization settings and how they can benefit crawl budget for larger websites\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization\" \/>\n<meta property=\"og:site_name\" content=\"Innovaxis B2B Marketing\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-17T17:00:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-04T18:20:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.innovaxisinc.com\/wp-content\/uploads\/2023\/10\/crawl-optimization.png\" \/>\n\t<meta property=\"og:image:width\" content=\"300\" \/>\n\t<meta property=\"og:image:height\" content=\"225\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Billy Altantulkhuur\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"Billy Altantulkhuur\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"4\u00a0Minuten\" \/>\n\t<meta name=\"twitter:label3\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data3\" content=\"Billy Altantulkhuur\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Demystifying Yoast Crawl Optimization Options | Innovaxis Marketing Consulting","description":"Learn about Yoast\u2019s Advanced Crawl Optimization settings and how they can benefit crawl budget for larger websites","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization","og_locale":"de_DE","og_type":"article","og_title":"Demystifying Yoast Crawl Optimization Options | Innovaxis Marketing Consulting","og_description":"Learn about Yoast\u2019s Advanced Crawl Optimization settings and how they can benefit crawl budget for larger websites","og_url":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization","og_site_name":"Innovaxis B2B Marketing","article_published_time":"2023-10-17T17:00:36+00:00","article_modified_time":"2023-12-04T18:20:13+00:00","og_image":[{"width":300,"height":225,"url":"https:\/\/www.innovaxisinc.com\/wp-content\/uploads\/2023\/10\/crawl-optimization.png","type":"image\/png"}],"author":"Billy Altantulkhuur","twitter_card":"summary_large_image","twitter_misc":{"Verfasst von":"Billy Altantulkhuur","Gesch\u00e4tzte Lesezeit":"4\u00a0Minuten","Written by":"Billy Altantulkhuur"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization#article","isPartOf":{"@id":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization"},"author":{"name":"Billy Altantulkhuur","@id":"https:\/\/www.innovaxisinc.com\/de\/#\/schema\/person\/1b46ac4d5605c943ed4cd09764f0df44"},"headline":"Demystify Yoast\u2019s Advanced Crawl Optimization","datePublished":"2023-10-17T17:00:36+00:00","dateModified":"2023-12-04T18:20:13+00:00","mainEntityOfPage":{"@id":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization"},"wordCount":845,"image":{"@id":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization#primaryimage"},"thumbnailUrl":"https:\/\/www.innovaxisinc.com\/wp-content\/uploads\/2023\/10\/crawl-optimization.png","keywords":["advanced crawl optimization","b2b best practices","crawl budget","yoast seo"],"articleSection":["Uncategorized"],"inLanguage":"de"},{"@type":"WebPage","@id":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization","url":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization","name":"Demystifying Yoast Crawl Optimization Options | Innovaxis Marketing Consulting","isPartOf":{"@id":"https:\/\/www.innovaxisinc.com\/de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization#primaryimage"},"image":{"@id":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization#primaryimage"},"thumbnailUrl":"https:\/\/www.innovaxisinc.com\/wp-content\/uploads\/2023\/10\/crawl-optimization.png","datePublished":"2023-10-17T17:00:36+00:00","dateModified":"2023-12-04T18:20:13+00:00","author":{"@id":"https:\/\/www.innovaxisinc.com\/de\/#\/schema\/person\/1b46ac4d5605c943ed4cd09764f0df44"},"description":"Learn about Yoast\u2019s Advanced Crawl Optimization settings and how they can benefit crawl budget for larger websites","breadcrumb":{"@id":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization#breadcrumb"},"inLanguage":"de","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization"]}]},{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization#primaryimage","url":"https:\/\/www.innovaxisinc.com\/wp-content\/uploads\/2023\/10\/crawl-optimization.png","contentUrl":"https:\/\/www.innovaxisinc.com\/wp-content\/uploads\/2023\/10\/crawl-optimization.png","width":300,"height":225},{"@type":"BreadcrumbList","@id":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/demystify-yoasts-advanced-crawl-optimization#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.innovaxisinc.com\/"},{"@type":"ListItem","position":2,"name":"Demystify Yoast\u2019s Advanced Crawl Optimization"}]},{"@type":"WebSite","@id":"https:\/\/www.innovaxisinc.com\/de\/#website","url":"https:\/\/www.innovaxisinc.com\/de\/","name":"Innovaxis B2B Marketing","description":"B2B Marketing Results","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.innovaxisinc.com\/de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"de"},{"@type":"Person","@id":"https:\/\/www.innovaxisinc.com\/de\/#\/schema\/person\/1b46ac4d5605c943ed4cd09764f0df44","name":"Billy Altantulkhuur","image":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/secure.gravatar.com\/avatar\/8a6239effa698579fbedf1b90a7fd6c800205e4748ae2f9059fb7714e8c7ad53?s=96&d=mm&r=gc32dec12579bbeea7a40444807f660b5","url":"https:\/\/secure.gravatar.com\/avatar\/8a6239effa698579fbedf1b90a7fd6c800205e4748ae2f9059fb7714e8c7ad53?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8a6239effa698579fbedf1b90a7fd6c800205e4748ae2f9059fb7714e8c7ad53?s=96&d=mm&r=g","caption":"Billy Altantulkhuur"},"url":"https:\/\/www.innovaxisinc.com\/de\/b2b-marketing-strategy-blog\/author\/baltantulkhuur"}]}},"_links":{"self":[{"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/posts\/4099","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/comments?post=4099"}],"version-history":[{"count":0,"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/posts\/4099\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/media\/4127"}],"wp:attachment":[{"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/media?parent=4099"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/categories?post=4099"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/tags?post=4099"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.innovaxisinc.com\/de\/wp-json\/wp\/v2\/coauthors?post=4099"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}