Cloudflare made AI crawlers choose a lane. Cloudflare split AI traffic into Search, Agent, and Training lanes, set September 15 defaults for ad-supported pages, and turned Verified bot status from a hall pass into conditional admission. Last week's Patreon crawler piece covered the first move: creator consent left robots.txt and moved into Cloudflare's edge enforcement. The new development is Cloudflare's July 1 bot-policy rollout becoming the general access model for all customers, with separate controls for Search, Agent, and Training traffic, September 15 defaults for ad-supported pages, and a narrower meaning for Verified bots. Cloudflare is making AI crawlers choose a lane. That sounds like dashboard housekeeping until you stare at the mechanism. The web's machine audience is being sorted by declared purpose, monetization context, crawler identity, and downstream content use. Search can still enter by default. Training and Agent traffic hit a harder boundary on pages with ads. Mixed-purpose crawlers inherit the strictest applicable rule. Cloudflare's old consumer-grade control was blunt: block AI bots. The new model splits automated traffic into three headline AI behaviors. Search means crawling that builds indexes or retrieval databases. Agent means user-directed automation visiting a page on behalf of a person. Training means collecting material to train or fine-tune models. Cloudflare's fuller taxonomy also tracks transaction bots, data collection, security testing, SEO, ads verification, social previews, feed fetching, and monitoring. That taxonomy matters because "AI bot" has become a junk drawer. A podcast client, search crawler, personal shopping agent, model-training scraper, uptime monitor, and link-preview fetcher all touch public pages with different social contracts. The old binary let vendors hide behind category fog. The new controls make purpose the unit of admission. Cloudflare says all customers, including Free-tier customers, can now tune AI traffic by Search, Agent, and Training. The operational bite arrives on September 15, 2026. New domains onboarding to Cloudflare will block Training and Agent bots on pages that display ads while allowing Search. Mixed-purpose crawlers that combine Search and Training will be handled by all of their behaviors, using the most restrictive applicable rules. The ad-page default is the nasty little systems detail. Cloudflare treats an ad as evidence that the site owner expected human attention to arrive there. That lets the network enforce different machine rules on monetized pages than on the rest of the site. The page's business model becomes part of the access decision. That is defensible. If an answer engine or training crawler consumes an ad-supported article without sending a reader, it drains the economic circuit that paid for the article. Search still gets default access because it plausibly returns visitors. Agent and Training traffic get fenced because their value path often bypasses the publisher. It also creates a fresh control surface. Ad detection, crawler classification, and default rule inheritance now decide whether automated traffic touches a page. A publisher can opt out, but the default writes the policy for whoever does nothing. Defaults are governance with a friendly settings label. Verified no longer means welcome. The sharpest change is the Verified-bot label. Cloudflare's updated docs define a Verified bot as one that identifies itself honestly and avoids abusing the access that honesty earns. Verification can come through cryptographic Web Bot Auth, a published IP list plus stable user-agent, or reverse DNS. Behavior still matters: robots directives, request rates, and owner preferences remain part of the bargain. Previously, Verified often functioned like default admission. Cloudflare's July language changes the meaning. Verified now makes a bot allowable inside its relevant category. If a site allows Search, a Verified Search crawler can enter. If the same crawler also performs Training and the site blocks Training, the stricter category can block it. Verification becomes passport control, then customs inspection. That solves one abuse pattern and introduces another political problem. The ability to revoke Verified status across Cloudflare's network gives crawler compliance real consequences. It also concentrates a huge amount of judgment inside BotBase, Cloudflare's directory of tracked bots and agents. A private infrastructure company is defining the recognized machine identities of the web. Cloudflare is also testing a use field for Content Signals in robots.txt: immediate, reference, or full. Immediate means interact without storing and reusing. Reference means index, excerpt, and link back. Full means summarize and reproduce. Cloudflare says managed robots.txt users who already publish search=yes,ai-train=no will receive use=reference. That is a weird little grammar for the next web. It gives publishers a way to say that indexing and linking are permitted while full reproduction is not. Like the rest of robots.txt, the signal is a preference until an enforcement layer acts. Cloudflare closes that loop by tracking content use in BotBase and threatening Verified-status loss for bots that abuse the signal. Today, bots that reproduce in full cannot have Verified status. There is a cleaner future hiding inside this messy rollout: portable content-use policy, signed bot identity, transparent crawler categories, and logs a site owner can inspect outside one CDN. The current version is useful because it works at the edge. It is dangerous for the same reason. Cloudflare's transitive-trust proposal points at the agentic web problem before the agentic web has standards sturdy enough to deserve the name. A request may come from a platform that hosts automation for thousands of downstream users. The site may trust the platform, distrust some operator riding through it, or want a use-level promise preserved across proxies. Cloudflare proposes using the existing Forwarded header to carry operator information and a content-use declaration such as Forwarded: for="openai";use="reference". That is elegant plumbing if everyone behaves. It is also exactly where the politics get ugly. The intermediary becomes the party expected to preserve identity, classify purpose, and attach consequences when a downstream operator lies. This is where machine access stops looking like crawling and starts looking like customs brokerage. Every request wants papers. The network wants a known operator. The publisher wants a use limit. The crawler wants enough ambiguity to keep access broad. The CDN becomes the checkpoint because the origin server cannot maintain the classifier alone. Cloudflare's move is directionally correct. Voluntary crawler etiquette collapsed once model training and answer interfaces broke the referral exchange. Publishers need enforceable distinctions between search, user-directed automation, training, summaries, feed fetching, ads verification, and outright scraping. Pretending all automated access has the same bargain is how extractive platforms get a free buffet while everyone else pays bandwidth and labor. The replacement cannot be a private black box with a nicer UI. Publishers need exportable policy, auditable decisions, appeal paths, portable bot identity, and logs that survive a CDN migration. Bot operators need purpose-specific crawlers instead of one mixed-use identity dragging search, RAG, training, and product features through the same pipe. Regulators should understand that crawler defaults are market structure now. Search access, agent access, and training access decide who receives traffic, who receives data, and who gets to build on the public web without permission. The old robots.txt bargain was weak because it lacked enforcement. Its virtues were legibility, portability, and origin control. Cloudflare is supplying enforcement. The fight now is making sure the new machine border keeps the old web's legibility before the dashboard becomes the constitution.

Promptwatch review 2026: features, pros & cons. Jul 25, 2026 This Promptwatch review evaluates the tool's features and ability to track brand presence across ChatGPT, Gemini, Claude, and Perplexity. While Promptwatch tracks how brands appear across ChatGPT, Claude, Gemini, and Perplexity, then benchmarks that visibility against competitors. Aside from other highlighted limitations by users, the price points may not be suitable for SMBs, freelancers, and agencies seeking a more accessible all-in-one solution. Hence, alternatives like Rankpilot have become a go-to option offering stronger rank tracking, audits, content optimisation, and a more budget-conscious pricing point. What is Promptwatch. Promptwatch is an AI search visibility and Generative Engine Optimisation (GEO) tool that tracks brand mentions, citations, and visibility across AI models like ChatGPT, Claude, Gemini, Perplexity, and Google AI Overviews. Promptwatch targets marketing teams, SEO professionals, and agencies who need to understand AI-driven search behaviour beyond traditional Google rankings. Its core value proposition centres on tracking prompts, analysing citations, and identifying content gaps across multiple AI platforms simultaneously. Core features of Promptwatch. * Prompt tracking: It monitors users' prompts and flags when AI engines mention the brand in their responses. * Citations analysis: The tool shows which sources AI engines cite when discussing a brand, including third-party mentions on Reddit and YouTube. * Agent analytics: Promptwatch tracks how AI agents and assistants interact with a brand's content, available on Professional and Business tiers. * Content agent: It generates AEO (answer engine optimisation) articles, though the number of articles is limited by tiers. * Sentiment analysis and content gap analysis: The tool evaluates how AI engines characterise a brand and identifies topics where visibility is weak * Integrations: Promptwatch also connects with Cloudflare, Fastly, Vercel, and other hosting/CDN providers. Pros and cons of Promptwatch. Pros. * It tracks AI visibility across ChatGPT, Claude, Gemini, and Perplexity. * It has a clean, intuitive interface (though some users reported otherwise) * It offers a free trial * It has a built-in article generation feature (5/month, 15/month, or 30/month depending on the pricing) * It offers country, state, and city-level tracking. Cons. * Users shared that the UI is complex and has a steep learning curve for new users * Reported UX bugs with limited escalation paths * Starting at $95/month, may be expensive for small businesses * Content generation quality rated as weak by users * The reporting depth and accuracy consistency require improvement * According to users, the Answer Gap report can be difficult for non-SEO professionals to interpret Promptwatch pricing. Promptwatch offers three primary subscription tiers alongside a 7-day free trial. For small businesses that don't need deep AI-driven visibility analytics or need more consistent content to support their growth, a $95 starter plan may offer less value than an all-in-one SEO/GEO tool like Rankpilot at $59/month. Promptwatch user testimonials. Positive reviews. Users highlight that the tool is very elaborate, giving very useful recommendations on how to improve visibility within AI tools. A user notes the content gap feature "showed us we weren't showing up in responses because we were missing specific topics" Negative review. On the other hand, a 60-day hands-on evaluation by Generate More (across 8 other SaaS clients) shared that: "Promptwatch has more UX errors and bugs than other solutions. There is currently no way to escalate and resolve them. We're seeing an increasing amount of bugs in the user interface that can't be dismissed or flagged. This means some core reports we share with customers are faulty." Other user testimonials share that the generated content/articles are weak. Promptwatch vs. Alternatives. Final verdict. Promptwatch offers excellent AI visibility tracking across multiple AI models, with unique features such as crawler log analysis and built-in content generation, though reported UI bugs and weak content warrant caution. For teams needing broad AI model coverage and technical crawl insights, Promptwatch justifies its mid-tier pricing. However, small businesses without dedicated SEO resources may find the learning curve and cost too much. Hence, a top Promptwatch alternative you can opt for is Rankpilot, which offers a more affordable all-in-one SEO/GEO suite including content automation and AI visibility tracking at $59/month.

Cloudflare Cache Response Rules: closing the post-origin performance gap. New Cloudflare rules let you fix cache-breaking headers like Set-Cookie after origin response, no code changes required. Performance boost. Cloudflare cache removes cookie header from origin. By Andres SEO Expert. Key takeaways. * Cache Response Rules run after origin reply but before cache write, fixing headers that break caching. * Strip Set-Cookie, ETag, Last-Modified; modify Cache-Control; manage cache tags without origin changes. * Available on all Cloudflare plans; complements existing Cache Rules for end-to-end cache control. Cloudflare's Cache Response Rules: the missing half of CDN caching control. Cloudflare has released Cache Response Rules, a new rule type that operates in the critical moment after an origin server responds but before Cloudflare writes the response to cache. This addresses a long-standing pain point: cache-breaking headers like Set-Cookie or misguided Cache-Control directives that originate from the server and undermine cache efficiency. With Cache Response Rules, organizations can strip, modify, or add headers and directives at this exact juncture, requiring zero changes to origin infrastructure. Table of contents. The core breakdown: how Cache Response Rules work. Cache Response Rules fill a specific timing gap. Traditional Cache Rules operate during the request phase, before Cloudflare contacts the origin. They decide whether to cache an object and under what key, using only request-time information. But many cache eligibility problems only become visible after the origin responds. The post-origin problem. When an origin returns a response, headers like Set-Cookie, incorrect Cache-Control directives, or overly aggressive ETags can make an otherwise cacheable asset ineligible for caching. This forces every subsequent request to bypass the cache and hit the origin, degrading performance and increasing costs. Cache Response Rules intercept these responses before Cloudflare's cache evaluates them, allowing teams to: * Strip headers like Set-Cookie, ETag, and Last-Modified that block caching. * Modify Cache-Control directives including max-age, s-maxage, stale-while-revalidate, and no-cache, with an option to apply changes only to Cloudflare's view (cloudflare_only). * Manage cache tags by setting, adding, or removing tags used for purge-by-tag operations, including translating tags from other CDN formats. Complementary, not replacement. According to Cloudflare's official announcement, Cache Response Rules do not replace Cache Rules. They add a response-phase layer that answers a different question: given the origin's response, should Andres SEO Expert LLC adjust how Andres SEO Expert LLC cache it? Cache Rules decide whether, what, and how to cache on the request. Cache Response Rules get the final word on the how and whether after seeing the response. When both rule types conflict, the response rule wins. Real-World use cases. Common scenarios include stripping Set-Cookie from static assets like JS, CSS, and fonts; overriding no-cache directives on known-static paths; and translating Surrogate-Keys headers from another CDN into Cloudflare's Cache-Tag format during migrations. Cloudflare provides example expressions and actions in their documentation. Strategic analysis: closing the gap in CDN performance. The introduction of Cache Response Rules represents a significant step toward true end-to-end cache control. Previously, teams had three workarounds for response-time cache issues: modify origin code, deploy a Cloudflare Worker to rewrite responses, or accept a lower cache hit ratio. Each option carried costs in engineering time, added latency, or ongoing operational expense. Cache Response Rules provide a fourth, far more efficient path. The official Cloudflare glossary defines fundamental cache concepts like cache hit, cache miss, and TTL, underscoring the platform's focus on caching performance. But the glossary also notes that Enterprise customers can adjust proxy read timeout using cache rules, hinting at deeper configuration possibilities for advanced users. Cache Response Rules extend this configurability to the response phase, giving all plan levels more granular control. Third-party ecosystem tools are already embracing the new capabilities. For instance, the Cache Tags for Cloudflare WordPress plugin, developed by Per Soderlind, demonstrates practical use of the Cache-Tag header and purge-by-tag API. The plugin emits Cache-Tag headers for posts and taxonomy terms, and performs batched purges respecting Cloudflare's 30-tag-per-call limit. According to the plugin documentation, the Cache-Tag header and purge-by-tag functionality are available on all Cloudflare plans, with only purge API rate limits scaling by plan tier. This shows that Cache Response Rules' cache tag management action can directly integrate with such existing workflows, making CDN migrations smoother and tag-based purging more accessible. For performance professionals, the implications are clear. Cache Response Rules eliminate the need to chase misconfigured headers through origin code, reducing time-to-fix from weeks (in large teams with separate origin and CDN management) to minutes. They also enable hybrid cache strategies where Cloudflare can hold assets longer than the browser, decoupling TTLs without complex origin setup. This is particularly valuable for sites with dynamic-static hybrid content, like ecommerce platforms where session cookies mistakenly attach to product images. However, teams must apply these rules judiciously. Stripping Set-Cookie from responses that genuinely require it can break functionality. Cloudflare advises scoping rules by path or extension, and being honest about what content is truly static. Conclusion: A new standard for Cache Control. Cache Response Rules close a critical gap in CDN caching architecture. By allowing modifications at the precise moment when response headers are about to affect cache eligibility, Cloudflare empowers organizations to reclaim lost cache efficiency without touching origin infrastructure. This innovation reinforces the principle that caching decisions should be as flexible and layered as the applications they serve. For teams looking to maximize CDN performance, integrating Cache Response Rules into your caching strategy is a logical next step. Andres SEO Expert helps businesses implement advanced caching configurations and performance engineering. Explore its WordPress Speed Engineering and Managed WordPress Cloud Hosting services to optimize your site's delivery. For a tailored caching audit, connect with Andres or learn more about Andres SEO Expert. Frequently asked questions. What are Cloudflare Cache Response Rules? How do Cache Response Rules differ from traditional Cache Rules? When should I use Cache Response Rules? Can Cache Response Rules replace Cloudflare Workers for cache control? What are the risks of using Cache Response Rules? How do Cache Response Rules work with cache tags? How do Cache Response Rules improve performance? July 23, 2026