Rise in Crawled – currently not indexed, preparing websites for agents, and how I’m using Antigravity.

Search News You Can Use May 12, 2026

Here are the topics I have been discussing with my clients recently.

1. A rise in crawled - currently not indexed pages

There has been significant chatter on social media about Google showing more pages in the Page indexing section of GSC that are marked as “crawled - currently not indexed.” These are pages that Google has crawled and has decided not to index.

Almost every site I monitor has seen a rise in crawled-currently-not indexed pages since March of 2026.

Note: I put these images in a folder and asked Antigravity to combine them into one image for me. It wrote Python to do this. 🦾

I’d highly recommend setting aside some time to read this document from Microsoft Bing called the Evolving role of the index: From ranking pages to supporting answers.

They say, “Search indexing was built to help humans decide what to read. Grounding indexing is being built to help AI systems decide what to say.”

I cannot prove this, but I believe that Google is being more selective in what they index as the index is shifting to primarily exist as grounding for AI. There are still some websites that people will want to visit, but if you are creating content that many others have already written on, there's only so much need for this to be in the index.

Check if your crawled-currently not indexed pages are commodity content

I found it interesting though that crawled-currently not indexed pages were discussed at the Google Search Central event which I attended in Toronto. They said that crawled - currently not indexed pages could be caused by technical issues, or “we looked and found it not to be good.”

There was also a lot of discussion at this event about commodity content.

I snapped this photo. I feel like Danny is judging me. 😂

I’d encourage you to try this exercise.

Go to GSC → Indexing → Pages and click on “Crawled - currently not indexed”

Some of these pages make sense to not be in the index, especially if they are feed pages, pagination or non-canonical variations such as pages with or without a trailing slash in the url.

Find some urls that you really do want to have in the index. Open them up in Chrome and open up Gemini in the sidebar and use this prompt:

is this commodity content? Is it likely many others on the web have a page with similar content?

 

You can then ask Gemini for ideas to improve it so that it is uniquely valuable and worthy of indexing.

2. Preparing for the agentic web

Google CEO Sundar Pichai recently stated that “Search will be an agent manager in which you’re doing a lot of things.”

“The agentic web” is an internet where autonomous AI agents, rather than humans, navigate websites, use tools and communicate with other agents to help humans achieve goals.

What does this mean for your website?

Try this prompt in your favourite LLM:

I've been reading about the web becoming agentic. Help me understand what this means for my business. My website is [url].

Our role as SEOs will shift from working to be surfaced by search engines, to instead, ensuring that AI agents choose our business to help their user complete a task.

We will also need to help determine how to turn those agentic visits into business opportunities. I do think there is so much opportunity here. At the same time, there is no playbook for us to follow.

This week Google published an important read on making our websites agent friendly. Agents currently have 3 ways of assessing web pages - via screenshots, via the HTML on the page and by using something called the accessibility tree. The accessibility tree is essentially a view of what’s important on a page, especially when it comes to taking action. It was designed for tools like screen readers, but also helps agents understand pages.

I used Antigravity to build a tool to evaluate a page in the eyes of everything that could be hindering agents from seeing or using your site that was recommended in Google’s blog post:

Test your site on my Agent-Readiness tool.

Once you get your results it will spit out a prompt you can use in your favourite LLM to help you determine whether the issues raised are worth working on.

Testing agentic Claude on client sites

In one of my client calls recently, we opened up Claude in the Chrome extension that puts it in the Chrome sidebar. Claude can currently take agentic action on a page. Gemini cannot, but I expect we will hear more about Gemini taking agentic action soon.

This page has a form which communicated information via an API and then provided an answer and a link to purchase, which is how this business monetizes their website. We asked Claude to get a quote. It asked us a few questions that were required by the form. It didn’t type on the screen, but instead it figured out what it needed to do to connect with the API. Very quickly it came back to us with an answer, along with the purchase link.

The client has now worked to implement WebMCP for this page. This will make it even easier for browser based agents to navigate and use. (Here is an excellent guide on implementing WebMCP published by Suganthan Mohanadasan.)

In another client call, we used Claude in the browser to help us choose a product and then add it to our cart. Claude recognized we had many options to choose from and shared those with us. In this case Claude failed. I prompted, "Were there technical issues with this site that could be improved so you could accomplish this task better?" Claude provided us with a big list of improvements including improving form dropdowns, fixing a broken link on the page and improving some other JavaScript driven actions that were difficult for the AI to execute.

This site is not currently rushing to implement WebMCP, but has joined the UCP waitlist as UCP will probably be more important for them than WebMCP.

I expect, but I don’t know for certain, that pages that are easy for agents to use to help accomplish goals will be more sought after by those agents. We don’t know, however, just how this will happen. Will there be a directory of WebMCP enabled pages? It makes sense to me that this may just be a part of the information Google stores on a page when they crawl it - whether it is agent-friendly. We will see!

3. Learning to Use Google’s Antigravity

I know it’s all the rage right now to use Claude Code, Claude Cowork or OpenAI’s Codex. I still really, really love Antigravity. I use it all day long.

I do not know this for sure, but I could see a world where we have a connection to Antigravity via the Chrome sidebar and other places where we can use Gemini. I think it is possible that Antigravity and Gemini are deeply embedded in everything that we do.

Here is what I did with Antigravity this week:

Gave it a bunch of screenshots and asked it to put them into one image for me.

Made a tool that connects with the GA4 MCP and gives me a daily update on which pages on my site people are engaging with. It opens a browser and reads the comments and reactions left on posts in the Search Bar. Then it summarizes what people have found helpful using Gemini 3 Flash. Over time, I hope to add agentic workflows that analyze what my audience is finding helpful and use this to guide me on better helping Search Bar members.

Built a system that takes stories I’ve marked as important and adds them to a Google doc along with tags for which clients I want to share those stories with. Now I can say to Antigravity, “Make the monthly newsletter for [clientname]” and it will make me a Google doc and remove their tags from the master document.

Used a workflow I created last month to Search queries on AI Mode and capture screenshots for a client report. It then uses Gemini 3 Flash to analyze what has changed in terms of SERP features or my client’s presence.

Used a workflow I created last month to analyze competitor sites and archive.org versions and report on what changes they made as well as what new blog posts they wrote.

Email triage via the Google Workspace CLI. This is so good.

Pulled data from LLM referrals from the GA4 MCP and popped it right into my client’s Google doc. It now takes me less than 5 seconds to do this part of my reports.

Built my Agent Readiness Tool, and deployed it to Google Cloud so it is live for all to use.

I finished this newsletter episode and said to Antigravity - “Put this on my WordPress site” and it used a workflow we created that connected with my WP API, optimized the images, and created a perfect draft for me to approve.

Played with using my webcam to make a HR and HRV sensor that uses the colour in your face. It is so freaking cool. I am trying to make a tool that will help people fine tune our bodies to have more joy. We will see if it works!

This is just some of the work I did with Antigravity. I’d be here all day if I included everything. I love it so much.

A few other stories I found interesting

Google announced updates to AI Mode including snippets from online discussions and more links to websites.

UCP has been spotted in the traditional Search results

"Gemini Intelligence" is coming to Android - it can automate tasks for you like booking things for example.

Shopify data says AI referred shoppers convert 50% better and spend 14% more

Papa John's is starting to offer drone delivery in partnership with Google's Wing

ROBOTS THAT CAN DO HOUSEWORK ARE COMING. Apologies for the shouting. I cannot wait till I have robots doing housework for me.

That's all for this week. There's so much more that happened though. If you miss my full report of everything that happens, you can find it here in the Search and AI News feed in my community, the Search Bar.

Sign up for the newsletter

If you want to be notified by email each time I publish a future newsletter like this, you can sign up here:

Or, join us in the Search Bar for up to the minute news on Search and AI. Or better yet, join my paid community where we have regular calls to brainstorm on using AI and I share tips and ideas as well.

btw...Antigravity helped me make code snippets to dynamically insert the best Search Bar conversations into my blog posts. 🦾

Comment  on this episode in the Search Bar.