How Boston Globe Media Used Conductor Monitoring to Improve Technical AEO at Scale
About Boston Globe Media
- Headquarters
- Boston, Massachusetts
- Company Size
- 5,000
- Industry
- Education
Boston Globe Media is one of New England's most prominent media companies, operating a portfolio of publications that includes The Boston Globe, Boston.com, Boston Magazine, and Stat News. Its properties span local and regional news, lifestyle and culture, and specialized science and health journalism, reaching millions of readers across digital and print every month.
As a publisher operating across multiple distinct brands and audiences, its digital presence is central to how readers discover, engage with, and return to its content. Search visibility plays a critical role in sustaining and growing that audience at scale.
The challenge
Boston Globe Media's search team was responsible for maintaining visibility across a complex portfolio of sites supported by a legacy, decentralized technology stack. Changes were happening constantly, but the data needed to understand their impact lived across multiple platforms. Identifying issues related to crawlability, indexing, rendering, or site performance often required significant manual investigation and still left gaps in visibility.
The stakes grew even higher as AI search emerged. Publishers suddenly needed to understand not only how traditional search engines crawled their content, but also how AI platforms were discovering and accessing it. Unlike search engines, answer engines offered little transparency into crawl behavior or content visibility, making it difficult to know whether technical issues were limiting performance in these new channels.
Boston Globe Media needed a way to monitor technical health in real time, connect data across its portfolio, and gain visibility into how both search engines and AI crawlers interacted with its content.
How Conductor helped
Conductor gave Boston Globe Media a single, unified view of technical site health and AI crawler behavior across its entire portfolio.
With all the data it needed in one place, it could move from reactive troubleshooting to proactive site management. Issues were caught the moment they happened, building the technical foundation needed to perform in traditional and AI-powered search.
- Real-time alerting across a complex, multi-property portfolio: Conductor Monitoring gave Boston Globe Media 24/7 visibility into site health across all four of its publications. With continuous crawling rather than scheduled snapshots, issues were surfaced within minutes of occurring rather than days or weeks later, making it possible to act before search visibility, revenue, or traffic took a hit.
- A research platform for the entire newsroom: Conductor Intelligence became the central platform for keyword and topic research across all four publications. AEO/SEO editors used it to identify hyperlocal opportunities, understand competitive positioning, and build data-driven plans for high-priority feature stories. For stories with lead time, custom keyword groups were set up to track performance over time and measure impact after publication.
- Full visibility into how Google and AI bots crawl their sites: Conductor's AI bot crawler activity data gave Boston Globe Media a direct view into how bots from OpenAI, Anthropic, and Google were accessing its content—and where their behaviors differed. Those insights helped the team identify technical improvements, better understand how AI crawlers navigate its sites, and reinforce the role strong technical foundations play in AI search visibility.
- Instant detection when development changes impact site performance: As new infrastructure was rolled out across its properties, Conductor's Core Web Vitals monitoring tracked performance changes in real time. By linking performance regressions to recent development changes, the team could quickly pinpoint what was causing the issue. That visibility made it easier to prioritize fixes before performance issues impacted search visibility or user experience.
Featured Conductor capabilities: 24/7 Monitoring, AI Crawler Activity, Real-time Alerting, Core Web Vitals, Log File Analysis, Research, AI Search Performance
Featured Conductor products: Conductor Monitoring, Conductor Intelligence
Before | After |
|---|---|
Technical issues discovered days or weeks after occurring | Critical site issues detected within minutes via real-time alerting |
Topic/keyword research and competitive analysis spread across separate platforms | Centralized research platform powering editorial planning across all four publications |
No visibility into how AI bots crawl the portfolio | Full visibility into crawl behavior across Google, OpenAI, and Anthropic bots |
Core Web Vitals regressions difficult to diagnose and attribute | Performance changes tracked in real time and tied to specific development work |
The results
A strong technical foundation and AI search visibility aren't separate goals. For Boston Globe Media, improvements to technical site health translated into stronger visibility across both traditional and AI-powered search.
Real-time monitoring prevented technical issues from impacting search performance
When a code push inadvertently reverted robots.txt files across Boston Globe Media's properties, Conductor’s 24/7 monitoring caught it within minutes. Real-time alerting surfaced the issue immediately, while log file analysis helped the team understand the potential impact and take action before search engine performance or AI crawler activity were affected.
![Marc Choquette, Senior Director, Search & AI Platforms, [object Object]](https://cdn.sanity.io/images/tkl0o0xu/production/941eded6542a888236a14d45abf9c91bb54ef065-800x800.jpg?fit=min&w=100&h=100&dpr=1&q=95)
Conductor Monitoring is the eyes and ears of our websites. When changes happen, or something breaks, we find out about it almost immediately, and that has been really valuable for us.
That speed mattered. Across four publications and a constant stream of development changes, technical issues rarely stayed isolated for long.
By continuously monitoring site health and crawl activity, the team identified issues before they could impact search visibility. As a result, content remained accessible to both traditional search engines and emerging AI platforms.
That attention to technical health paid off. The Boston Globe saw a 3% increase in site health, with its health score rising from 860 to 885. That's what two years of consistently catching and resolving issues looks like on a site that reaches millions of readers each month.
AI crawl data uncovered opportunities to improve AI visibility
Conductor's log file analysis gave Boston Globe Media a real-time view of how Google and AI bots crawl its sites, and how differently they behave. Google bots follow site hierarchy methodically, moving from section fronts through individual pages in a predictable, structured pattern. AI bots are far less consistent, frequently landing on 404 pages and bypassing the navigational signals that guide traditional search crawlers.
Those crawl insights made it possible to identify exactly where AI bots were encountering barriers and prioritize the technical fixes most likely to improve AI crawlability and, in turn, AI visibility.
Conductor has given us a window into how we show up in these spaces, especially from a crawl perspective. We can see in real time how different bots are moving, and it shows us what each sees as valuable content on our sites.
As those improvements were made, crawl behavior improved alongside them. The outcome was a stronger technical foundation that drove measurable gains in AI search visibility across all four publications.
Conductor’s site monitoring capabilities have reinforced that having strong technical fundamentals—whether it's server-side rendering or strong site performance—the two dovetail. We've definitely seen improved crawling behavior the more we've optimized certain sections of our sites.
Topic, keyword, and competitive research uncovered high-value content opportunities
Conductor Intelligence gave Boston Globe Media's editorial teams a single platform for topic/keyword research, competitive analysis, and content strategy. For high-priority feature stories, that research directly shaped publishing decisions: which topics to prioritize, how to structure coverage, and when to publish for maximum impact.
Competitive analysis surfaced hyperlocal opportunities that had not previously been prioritized, and custom keyword groups tracked how those stories performed after publication. That feedback loop between editorial decisions and search outcomes paid off.
Both The Boston Globe and Stat News grew organic traffic by double digits—results driven by a precise, data-driven approach to content planning across the portfolio.
Conductor has really helped us from a research and monitoring perspective to identify trends, track issues, and ensure we're moving in the right direction. We've managed to grow traffic to a couple of our sites, specifically Boston Globe and Stat News, in the last couple of years by double digits.
