I am Kazushi Nagayama, the founder of JADE. Today, as the title suggests, I'd like to discuss how "product improvement and advanced SEO go hand in hand.”
Today’s agenda:
- About the Search Interaction Model
- Viewing SEO as feedback for product improvement
- Practical examples (Crawl, Index, Click)
About the Search Interaction Model
In the past years, we have made several presentations based on JADE's unique Search Interaction Model (DCIR—QCLS) framework. We’ve been conducting analyses and making proposals based on this framework since 2021, and we shifted towards more actively promoting this model externally in 2022.
This framework was created to redefine SEO and clearly outline its scope. The core issue was the growing confusion in the 2020s about what "search engine optimization" truly encompasses, its objectives, and the necessary actions. With the end of the era characterized by spam and search engine hacks—where rankings were artificially boosted by manipulating signals in violation of search engine guidelines—a lack of consensus emerged on what should be done going forward. As a result, SEO lost its clear boundaries, leading to strategies based on misconceptions or those that failed to deliver meaningful results.
This framework aims to:
- Clearly redefine the scope of search engine optimization
- Offer a structured approach for implementing strategies
It aims to address the existing challenges and propose a sustainable and growth-oriented model of search engine optimization.
For more information about this model, please refer to the corresponding article. The main goal of this model is to enhance the experience of users who visit a site via search engines. This is accomplished through search experience optimization, which is essential in helping search engines accurately understand your website. Until now, the primary objective of search engine optimization has been to improve rankings in search results. Whether users' behaviour can be considered ranking signals is a common point of discussion. This transformation has fundamentally shifted the debate towards a user-first perspective. It asserts that the primary goal of SEO is to enhance the overall search experience, which is a highly significant shift.
Even within this paradigm, it remains crucial to enable search engines to crawl and index the site and design it so that signals are appropriately evaluated in rankings. We can move beyond simply discussing "search rankings" by dividing the process into phases. This allows for a more detailed and thorough conversation rather than asking if something contributes to rankings.
The terms "good or bad for SEO" are often used, but they lack precision and are not recommended by JADE. By using expressions like "good for index rate" or "good for the landing experience," we can better understand the objectives of a particular strategy.
While such considerations were already common knowledge among those practising high-level SEO as of 2021, the novel aspect is that we have articulated, systematized, and formalized these ideas into a teachable framework.
SEO as feedback for product improvement
Using this framework and adopting a fundamentally user-first approach allows us to reassess existing SEO strategies based on their intended goals and identify any areas that may be lacking. A key development is the introduction of the Surf phase, in addition to the Land phase, which focuses on the user experience at the entry point. This addition enables us to comprehensively analyze the overall user experience and explore ways to enhance it. One of our colleagues, Mr. Yamada, has written an article that is a great example of this approach (currently available only in Japanese).
Through this transformation, search engine evaluations can be seen as valuable service or website feedback. Using data, we can strive for sustainable growth that prioritizes improving the product rather than relying on shortcuts or hacks.
Use the framework to select metrics, collect data, analyze, develop hypotheses, propose improvements, and implement them to measure changes. Adjust the hypotheses and improve the strategies if the expected outcome is not achieved. JADE believes that by continuously cycling through this process and aiming for product growth, it can exceed traditional SEO and make improvements.
Practical Examples of Measures
Here are examples of using data to improve products in phases based on our framework.
Crawl: Creating a Crawler Dashboard to Monitor Status
When JADE starts working on a new client's website, one of the first things it does for large-scale sites is create a crawler dashboard. We collect crawler logs in BigQuery and create an optimized Looker Studio dashboard using tools like Materialized Views. This allows us to analyze requests from different User-Agents and the patterns of URLs being requested.
In one case, we found that many unwanted crawls were coming to a set of URLs with unnecessary parameters via external links. We took action by blocking these links with robots.txt. As a result, the number of unwanted crawls was successfully reduced by several million daily without affecting the indexation rate.
Furthermore, we employ BigQuery ML to train ARIMA models for predicting request counts by UserAgent type, URL pattern, and HTTP status. We have implemented a pipeline that triggers alerts when actual request numbers deviate significantly from the predicted values. This allows us to respond more rapidly to issues such as sudden spikes in 50x errors. We are also developing specialized tools to streamline this analysis.
Index: Evaluating Page Groups by Checking Index Rates in a Directory
When implementing crawl reduction measures, ensuring they do not negatively affect indexing or rankings is crucial. However, many people do not keep track of the percentage of URLs within a subdirectory that are indexed.
JADE has developed a tool to address this and monitor index rates at specific intervals. This tool creates groups by randomly sampling URLs from the targeted subdirectory or URL patterns, automatically checks their indexing status, and reports the index rate.
This allows for setting goals such as "improving the index rate of this subdirectory" and implementing targeted measures. The graph here shows a case where focused efforts led to a significant improvement in the index rate of a subdirectory within about six months of starting the initiative.
We have been continuously developing this tool, and its features have been gradually expanding. For example, it now includes functionalities like "random sampling from existing XML sitemaps to verify index rates."
It was launched as a Public Beta last year.
Click: Determine the best title by using a data-driven approach
When searching for better titles, you might want to conduct A/B tests. However, it takes more work to fully control which titles appear in search results, making validating results from large-scale tests difficult. While simple tests can be conducted by changing titles in specific subdirectories, user responses may vary by subdirectory, making it hard to apply results universally.
In an example conducted by JADE, for a large site, we used stratified sampling to separate the targets and then CausalImpact to verify the data. Initially, we planned to use a regular randomization function for grouping. However, when we examined the generated groups from various perspectives, we found significant differences in traffic between the groups. Therefore, to ensure that the traffic between the test groups was somewhat balanced, we performed stratification based on traffic volume to create the groups.
At a certain point, we change each group's title generation patterns. Since it takes a little time for all pages to be reindexed by Google, we monitor whether the URLs of each group are being crawled while visualizing the crawl logs in Looker Studio. In this case, there were no issues because we knew in advance that the crawler's behaviour was stable due to measures taken beforehand. However, if the crawler's behavior is unstable or lacks information about the crawler's status, you should first analyze and address these issues. Additionally, if the titles specified on the pages are not being used, you must prioritize analyzing and addressing the causes.
After initiating the changes, we ran CausalImpact on each group about a week later to assess the impact on clicks. The results show that the cumulative Clicks for the Treatment group are taking off.
With the implementation of this method, JADE can now effectively identify the best possible title for extensive websites. In a particular instance, they formed three treatment groups and tested them simultaneously, leading to a 20% rise in CTR.
Making the Internet better through product improvement
At JADE, we see SEO as a way to genuinely engage with users, crossing the boundaries between marketing and engineering to enhance products comprehensively. By accurately understanding the dynamics of search engines and search users and providing precise strategies as fertilizers to this foundation, the fruit of growth can be harvested successfully. The era of artificially boosting signals like links and content to achieve results is over.
JADE’s consulting focuses on "maximizing data use" and "strategic appropriateness" to support clients in achieving natural, growth-oriented outcomes rather than mere hacks. Our focus is on sustainable business growth rather than quick hacks. Our clients operate sites and services of exceptional value. We believe that the internet will become a better place as these outstanding products grow.
