Web scraping offers immense potential for data collection, insights, and analytics, but sites like Walmart.com have stringent security measures to protect their data. In this blog post, I’ll delve into the challenges faced, methods employed to bypass advanced security protocols, and some insights into the data scraped from Walmart.com.

Project Overview

The project aimed to scrape Walmart.com, an e-commerce giant, to gather a range of data, including product details, pricing, availability, reviews, and seller information. The complexity of the project stemmed from Walmart's advanced security measures and the need to circumvent them ethically without disrupting the website’s functionality.

Challenges Faced

As expected, scraping Walmart.com presented unique challenges. Here are the key obstacles and solutions I used to achieve a successful scrape:

1. IP Blocking

Walmart.com uses IP blocking to prevent continuous or suspicious access from single IPs. This measure limits how frequently one can make requests, as high-frequency requests from the same IP address often signal a bot or web scraper.

Solution: To avoid triggering IP blocks, I implemented a rotating proxy system. By using multiple proxies, I was able to distribute requests across different IP addresses, thereby mimicking human-like behavior and avoiding detection. Additionally, I added randomized delays between requests to further reduce the likelihood of being flagged.

2. "Press & Hold" CAPTCHA

CAPTCHA tests are designed to distinguish between humans and bots, and Walmart.com employs the "Press & Hold" CAPTCHA to prevent automated access.

Solution: Solving this CAPTCHA type involved emulating real user behavior with automated tools. I adjusted the request headers and user-agent strings to closely match those of legitimate browsers, making it less likely to trigger CAPTCHA tests. For instances where CAPTCHAs still appeared, I employed image recognition techniques and manual CAPTCHA-solving services, ensuring the scraping process continued without interruption.

3. Advanced Security Measures: PerimeterX and Akamai Bot Manager

Walmart.com deploys PerimeterX and Akamai Bot Manager to detect and block bots by analyzing behavioral patterns, request headers, and browsing fingerprints.

Solution: Bypassing these systems required precision. I carefully configured each request’s headers to include randomized elements (such as user-agent, language settings, and accepted content types) that closely mimic a human browsing session. Additionally, I utilized browser automation tools to simulate authentic browsing behavior, including mouse movements, scrolls, and clicks, ensuring the requests appeared as though they were made by an actual user.

Data Collected

With these solutions in place, I was able to scrape a comprehensive dataset from Walmart.com, including detailed product information. Below is an example of the data fields collected:

{
    "ProductId": "2UW4JTE2A5UZ",
    "ProductUrl": "https://www.walmart.com/ip/Men-s-1-10-ctw-Black-Diamond-Black-Tungsten-Grooved-8MM-Wedding-Band-Men-s-Ring/949552322?classType=VARIANT",
    "ProductName": "Brilliance Fine Jewelry Men's 1/10 Ctw Black Diamond Black Tungsten Grooved 8MM Wedding Band - Men's Ring",
    "category": "Jewelry",
    "ProductPrice": "$98",
    "wasPrice": "",
    "savingsAmount": "",
    "sellerName": "Walmart.com",
    "availabilityStatus": "In stock",
    "rating": 4.4,
    "numberOfReviews": 112,
    "brand": "Brilliance Fine Jewelry",
    "manufacturerProductId": "TG17204",
    "returnPolicy": {
        "returnable": true,
        "freeReturns": true,
        "returnWindow": {
            "value": 90,
            "unitType": "Day"
        },
        "returnPolicyText": "Free Holiday returns until Jan 31",
        "returnPolicyTextCode": {
            "code": "PRODUCT_HOLIDAY_RETURN",
            "data": null
        },
        "returnPolicyCondition": null,
        "holidayReturnEnabled": true
    },
    "description": "Bold and elegant black diamonds make a striking statement in this black ion-plated tungsten men's wedding ring. Step away from traditional wedding band choices and select a unique ring that truly reflects your style and personality.  This modern design is perfect for men of all ages\u2014whether you\u2019re a teenager looking to make your mark, a young adult ready to commit, or a senior celebrating a lifetime of love. The eye-catching black diamonds are complemented by the sleek tungsten band, creating a distinctive look that stands out on any occasion.  Embrace your individuality with this stylish black ion-plated tungsten ring, a perfect choice for anyone seeking a contemporary and sophisticated wedding band.",
    "ProductImages": [
        {
            "url": "https://i5.walmartimages.com/seo/Men-s-1-10-ctw-Black-Diamond-Black-Tungsten-Grooved-8MM-Wedding-Band-Men-s-Ring_1b8b58e3-3119-4020-a8c1-1bcc4a8c7da0.c487b34ddd2267ec0018e877b2308a83.jpeg"
        },
        {
            "url": "https://i5.walmartimages.com/asr/10fe3fca-a156-43de-8071-6172d3420909.9764ef731c98a7230689f8ead754e6c2.jpeg"
        },
        {
            "url": "https://i5.walmartimages.com/asr/0cd08853-3632-4c78-8953-2f0c0d6a4223.18f699126bb6fc945be679dcd56dc442.jpeg"
        },
        {
            "url": "https://i5.walmartimages.com/asr/4c26bec3-7742-44fa-8d47-88672992c112.2d9efab43a6c248b0d6f5ae8be6a5197.jpeg"
        }
    ],
    "reviews": {
        "customerReviews": [
            {
                "reviewId": "360058301",
                "rating": 5,
                "reviewSubmissionTime": "10/12/2024",
                "reviewText": "This ring is amazing quality for the price. My husband loves it. He never takes it off and it looks the same way it did when I purchased it.",
                "reviewTitle": "Great quality!",
                "negativeFeedback": 0,
                "positiveFeedback": 0,
                "userNickname": "Sav",
                "fulfilledBy": "Walmart",
                "status": null,
                "sellerName": "Walmart.com",
                "media": null,
                "photos": [],
                "badges": [
                    {
                        "badgeType": "Custom",
                        "id": "VerifiedPurchaser",
                        "contentType": "REVIEW",
                        "glassBadge": {
                            "id": "VerifiedPurchaser",
                            "text": "Verified Purchase"
                        }
                    }
                ],
                "clientResponses": null,
                "syndicationSource": null,
                "snippetFromTitle": null,
                "features": [
                    {
                        "name": "Ring size",
                        "value": "7"
                    }
                ]
            }
        ],
        "topNegativeReview": {
            "reviewId": "359851258",
            "rating": 2,
            "reviewSubmissionTime": "10/10/2024",
            "userNickname": "Laura",
            "negativeFeedback": 0,
            "positiveFeedback": 0,
            "reviewText": "I had this as a gift to my husband. I wrote a special note to come with it and they didn't put it in there so it ruined my vow renewal question ring is perfect but ruined everything that was planned.",
            "reviewTitle": null,
            "badges": [
                {
                    "badgeType": "Custom",
                    "id": "VerifiedPurchaser",
                    "contentType": "REVIEW",
                    "glassBadge": {
                        "id": "VerifiedPurchaser",
                        "text": "Verified Purchase"
                    }
                }
            ],
            "clientResponses": null,
            "syndicationSource": null,
            "snippetFromTitle": null,
            "media": null
        },
        "topPositiveReview": {
            "reviewId": "360058301",
            "rating": 5,
            "reviewSubmissionTime": "10/12/2024",
            "userNickname": "Sav",
            "negativeFeedback": 0,
            "positiveFeedback": 0,
            "reviewText": "This ring is amazing quality for the price. My husband loves it. He never takes it off and it looks the same way it did when I purchased it.",
            "reviewTitle": "Great quality!",
            "badges": [
                {
                    "badgeType": "Custom",
                    "id": "VerifiedPurchaser",
                    "contentType": "REVIEW",
                    "glassBadge": {
                        "id": "VerifiedPurchaser",
                        "text": "Verified Purchase"
                    }
                }
            ],
            "syndicationSource": null,
            "snippetFromTitle": null,
            "clientResponses": null,
            "media": null
        }
    }
}

Sample Data

A sample dataset is available on my GitHub repository. The JSON file (walmart.com_sample_data.json) showcases the structure and details captured for each product, providing insight into the data points available for analysis.

Key Insights and Analysis

The data collected from Walmart.com provides extensive information about each product, enabling rich insights, including:

• Pricing Trends: Comparison between current prices, discounts, and the original price.
• Consumer Feedback: Rating and review analysis help understand product reception and common customer concerns.
• Availability Status: Real-time stock information for timely purchasing or recommendations.

Lessons Learned and Takeaways

Through this project, I developed new strategies for overcoming cutting-edge security measures while remaining compliant with legal and ethical standards. This experience not only strengthened my expertise in web scraping but also reinforced the importance of responsible data collection practices.

Conclusion

Scraping websites like Walmart.com is both challenging and rewarding, especially with advanced security protocols in place. Overcoming IP blocks, CAPTCHAs, and bot managers required a combination of technical expertise, strategic planning, and ethical responsibility. The data gathered has significant value in e-commerce analysis and consumer behavior insights.

🔗 Explore the Project and Connect

If you’re interested in learning more or have similar projects in mind, check out the full project on GitHub: walmart.com Scraper. I’d love to hear your feedback and connect with fellow developers!

For inquiries or service requests, feel free to reach out via LinkedIn or visit my portfolio at mominur.dev.

Are you ready to leverage the future of data scraping for your business? Contact me today to explore innovative data solutions that can transform your organization!