Why Automated Price Monitoring Matters
In competitive e-commerce markets, prices change constantly. A competitor might drop their price by 5% at 2 AM, and by the time you notice, you have already lost a day of sales. Automated price monitoring eliminates this blind spot by continuously tracking competitor prices and alerting you to changes in real time.
Whether you are a retailer adjusting prices to stay competitive, a brand monitoring MAP (Minimum Advertised Price) compliance, or an analyst tracking market trends, a well-built price monitoring system pays for itself quickly. The key ingredient that makes it all work reliably is a robust proxy infrastructure — without it, your monitoring requests get blocked within hours. For a broader look at e-commerce data collection, see our e-commerce data scraping guide.
Architecture of a Price Monitoring System
A production-grade price monitoring system has four main components: a URL manager, a scraping engine, a data store, and an alerting layer.
| Component | Responsibility | Technologies |
|---|---|---|
| URL Manager | Stores target URLs, scheduling metadata, and scraping frequency | PostgreSQL, Redis |
| Scraping Engine | Fetches pages through proxies, extracts prices | Python/Node.js, ProxyHat, BeautifulSoup/Cheerio |
| Data Store | Stores price history with timestamps | PostgreSQL, TimescaleDB, ClickHouse |
| Alert System | Detects changes, sends notifications | Webhooks, Slack, Email, SMS |
Scheduling Strategy
Not all products need the same monitoring frequency. High-priority items (your top 100 SKUs, direct competitor products) might need hourly checks, while long-tail items can be checked daily. Prioritize based on:
- Price volatility: Products that change prices frequently need more frequent checks.
- Revenue impact: Your bestsellers deserve higher monitoring priority.
- Competitive density: Categories with many competitors need tighter monitoring.
Setting Up Proxy Rotation for Monitoring
Price monitoring means hitting the same URLs repeatedly over days, weeks, and months. This pattern is exactly what anti-bot systems are designed to detect. Residential proxies with automatic rotation are essential.
ProxyHat Configuration
# Standard rotating proxy (new IP per request)
http://USERNAME:PASSWORD@gate.proxyhat.com:8080
# Geo-targeted for regional pricing (e.g., US prices)
http://USERNAME-country-US:PASSWORD@gate.proxyhat.com:8080
# Session-based for multi-page price checks
http://USERNAME-session-price001:PASSWORD@gate.proxyhat.com:8080For price monitoring, per-request rotation works best because each price check is an independent operation. Use geo-targeted proxies when monitoring regional pricing differences.
Python Implementation
Here is a complete price monitoring system built with Python, using ProxyHat's Python SDK.
Price Scraper Module
import requests
from bs4 import BeautifulSoup
import json
import time
import random
from datetime import datetime
from dataclasses import dataclass, asdict
PROXY_URL = "http://USERNAME:PASSWORD@gate.proxyhat.com:8080"
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/124.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/124.0.0.0 Safari/537.36",
]
@dataclass
class PriceResult:
url: str
price: float | None
currency: str | None
in_stock: bool
scraped_at: str
seller: str | None = None
def scrape_price(url: str, selectors: dict) -> PriceResult:
"""Scrape a product price from any e-commerce site."""
headers = {
"User-Agent": random.choice(USER_AGENTS),
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
}
proxies = {"http": PROXY_URL, "https": PROXY_URL}
try:
response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
response.raise_for_status()
except requests.RequestException as e:
return PriceResult(
url=url, price=None, currency=None,
in_stock=False, scraped_at=datetime.utcnow().isoformat()
)
soup = BeautifulSoup(response.text, "html.parser")
price = extract_price(soup, selectors.get("price"))
currency = selectors.get("currency", "USD")
in_stock = check_stock(soup, selectors.get("stock"))
return PriceResult(
url=url,
price=price,
currency=currency,
in_stock=in_stock,
scraped_at=datetime.utcnow().isoformat(),
)
def extract_price(soup, selector: str) -> float | None:
"""Extract and parse price from a CSS selector."""
if not selector:
return None
el = soup.select_one(selector)
if not el:
return None
text = el.get_text(strip=True)
# Remove currency symbols, commas, spaces
cleaned = "".join(c for c in text if c.isdigit() or c == ".")
try:
return float(cleaned)
except ValueError:
return None
def check_stock(soup, selector: str) -> bool:
"""Check if product is in stock."""
if not selector:
return True
el = soup.select_one(selector)
if not el:
return False
text = el.get_text(strip=True).lower()
return "in stock" in text or "available" in text
# Site-specific selector configurations
SITE_SELECTORS = {
"amazon.com": {
"price": "span.a-price-whole",
"stock": "#availability span",
"currency": "USD",
},
"walmart.com": {
"price": "[data-testid='price-wrap'] span.f2",
"stock": "[data-testid='fulfillment-badge']",
"currency": "USD",
},
"target.com": {
"price": "[data-test='product-price']",
"stock": "[data-test='fulfillmentSection']",
"currency": "USD",
},
}Monitoring Scheduler
import schedule
import threading
from collections import defaultdict
class PriceMonitor:
def __init__(self, db_connection):
self.db = db_connection
self.price_history = defaultdict(list)
def add_product(self, url: str, site: str, check_interval_minutes: int = 60):
"""Register a product for monitoring."""
selectors = SITE_SELECTORS.get(site, {})
def check():
result = scrape_price(url, selectors)
self.price_history[url].append(result)
self.store_result(result)
self.check_alerts(url, result)
time.sleep(random.uniform(1, 3))
schedule.every(check_interval_minutes).minutes.do(check)
def store_result(self, result: PriceResult):
"""Store price result in database."""
# Insert into price_history table
self.db.execute(
"INSERT INTO price_history (url, price, currency, in_stock, scraped_at) "
"VALUES (%s, %s, %s, %s, %s)",
(result.url, result.price, result.currency,
result.in_stock, result.scraped_at)
)
def check_alerts(self, url: str, result: PriceResult):
"""Check if price change triggers an alert."""
history = self.price_history[url]
if len(history) < 2:
return
prev = history[-2]
curr = history[-1]
if prev.price and curr.price:
change_pct = ((curr.price - prev.price) / prev.price) * 100
if abs(change_pct) >= 5: # 5% threshold
self.send_alert(url, prev.price, curr.price, change_pct)
# Stock status change
if prev.in_stock and not curr.in_stock:
self.send_alert(url, msg="Product went out of stock")
elif not prev.in_stock and curr.in_stock:
self.send_alert(url, msg="Product back in stock")
def send_alert(self, url, old_price=None, new_price=None,
change_pct=None, msg=None):
"""Send price change notification."""
if msg:
print(f"ALERT [{url}]: {msg}")
else:
direction = "dropped" if change_pct < 0 else "increased"
print(f"ALERT [{url}]: Price {direction} {abs(change_pct):.1f}% "
f"(${old_price} -> ${new_price})")
def run(self):
"""Start the monitoring loop."""
while True:
schedule.run_pending()
time.sleep(1)
# Usage
if __name__ == "__main__":
monitor = PriceMonitor(db_connection=None) # Replace with actual DB
# Monitor competitor products
monitor.add_product(
"https://www.amazon.com/dp/B0CHX3QBCH",
site="amazon.com",
check_interval_minutes=60,
)
monitor.add_product(
"https://www.amazon.com/dp/B0D5BKRY4R",
site="amazon.com",
check_interval_minutes=30, # Higher priority
)
monitor.run()Node.js Implementation
For teams using Node.js, here is an equivalent monitoring setup using ProxyHat's Node SDK.
const axios = require("axios");
const cheerio = require("cheerio");
const { HttpsProxyAgent } = require("https-proxy-agent");
const cron = require("node-cron");
const PROXY_URL = "http://USERNAME:PASSWORD@gate.proxyhat.com:8080";
const agent = new HttpsProxyAgent(PROXY_URL);
const USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/124.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/124.0.0.0 Safari/537.36",
];
async function scrapePrice(url, selectors) {
try {
const { data } = await axios.get(url, {
httpsAgent: agent,
headers: {
"User-Agent": USER_AGENTS[Math.floor(Math.random() * USER_AGENTS.length)],
"Accept-Language": "en-US,en;q=0.9",
},
timeout: 30000,
});
const $ = cheerio.load(data);
const priceText = $(selectors.price).first().text().trim();
const price = parseFloat(priceText.replace(/[^0-9.]/g, "")) || null;
return {
url,
price,
currency: selectors.currency || "USD",
inStock: $(selectors.stock).text().toLowerCase().includes("in stock"),
scrapedAt: new Date().toISOString(),
};
} catch (err) {
return { url, price: null, currency: null, inStock: false, scrapedAt: new Date().toISOString() };
}
}
class PriceMonitor {
constructor() {
this.products = [];
this.history = new Map();
}
addProduct(url, selectors, cronExpression = "0 * * * *") {
this.products.push({ url, selectors, cronExpression });
this.history.set(url, []);
cron.schedule(cronExpression, async () => {
const result = await scrapePrice(url, selectors);
const prev = this.history.get(url);
prev.push(result);
if (prev.length >= 2) {
const last = prev[prev.length - 2];
if (last.price && result.price) {
const changePct = ((result.price - last.price) / last.price) * 100;
if (Math.abs(changePct) >= 5) {
console.log(`ALERT [${url}]: Price changed ${changePct.toFixed(1)}%`);
}
}
}
console.log(`Checked ${url}: $${result.price} (${result.inStock ? "in stock" : "out of stock"})`);
});
}
}
// Usage
const monitor = new PriceMonitor();
monitor.addProduct(
"https://www.amazon.com/dp/B0CHX3QBCH",
{ price: "span.a-price-whole", stock: "#availability span", currency: "USD" },
"0 * * * *" // Every hour
);
monitor.addProduct(
"https://www.amazon.com/dp/B0D5BKRY4R",
{ price: "span.a-price-whole", stock: "#availability span", currency: "USD" },
"*/30 * * * *" // Every 30 minutes
);Data Storage and Analysis
Raw price data becomes valuable when you can analyze trends over time.
Database Schema
CREATE TABLE monitored_products (
id SERIAL PRIMARY KEY,
url TEXT NOT NULL,
site VARCHAR(100) NOT NULL,
product_name VARCHAR(500),
our_sku VARCHAR(100),
check_interval_minutes INT DEFAULT 60,
is_active BOOLEAN DEFAULT true,
created_at TIMESTAMPTZ DEFAULT now()
);
CREATE TABLE price_history (
id SERIAL PRIMARY KEY,
product_id INT REFERENCES monitored_products(id),
price DECIMAL(10, 2),
currency VARCHAR(3) DEFAULT 'USD',
in_stock BOOLEAN,
scraped_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX idx_price_history_product_time
ON price_history (product_id, scraped_at DESC);Price Trend Queries
-- Average daily price for the last 30 days
SELECT
date_trunc('day', scraped_at) AS day,
AVG(price) AS avg_price,
MIN(price) AS min_price,
MAX(price) AS max_price
FROM price_history
WHERE product_id = 1
AND scraped_at >= now() - INTERVAL '30 days'
GROUP BY day
ORDER BY day;
-- Products with price drops > 10% in the last 24 hours
SELECT
mp.product_name,
mp.url,
old_prices.avg_price AS price_yesterday,
new_prices.avg_price AS price_today,
((new_prices.avg_price - old_prices.avg_price) / old_prices.avg_price * 100) AS change_pct
FROM monitored_products mp
JOIN LATERAL (
SELECT AVG(price) AS avg_price
FROM price_history
WHERE product_id = mp.id
AND scraped_at BETWEEN now() - INTERVAL '48 hours' AND now() - INTERVAL '24 hours'
) old_prices ON true
JOIN LATERAL (
SELECT AVG(price) AS avg_price
FROM price_history
WHERE product_id = mp.id
AND scraped_at >= now() - INTERVAL '24 hours'
) new_prices ON true
WHERE ((new_prices.avg_price - old_prices.avg_price) / old_prices.avg_price * 100) < -10;Alerting and Notifications
Automated alerts ensure you react to price changes quickly. Common notification channels include:
- Slack webhooks: Ideal for team-wide visibility. Send structured messages with price change details.
- Email digests: Daily or hourly summaries of all price changes above your threshold.
- Webhook callbacks: Trigger your repricing engine or other automation when prices change.
- Dashboard: Real-time visualization of price trends across all monitored products.
Alert Thresholds
Configure different alert thresholds for different scenarios:
| Scenario | Threshold | Action |
|---|---|---|
| Competitor price drop > 5% | 5% | Slack notification |
| Competitor price drop > 15% | 15% | Email to pricing team + auto-reprice |
| Product goes out of stock | Stock change | Opportunity alert |
| Price below MAP | Below MAP value | Compliance alert |
Proxy Best Practices for Monitoring
Continuous monitoring creates unique challenges for proxy management compared to one-time scraping.
- Distribute requests over time: Instead of checking all 10,000 products at midnight, spread checks across the entire interval. This creates a steady, low-profile request pattern.
- Use residential proxies: Residential proxies are essential for long-running monitoring because the same datacenter IPs hitting the same sites daily will get banned.
- Match geo-location: When monitoring regional pricing, use proxies from the target region. A US IP checking German prices will see the wrong data or get redirected.
- Handle failures gracefully: If a request fails, wait and retry with exponential backoff rather than immediately re-requesting. Monitor your success rate and reduce concurrency if it drops.
- Cache and deduplicate: If a price has not changed, do not store a duplicate record. This keeps your database lean and makes analysis faster.
Key takeaway: Price monitoring is a marathon, not a sprint. Design your system for steady, sustainable request patterns rather than burst scraping.
Scaling Your Monitoring System
As your product catalog grows, scaling becomes critical. Here are the patterns that work:
- Worker pool: Use multiple workers pulling from a job queue (Redis, RabbitMQ). Each worker has its own proxy connections and operates independently.
- Priority queues: High-value products get checked first and more frequently. Low-priority items fill remaining capacity.
- Adaptive scheduling: If a product's price has not changed in 7 days, reduce check frequency automatically. If it changed twice today, increase frequency.
- Rate limiting per site: Respect each target site's rate limits. Amazon, Walmart, and niche stores all have different tolerances.
For more on scaling scraping operations, check our guide on best proxies for web scraping in 2026 and explore ProxyHat's pricing plans for high-volume monitoring.
Key Takeaways
- Automated price monitoring requires a robust architecture: URL manager, scraping engine, data store, and alert system.
- Residential proxies with per-request rotation are essential for sustained monitoring without blocks.
- Schedule checks based on priority — not all products need hourly monitoring.
- Store price history in a time-series-friendly schema for trend analysis.
- Configure tiered alert thresholds to balance responsiveness with noise reduction.
- Distribute requests evenly over time for a sustainable, low-profile scraping pattern.
Ready to build your price monitoring system? Start with ProxyHat's residential proxies and read our e-commerce scraping guide for the full strategy. For technical implementation details, see our guides on using proxies in Python and using proxies in Node.js.
