Autonomous daily job scraping, scoring, and email delivery pipeline. Built as a cloud-native workflow that scans 8 platforms, ranks 50 roles, and keeps running cost at INR 0/month.
Pipeline at a glance
08:15 IST dual trigger with idempotent guard.
8 platforms, about 380 raw roles per day.
URL and semantic cross-platform collapse.
Hard filters, deterministic rules, then LLM.
Score bands, queue merge, top-50 cap.
Excel attachment and email digest around 08:18 IST.
Daily trigger
cron-job.org is the primary scheduler and GitHub cron is the fallback. Both land in the same GitHub Actions workflow, and a pre-flight check exits quickly if the digest already ran that day.
cron-job.org triggers the GitHub workflow at 08:15 IST through the workflow dispatch API.
Native scheduled workflow runs later as a zero-cost safety net if the primary trigger fails.
A run-history check makes double-firing safe. The second run exits without resending.
Step 1 - Scrape
Each scraper has platform-specific constraints, freshness caps, seniority filters, and error isolation. A crash in one source does not abort the run.
Paginated job search with experience filter and 72-hour freshness cap.
IT, systems, and consulting verticals with post-fetch seniority filtering.
Multi-board aggregator across several query combinations.
Managed actor with structured output and PM-title filtering.
Login, session cookies, opportunities API, then seniority filtering.
Headless Chrome across product-management role slugs.
RSS pull, freshness cap, language checks, and liveness validation.
Residential proxy path with structured extraction and global-hiring checks.
Step 2 - Deduplicate
The database stores seen jobs, overflow queue items, and cached scores. URL matching catches obvious repeats. Semantic company-role keys catch the same role posted across different platforms.
Every URL already emailed plus a normalized company-role key. 30-day rolling window.
Overflow beyond the top-50 cap and deferred items for leaner days.
Persisted LLM scores keyed by URL to save quota and avoid repeated scoring.
Step 3 - Score
Deterministic filters run first, LLM only for the ambiguous tail. This keeps the system predictable and protects free-tier quota.
Seniority keywords, freshness caps, title filters, and geography exclusions before a job enters the scoring pool.
Weighted product fit, AI/GenAI relevance, seniority match, and location/work-mode signals.
Groq scoring for the unclear jobs where language nuance matters.
Steps 4 and 5
Score 65 and above. Highest fit against the profile, shown as the top tier.
Score 55 to 64. Borderline fit, used when the top tier is not already full.
Anything beyond 50 moves to the queue and can re-enter on future runs.
Rank, Score, Role, Company, Platform, Location, Work Mode, Posted, Reason, and Apply link.
Stats, platform breakdown, and a clean call-to-action to open the workbook.
Resend email delivery plus a separate health report so failures are visible.
External services
LLM scoring with llama-3.3-70b-versatile. About 5 RPM used against 30 RPM free.
Transactional email and attachments. About 60 emails/month against 3,000/month free.
Aggregator API. About 60 calls used against 1,000/month free.
Managed Indeed actor, currently below the monthly platform credit.
Residential proxy path for Wellfound where direct access can fail.
Runner, dispatch, state commit-back, and operational logs.
Operations
The agent is designed to run unattended, but not blindly. It has explicit modes for testing, safety checks around scraped content, and evaluation signals that show whether the daily digest is still useful.
Reliability layer
The architecture is not only scrape, score, send. It also checks whether the input is safe, whether the run is healthy, whether the ranking is relevant, and whether the final delivery is complete.
Escapes spreadsheet formulas, normalizes source text, keeps profile data out of plaintext logs, and avoids trusting scraped fields blindly.
Each scraper can fail independently, external calls have time bounds, and the daily idempotency check prevents duplicate sends.
Apply rate, rejected roles, duplicate misses, and weak-fit reasons show whether the scoring weights need adjustment.
Health reporting checks source counts, top-50 packaging, attachment generation, and email delivery so failures are visible.
At a glance
Forkable pattern
To adapt this for another person or role, swap the profile, platforms, scoring weights, and delivery channel. The load-bearing choices are the dual trigger, three-layer scoring, SQLite-in-repo state, dedup strategy, and free-tier discipline.