API-Based Delivery
Most modern vendors provide REST APIs. You authenticate with an API token, make HTTP requests, and receive JSON responses. Benefits:- Query exactly what you need (don’t download everything)
- Easy to integrate into applications (CRM enrichment, sourcing tools)
- Can build interactive features (search, live updates)
- Rate limits constrain how fast you can query
- Per-request or per-entity costs (each API call might cost money)
- Need to handle failures, retries, validation
- API schemas change over time
File-Based Delivery
Before writing the importer, get two consecutive deliveries. Check whether each contains the whole dataset or only rows that changed. A company missing from a full export could have disappeared from the source. A company missing from a list of changes may simply have nothing new to report. Ask how deletions and corrections are represented. Almost all vendors also provide data as file exports: CSV, Parquet, or JSON files that you download or they upload to your S3 bucket. This is common for bulk data (entire company database, historical funding data, periodic dumps). However, this is often in the “premium” tier, which is usually many times more expensive than API access. Benefits:- Cheaper than API calls if you need the full dataset
- Get everything at once (good for analytics, data warehouse loading)
- Predictable costs (usually flat fee for the subscription)
- No rate limits once you have the file
- Data is a snapshot (might be stale compared to API data)
- Need to process and load files
- Need to handle incremental updates
Prefer Parquet Over CSV
Vendors will sometimes give you the option of CSV, TSV, JSON/JSONL or Parquet. Always choose Parquet. It includes schema information (you know data types without guessing), compresses well (smaller files), and loads much faster into data warehouses. JSON/JSONL is also a decent choice, but typically more cumbersome to work with. CSV/TSV files require parsing, have encoding issues, no schema, and are slower to work with.Hybrid Approaches
Some vendors offer both APIs (for real-time enrichment) and bulk exports (for loading your data warehouse). This is ideal: use the API for interactive features, use bulk exports to populate your data warehouse efficiently. Check that the API and the export use the same identifiers. If a partner finds a company in the live search, you want to open its record in yesterday’s warehouse data. Matching on an ID makes that a lookup. Matching on the name gives you an entity-resolution project.File Delivery Authentication
For file-based vendors, there are three common options:- They upload files to their own S3 bucket that you can access (they provide AWS credentials)
- They upload files directly to your S3 bucket (you provide them with write-only credentials)
- They provide download links through their portal (you download manually or via script)