Skip to main content
Use an API to enrich one company while a partner is looking at it. If you need to rebuild a market map across the whole dataset every night, ask the vendor about bulk delivery before pricing it by the request.

API-Based Delivery

Most modern vendors provide REST APIs. You authenticate with an API token, make HTTP requests, and receive JSON responses. Benefits:
  • Query exactly what you need (don’t download everything)
  • Easy to integrate into applications (CRM enrichment, sourcing tools)
  • Can build interactive features (search, live updates)
Considerations:
  • Rate limits constrain how fast you can query
  • Per-request or per-entity costs (each API call might cost money)
  • Need to handle failures, retries, validation
  • API schemas change over time

File-Based Delivery

Before writing the importer, get two consecutive deliveries. Check whether each contains the whole dataset or only rows that changed. A company missing from a full export could have disappeared from the source. A company missing from a list of changes may simply have nothing new to report. Ask how deletions and corrections are represented. Almost all vendors also provide data as file exports: CSV, Parquet, or JSON files that you download or they upload to your S3 bucket. This is common for bulk data (entire company database, historical funding data, periodic dumps). However, this is often in the “premium” tier, which is usually many times more expensive than API access. Benefits:
  • Cheaper than API calls if you need the full dataset
  • Get everything at once (good for analytics, data warehouse loading)
  • Predictable costs (usually flat fee for the subscription)
  • No rate limits once you have the file
Considerations:
  • Data is a snapshot (might be stale compared to API data)
  • Need to process and load files
  • Need to handle incremental updates

Prefer Parquet Over CSV

Vendors will sometimes give you the option of CSV, TSV, JSON/JSONL or Parquet. Always choose Parquet. It includes schema information (you know data types without guessing), compresses well (smaller files), and loads much faster into data warehouses. JSON/JSONL is also a decent choice, but typically more cumbersome to work with. CSV/TSV files require parsing, have encoding issues, no schema, and are slower to work with.

Hybrid Approaches

Some vendors offer both APIs (for real-time enrichment) and bulk exports (for loading your data warehouse). This is ideal: use the API for interactive features, use bulk exports to populate your data warehouse efficiently. Check that the API and the export use the same identifiers. If a partner finds a company in the live search, you want to open its record in yesterday’s warehouse data. Matching on an ID makes that a lookup. Matching on the name gives you an entity-resolution project.

File Delivery Authentication

For file-based vendors, there are three common options:
  • They upload files to their own S3 bucket that you can access (they provide AWS credentials)
  • They upload files directly to your S3 bucket (you provide them with write-only credentials)
  • They provide download links through their portal (you download manually or via script)
The third option (manual download) doesn’t scale well. Prefer vendors who can integrate with your cloud storage.

Working with APIs

For details on API authentication, validation, rate limits, error handling, and other integration patterns, see Integrations.