Skip to main content

​
Overview

Data providers deliver data in two main ways: APIs and files. Each has different implications for how you build integrations and what you pay.

​
API-Based Delivery

Most modern vendors provide REST APIs. You authenticate with an API token, make HTTP requests, and receive JSON responses. Benefits:
  • Query exactly what you need (don’t download everything)
  • Easy to integrate into applications (CRM enrichment, sourcing tools)
  • Can build interactive features (search, live updates)
Considerations:
  • Rate limits constrain how fast you can query
  • Per-request or per-entity costs (each API call might cost money)
  • Need to handle failures, retries, validation
  • API schemas change over time

​
File-Based Delivery

Almost all vendors also provide data as file exports: CSV, Parquet, or JSON files that you download or they upload to your S3 bucket. This is common for bulk data (entire company database, historical funding data, periodic dumps). However, this is often in the “premium” tier, which is usually many times more expensive than API access. Benefits:
  • Cheaper than API calls if you need the full dataset
  • Get everything at once (good for analytics, data warehouse loading)
  • Predictable costs (usually flat fee for the subscription)
  • No rate limits once you have the file
Considerations:
  • Data is a snapshot (might be stale compared to API data)
  • Need to process and load files
  • Need to handle incremental updates

​
Prefer Parquet Over CSV

Vendors will sometimes give you the option of CSV, TSV, JSON/JSONL or Parquet. Always choose Parquet. It includes schema information (you know data types without guessing), compresses well (smaller files), and loads much faster into data warehouses. JSON/JSONL is also a decent choice, but typically more cumbersome to work with. CSV/TSV files require parsing, have encoding issues, no schema, and are slower to work with. Ask vendors to provide Parquet if they don’t already. Most modern vendors support it.

​
Hybrid Approaches

Some vendors offer both APIs (for real-time enrichment) and bulk exports (for loading your data warehouse). This is ideal: use the API for interactive features, use bulk exports to populate your data warehouse efficiently.

​
File Delivery Authentication

For file-based vendors, there are three common options:
  • They upload files to their own S3 bucket that you can access (they provide AWS credentials)
  • They upload files directly to your S3 bucket (you provide them with write-only credentials)
  • They provide download links through their portal (you download manually or via script)
The third option (manual download) doesn’t scale well. Prefer vendors who can integrate with your cloud storage.

​
Working with APIs

For details on API authentication, validation, rate limits, error handling, and other integration patterns, see Integrations.