GeminIQ
Subscribe

How to Pull SEC Financial Data With Python

Chad Hartman

By

Published · Last updated

The case against paying for fundamental data is always the same, and it is a fair one: the SEC gives this away. Three public XBRL endpoints, no key, no signup, no cost, every number in almost every 10-K and 10-Q sitting behind a URL. A hundred lines of Python and the problem is solved.

The script part of that is true. A Python script against SEC EDGAR's API will run cleanly, return a 200 status code, parse into valid JSON, and still be wrong — because a script that only handles the happy path silently mishandles the situations EDGAR's own data structure creates: a filing history too long for one response, a rate limit that throttles without warning, a period parameter that 404s if it is off by one character, a fiscal quarter that appears twice, and a fiscal calendar that doesn't line up with the calendar year the script assumes.

This guide builds the whole thing from the ground up, calling out each trap at the exact point in the code where it has to be handled. Then it does the more useful thing: it shows what is still broken after the script is finished and correct, because those last two problems live in the filings, not in the code.

Start your 7-day free trial →

Table of Contents

The Endpoints You Are Working With

The SEC's XBRL data lives behind a small number of RESTful endpoints at data.sec.gov, all free, all returning JSON, none requiring an API key. Three of them serve XBRL facts and one serves filing metadata, and choosing the wrong one is the difference between a single fast call and downloading a company's entire reporting history to read one line off it.

Endpoint URL pattern What it returns Call it when
submissions /submissions/CIK{10-digit}.json A company's filing history — form types, dates, accession numbers You need the list of filings, not the numbers inside them
companyfacts /api/xbrl/companyfacts/CIK{10-digit}.json Every XBRL fact one company has ever reported, across every concept and filing You are building a full model across many line items for one company
companyconcept /api/xbrl/companyconcept/CIK{10-digit}/{taxonomy}/{tag}.json One concept for one company across its whole filing history You already know the tag and want a single trend line
frames /api/xbrl/frames/{taxonomy}/{tag}/{unit}/{period}.json One concept across every company that reported it for one period You want a cross-sectional comparison rather than one company

companyfacts pulls back everything a company has ever tagged — often several hundred distinct concepts across us-gaap, plus a much thinner dei (Document and Entity Information) taxonomy carrying entity-level facts like shares outstanding. Requesting a full facts file for a single metric like revenue means downloading and discarding almost everything in the response. frames inverts the whole model: instead of one company across many concepts, one concept across every company. It is the only cross-sectional endpoint the SEC offers.

One formatting rule applies to every URL above: the CIK — the Central Index Key, the permanent numeric identifier the SEC assigns to each filer — must be zero-padded to exactly 10 digits, with the literal string CIK prefixed onto it. Apple's CIK is 320193, but the URL requires CIK0000320193. Passing the unpadded number returns a 404 rather than resolving gracefully, and it is the single most common first mistake. The full padding rules, including where EDGAR is more forgiving than this, are in EDGAR Accession Numbers and Filing URL Structure.

Step 1: Set Up Request Headers

Every request to data.sec.gov needs a descriptive User-Agent header identifying the caller and a contact method — a request without one gets a 403, not a graceful fallback, and it is the second most common first mistake after CIK padding.

import requests
import time

HEADERS = {"User-Agent": "YourApp your-email@example.com"}
BASE = "https://data.sec.gov"

Replace the placeholder with a real app name and a real contact email. The SEC doesn't validate the string beyond checking that a User-Agent header is present at all, but an accurately descriptive one is what the fair-access policy actually asks for.

Step 2: Handle Submissions Pagination

The submissions endpoint returns a company's filing history — but not all of it in one response. The filings.recent block covers at least one year of filings or the 1,000 most recent, whichever is larger. Anything older lives in separate JSON files, referenced under filings.files, each with the same columnar structure as recent. A script that reads only filings.recent silently truncates the filing history of any company with more than a year or two of activity — which is most established companies.

def sec_get(url, headers=HEADERS, max_retries=3):
    for attempt in range(max_retries):
        resp = requests.get(url, headers=headers, timeout=20)
        if resp.status_code == 200:
            time.sleep(0.12)  # stay comfortably under 10 requests/second
            return resp.json()
        if resp.status_code == 403:
            raise RuntimeError("403 — check that your User-Agent header is set and descriptive.")
        if resp.status_code == 429:
            time.sleep(10)
            continue
        resp.raise_for_status()
    raise RuntimeError(f"Failed after {max_retries} retries: {url}")


def get_full_submissions(cik):
    cik_padded = str(cik).zfill(10)
    data = sec_get(f"{BASE}/submissions/CIK{cik_padded}.json")
    recent = dict(data["filings"]["recent"])  # copy so we don't mutate the response

    for older_file in data["filings"].get("files", []):
        older = sec_get(f"{BASE}/submissions/{older_file['name']}")
        for key in recent:
            if key in older:
                recent[key] = older[key] + recent[key]

    return data, recent

Each entry in filings.files carries its own name — the exact filename to request next — so the code never has to guess a URL pattern for the older archives. It just follows the pointer the SEC's own response gives it.

Step 3: Respect the Rate Limit

data.sec.gov caps requests at 10 per second per IP address, enforced regardless of how many machines are making the calls. Exceeding it triggers a temporary block — typically around 10 minutes — that gets extended if requests keep arriving during the timeout, rather than a clean error on each request after the limit.

The sec_get function above already sleeps 0.12 seconds after every successful call, which keeps a sequential script comfortably under the ceiling without a more elaborate rate limiter. For anything pulling data across more than a handful of companies, that per-request delay adds up in wall-clock time faster than it looks on paper. A job touching a few hundred companies is a run measured in minutes, not seconds — and that's by design, not a bug in the script. For jobs at that scale, the SEC's own bulk companyfacts.zip archive — republished nightly around 3:00 a.m. ET, containing the same data the companyfacts and frames APIs serve — is faster than looping through individual calls.

Step 4: Pull One Company's Concept History

With headers and pagination handled, pulling an actual number looks like the easy part. It is the part where most scripts quietly break, and the break shows up as a 404.

REVENUE_TAG_CANDIDATES = [
    "RevenueFromContractWithCustomerExcludingAssessedTax",
    "Revenues",
]


def get_concept_series(cik, tag, taxonomy="us-gaap"):
    """
    Return the units dict for one concept, or None if this company has never
    reported under that element. A 404 here is data, not a failure.
    """
    cik_padded = str(cik).zfill(10)
    url = f"{BASE}/api/xbrl/companyconcept/CIK{cik_padded}/{taxonomy}/{tag}.json"
    try:
        return sec_get(url)["units"]
    except requests.HTTPError as err:
        if err.response is not None and err.response.status_code == 404:
            return None
        raise


def get_revenue_series(cik, candidates=REVENUE_TAG_CANDIDATES):
    for tag in candidates:
        units = get_concept_series(cik, tag)
        if units and "USD" in units:
            return tag, units["USD"]
    raise RuntimeError(
        f"None of {candidates} returned data for CIK {cik}. "
        "Check the company's own filed tags before assuming the element exists."
    )

A 404 on a companyconcept request does not mean the endpoint is broken or the CIK is wrong. Most often it means exactly what it says: this company has never reported anything under that element. Hard-coding us-gaap/Revenues and treating a 404 as an outage is the single most common way a working script produces an empty dataset for a real company.

The reason there is more than one candidate to try is a taxonomy change, not a quirk of the API. The element that carried service revenue on income statements for years was deprecated in the 2018 taxonomy alongside ASC 606 and superseded by RevenueFromContractWithCustomerExcludingAssessedTax, and a large share of filers moved to the newer element while others kept the older, simpler one. Which element a given company uses depends on the filer and the period, so the only safe pattern is to try each candidate and let the 404 tell you. The taxonomy post covers why the vocabulary moves at all.

The response for a resolved concept is one time series and nothing else:

{
  "cik": 320193,
  "entityName": "Apple Inc.",
  "units": {
    "USD": [
      {
        "end": "2024-09-28",
        "val": 391035000000,
        "accn": "0000320193-24-000123",
        "fy": 2024,
        "fp": "FY",
        "form": "10-K",
        "filed": "2024-11-01"
      },
      {
        "end": "2023-09-30",
        "val": 383285000000,
        "accn": "0000320193-23-000106",
        "fy": 2023,
        "fp": "FY",
        "form": "10-K",
        "filed": "2023-11-03"
      }
    ]
  }
}

Each entry in the units.USD array is one reported value: the period end date, the value itself (in whole dollars, never thousands or millions), the accession number of the filing it came from, the fiscal year and period, the form type, and the date it was filed. That accn field is the reason any figure derived from this endpoint stays checkable — it resolves to one exact filing, and nothing else.

Step 5: Query Every Company at Once With Frames

frames answers a different question: not "what has this company reported," but "what did every company report." A single call to:

GET https://data.sec.gov/api/xbrl/frames/us-gaap/AccountsPayableCurrent/USD/CY2024Q1I.json

returns one JSON object with a data array containing one entry per company that reported Accounts Payable, Current as of that period — thousands of rows from a single request, in place of thousands of individual companyconcept calls:

{
  "taxonomy": "us-gaap",
  "tag": "AccountsPayableCurrent",
  "ccp": "CY2024Q1I",
  "uom": "USD",
  "label": "Accounts Payable, Current",
  "description": "Carrying value as of the balance sheet date...",
  "pts": 3440,
  "data": [
    {
      "accn": "0001104659-24-037408",
      "cik": 1750,
      "entityName": "AAR CORP",
      "loc": "US-IL",
      "end": "2024-02-29",
      "val": 230300000
    }
  ]
}

The pts field states the row count up front — always equal to the length of data — which is worth checking before assuming the response includes every company you expected. A materially lower pts count than expected usually means the tag itself isn't the one most companies in your universe actually use.

Building the Period Parameter

The single most common frames failure is a malformed period parameter, and getting it right requires knowing whether the concept you are requesting is a duration fact or an instant fact.

Period format Shape Concept type Example Use for
CY#### Full calendar year (a duration of roughly 365 days) Duration CY2024 Annual income statement and cash flow concepts
CY####Q# Calendar quarter (a duration of roughly 91 days) Duration CY2024Q1 Quarterly income statement and cash flow concepts
CY####Q#I Point in time, as of a specific date (I for instantaneous) Instant CY2024Q1I Balance sheet concepts

Balance sheet concepts — Assets, Liabilities, StockholdersEquity, AccountsPayableCurrent — are instant facts and require the I suffix. Income statement and cash flow concepts — Revenues, NetIncomeLoss, OperatingIncomeLoss — are duration facts and use the plain form. Appending I to a duration concept, or omitting it from an instant one, doesn't return a helpful error. It returns a 404, which looks identical to a typo in the tag name and sends most people debugging the wrong part of the URL.

This isn't something a filer decides fact by fact, which is why the table above is reliable rather than a rule of thumb. Every concept in the US-GAAP taxonomy carries its own declared period type — instant or duration — set once, centrally, when the concept was defined. us-gaap:Assets is declared instant, because a balance sheet reports what a company owns as of a date. us-gaap:Revenues is declared duration, because revenue only means something over a span of time. A filer cannot legally attach a duration context to us-gaap:Assets, or an instant context to us-gaap:Revenues — validation software rejects the mismatch before the filing is ever accepted.

One convention to know before doing date arithmetic on what comes back. XBRL interprets a startDate as the very beginning of that calendar day and an endDate as the very end of that calendar day. A period written in a filing as "April 1, 2023 to March 31, 2024" is encoded with a start of 2023-04-01 and an end of 2024-03-31, but the underlying instant those two dates actually bound runs from midnight opening April 1 to midnight closing March 31. XBRL's own published guidance describes the resulting confusion plainly: the same calendar date means a different literal instant depending on whether it appears in a startDate, an endDate, or an instant element, and that asymmetry has caused, in the specification's own words, "a lot of confusion and a lot of tagging errors." A pipeline computing period length from those two dates has to know the convention to get the day count right.

Step 6: Deduplicate Repeated Rows in a Response

The same fiscal period can appear more than once in a companyconcept response. When a company reports a prior period again — as a comparative figure in a later filing, or as a corrected number after a restatement — both entries stay in the historical record, tied to different accession numbers and filed dates. A script that takes the first match for a period, or averages the two, gets a number nobody actually reported.

def dedupe_latest_filed(facts):
    """
    Keep only the most recently filed value for each (fy, fp, end) period.
    """
    latest = {}
    for fact in facts:
        key = (fact["fy"], fact["fp"], fact["end"])
        if key not in latest or fact["filed"] > latest[key]["filed"]:
            latest[key] = fact
    return sorted(latest.values(), key=lambda f: f["end"])

Keying on (fy, fp, end) rather than end alone matters too — the same period-end date can occasionally carry more than one fp designation across filing types, and treating those as one key would silently merge two different things.

Worth being precise about scope: this handles duplicate rows in an API response, which is a data-shape problem. It is not a way to detect that a restatement happened, what changed, or whether it matters — that is a research question, and it is covered in How to Find Restatements in Filings. All this function does is pick one row per period.

The frames endpoint doesn't need this. Its own documentation states it aggregates one fact for each reporting entity that is "last filed" for the requested period, meaning the SEC already performs this selection server-side. companyconcept and companyfacts make no such guarantee, and the SEC's own field descriptions don't warn you to add the step.

Step 7: Align Fiscal Periods Correctly

A company's end date doesn't tell a script which calendar quarter it covers — only the fp field does. Apple's fiscal Q1 ends in December; Microsoft's fiscal Q1 ends in September. A script that buckets facts into calendar quarters by inspecting the month of end gets Apple and Microsoft's fiscal years misaligned relative to each other, even though both companies' own data is internally correct.

def interim_periods_only(facts):
    """
    Use fp (Q1/Q2/Q3/FY) — the company's own fiscal-period label — never
    infer the period from the calendar month of 'end'.
    """
    return [f for f in facts if f["fp"] in ("Q1", "Q2", "Q3")]

This also matters for annual figures: a 10-K's fp value is FY, and filtering it out here (or including it, for an annual series) has to be deliberate rather than inferred from the reporting date.

The second half of getting periods right is knowing what an interim figure actually contains. A 10-Q reports flow concepts like revenue as year-to-date figures, not quarter-only. A companyconcept response for a Q3 filing shows nine months of revenue, not three. Deriving the standalone third-quarter number requires subtracting the prior quarter's year-to-date figure from the current one — a calculation the API never performs for you, and one this script deliberately does not perform either, so that everything it prints is exactly what the filing said.

Step 8: Put It Together

if __name__ == "__main__":
    cik = 320193  # Apple

    submissions, recent = get_full_submissions(cik)
    print(f"{submissions['name']}: {len(recent['accessionNumber'])} total filings retrieved")

    tag, revenue = get_revenue_series(cik)
    print(f"Revenue element resolved to: us-gaap:{tag}")

    clean = dedupe_latest_filed(revenue)
    interim = interim_periods_only(clean)

    for f in interim[-8:]:
        print(f"{f['end']}  {f['fp']} {f['fy']}  ${f['val']:,}  "
              f"(as filed — year-to-date, not a standalone quarter)  accn {f['accn']}")

Run end to end, this pulls Apple's complete filing history — not just the most recent year — resolves which revenue element the company actually reports under rather than assuming one, and prints the last eight deduplicated, correctly fiscal-labelled interim revenue figures, each with the accession number of the filing it came from.

Read that output label carefully, because it is the difference between a correct script and a wrong dataset. Those are not eight quarters of revenue. They are eight year-to-date figures, exactly as the filings report them. Turning them into standalone quarters is a subtraction the script above leaves to you on purpose, and a script that skips that step while calling its output "quarterly revenue" is producing wrong numbers with a straight face.

What This Script Doesn't Solve

Everything above is real work, and none of it is wasted: the script is correct, the endpoints are free, and the data is authoritative. That is the honest version of the "just script it" argument. Here is the other half.

Two problems survive every line of the code above, because they aren't bugs in the request logic — they are properties of the underlying filings.

The first is a tag that is present and wrong. If a company's Revenues element captured one component of revenue rather than the total in a particular year, no amount of correct pagination, deduplication, or fiscal alignment recovers the right number. The script returns exactly what was filed, which is the whole point, and exactly what was filed is misleading. That is a tag-drift problem, and the only defense is checking the tag against the filing.

The second is a tag that isn't there at all. If a company reported a line item under a custom extension tag it defined itself, rather than the standard us-gaap element this script queries, the value is invisible to the query — no error, no warning, no row. The candidate-list pattern in Step 4 handles standard-element aliases, not company-invented ones, and nothing generic can, because a custom tag has no meaning outside that one filer's own taxonomy. The taxonomy post covers when a filer is permitted to create one and what every platform downstream has to guess as a result.

Both require reading the actual tag value a company used, not just querying the concept a script assumes it used. That is a research step, not a code fix, and it is the step the pipeline never finishes. This is also the honest way to think about the build-versus-buy question: the script is a weekend. Keeping it correct across tag aliases, taxonomy revisions, custom extensions, and restated periods, for every company you care about, is not. Platforms that attach a source citation to an extracted figure — Finbox among them — remove part of that maintenance burden while still handing back a modeled number. GeminIQ's Financial Statements go the other way: every value keeps the tag and the accession number it was filed under, so the check the script above can't perform is one you can run on any figure in seconds. If what you want is the number the company actually reported, with the tag still attached, that is what the paid and free plans are for.

Frequently Asked Questions

Why does my SEC EDGAR script get a 403 error?

Almost always a missing or generic User-Agent header. data.sec.gov requires a descriptive header identifying the requester and a contact method on every request, including the first one.

Why does a companyconcept or frames request return a 404?

Three common causes, in order. An unpadded CIK — the API requires exactly 10 digits with a CIK prefix. A mismatched frames period suffix — an instant concept like Assets requested without the trailing I, or a duration concept like Revenues requested with it. Or the company simply never reported under the element you asked for, which is data rather than an error and should be handled as a fallback rather than a crash.

What's the difference between companyfacts, companyconcept, and frames?

companyfacts returns every concept for one company. companyconcept returns one concept for one company across all periods. frames returns one concept across every company for a single period. The choice depends on whether you're building a dataset for one company or one metric across the market.

How do I know if a tag should be instant or duration?

The taxonomy declares it. Every concept in the US-GAAP taxonomy has a fixed period type set when the concept was defined — balance sheet items like Assets are instant, income statement and cash flow items like Revenues are duration — and a filer cannot legally attach the wrong type of context to a concept.

Why is my filing history missing older filings?

The submissions endpoint's filings.recent block only covers roughly the last year or 1,000 filings, whichever is larger. Older filings are referenced separately under filings.files and need to be fetched and merged in.

Why do I get two values for the same fiscal quarter?

A period reported again in a later filing — as a comparative figure or a corrected number — stays in the record alongside the original, tied to a different accession number. Deduplicate by keeping the entry with the most recent filed date for each (fy, fp, end) combination.

Can I use the end date to figure out which fiscal quarter a filing covers?

No, not reliably. Companies with non-calendar fiscal years — Apple, Microsoft, and many others — have an end date that doesn't align with calendar quarters. Use the fp field (Q1, Q2, Q3, FY) instead.

Do I need an API key for the SEC EDGAR XBRL API?

No. None of the endpoints require registration or a key. They do require a descriptive User-Agent header with contact information, and requests are capped at 10 per second per IP address.

A script that runs without errors and a script that returns correct data are two different bars to clear. Everything in this guide is aimed at the second one — and at being clear about where the second one still isn't enough.

Start your 7-day free trial →

Wall Street's data. Main Street's price.

Institutional terminals charge thousands a year for as-filed accuracy. GeminIQ gives you the same thing for a fraction of the cost: financials built directly from raw SEC EDGAR filings, not third-party APIs, with full XBRL traceability back to the original 10-K or 10-Q. No normalized guesswork, just calculated metrics, charts, screeners, and watchlists built on numbers exactly as the company reported them. Start researching now at GeminIQ.com.

Data Used / Sources

  • Endpoint URL patterns, response fields, the 10-requests-per-second rate limit, the User-Agent requirement, the frames "last filed" aggregation rule, and the nightly bulk companyfacts.zip archive are all as documented at data.sec.gov and sec.gov/os/accessing-edgar-data.
  • The startDate / endDate boundary convention and the quoted language on tagging errors are from XBRL's own published guidance on period contexts.
  • RevenueFromContractWithCustomerExcludingAssessedTax and the 2018 taxonomy / ASC 606 transition referenced from the FASB US GAAP Financial Reporting Taxonomy release notes.
  • Sample JSON responses are illustrative of each endpoint's structure; values shown reference Apple Inc. and AAR Corp filings via SEC EDGAR.

Disclaimer: The content in this blog is for educational and entertainment purposes only and does not constitute financial, legal, or tax advice. Investing involves risk, including the loss of principal. The views expressed are my own and not intended as financial advice or a guarantee of future performance.