Bruh, this is literally me rn. My buddy said it'd be cool if I could parse SEC docs, started with one, it was going great, until I checked another filing and realized nobody uses the same wording and therefore my program was great for a single corporation lol. Now I am here!
@@misterespresso This is also 'literally me' rn (right now). Have you made progress on your script? Would be nice to be in touch with someone working on a similar project. Determining sufficient 12mo runway feels like looking at Everest from below, as like you said, the language is entirely different in each filing.
What's up Miguel - it's hard to say without seeing your code. You can find all the code from the video on my github. This way you can follow along. github.com/AdamGetbags/secAPI/blob/main/secFilingScraper.py DM me on IG, Twitter or send me an email (you can find my email on the about section on my YT channel) if you need to book a technical consultation.
@@damgetbags I fixed it. I was following another video but I didn't realize the video was made for the old SEC structure. I followed yours and everything worked perfectly. Now I'm just trying to figure out how to pull only the relevant data. Thanks for the great video.
Are you ready to get this bag, Y/N? THROW SOME "Y"s IN THE COMMENTS IF YOU GETTING THIS BAG. Subscribe to join our community and receive value on all things finance and technology! If you love the content, buy me a coffee! www.buymeacoffee.com/adamgetbags Keep up with me! instagram.com/damgetbags twitter.com/damgetbags
ADAM, dude! This tutorial is INCREDIBLE! I cannot tell you how much I am thankful for the one-time chance to having found it! I subscribed, liked the vid and would be so much looking forward for more of your know-how to be shared with the community! The coffee was the least we could spend to say thank you!
Yo Adam great video. I don't often comment on videos but I just had to complement this, it's really good. The only thing I wish you did was maybe move through a bit slower, you were kinda rushing everything. But that's not really the most important thing I can decrease the speed when I need to. Other than that great video. Straight to the point. Very good audio quality. Charismatic dude leading through the subject matter in consistent, linear fashion. Keep it up!
sup cedomir, thank you for the feedback. video speed is one of those things i kinda have to bite the bullet on. most people have tiny attention spans, so I try to keep the videos moving to keep them engaging (as engaging as technical finace videos can be). can't lie, it would be pretty funny to show an unedited video - there are usually random errors and angry flying curse words.
@@damgetbags Please do show your unedited video. There are some things that don't work out perfectly that you might have edited out of this video. But similar or the exact problems might be faced by the person following your video. And if you find the solution in that video for that problem, other people will also do so by following your instructions in the longer video. Anyways thanks for this video. It is a great starting point for what I have cooking in my head.
Great video. Thank you. How can you get general filing updates from the API without using a specific CIK number? As far as I know, you can only get ticker data if you have the cik number.
I'm facing challenges in processing financial data from multiple companies at once, as each company uses its own unique tag for the same financial items. For instance, NVIDIA (NVDA) refers to its sales as 'Revenues', while Apple (AAPL) uses 'RevenueFromContractWithCustomerTextBlock'. This variation in the naming of financial tags across different companies makes it challenging to aggregate their financial data effectively. Is there a way to gather and use revenue data from various companies, considering that each company might refer to revenue using different terms? Can we retrieve this specific financial information (like revenue) from multiple companies in a unified manner?
this is a common feedback i hear about this video - you might be able to access more uniform financial statements here www.sec.gov/dera/data/financial-statement-data-sets there's also a github repo for accessing the data in the above link. but you might want to verify the author / repo safety github.com/HansjoergW/sec-fincancial-statement-data-set
Legend brother! To be honest after SMCI crash (huge money loss for me) I am thinking of trying to do this with a WEBSOCKET type connection and how to trade based on that with LLM.
Thank you for your effort. I have a question, please. I always receive information with a delay of about a minute. For example, the 8-K file appeared on the sec.gov page at 2024-05-10 21:58:14, but it only showed up one minute later, whether I used Python or even when I refreshed the page every 3 seconds. What could be the problem, and how can I obtain real-time information?
of course bro, under the heading 'update schedule' they have some details. could be some delay, but depends how you're polling the API www.sec.gov/edgar/sec-api-documentation
@AdamGetbaggs How come when I run this code, at the end, my "assets10Q" dataframe only has 132 rows and 8 columns? I believe you also had a low amount of rows...Shouldnt this pull all companies? I am confused... Also why doesnt CIK show as one of the columns?
This was very helpful, great work I also wanted to pull data for RSU's and PSU's granted but am unable to do it. Is there a way to pull it using SEC Edgar API
Thanks Adam. Data filtration question: Using the Edgar API, what filter do I use in order to get the quarterly earnings date? I'm getting a lot of dates that are not quarterly earnings date related (e.g., Director changes, etc.).
glad to help. without getting into too much detail, i would probably filter by form to get the 10Q forms. If you have the accession number for the 10Q forms, you should be able to find the dates related to that form/accession number
@@damgetbags Thanks, Adam. I did set the form to be "10-Q", but that only lists 3 quarters per year. The "10-K" lists the 4th one (I assume annual is the same as the 4th quarter), but its date is different than the actual earnings announcement date.
Awesome video! Great content and easy to follow along especially for a python newbie like myself. I was wondering how you were able to format the console to return the data in the format the way it did? I'm current using vs code and it doesn't look that way.
Hey Jervin, On the API docs there is a section on the left side nav bar that takes you to latest filings. Here's the link. www.sec.gov/cgi-bin/browse-edgar?action=getcurrent You could scrape this data, or use polling to check ALL the CIKs for new filings. However, there may be a better way to get what you're looking for.
Super helpful, what if you want it as it comes in live, do you just keep polling the API, if for example earnings is coming in at 4:00 pm EST you just keep hitting the api real time?
sup natty, without putting to much thought into infra, polling would probably be the way. however, there are some other services ($$$) that have RSS feeds/websockets. but it is entirely possibly they source the data through polling too. let me know how your build goes. AG
@@damgetbags I did it, but it gave me reports of 10-q only from 2013 for Apple, the code is same till the point I filtered allForms where form=10-q. Can I ask why?
I am trying to extract historical 5 years of revenue data for APPL from 10-K forms, do you have a tutorial for parsing 10-K forms? In the company-concept and company-fact dictionary, I can only see 2018 numbers, not sure why.
The one problem I found with the SEC API is custom XBRL (non us-GAAP) line items are not found in the JSON files. If you want to reconstruct the financial statements you may also have to parse the XML. Working through the best way to do that now. Any ideas you have would be great.
Sup Jerry. Solid observation. This API is very nuanced, and there is a lot of detail in the documentation. I'm working on a SEC form 4 xml parser. When I make the video, you can take the general idea from there. However, form 4 doesn't change 'shape', but financial statements vary from issuer to issuer. You'll see what I mean once I publish the video.
Maybe you addressed this in the video and I missed it, but I am trying to pull one "fact" for all companies and keep failing. Ex. Pulling 'Revenues' data for all companies into a single table for querying. Any idea on how to accomplish this?
around 7:00 i go into detail about how some fact/keys do not return too much data and how keys can be deprecated. using the method from this video, the first thing i would check is to see if the ~naming~ of fact/keys vary from company to company. for example, sales, revenue, revenues. i would take a careful look at each company's keys to determine how you can standardize your data ingestion. i should be making a video updating this information soon, stay tuned.
What do you do if the filing data is seriously insufficient. For example, PANW only has revenue facts up to 2017! Do you have another tutorial that can show us how to get data that is garuanteed to be complete (like all the stuff compnaies have to file).
Hey, it looks like he has an output terminal fully opened on the right and he’s block executing code by using the a command that executes the highlighted code. Some people have different shortcut keys for it so it’s best to look into what your code editor uses
Is there a way to get the data more efficiently? Meaning, say you have 300 companies you want to track Insider Trading for (Form 4), it looks like each time you have to pull all filings for each company (4, 8-K, 10K/Q etc) first, then filter specifically for Form 4 and probably at the end for the last X days if you already have previously reported trades.
Hi Matt, Thanks for subscribing. I get what you mean, and yes that appears to be the brute force solution. The short answer is you can get the data more efficiently, but it might not be the MOST efficient way. Take the below idea - First, poll and scrape the results page from the below link, filter for is form 4 AND is company on your list. Then, you only request data for the companies that pass the filter. So that way you don't have to poll all 300 - just the ones that show up. www.sec.gov/edgar/searchedgar/currentevents
On the last data frame "assets10Q", why your cik is not consistent? You only selected cik for Apple which is 320193! But initial rows contain 1193125! clarification will be appreciated!
sup syed. i see what you mean, the accession number on the earlier rows have a different prefix than the later accession numbers, which contain the CIK 320193. im not sure why this is the case. be aware that CIK and accession number are different. accession numbers are unique to the form submission, while CIK are more to do with identifying the company - rather than a identifying a form.
Thanks Depth Wish, glad you are part of the finance family. For now, I focus on data processing, technical skills, and quantitative methods that will help people get better jobs and automate their businesses. The type of content you mention may or may not be "Phase Two" for this channel. Stay tuned.
i dont think so according to the link below, Currently included in the APIs are the submissions history by filer and the XBRL data from financial statements (forms 10-Q, 10-K,8-K, 20-F, 40-F, 6-K, and their variants). www.sec.gov/edgar/sec-api-documentation
Hi Adam, love your content. It was really helpful. I just had a small question, I am pulling revenues data for a bunch of companies and even though they have reported revenues for a particular year, the api can't seem to find it. Can you please help me with this?
Hey Dhruv, you may find the line items from the actual 10K and 10Q reports can be different than the line items from the API. Reach out to me on IG, Twitter, or my email in the about section of my channel if you'd like to book a consultation!
Hey adam, is there way of automatize the code so I can input the company name or ticker in one part of the code INSTEAD of the key, In a way it prints all the following data based on that input
sup francisco, so around 1:56 where I show the CIK mappings, you can use the mappings to map CIK to ticker. so, the opposite is true. perhaps write a function that, given a ticker, returns a CIK. then, use the CIK to grab the data you need. you see?
@@damgetbags Yes. I get positions from url (f'data.sec.gov/api/xbrl/companyfacts/CIK{cik}.json',) and which one is balance sheet, cashflow, P&L? Is there any indicator or I should map each company line-item manually to those 3 statements? Short example: AccountsPayableAndAccruedLiabilitiesCurrent 10-Q 2024 Q2 44 Mln 2024-05-09 CY2024Q1I AccountsReceivableGross 10-Q 2024 Q2 0 Mln 2024-05-09 CY2024Q1I AccountsReceivableNet 10-Q 2024 Q2 0 Mln 2024-05-09 CY2024Q1I AccountsReceivableNetCurrent 10-Q 2024 Q2 0 Mln 2024-05-09 CY2024Q1I AccumulatedDepreciationDepletionAndAmortizationPropertyPlantAndEquipment 10-Q 2024 Q2 223 Mln 2024-05-09 CY2024Q1I AccumulatedOtherComprehensiveIncomeLossNetOfTax 10-Q 2024 Q2 0 Mln 2024-05-09 CY2024Q1I AdditionalPaidInCapital 10-Q 2024 Q2 1,642 Mln 2024-05-09 CY2024Q1I AllowanceForDoubtfulAccountsReceivableCurrent 10-Q 2024 Q2 0 Mln 2024-05-09 CY2024Q1I Assets 10-Q 2024 Q2 1,537 Mln 2024-05-09 CY2024Q1I
Thank you for making a wonderful instructional video, Adam. I think the SEC publishes a list of CIKs of companies reporting each day. Do you know how to get this list via Pythong? Thanks again!!!
I haven't looked very hard, but on the API docs there is a section on the left side nav bar that takes you to latest filings. Here's the link. www.sec.gov/cgi-bin/browse-edgar?action=getcurrent You could scrape this data, or use polling to check ALL the CIKs for new filings. However, there may be a better way to get what you're looking for.
Hi Adam. Just wondering if there is a way to get more specific info than just the tag? Eg instead of just us-gaap: RevenueFromContractWithCustomerExcludingAssessedTax, could we get the Member or Axis too (to distinguish between revenues from different things)? Thank you!
Hi Dr. Unea, Parse through all the CIK ticker data to find Mastercard stock ticker, then take the CIK number from that entry, and pull that data down. DM me on Twitter, IG, or send me an email (you can find my email on the about section of my YT channel) to book a technical consultation. -AG
@@damgetbags COGS for Mastercard takes into account what you will find in the SEC under the "Personnel" key. For the year 2022, for example, the value is 5,263. You will not find this parameter in the key list. I asked the question specifically to emphasize that not all parameters are available to automatically import financial data. What we see on certain sites (such as yahoo finance or other guys you say make a lot of money by exploiting this data) are data collected both from the SEC API but also through the particular interpretation of the reports in html or pdf format. Most use AI to gather this data. The conclusion would be that SEC API is not enough to create a financial situation.
Hey Adam, i have seen your video on SEC Edgar and i want help from you on that i want to fetch security shareholders and security stockholders from form S-1 and 1-A i am a new developer and this my first client so i am on the edge to lost my first work could you please help me.
Hi Adam, thank you for sharing the information. I noticed that there is an issue with the data. It appears that there is missing data, particularly when it comes to Apple's quarterly revenue. The available data only goes up to 2018Q3. I used the code of pd.DataFrame.from_dict((companyFacts.json()['facts']['us-gaap']['Revenues']['units']['USD']))
Thanks for watching - I know what you mean. I believe if the issuer changes the name of the line item on their financials, then the data will have a different key name in the dictionary. I'm not sure if this is the case for Apple; but for example, if the name goes from revenue to sales, then I think that revenue key will not have new values, but sales key will have the new values. You feel?
Im getting this error "line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) JSONDecodeError: Expecting value" when we go through how to review the JSON data I added sec_response = requests.get(filingMetadata, headers=headers), and it looks like there is a 403 (forbidden access) error is anyone else getting this? I'm guessing its an error with the header, this is mine headers = {'Actual Name': "Actual Email"} is anyone else getting this? I am able to pull the CIK data without any problem, does that mean it's not an issue with the header?
Hey Connor, Projects can be frustrating, I do technical tutoring if needed. You can send me an email w/ details (you can find email in the about section on my channel) or a DM on twitter or instagram!
bro this saved me so much time. was about to start writing this from scratch parsing excel files and wanted to die
hell yeah, that's how we do it around here.
Bruh, this is literally me rn. My buddy said it'd be cool if I could parse SEC docs, started with one, it was going great, until I checked another filing and realized nobody uses the same wording and therefore my program was great for a single corporation lol. Now I am here!
@@misterespresso This is also 'literally me' rn (right now). Have you made progress on your script? Would be nice to be in touch with someone working on a similar project. Determining sufficient 12mo runway feels like looking at Everest from below, as like you said, the language is entirely different in each filing.
Nicely done! Love this video. Keep up the great work.
appreciate you
I came from another video looking for ways to fix the "403" error.
Thank you!
Your channel is a gold mine.
What's up Miguel - it's hard to say without seeing your code.
You can find all the code from the video on my github. This way you can follow along.
github.com/AdamGetbags/secAPI/blob/main/secFilingScraper.py
DM me on IG, Twitter or send me an email (you can find my email on the about section on my YT channel) if you need to book a technical consultation.
@@damgetbags I fixed it.
I was following another video but I didn't realize the video was made for the old SEC structure.
I followed yours and everything worked perfectly. Now I'm just trying to figure out how to pull only the relevant data.
Thanks for the great video.
Are you ready to get this bag, Y/N?
THROW SOME "Y"s IN THE COMMENTS IF YOU GETTING THIS BAG.
Subscribe to join our community and receive value on all things finance and technology!
If you love the content, buy me a coffee!
www.buymeacoffee.com/adamgetbags
Keep up with me!
instagram.com/damgetbags
twitter.com/damgetbags
ADAM, dude!
This tutorial is INCREDIBLE! I cannot tell you how much I am thankful for the one-time chance to having found it! I subscribed, liked the vid and would be so much looking forward for more of your know-how to be shared with the community! The coffee was the least we could spend to say thank you!
@@justtohost3431 Great to hear from you, and thank you so much for the coffee. v delicious.
Grateful for the support - more content coming your way!
Thank you Adam for this information, super helpful.
cheers mate
Keep up the work brother, this helped a lot.
Awesome, loved the way you explained
Yo Adam great video. I don't often comment on videos but I just had to complement this, it's really good. The only thing I wish you did was maybe move through a bit slower, you were kinda rushing everything. But that's not really the most important thing I can decrease the speed when I need to.
Other than that great video. Straight to the point. Very good audio quality. Charismatic dude leading through the subject matter in consistent, linear fashion. Keep it up!
sup cedomir, thank you for the feedback.
video speed is one of those things i kinda have to bite the bullet on. most people have tiny attention spans, so I try to keep the videos moving to keep them engaging (as engaging as technical finace videos can be).
can't lie, it would be pretty funny to show an unedited video - there are usually random errors and angry flying curse words.
@@damgetbags Please do show your unedited video. There are some things that don't work out perfectly that you might have edited out of this video. But similar or the exact problems might be faced by the person following your video. And if you find the solution in that video for that problem, other people will also do so by following your instructions in the longer video.
Anyways thanks for this video. It is a great starting point for what I have cooking in my head.
Great video. Thank you. How can you get general filing updates from the API without using a specific CIK number? As far as I know, you can only get ticker data if you have the cik number.
Thank you Adam for this information, its very super helpful)
I'm facing challenges in processing financial data from multiple companies at once, as each company uses its own unique tag for the same financial items. For instance, NVIDIA (NVDA) refers to its sales as 'Revenues', while Apple (AAPL) uses 'RevenueFromContractWithCustomerTextBlock'. This variation in the naming of financial tags across different companies makes it challenging to aggregate their financial data effectively.
Is there a way to gather and use revenue data from various companies, considering that each company might refer to revenue using different terms? Can we retrieve this specific financial information (like revenue) from multiple companies in a unified manner?
this is a common feedback i hear about this video -
you might be able to access more uniform financial statements here
www.sec.gov/dera/data/financial-statement-data-sets
there's also a github repo for accessing the data in the above link.
but you might want to verify the author / repo safety
github.com/HansjoergW/sec-fincancial-statement-data-set
Hey buddy! I love your videos! Thanks a bunch!
Awesome! Could you help download the bulk of Accounting and Auditing Enforcement releases AAER from the SEC.
you can download just about anything with code.
send me an email if you need a custom script, thanks!
Thanks for this. Subscribed!
great video, you are a smart dude
thanks fam, more vids on the way
thanks for the video! how do you keep track of split stocks? when I look at old data everything is non-realigned
it depends, but a lot of time series data providers will have stock split data.
i'd say it really depends on the time series you are looking at.
This is similar to how I do it, but I often use the ticker symbol with frames.
I see what you mean, definitely want to keep as much info as you can.
Much respect, I appreciate you homie🤘🏻
100% fam, good to have you around!
Legend brother! To be honest after SMCI crash (huge money loss for me) I am thinking of trying to do this with a WEBSOCKET type connection and how to trade based on that with LLM.
very nice , do you have any idea on ml project way of auto tagging previous filing data to current filing period .
impressive, very nice.
sup baskarp, your question is not entirely clear.
what do you mean?
Thank you for your effort. I have a question, please. I always receive information with a delay of about a minute. For example, the 8-K file appeared on the sec.gov page at 2024-05-10 21:58:14, but it only showed up one minute later, whether I used Python or even when I refreshed the page every 3 seconds. What could be the problem, and how can I obtain real-time information?
of course bro,
under the heading 'update schedule' they have some details.
could be some delay, but depends how you're polling the API
www.sec.gov/edgar/sec-api-documentation
Can you do a video specifically for Form 4?
Hey Kivilig, might look into that.
I think that could be pretty valuable for an app build!!
How can I standardize the statements hierarchy? For example, where do I find what accounts are “Revenues”?
How do I search for specific word or phrase within filing documents for filing types 6-K and 8-K?
Many thanks for your kind support
@AdamGetbaggs How come when I run this code, at the end, my "assets10Q" dataframe only has 132 rows and 8 columns? I believe you also had a low amount of rows...Shouldnt this pull all companies? I am confused... Also why doesnt CIK show as one of the columns?
Great video man. Thank you
more vids coming fam
This was very helpful, great work
I also wanted to pull data for RSU's and PSU's granted but am unable to do it.
Is there a way to pull it using SEC Edgar API
Thanks Adam. Data filtration question: Using the Edgar API, what filter do I use in order to get the quarterly earnings date? I'm getting a lot of dates that are not quarterly earnings date related (e.g., Director changes, etc.).
glad to help.
without getting into too much detail, i would probably filter by form to get the 10Q forms.
If you have the accession number for the 10Q forms, you should be able to find the dates related to that form/accession number
@@damgetbags Thanks, Adam. I did set the form to be "10-Q", but that only lists 3 quarters per year. The "10-K" lists the 4th one (I assume annual is the same as the 4th quarter), but its date is different than the actual earnings announcement date.
Awesome video! Great content and easy to follow along especially for a python newbie like myself. I was wondering how you were able to format the console to return the data in the format the way it did? I'm current using vs code and it doesn't look that way.
I was using spyder, from anaconda navigator to write this code.
One of my first videos, enjoy.
ruclips.net/video/Nns9Zb8m0m8/видео.html
Hi Adam, Amazing work. I wonder whether we can get the narrative parts of the 10-K such as Item 1, Item 1A ... using this API?
that would be pretty tight for NLP / LLM apps
it looks like this topic is pretty popular, might have to follow up
Damn, you really know your stuff!
Zenon - glad you are enjoying the content. Thank you for joining us!
hi adam, this is a nice video. i'm just wondering if there is a way to pull latest filling without cik? just latest fillings by whichever company.
Hey Jervin,
On the API docs there is a section on the left side nav bar that takes you to latest filings.
Here's the link.
www.sec.gov/cgi-bin/browse-edgar?action=getcurrent
You could scrape this data, or use polling to check ALL the CIKs for new filings. However, there may be a better way to get what you're looking for.
Thank you for the tutorial
Super helpful, what if you want it as it comes in live, do you just keep polling the API, if for example earnings is coming in at 4:00 pm EST you just keep hitting the api real time?
sup natty,
without putting to much thought into infra, polling would probably be the way.
however, there are some other services ($$$) that have RSS feeds/websockets. but it is entirely possibly they source the data through polling too.
let me know how your build goes.
AG
This tutorial has me cheesing like it's picture day
Hell ya, we getting this cheddar.
Hey Adam, nice video. Is there a video or a snippet you can add on how to access the 10-K/10-Q
Hi Rajesh,
I would probably sort the data you got by "Form" and "Accession Number". So, you can look at just the data from a specific form.
@@damgetbags I did it, but it gave me reports of 10-q only from 2013 for Apple, the code is same till the point I filtered allForms where form=10-q. Can I ask why?
I am trying to extract historical 5 years of revenue data for APPL from 10-K forms, do you have a tutorial for parsing 10-K forms? In the company-concept and company-fact dictionary, I can only see 2018 numbers, not sure why.
hey eliza,
check out my response to the comment from user-lt1mc3zr6f
hopefully thats helpful
The one problem I found with the SEC API is custom XBRL (non us-GAAP) line items are not found in the JSON files. If you want to reconstruct the financial statements you may also have to parse the XML. Working through the best way to do that now. Any ideas you have would be great.
Sup Jerry. Solid observation.
This API is very nuanced, and there is a lot of detail in the documentation.
I'm working on a SEC form 4 xml parser.
When I make the video, you can take the general idea from there.
However, form 4 doesn't change 'shape', but financial statements vary from issuer to issuer.
You'll see what I mean once I publish the video.
This is awesome!
lets go!
Maybe you addressed this in the video and I missed it, but I am trying to pull one "fact" for all companies and keep failing. Ex. Pulling 'Revenues' data for all companies into a single table for querying. Any idea on how to accomplish this?
around 7:00 i go into detail about how some fact/keys do not return too much data and how keys can be deprecated.
using the method from this video, the first thing i would check is to see if the ~naming~ of fact/keys vary from company to company.
for example, sales, revenue, revenues.
i would take a careful look at each company's keys to determine how you can standardize your data ingestion.
i should be making a video updating this information soon, stay tuned.
What do you do if the filing data is seriously insufficient. For example, PANW only has revenue facts up to 2017! Do you have another tutorial that can show us how to get data that is garuanteed to be complete (like all the stuff compnaies have to file).
Would it also be possible to have real-time data for approvals of ETF fillings ?
Hey ComanderLoris,
If the SEC API has a form for it, you will be able to see it here.
Great video Adam! Can you tell me what's the right panel that executes code block by block?
Hey, it looks like he has an output terminal fully opened on the right and he’s block executing code by using the a command that executes the highlighted code. Some people have different shortcut keys for it so it’s best to look into what your code editor uses
@@joshnijjar978 Thanks for the answer. I used to not know what Jupyter notebook was.
yeah boy. well done. keep udpate new videos!
Hell yeah, stay tuned!
Is there a way to get the data more efficiently? Meaning, say you have 300 companies you want to track Insider Trading for (Form 4), it looks like each time you have to pull all filings for each company (4, 8-K, 10K/Q etc) first, then filter specifically for Form 4 and probably at the end for the last X days if you already have previously reported trades.
Hi Matt, Thanks for subscribing.
I get what you mean, and yes that appears to be the brute force solution.
The short answer is you can get the data more efficiently, but it might not be the MOST efficient way. Take the below idea -
First, poll and scrape the results page from the below link, filter for is form 4 AND is company on your list. Then, you only request data for the companies that pass the filter. So that way you don't have to poll all 300 - just the ones that show up.
www.sec.gov/edgar/searchedgar/currentevents
On the last data frame "assets10Q", why your cik is not consistent? You only selected cik for Apple which is 320193! But initial rows contain 1193125!
clarification will be appreciated!
sup syed. i see what you mean, the accession number on the earlier rows have a different prefix than the later accession numbers, which contain the CIK 320193.
im not sure why this is the case.
be aware that CIK and accession number are different.
accession numbers are unique to the form submission, while CIK are more to do with identifying the company - rather than a identifying a form.
How's it going Adam. Are you doing any algo trading? It would be interesting to discuss strategies and using python/apis to do it. Great videos btw!
Thanks Depth Wish, glad you are part of the finance family.
For now, I focus on data processing, technical skills, and quantitative methods that will help people get better jobs and automate their businesses.
The type of content you mention may or may not be "Phase Two" for this channel. Stay tuned.
how to get vendors contract with company data
Is there a way to get 13F filings using this API, too?
i dont think so
according to the link below,
Currently included in the APIs are the submissions history by filer and the XBRL data from financial statements (forms 10-Q, 10-K,8-K, 20-F, 40-F, 6-K, and their variants).
www.sec.gov/edgar/sec-api-documentation
Really cool. Thank you
Hey Jeremy, glad you like the content and thx for subscribing!
Hi Adam, love your content. It was really helpful. I just had a small question, I am pulling revenues data for a bunch of companies and even though they have reported revenues for a particular year, the api can't seem to find it. Can you please help me with this?
Hey Dhruv, you may find the line items from the actual 10K and 10Q reports can be different than the line items from the API.
Reach out to me on IG, Twitter, or my email in the about section of my channel if you'd like to book a consultation!
Hey adam, is there way of automatize the code so I can input the company name or ticker in one part of the code INSTEAD of the key, In a way it prints all the following data based on that input
sup francisco, so around 1:56 where I show the CIK mappings, you can use the mappings to map CIK to ticker. so, the opposite is true.
perhaps write a function that, given a ticker, returns a CIK. then, use the CIK to grab the data you need. you see?
Is there any indication of statement positions like 'Assets' that it belong to Balance Sheet statement? They are stored there without any logic.
what do you mean? this video uses assets from the balance sheets.
do you mean what sub categories are part of assets, liabilities, etc?
@@damgetbags Yes. I get positions from url (f'data.sec.gov/api/xbrl/companyfacts/CIK{cik}.json',) and which one is balance sheet, cashflow, P&L? Is there any indicator or I should map each company line-item manually to those 3 statements? Short example: AccountsPayableAndAccruedLiabilitiesCurrent 10-Q 2024 Q2 44 Mln 2024-05-09 CY2024Q1I
AccountsReceivableGross 10-Q 2024 Q2 0 Mln 2024-05-09 CY2024Q1I
AccountsReceivableNet 10-Q 2024 Q2 0 Mln 2024-05-09 CY2024Q1I
AccountsReceivableNetCurrent 10-Q 2024 Q2 0 Mln 2024-05-09 CY2024Q1I
AccumulatedDepreciationDepletionAndAmortizationPropertyPlantAndEquipment 10-Q 2024 Q2 223 Mln 2024-05-09 CY2024Q1I
AccumulatedOtherComprehensiveIncomeLossNetOfTax 10-Q 2024 Q2 0 Mln 2024-05-09 CY2024Q1I
AdditionalPaidInCapital 10-Q 2024 Q2 1,642 Mln 2024-05-09 CY2024Q1I
AllowanceForDoubtfulAccountsReceivableCurrent 10-Q 2024 Q2 0 Mln 2024-05-09 CY2024Q1I
Assets 10-Q 2024 Q2 1,537 Mln 2024-05-09 CY2024Q1I
Thank you for making a wonderful instructional video, Adam. I think the SEC publishes a list of CIKs of companies reporting each day. Do you know how to get this list via Pythong? Thanks again!!!
I haven't looked very hard, but on the API docs there is a section on the left side nav bar that takes you to latest filings.
Here's the link.
www.sec.gov/cgi-bin/browse-edgar?action=getcurrent
You could scrape this data, or use polling to check ALL the CIKs for new filings. However, there may be a better way to get what you're looking for.
Hi Tim - Also, see my reply to Matt's comment on this thread. It has what you're looking for.
Hi Adam. Just wondering if there is a way to get more specific info than just the tag? Eg instead of just us-gaap: RevenueFromContractWithCustomerExcludingAssessedTax, could we get the Member or Axis too (to distinguish between revenues from different things)? Thank you!
very helpful
How do you calculate COGS using this API for Mastercard?
Hi Dr. Unea,
Parse through all the CIK ticker data to find Mastercard stock ticker, then take the CIK number from that entry, and pull that data down.
DM me on Twitter, IG, or send me an email (you can find my email on the about section of my YT channel) to book a technical consultation.
-AG
@@damgetbags COGS for Mastercard takes into account what you will find in the SEC under the "Personnel" key. For the year 2022, for example, the value is 5,263.
You will not find this parameter in the key list.
I asked the question specifically to emphasize that not all parameters are available to automatically import financial data. What we see on certain sites (such as yahoo finance or other guys you say make a lot of money by exploiting this data) are data collected both from the SEC API but also through the particular interpretation of the reports in html or pdf format. Most use AI to gather this data.
The conclusion would be that SEC API is not enough to create a financial situation.
@@humidorro good observation!
Hi can you help me scrap financial sec data from website gm
gm, of course thats why i made the video.
Hey Adam, i have seen your video on SEC Edgar and i want help from you on that i want to fetch security shareholders and security stockholders from form S-1 and 1-A i am a new developer and this my first client so i am on the edge to lost my first work could you please help me.
i’ll send you a DM on twitter later.
1:35
hell yeah brother
2:09
Wouw and I thought you could only get this by paling for a subscription
Hi Bram,
Glad to help out.
You can always buy me a coffee with the money you saved, HA.
www.buymeacoffee.com/adamgetbags
Thank you : )
Of course finance fam!
Nice!
Right?! This one takes a bit of navigating, but the data is pretty rich.
There's no good libraries for this, I've modified your script to have functions and stuff and am considering uploading it to pypi
Hi Adam, thank you for sharing the information. I noticed that there is an issue with the data. It appears that there is missing data, particularly when it comes to Apple's quarterly revenue. The available data only goes up to 2018Q3. I used the code of pd.DataFrame.from_dict((companyFacts.json()['facts']['us-gaap']['Revenues']['units']['USD']))
Thanks for watching - I know what you mean.
I believe if the issuer changes the name of the line item on their financials, then the data will have a different key name in the dictionary.
I'm not sure if this is the case for Apple; but for example, if the name goes from revenue to sales, then I think that revenue key will not have new values, but sales key will have the new values. You feel?
Im getting this error
"line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
JSONDecodeError: Expecting value"
when we go through how to review the JSON data
I added sec_response = requests.get(filingMetadata, headers=headers), and it looks like there is a 403 (forbidden access) error
is anyone else getting this? I'm guessing its an error with the header, this is mine
headers = {'Actual Name': "Actual Email"}
is anyone else getting this? I am able to pull the CIK data without any problem, does that mean it's not an issue with the header?
Hi Connor, not sure if you got this resolved, but you can get the exact code from the video from my Github link near the top of the description!
@@damgetbags thanks, i was getting the error using the code from your github, still trying to figure this one out
Hey Connor,
Projects can be frustrating, I do technical tutoring if needed.
You can send me an email w/ details (you can find email in the about section on my channel) or a DM on twitter or instagram!
@@damgetbags figured it out, i didnt realize "User Agent" wasnt supposed to be my name...