Extracting time series data from MULTIPLE netCDF files into a CSV (Part 6)

Поделиться
HTML-код
  • Опубликовано: 26 ноя 2024

Комментарии • 99

  • @geodeltalabs
    @geodeltalabs  4 года назад +1

    Link to Part 7 - ruclips.net/video/FzMQs8oTS3Q/видео.html

    • @engr.abdullahazzamsafi4515
      @engr.abdullahazzamsafi4515 4 года назад

      Hi, i have created my own code with help of this video but the problem in data is that t_index is 60225 but actual df column rows for 01/01/1850 to 31/12/2014 is 60265. Hence, it displaying error: index exceeds dimension bounds. We would be very thankful to you to solve our problem. Thanking you in anticipation

    • @basemel-shafei6452
      @basemel-shafei6452 4 года назад +1

      Hello there,
      Thank you very much for this content it is exactly what i have been looking for. However, as I was applying this to my files, i realised my Time dimensions doesn't have units. any suggestions on how to carry the same task when there is no unit for time.
      Thank you.

    • @engr.abdullahazzamsafi4515
      @engr.abdullahazzamsafi4515 4 года назад

      @@basemel-shafei6452 Hi Dear Brother, I have been working on climate data and i have faced such kind of challenges. It is actually the data problem. It doesn't have data of leap year day i.e. 29th of Feb after each 4 years. Therefore, it has such kind of problem. But it easy to solve it by one way to download monthly data if monthly data meets your research objectives.

  • @lngl2786
    @lngl2786 4 года назад +11

    Man. I have no words. You're just amazing. Not only providing a unique topics but also your slow and clear approach makes you a wonderful teacher. Glad to know you and pray that you may stay safe!

  • @pacomermela6497
    @pacomermela6497 Год назад +1

    Even better than ChatGPT. Amazing tutorial. This is how a data scientist works... Printing variables, testing iterations, ensuring data types, printing whats happening inside loops and testing every step. I feel very identified with this way of coding... It is very intuitive. After that you can simplify and optimize your Code. Congrats Sr.

  • @alimkotlar9482
    @alimkotlar9482 2 года назад +1

    in the sixth minute and cant wait to thank you for such a wonderful explanation. Keep posting👍

  • @sandeepbhatti3124
    @sandeepbhatti3124 Год назад

    Thank you so much, my friend! Love your content and how you explain each line of code super clearly.

  • @hafeezo4439
    @hafeezo4439 4 года назад

    Nice one, Your tutorials have been amazing, they are one in a million. Looking forward to more.

  • @engr.abdullahazzamsafi4515
    @engr.abdullahazzamsafi4515 4 года назад

    You are amazing really a very great job. I have modified the script for precipitation by myself. Millions thanks

  • @ZULKIFLIRAMADHAN
    @ZULKIFLIRAMADHAN 2 года назад

    bruh million thanks for you, you're my savior I'm absolutely gonna subscribe this channel it's a really helpful tutorial

  • @IWayanAndiYuda
    @IWayanAndiYuda 4 года назад

    thank you very much. i tried to plot rainfall.nc from TRMM (multiples files). and here i got the way how to do that efficiently. thank you very much

  • @wellmanlake
    @wellmanlake 3 года назад +2

    Spectacular series!!! I followed along the tutorial with my own data and was able to create a time series. Thanks a lot!!!! Would you consider making a tutorial on handling .HDF5 files? The IMERG data is in that format, and I tried to extract a time series with the h5py python package and NASA's IMERG data but I couldnt. Anyway thanks a lot for your amazing work! Cheers from Santo Domingo!

  • @muhammadanjum524
    @muhammadanjum524 4 года назад +1

    Great work

  • @adetayoeyelade3912
    @adetayoeyelade3912 Год назад

    Great tutorials, what about if I want to average my variables at 1min intervals for the whole year under consideration?

  • @limbermontenegro5201
    @limbermontenegro5201 4 года назад +1

    Thank you so much. How would you go about extracting data from a single file containing 36 years

  • @kofiopare4111
    @kofiopare4111 Год назад

    Thank you very much for this tutorial. I have followed through and been able to do everything until the last line which is giving me an error that index exceeds dimension bounds.

  • @chenya6875
    @chenya6875 3 года назад +1

    Thanks for a nice tutorial. Are there any ways to query variables and attribute information and assign them to variables to loop. Something similar like disp() in matlab?

  • @imaneelkasimi1029
    @imaneelkasimi1029 Год назад

    Thank you for the wonderful explanation! I wrote my script in a folder and i copied it to a new folder to use it for a new set of data, but the script doesn't seem to work except in the original folder. Am i doing something wrong?

  • @_Muser23
    @_Muser23 4 года назад

    First, I'd like to say you're amazing at this. You explain everything clearly and it is easy to follow along. Thank you!
    Second, I understand that this tutorial is specifically for one location. But let's say I am trying to extract daily data (temperature or precipitation, for example) for an entire country; how much would that change the script you set up as an example?

    • @geodeltalabs
      @geodeltalabs  4 года назад

      Hi, thanks. Did you mean as in extracting data at different point locations? if that's what you meant, you can check out the part 7 of the tutorial. I have explained how to retreive the data at multiple points, given that you provide the corresponding lat, lon data

    • @_Muser23
      @_Muser23 4 года назад

      @@geodeltalabs I meant as in extracting data from one entire country, for example, instead of just extracting data for Kathmandu, how would it be if we extracted data for the entire country of Nepal (in which case we would need the lat and lon of the entire country)? I hope that made sense. I did watch part 7, but that is multiple points from different countries. Thanks for replying!

    • @geodeltalabs
      @geodeltalabs  4 года назад

      Hi, if I am understanding you correctly, when you mean the whole country, you still need to have some sort of a realistic spatial resolution , which again could be viewed as points covering an entire area. So in that case, I guess you would need to again have the points of a particular resolution with known lat lons within the country and then sort of extract the values of each point I guess?
      Maybe you can tell what your objective finally is by extracting the data as such a fine scale for the entire country? Maybe that might help to shed some light and see if there is another easy method to achieve this

  • @cmushi4217
    @cmushi4217 4 года назад

    Hi, thanks for the great videos.

  • @harshupadhyay3207
    @harshupadhyay3207 Год назад

    Hi. I am getting issue with the for loop that the same year gets appended to the list. So it seems like the loop is not iterating as it should and I end up getting a list containing the same years, e.g., ['1970', '1970', '1970', ..., '1970']

    • @harshupadhyay3207
      @harshupadhyay3207 Месяц назад

      @@ernestbaragamba7362It has been a year since I asked that question and honestly, I don't remember what was the original problem I had. But if you can wait till monday, i will just straight up copy paste the entire script that I use here.
      Edit: if you need it urgently, I'd suggest using xarray library with xr.open_mfdataset function. I prefer it over glob in this specific case. You can check its syntax and work around it to write the rest of the script.

    • @harshupadhyay3207
      @harshupadhyay3207 Месяц назад

      @@ernestbaragamba7362 Can you tell me what is the data source?

    • @harshupadhyay3207
      @harshupadhyay3207 Месяц назад

      @@ernestbaragamba7362 Can you mention the dataset source you're using?

  • @fuadmutasim4276
    @fuadmutasim4276 4 года назад

    Great Work. Many thanks

  • @kourosh39
    @kourosh39 4 года назад

    Thank you. Great job.

  • @agirl3517
    @agirl3517 2 года назад

    Sir I have one file for 1998 to 2013 it is monthly distributed but when I am opening it, it is only showing 12 months and no year-wise distribution, kindly help

  • @litiagaunavou2002
    @litiagaunavou2002 4 года назад

    Great explanation and very detail and very easy to follow along especially for as a beginner. What about if the netcdf files have time in Julian days? Is there a tutorial that converts to normal days?

  • @amineouhechou1754
    @amineouhechou1754 3 года назад

    Thank you very very much

  • @navneetsrivastava688
    @navneetsrivastava688 2 года назад

    I don't know if you will answer this but anyways my question is how will one evaluate the time variable and corresponding tave if the unit of time is days.

  • @asmitasubedi8215
    @asmitasubedi8215 3 года назад +1

    Thank you for the tutorial!
    At 12:40, I get the same result for max and min of all years. The years listed under (all_years) are also same. I have tried extracting csv file from individual .nc files. The results are different. However, I could not get the result for multiple years while extracting the files. Could you please let me know what the problem can be? Thank you!

  • @turning_point96
    @turning_point96 4 года назад

    I have several NetCDF files (year wise) for a specific region. Could you tell me how can I merge them to get only one file? Thanks.

  • @jpolanco2004
    @jpolanco2004 2 года назад

    Hello. I would like to read from multiple netcdf files, without using mfdataset (b/c it doesn’t work with netcdf4), and merging the arrays removing the netcdf structure (plain lists/arrays). Can you make a tutorial of this? Thank you in advance.

  • @IcYcR3aMiShSw33t
    @IcYcR3aMiShSw33t 4 года назад

    Hi thank you so much for this video. I am a student at college trying to read netCDF file for a research project. I am wondering why you did longitude and latitude squared. What is the mathematical purpose of that?

  • @MuhammadNawaz-iq4gt
    @MuhammadNawaz-iq4gt 4 года назад

    I am getting an error with the date of the nc raster as it shows 1899 instead of 1999 when I am applying this code, what can be the reason?

  • @bivargarceschavango3383
    @bivargarceschavango3383 2 года назад

    Hello, I appreciate you the work that you have been done so far with your videos, and I'm thankful with your help.
    I try to use this tutorial but when I run for my datas the out put of time is not the same with the period of time that my datas belong. But when I use the 'xarry' library I can have the exact time domain of my datas.
    Then I don't know what to do because I want to glob my datas. Thanks!

  • @maiya_angel
    @maiya_angel 3 года назад

    Please can you explain how to apply bias correction for downloaded netcdf data and get output also in netcdf format so that later I can convert ito to csv format following the method you explained in this video. Please Please reply. I have very little time for my thesis submission.

  • @isabelbrandao6096
    @isabelbrandao6096 3 года назад

    Great tutorial! How about if I have the name of my locations saved as .nc and these have a time series (10 years) of many variables? I want to extract for example two variables (temperature and salinity) considering my time variable which is about 10 years.

  • @sujathayadala6783
    @sujathayadala6783 2 года назад

    Hi Sir, I want to make time series plot of U wind for each season (MAM,JJAS,ON,DJF) from 1979-2021(Daily data). Could you please tell me if there is any tutorial related to this,Thank you.

  • @Jammie_5621
    @Jammie_5621 Год назад

    Sir you are taking years data and I want to work on a single day's data where my time series consits of hours and minutes instead of years. How do I deal with that?

    • @Jammie_5621
      @Jammie_5621 Год назад

      Also when I'm printing the all_time (appended times) the list is coming as 00:00 UTC only for all time stamps

  • @nikhilteja91
    @nikhilteja91 4 года назад

    Thanks a lot. That's a great explanation. Also, could you please show us how to extract data from Gridded area instead of a single point that you have shown above?

    • @geodeltalabs
      @geodeltalabs  4 года назад

      Hi, for that you can check out our zonal statistics video. We have done 2 tutorial videos on spatial data extraction from rasters

    • @nikhilteja91
      @nikhilteja91 4 года назад

      GeoDelta Labs what I actually want is the extraction of data from gridded area from NC files to CSV. i.e. I need all the gridded points (lat,lon) on x-axis, dates in index, and values for each grid point.

  • @pubalimukherjee4793
    @pubalimukherjee4793 3 года назад

    Hi, the video has been beneficial, but I have NC files varying in month and hours basically 6 hourly files over three months. I am finding it difficult to iterate through files varying in month and hours. Please help.

  • @saulcazaresmartinez2990
    @saulcazaresmartinez2990 3 года назад

    Hello friend, excellent tutorials, I'm doing the same as you do in this tutorial, but I can't find how to do it for a period of 40 years, could you help me?

  • @hadikarimi2818
    @hadikarimi2818 2 года назад

    Thank you for your great videos, I'm trying to extract Time series data from precipitation. Is there any link that you could help me or any videos. I did your wat for temperature but at the end I got error.
    Thank you in advance

  • @vihanweeraratne1476
    @vihanweeraratne1476 4 года назад

    Thanks for your awesome tutorial! How would I combine multiple netCDF files into one file (I have a few netCDF files, 1 file per day and I wish to combine these into 1 netCDF file)? Any advice would be much appreciated.

  • @geosciences5097
    @geosciences5097 11 месяцев назад

    Pourquoi on faire la différence de lat et l’on?

  • @chrisspeter2866
    @chrisspeter2866 3 года назад

    what if the ncdf file time variable is not defined as yearly is given in range eg 1981- 1985

  • @someshkumar495
    @someshkumar495 2 года назад

    how to give the range in lat and lon
    ex- lat 27.36 to 32.31
    lon 85.12 to 90.12

  • @waheed11ful
    @waheed11ful 4 года назад

    what about the .nc file which is multi-year eg 10 year nc file

  • @elmonovagales2929
    @elmonovagales2929 4 года назад

    Hello! thanks for your video! Can you help me out with something:
    I am working with python version 3.6.2
    I got an error in min 17:47, While creating the date_range. I followed your steps and it gave me an error, 'could not convert string to Timestamp', u' occurred at index 0'). I tried several options to fix this, and I ended up writing the "start" and "end" date manually.
    pd.date_range(start = '2011-01-01',
    end= '2017-12-31',
    freq ='D')
    in this way it worked, however,
    in the minute 39:27 in the last piece of code corresponding to:
    for t_index in np.arange(0, len(d_range)):
    df.loc[d_range[t_index]]['Temparature'] = temp[t_index, min_index_lat, min_index_lon]
    it gives me an error:
    df.loc SyntaxError. invalid syntax
    I copied paste your code, however, I think it is related to the fact that python doesn't allow me to specify a time range with strings.
    Any hint of how to solve this issue?

    • @elmonovagales2929
      @elmonovagales2929 4 года назад +1

      Just in case you want to know, I was working with the original name of the files, (it was too long), apparently, I managed to make it work, but at the end, the easiest way to handle the archives it was to change the name of the files to the year. In this way, it works.

  • @engr.abdullahazzamsafi4515
    @engr.abdullahazzamsafi4515 4 года назад

    Can please tell us the Key that you are bringing "#" before the selected script so that it wouldn't run. Thank you

    • @geodeltalabs
      @geodeltalabs  4 года назад +2

      If you are using Spyder IDE, then it's Ctrl + 1

    • @engr.abdullahazzamsafi4515
      @engr.abdullahazzamsafi4515 4 года назад

      @@geodeltalabs If we have a data file of 10 years instead of 1 year. For example 1950-1959, 1960-1969, ............. etc so what would we do in this case. Thank you in advance

  • @shankarbhattarai3251
    @shankarbhattarai3251 4 года назад

    Hello sir! I really appreciate your video which is very useful for the people like us who do not know anything in python. I am trying to use your code to extract NetCDF file. I successfully did in part one. But encountered problems in part 6. I have not try yet with part 7 as I have not been able to proceed from part 6. My knowledge and skill in python is almost like zero.
    My problem appeared as follows:
    File "C:\My Document\SCHOOL\Python\GCM_data_py\Try_gcm_w.part6\Try_gcm_w.part6.py", line 31, in
    data = Dataset(str(year)+'.nc', 'r')
    File "netCDF4\_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.__init__
    File "netCDF4\_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success
    [Errno 2] No such file or directory: b'2006.nc'
    I would be grateful to you for your support to solve this problem.

  • @ladiran
    @ladiran 4 года назад

    Thanks for this, please could you also provide back link for the other parts here, I.e., parts 1, 2,3,4 and 5.

    • @geodeltalabs
      @geodeltalabs  4 года назад +1

      Hi, here is the playlist
      Handling netCDF Files using Python: ruclips.net/p/PLLxyyob7YmEE8S3QDs1PZQkiBxA4zn_Gx

  • @sohel.env46
    @sohel.env46 3 года назад

    It would be very nice if make a tutorial on how to extract data for ploting time series from Sentinel 5p

    • @geodeltalabs
      @geodeltalabs  3 года назад

      Hi, you may check this tutorial. Even though the satellite product is different, you can use the same approach to extract raster data
      ruclips.net/video/_uaVrSeLFmA/видео.html

  • @edgarvillamarin
    @edgarvillamarin 3 года назад

    Someone know about use a wget command to download file from a private serves?

  • @abdurrub630
    @abdurrub630 4 года назад

    You made the time.units cropping the details of the time variable. Unfortunately for all my files I have the units “hours since 1900-01-01” for each year file. How can I specify the year variable in this case as I cannot make the time.units variable. Your urgent is highly appreciated. I am stuck on this thing for days now.

    • @sashacrighton7359
      @sashacrighton7359 3 года назад

      I have the same problem. Did you find a solution?

    • @siljehelenejohnsen7509
      @siljehelenejohnsen7509 3 года назад

      ​@@sashacrighton7359 Hi! I'm also stuck on the same issue. Did any of you find a solution for this?

    • @ernestbaragamba7362
      @ernestbaragamba7362 Месяц назад

      Hi, how did you solve it?

  • @adekunbifalilu5391
    @adekunbifalilu5391 4 года назад

    Hi everyone can anyone tell me what program to open the python script sent in the google drive folder. I tried to open with Jupyter notebook it was successful. Please help

  • @Jammie_5621
    @Jammie_5621 Год назад

    when I say print (file) it is not printing

  • @binataroy9490
    @binataroy9490 3 года назад

    I have downloaded CMIP6 daily precipitation data from 2015-2034. But while printing its unit, it is printed as units: days since 1850-01-01 00:00:00
    So, how can I use the line "year=time.units[14:18]" in the code to get the starting date and end date?
    for file in glob.glob('*.nc'):
    print (file)
    data=Dataset(file, 'r')
    time=data.variables['time']
    year=time.units[14:18]
    print (year)
    all_year.append(year)
    print (all_year)

  • @davendersethi5831
    @davendersethi5831 2 года назад

    How to extract the time from the file names of the list of netdf4 files using the for loop in python. Please help any one who knows

  • @stephenbarry9288
    @stephenbarry9288 4 года назад

    Hi, thanks for this amazing tutorial. I keep getting the following error in the code - FileNotFoundError: [Errno 2] No such file or directory: b'2010.nc'
    This occurs after setting up the for loop:
    for yr in all_years:
    data = (Dataset(str(yr) +'.nc', 'r'))
    Any advice would be appreciated!

    • @yuvii007
      @yuvii007 4 года назад

      you need to provide complete path if your files are not in the same folder. for eg
      data = Dataset('D:/PATH/'+str(yr) +'.nc','r')

    • @aggarnica
      @aggarnica 4 года назад

      Yuvraj Sinh Rajput hi I also have the same error, my data and .py file are in the same directory. However, it still gives me the same error. Your reply would be highly appreciated

    • @yuvii007
      @yuvii007 4 года назад

      @@aggarnica Please send me the screen shots of your program and your windows folder to verify the error. You can send email on yuvi.tashan@gmail.com

    • @aggarnica
      @aggarnica 4 года назад

      Yuvraj Sinh Rajput Hello, thank you so much, I have already sent the screenshots

    • @Machetyman
      @Machetyman Год назад

      @@aggarnica I have the same issue. Were you ever able to get this issue resolved?

  • @agumasekindie7672
    @agumasekindie7672 3 года назад

    i am using spyder, i tried to extract time series data from netcdf4(2000-2003). when i run all_years, the output is the same 2000
    how can i solve
    here is my code
    import glob
    from netCDF4 import Dataset
    all_years = []
    for file in glob.glob('*.nc4'):
    print(file)
    data=Dataset(file, 'r')
    time = data.variables['time']
    year = time.units[11:15]
    all_years.append(year)
    all_years
    Out[136]: ['2000', '2000', '2000', '2000']

  • @hindokush3728
    @hindokush3728 4 года назад

    kindly check your inbox sir, i got an error and i sent it.

  • @engr.abdullahazzamsafi4515
    @engr.abdullahazzamsafi4515 4 года назад

    We have learned with you a lot. thank you so much. I have problem of extracting the historical data of scenario (tasmin_day_CanESM5_historical_r10i1p1f1_gn_18500101-20141231.nc). Can you please share the script or video how to extract such kind of data. I will be very grateful to you. many thanks

    • @shankarbhattarai3251
      @shankarbhattarai3251 4 года назад

      Hi Abdullah! I am also looking the solution for the same. Did you find it? if you have found it, would you like to share? this would be of great help.

    • @engr.abdullahazzamsafi4515
      @engr.abdullahazzamsafi4515 4 года назад

      @@shankarbhattarai3251 Hi, i have created my own code with help of this video but the problem in data is that t_index is 60225 but actual df column rows for 01/01/1850 to 31/12/2014 is 60265. Hence, it displaying error: index exceeds dimension bounds

    • @Hydroclim
      @Hydroclim 4 года назад +1

      @@engr.abdullahazzamsafi4515 Hi there, I am also having this issue. Actually, mine does not show any error during run, but it hangs and stop functioning completely. Another problem I found that even when I extracted data from one single file, it does not show same result as I got from another software (CMhyd). I wonder which one to use. Do you have any idea.

    • @abdullahsafi4747
      @abdullahsafi4747 4 года назад

      i have modified the script for my own data but unfortunately for some stations data is correct but some stations data isn't correct. I used another way to solve my problem but @GeoDelta Labs did a very great job always.

    • @nataliacardenas6358
      @nataliacardenas6358 2 года назад

      @@engr.abdullahazzamsafi4515 Hello, I have the same error (IndexError: index exceeds dimension limits), but in my case it is because the .nc files contain only one time step and I have not managed to adjust the code.
      How did you solve it in your case?