AWS Lambda : load JSON file from S3 and put in dynamodb
HTML-код
- Опубликовано: 26 июл 2024
- www.udemy.com/course/masterin...
for online/classroom trainings contact +91988661111
join udemy course Mastering AWS CloudFormation www.udemy.com/mastering-aws-c...
![MAF Teeski - Whole Thing (feat. VonOff1700) [Official Music Video]](http://i.ytimg.com/vi/SHWEj-ndz6g/mqdefault.jpg)
contact us online classroom training's & project support please contact phone number +919886611117
why do you need to upload the json to an s3 bucket ? is it possible to upload the json directly to the dynamodb. Apologies i do not know much about AWS as im just learning
I am getting an error of....
[ERROR] ClientError: An error occurred (ValidationException) when calling the PutItem operation: One or more parameter values were invalid: Missing the key object_id in the item
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 14, in lambda_handler
table.put_item(Item = jsonDict)
File "/var/runtime/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/var/runtime/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/var/runtime/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
The way you explain makes it appear that it is a piece of cake. Looking forward for more of these.
This is one of the cleanest solutions I have seen
Thanks very much, was very clear and easy to follow, definitely a concept that any one wanting to write to dynamodb from a json file should follow. great stuff.
Thank you! I've been trying to split my s3 object all day! Lifesaver
I know this is 3 years ago but pls watch this, this save my whole semester :))
Thank you
Thanks for uploading. The tutorial explains concepts very clearly. I believe its AWS which has modified the event schema. I had to modify 2 lines as on date to get the bucket name and json file name. ie. bucket = event['Records'][0]['s3']['bucket']['name']
json_file_name = event['Records'][0]['s3']['object']['key'].
Can't thank you enough for posting the correction
Dear @@ronnic1 , thanks for taking out time and sharing your knowledge. Also, thank you for acknowledging my bit of efforts. Regards
Good Explanation.. Thanks for sharing the knowledge !!
Crystal clear. Please Upload a video to Import multiple JSON into Dynamo DB
AWESOME VIDEO! extremely clear and well informed instruction. thank you!
Very good and clear! Very easy to follow !
Good tutorial and use case shown👍
You are a God sent... I am very grateful to you for clearing so many concepts together...
You are very welcome
You are so good at explaining things! Very grateful for your generosity!
You are so welcome!
Thanks for the video. Easy to follow and worked as advertised. Thanks
Thank you
Really great explanation!
Thanks for the nice explanation!
Thanks you sir. That was very helpful.
Fantastic video! Thank you so much! You are a genius!
Video is of great help....thx
Thank you so much! This was such a clear and informative video and really helped me with my project.
Glad it helped!
Great information, how would I use Athena here to read from S3 & send that information to Aurora ? Also how would you handle multiple records
Hello,
crystal clear explanation. Thank you for this.
Do you have video series on Lamda function with python!
I got it to work! Thanks.
You're welcome!
Thank you very much. It is very clear.
You are welcome!
Thank you
You are awesome
This work for local json file , what's of I need to do that for On-line json file and API ?
When you create the lambda function, your destinations to CloudWatch, DynamoDB, and S3 auto-populate. When I create my function they are not included in the flowchart and my data doesn't transmit. How do you have those destinations populate on the creation of the function?
How is the data getting into the Employees DB when we didn't create it?
Any suggestion on how this lambda can be implemented in Java as well? That'd be great :)
Hi Hari, I have one question if I want to trigger Terraform script to create a ecs cluster for new products automatically using lambda, SNS topic message as a input parameter into lambda function how can I do it ?
How can we insert bulk data json file to dynamo db
I would also like to know how to insert multiple rows from a json file into dynamodb from S3 using similar lambda function
Some solution? In mi case have un flat file with a size of 7000 records. How can load with calls minimum to DynamoDB?
If anyone is still looking for a way to loop and persist:
for line in jsonDict:
print(line)
table.put_item(Item=line)
getting Extra data error;
Extra data: line 2 column 1 (char 521): JSONDecodeError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 12, in lambda_handler
jsonDict = json.loads(jsonFileReader)
File "/var/lang/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/var/lang/lib/python3.6/json/decoder.py", line 342, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 521)
Hey were you able to resolve this ?
Today (dec-2019) seems that things are not the same in AWS. Followed the tutorial but I cant see the list of rerouces on te riigth side of the lambda function, like in 5:21. What is going on? Any idea∑
same for me.
constantly get the error in cloudwatch even though replicating the exact steps and code. not sure what is wrong out here.
table.put_Item(Item=jsonDict)
AttributeError: 'dynamodb.Table' object has no attribute 'put_Item'
import boto3
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('employees')
table.put_item(Item=jsonDict)
Thanks man
You're welcome!
Very good. Thx a lot. Question: My Bucket is in US-West and apparently my Lambda Function is in US-East. Apparently, both must be in the same region. I can't find the way to assign my function to US-West. Any idea ? Thx
GloDBSec
In lambda code while getting s3 client mention the region of your s3 bucket , hopefully it should work.
Who is the limit form rows?
can you please provide me the code to load csv files from s3 to dynamo db thanks
hi great work! , i have 1 question like i want to store data from exteranal api in aws and want show some data in view how it is possible can you help me?
Can you add more to your question
@@JavaHomeCloud , i want to fetch data from different external api and want to store it in dynamodb or any other database so how i can do it?i will have to use lambda with S3 or how it is possible?
Wheeeew! Why so complex compared to other cloud solutions??
Any idea on how to do the same thing using YML file?
I am getting below error
n error occurred (ValidationException) when calling the PutItem operation: One or more parameter values were invalid: Type mismatch for key line expected: S actual: M: ClientError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 13, in lambda_handler
table.put_item(Item=jsonDict)
File "/var/runtime/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/var/runtime/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/var/runtime/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the PutItem operation: One or more parameter values were invalid: Type mismatch for key line expected: S actual: M
Its expecting String and you are passing Map.
"more parameter values were invalid: Type mismatch for key line expected: S actual: M"
Verify your code
i am getting error
{
"errorMessage": "'str' object has no attribute 'loads'",
"errorType": "AttributeError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 12, in lambda_handler
jsonDict = jsonFileReader.loads('jsonFileReader')
"
]
}
Good video.
I have a question. what if the file is huge in size, so lambda can not process in 15minutes, it will give timeout error. then how can we process and save the file information into DB?
We have to choose different approach for example
1. AWS EMR
2. Run on EC2
3. AWS Batch,
4. etc...
what we need to change if we want to import csv files from s3 to dynamodb
Walid Sliti
we have to add csv python module to the code and read data from csv file
i tried but it doesn't work
Can you please provide me information for how to transfer data from DynamoDB to S3 using lambda?
I set the permissions the same way, but the destination is empty. How can I add it?
same here as well.
Thanks for your sharing! I follow those step by step, all the log info is right. But I still can not see the data dynamodb. why is that?
check you lambda logs for errors
In mi case have un flat file with a size of 7000 records. How can load with calls minimum to DynamoDB?
use DynamoDB bath write
json example at 8:46
Thank u
getting error object not defined on this line json_obj = s3_client.get_object(Bucket=bucketname,Key=filename)
print(Json_obj)
same here
I am unable to find the logs in cloud watch. is there any additional configuration required?
Sivaprasad ML
no additional configuration required, check your lambda has permissions to send logs to cloudwatch
It helped me to assign the role CloudWatchLogsFullAccess policy.
Is anyone else getting " [ERROR] KeyError: 'Records" while running this?
Can anyone help please? Code is exactly similar as in video.
check your IAM policy and role, looks like lambda could not connect to S3
At your lamdba function page, you need to click on the drop down of the button **Test** and choose to **Configure test event.** Over there you need to create a new test event with the following data inside:
{
"Records": [
{
"s3": {
"bucket": {
"name": "your-bucket",
"arn": "arn:aws:s3:::your-bucket"
},
"object": {
"key": "your_data.json"
}
}
}
]
}
where:
1. *your-bucket* is the name of the bucket you have created in S3
2. *your_data.json* is a json file that you have uploaded in your S3 bucket
I am getting error while put the dictionary in the dynamodb
b'{
"emp_id":"3",
\t"Name":"Hari",
\t"Age""26,
\t"Location":["USA"]
}'
error:- Expecting ':' delimiter: line 4 column 7 (char 42)
':' is missing after "Age" in your JSON file
can you please let me know how to load data to redshift from s3, here showing dynamo DB i want same thing videos in redshift. kindly help us
Hi Hari, Multiple options are there for your requirement, either use Lambda or Glue
is it possible to upload CSV file instead of jason
yeah
Is there any way to copy zip files of cloudtrail logs from s3 to dynamodb?
Yes
Would you please record a session on how to unzip and read cloud trail log files in s3 if possible.
Thnaks in advance
In above video he showed us how to insert only one item.but I have to multiple items.how can we do that.please help me how to do this through iteration.
1. Iterate those objects using a for loop and insert each one OR
2. Try to find if there is any method load_many like put_item
I need load many record (5000 records) in one request only..... is possible?
Hey did you get through this ? Getting the same issue
Is this code only for a single entry or multiple entries in json file?
This is for single entry
@@JavaHomeCloud Can u please send the code for multiple entries??
I followed the same but I am getting this error:
File "/var/task/lambda_function.py", line 6, in lambda_handler
bucket = event['Records'][0]['s3']['bucket']['name']
KeyError: 'Records'
Getting the same error, were you able to figure it out?
Same error here..
I found out that the problem is related to the bucket name and file name When I explicitly inform both it works fine. I gotta fix this problem in order to get the file names dynamically.
here are the edited lines:
bucket = 'my-bucket-name' #event['Records'][0]['s3']['bucket']['name']
json_file_name = 'my-file-name.json' #event['Records'][0]['object']['key']
At your lamdba function page, you need to click on the drop down of the button **Test** and choose to **Configure test event.** Over there you need to create a new test event with the following data inside:
{
"Records": [
{
"s3": {
"bucket": {
"name": "your-bucket",
"arn": "arn:aws:s3:::your-bucket"
},
"object": {
"key": "your_data.json"
}
}
}
]
}
where:
1. *your-bucket* is the name of the bucket you have created in S3
2. *your_data.json* is a json file that you have uploaded in your S3 bucket
Getting the below error
Parameter validation failed:
Missing required parameter in input: "Key"
Unknown parameter in input: "key", must be one of: Bucket, IfMatch, IfModifiedSince, IfNoneMatch, IfUnmodifiedSince, Key, Range, ResponseCacheControl, ResponseContentDisposition, ResponseContentEncoding, ResponseContentLanguage, ResponseContentType, ResponseExpires, VersionId, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, RequestPayer, PartNumber: ParamValidationError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 8, in lambda_handler
json_object = s3_client.get_object(Bucket=bucket, key=json_file_name)
File "/var/runtime/botocore/client.py", line 272, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 549, in _make_api_call
api_params, operation_model, context=request_context)
File "/var/runtime/botocore/client.py", line 597, in _convert_to_request_dict
api_params, operation_model)
File "/var/runtime/botocore/validate.py", line 297, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Missing required parameter in input: "Key"
Unknown parameter in input: "key", must be one of: Bucket, IfMatch, IfModifiedSince, IfNoneMatch, IfUnmodifiedSince, Key, Range, ResponseCacheControl, ResponseContentDisposition, ResponseContentEncoding, ResponseContentLanguage, ResponseContentType, ResponseExpires, VersionId, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, RequestPayer, PartNumber
Why this error is coming and how to resolve it ?
resolve this error
bcz i am getting same error
I know ur name sir Hari...
hello sir i m facing an error , while creating IAM policy , the error is " parser error message " an i m not able to create policy ,
here is the application
aws.amazon.com/blogs/machine-learning/build-your-own-text-to-speech-applications-with-amazon-polly/
I'll be very thankful for your help , plzz do help sir
Hello
I need python along with AWS services
Through lamda, now also your doing classes
yeah