AWS OpenSearch | Creating an OpenSearch domain within a VPC and accessing it using Proxy API
HTML-код
- Опубликовано: 7 сен 2024
- This video demonstrates how to create an OpenSearch domain from scratch and secure it within a VPC.
This also provisions a Lambda and API gateway API to access the Kibana dashaboard.
Git repository: github.com/lis...
Thanks you so much, this is what I was looking for..the way to access OS within VPC
You are welcome! :) and thanks for your support!
You just save my so much time, I have been looking this for a week. Thanks a lot. Keep the good job and your video are well explained.
Glad it helped :) you are welcome!
Excellent Video for OpenSearch with VPC
Thanks Haneep!
Very interested method to replace additional instances with reverse proxy to services on private networks with Lambda Proxy.
thanks!
Thanks for the video, it was useful. Would you share some details or post a video on AWS DMS target endpoint as OpenSearch creations, as it involves user role mapping.
you are welcome :) thanks. I haven't really worked with DMS endpoints. I will try to do some research around it and get back.
The api gateway gave me this error "Missing Authentication Token" when no query parameters were given, with any query string the gateway gave this root cause error type: "index_not_found_exception",
Could you please paste the URL you are trying to access?
Please make sure you are hitting the right url:
//_dashboards/app/dev_tools#/console
I’m hitting this url and it still doesn’t help. It gives me an error about checking my aws access secret.
Getting Internal server error. In the lambda handler event is coming as empty checked in CloudWatch logs
Please try to check you api gateway setup and the version of opensearch
pls do make a video also about creating a domain with public access and how to stream the logs to opensearch using a lambda function and visualize it in kibana dashboard, it will be more helpful if you do.
Thanks in advance
Hi Benny,
ruclips.net/video/06a3NJwM1VU/видео.html - this video demonstrates setting up a public access domain and stream s3 data using lambda. It also shows how to access that data in kibana. You could customise it to stream log data instead of s3.
Thanks.
Thanks it is so useful, glad about your work 👍.
Once the domain and the indexes are created, shouldn't that lambda proxy function be removed? I mean maybe change it in a way that it'll only expose the search api that way you can hide it behind an auth provider... otherwise the whole thing would just be publicly available for everyone. I'm not much of an expert on this that's why I'm writing this comment trying to get some guidance on the matter. What do you think?
Hello, Since the entire dashboard in exposed using api gateway, setting up proper authentication and controlling access to the api will automatically limit the access to opensearch dashboard. There are various ways to control api gateway access - docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-integrate-with-cognito.html
if you are interested, here is the video about lambda authorisers -
ruclips.net/video/Q5RwxhCONy8/видео.html
Very Helpful. Try hard finally was able to get with /_dashboards/app/dev_tools#/console
great!
Good solution to access opensearch dashboard. Any ways to provide authentication with the dashboard? With current configuration, it is using lambda role.
Hello, Since the entire dashboard in exposed using api gateway, setting up proper authentication and controlling access to the api will automatically limit the access to opensearch dashboard. There are various ways to control api gateway access - docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-integrate-with-cognito.html
if you are interested, here is the video about lambda authorisers -
ruclips.net/video/Q5RwxhCONy8/видео.html
@@listentolearn2363 Thanks. Will check that out.
Thank you for the tutorial. I am getting this error though:
"errorMessage": "'NoneType' object has no attribute 'upper'",
"errorType": "AttributeError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 95, in lambda_handler
'method': method.upper(),
"
Hello,
Looks like the event object is missing or getting passed as None. Please check your api gateway setup and try triggering a test event from api gateway.
I am having error for {missing Authentication Token} when hitting API URL. did anyone have same issue?
Could you please paste the URL you are trying to access?
Please make sure you are hitting the right url:
//_dashboards/app/dev_tools#/console
Can i use internal user database to login the dashboard?
nice. Neatly explained
thanks for your support :)
I have done all same exact still I have got "Internal server error"
Hi Sapnoka,
Could you please paste the URL you are trying to access?
Please make sure you are hitting the right url:
//_dashboards/app/dev_tools#/console
If you are seeing any specific errors in cloudwatch, can you share the error?
Hi, thanks for the tutorial but I’m having trouble opening the open search dashboard after following your video. Can you provide more details on the HOST?
please use below url:
//_dashboards/app/dev_tools#/console
you can find the host and api-stage-name in apigateway.
Hey
I followed all the steps but in the last step when I used my API gateway url, it says “OpenSearch Dashboards did not load properly. Check the server output for more information.”
I would suggest seeing the cloudwatch logs to check for any errors. If that looks good, then you can try enabling the api gateway logs and check for any errors there.
Same error for me to. Dashboard connected. An same error on 10 second of downloading. The error explained on stackoverflow as limitation for Lambda to 6 MB. JSON send in full size in request.
@@listentolearn2363 Errors in browser: Refused to execute inline script because it violates the following Content Security Policy directive: script-src unsafe-eval self.
@@listentolearn2363 CloudWatch. RuntimeError: Failed to post invocation response. LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413.
After having a look at the API Gateway Cloudwatch logs, it seems the error is
"Lambda execution failed with status 200 due to customer function error: Response payload size exceeded maximum allowed payload size"
Great content, but the audio is too low
Thanks. Will make it better in future videos.
Hello! great video and well explained, but I have the following doubt: doing this I am removing network protection from my dashboard and it becomes public, or do am I missing something? What are the benefits of doing this instead of removing the VPC from my open search domain directly? Thank you!
Hi Tobias, Thanks for your interest and to answer your question, open search domain within the VPC is still protected and you are allowing only lambda to access the open search domain. You need to add some type of auth mechanism to api gateway in order to restrict access to the dashboard.
If you like, please checkout this video that explains about api gateway authorisers - ruclips.net/video/Q5RwxhCONy8/видео.html
@@listentolearn2363 thank you for your response! I understand your point, but if you end up protecting your api gateway with cognito for example, why don’t you do the same with open search and avoid the vpc? I’m getting into open search (I have already worked a lot with api gateway), so I would like to understand the pros and cons of the solution, or if there is any other advantage that I might not be seeing. Thank you!
VPC provides an extra layer of security. You can give this a read to understand the pros and cons - docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html
When im trying to access api gateway endpoint I'm getting "OpenSearch Dashboards did not load properly. Check the server output for more information." And in lambda logs i see that LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413, probably due to payload limit threshold ? Im wondering why it did not happen in your video ?
Hello, thanks for giving it a try. Could compare the versions of opensearch domain and python used pls?
getting the same error ; how did you resolve ?
If you do same witch cdk, that would be nice too :)
yeah, will try to automate.. thanks!
I'm getting below error while executing the lambda function, any idea ?
"errorMessage": "unsupported operand type(s) for +: 'NoneType' and 'str'",
Hi Prasada, what is the url that you are trying to access? Does the page load?
@@listentolearn2363 i setup everything like you explained, and after that, when i hit the API GW URL, i got the internal server error, so i thought i could test the lambda directly. When I was testing the lambda, i saw the error that i mentioned above.
ah okay, cant run standalone test on this lambda as its tied to the url.. Are you seeing any errors in cloudwatch logs when you got the internal server error?
Super... 👍🏽👍🏽👍🏽👌👌👌😊
Thank you 🙂
I am gettting following error while making request using API
```message "Missing Authentication Token"
```
Hi Ketul,
Could you please paste the URL you are trying to access?
Please make sure you are hitting the right url:
//_dashboards/app/dev_tools#/console
Still unable to access the web page
. Any changes need to make in the code if we are changing the region? I'm not proficient in python.
Hi Reshma,
The region is taken from AWS session, so this should work in a different region as well.
Could you please share the error that you are seeing?
It would be nice if you can share the cloudwatch logs as well.
Very good video !! thanks for info. I have a question how are you opening the Open search dashboard what is the exact URL
Hi Ramya,
thanks. glad you found it informative.
please use below url:
//_dashboards/app/dev_tools#/console
Thanks.
This is not working with OS 2.7. It keeps giving signature error
"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The Canonical String for this request should have been
'GET
/_dashboards/app/home
content-type:......
Same error with Python 3.8 or 3.11
go with OS 1.3, thats what I am doing.
I have it tested only in OS 1.3. Can you try with it? 2.7 might need few code update. I have to look into it.
hi, after some hit and trial. I'm getting this page and it seems like not working properly and why it is only redirecting towards only this dev tools. Can you please suggest something so its start working properly. Your help is highly appreciated
OpenSearch Dashboards logo is not available
Getting erros like this Expected ',' or ']' after array element in JSON at position 324
Hi Ajmal,
The current python implementation only supports dev tools. If you would like to access other sections of the dashboard, please feel free to extend the code.
@@listentolearn2363 Thanks, but why opensearch page is getting broken? l was assuming we will be able to access the opensearch via this process but unable to do that. This is not the correct way to access the opensearch i beleive. Pls suggest anything else.
this is not working if you choose to user elasticsearch core in aws opensearch... you could create another video for that, maybe i just changing the py code
thanks for giving it a try. you are right, the code is specific to opensearch.
however, we can get it working with elasticsearch by making few changes to the code. I will try to add it to the repo.
followed the whole process but unable to access the opensearch UI...can you please help
Hi Ajmal,
Could you please paste the URL you are trying to access?
Please make sure you are hitting the right url:
//_dashboards/app/dev_tools#/console
If you are seeing any specific errors, can you share the error?
I’ve been getting the following error: “Request must contain a osd-xsrf header. The cloud watch logs don’t indicate anything either.
Hi Alvin,
What version of OpenSearch are you using?
And when are you seeing this error? Is it while loading the first page or while running any specific commands?
@@listentolearn2363 I’m using open search 2.3. And yes, it is while loading the first page
@@listentolearn2363 Would be great if you could help me asap as I’ve been stuck on this for more than a week
If using a lower version is not a problem, can you try with 1.3? as I haven't tested it with 2.3 yet.
I think the header is causing a problem in 2.3 but am not sure yet. see opensearch.org/docs/latest/troubleshoot/index/
@@listentolearn2363 the domain had been defined for quite some time. Was using en ec2 instance before to access it outside its vpc. So not possible to try it with 1.3. Also I tried adding the header in the lambda function, but it keeps giving internal server error. Also I’m unable to debug the lambda function at all since adding any print or log statement results in an error
does not work...{"message": "Internal server error"} OR Token error...
Ensure you are accessing the correct URL. It is in the description of the video. The proxy works only for kibana dashboard.
Please help me it shows
"Message:Internal server error"
can you share the errors from cloudwatch logs?
I got "internal server error" because not changed every occurrence of AWS region in all listed policies first time and because of not correct URL in second time: /_dashboards/app/dev_tools#/console
same problem@@vladgursky149
no luck, trying make it run since yesterday...
Can you please provide the error details?