This is such really great tutorial !!! You have deep understanding of both AWS EKS and Opensearch Services. Hence your simple explanation totally makes sense about this integration. There a lot😊😊😊
Greate video!! How could I add more than one EKS clusters in same Opensearch. In my case, first I configured opensearch for staging cluster it was fine. I did the same thing for production cluster (difference VPC) and production works fine, then the staging has no permission and could not get log anymore. Thank you.
Thank you for watching my videos. Indeed this is great way to use the AWS resource, as you want point multiple eks cluster to a same OpenSearch cluster. It should support this until you have required access
@@cloudquicklabs Thank for you reply, I have found the solution for my case, just add the arn IAM for every cluster and that should get logs for multiple EKS cluster
Hi, can you please tell me the why does the curl command you ran to link sa with internal users was required? Is it because there are internal users in the database?
Thank you for watching my videos. I have shown how to creat log index using OpenSearch UI, you could follow the same. But how to do it with CLI if any I need to explore it.
Thank you for watching my videos. Glad that it helped you. Forwarding control plane logs to AWS OpenSearch can be configured at fluentbit specification file.
Thank you for watching my videos. With fluent-bit-Policy we are giving fluentbit access on Open Search service to push the logs. With Resource being mentioned in policy, we narrow down the permissions to a specific elastic domain name service where we are actually dumping the logs.
Thank you for watching my videos. Indeed it would not support for non-eks cluster if you follow same steps. There should be different ways for different cluster, may I know which cluster that you are trying
Self managed k8s cluster I have modified the yamls of fb it's throws with 401 i have added the role with os domain full permission on it to make aws signed request still didn't working
@@cloudquicklabs [2023/04/07 15:56:42] [error] [output:es:es.0] HTTP status=400 URI=/_bulk, response: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400} im getting the same, ive followed all the steps. deployed the fluent bit, checked the logs and this is what i get
This is such really great tutorial !!! You have deep understanding of both AWS EKS and Opensearch Services. Hence your simple explanation totally makes sense about this integration.
There a lot😊😊😊
Thank you for watching my videos.
Glad that it helped you.
Greate video!! How could I add more than one EKS clusters in same Opensearch. In my case, first I configured opensearch for staging cluster it was fine. I did the same thing for production cluster (difference VPC) and production works fine, then the staging has no permission and could not get log anymore. Thank you.
Thank you for watching my videos.
Indeed this is great way to use the AWS resource, as you want point multiple eks cluster to a same OpenSearch cluster. It should support this until you have required access
@@cloudquicklabs Thank for you reply, I have found the solution for my case, just add the arn IAM for every cluster and that should get logs for multiple EKS cluster
Hi, can you please tell me the why does the curl command you ran to link sa with internal users was required? Is it because there are internal users in the database?
Thank you for watching my videos.
This is to provide EKS OIDC role permissions on OpenSearch (es) to push the logs from EKS to es.
how to create index for K8 logs what command did you run in dev console
Thank you for watching my videos.
I have shown how to creat log index using OpenSearch UI, you could follow the same. But how to do it with CLI if any I need to explore it.
Thank you for making a nice educational video. How can I configure fluenbit to send eks control planes logs to OpenSearch?
Thank you for watching my videos.
Glad that it helped you. Forwarding control plane logs to AWS OpenSearch can be configured at fluentbit specification file.
How did you come up with the fluent policy at 18:55? Where does the "Resource" come from and what makes up that file?
Thank you for watching my videos.
With fluent-bit-Policy we are giving fluentbit access on Open Search service to push the logs. With Resource being mentioned in policy, we narrow down the permissions to a specific elastic domain name service where we are actually dumping the logs.
Nice❤
Thank you for watching my videos.
Glad that it helped you.
Subscribed
Thank you, keep watching my videos.
Thanks for Good video. if you possible, please show me this Github repo URL this lecture. thanks.
Thank you for watching videos.
You can find GitHub repo at this video's description, please find from there.
Unable to fetch cronjobs logs using fluent bit
Thank you for watching my videos.
Check out the path where this log is stored and then that should be configured log scrapers.
Liked
Thank you so much.
Its not working on non eks k8s cluster
Thank you for watching my videos.
Indeed it would not support for non-eks cluster if you follow same steps. There should be different ways for different cluster, may I know which cluster that you are trying
Self managed k8s cluster
I have modified the yamls of fb it's throws with 401 i have added the role with os domain full permission on it to make aws signed request still didn't working
I followed however the k8s demon sets emitting errors as :
[2022/11/29 07:42:41] [ warn] [engine] failed to flush chunk '1-1669707737.413578655.flb', retry in 106 seconds: task_id=2, input=tail.0 > output=es.0
[2022/11/29 07:42:41] [error] [output:es:es.0] HTTP status=400 URI=/_bulk, response:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
Thank you for watching my videos.
This looks to be syntactical issue.
Could you please let me know in which step you are experiencing this issue
@@cloudquicklabs [2023/04/07 15:56:42] [error] [output:es:es.0] HTTP status=400 URI=/_bulk, response:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
im getting the same, ive followed all the steps. deployed the fluent bit, checked the logs and this is what i get
As '_type' parameter is removed from latest ES or Open search versions you need to set "Suppress_Type_Name" to On in fluentbit.yml in Output section
Yes @@carlpei6570 that's true @work8597 you need to set [ suppress_type_name On ] in output section of fluentbit configuration file.