Pub/Sub tips and tricks

Поделиться
HTML-код
  • Опубликовано: 16 янв 2025

Комментарии • 35

  • @googlecloudtech
    @googlecloudtech  3 года назад +4

    Tune into our #AskGoogleCloud premiere on Friday, March 12 10AM PT for a chance to ask Google Cloud’s serverless experts any questions regarding serverless architectures → goo.gle/3evqXcU
    Get $300 and start running workloads for free → goo.gle/2Zvv4wE

  • @aaronbelchamber5007
    @aaronbelchamber5007 Год назад +2

    A great example used, very clear! Nice job!

  • @JosephLust
    @JosephLust 2 года назад +1

    Great presentation, Martin & Wes!!

  • @laskdkmgful
    @laskdkmgful 3 года назад +5

    Ordering is important. Might try to share more in detail next time. Thx. 👍

    • @Tenelia
      @Tenelia 3 года назад

      I also want to see ordering!

    • @MartinOmander
      @MartinOmander 3 года назад +2

      @@Tenelia More details about ordering -- got it! We are adding this to our list of new episode ideas.

  • @GabrielAmyot-q3e
    @GabrielAmyot-q3e Год назад +1

    Thank for the tips. Love this format. Thanks

  • @ganeshsonawane4276
    @ganeshsonawane4276 2 года назад +1

    Excellent idea. Got more insights about pub/sub through this.

  • @AntonioCachuanAlipazaga
    @AntonioCachuanAlipazaga 3 года назад +8

    Excellent tips! Please, a Dataflow tips and tricks would be aweosome and a video about real time analytics (Pub/Sub + Dataflow + Tensorflow)

    • @tianzicai9964
      @tianzicai9964 3 года назад +1

      There's a streaming example that uses Pub/Sub + Dataflow + Tensorflow, which it is part of a longer tutorial that showcases batch processing.
      Try to search for "molecules-walkthrough" or "Machine Learning with Apache Beam and TensorFlow" in the GCP docs.
      Hope you find that tutorial useful, and thank you for watching and for your comment.

    • @AntonioCachuanAlipazaga
      @AntonioCachuanAlipazaga 3 года назад +1

      Thanks Tianzi! I found the example cloud.google.com/dataflow/docs/samples/molecules-walkthrough I'll try it.

  • @pinkanrou7167
    @pinkanrou7167 3 года назад +2

    Excellent one!!! Thanks @Tianzi :)
    I tried the dead letter topic one but with mixed luck :(
    It worked when I tested without any subscriber . I pushed a message and checked the delivery attempt after 5 times it pushed the message to dead letter one.
    When I used dataflow as a subscriber and intentionally threw exception from subscriber code it did not work. The message is not getting acknowledged as I can see from Unacked message count ,but it never reached to dead letter topic. So the error count keeps on increasing in dataflow.I have not written any code inside subscriber to push the message to dead letter as I think It should be pushed automatically. Am I missing anything ?

    • @tianzicai9964
      @tianzicai9964 3 года назад +1

      Thank you Pinkan. So glad to know that you gave it a try. And thank you for the question.
      Mind I ask you how you "intentionally threw exception from subscriber code" in your pipeline? `PubsubIO.readMessages()` returns `org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage.PubsubMessage`, which is unlike `com.google.pubsub.v1.PubsubMessage` and does not have an `ack()` or `nack()` method.
      I think it's for this reason that all Pub/Sub messages arriving in the Beam/Dataflow will be considered acknowledged. "The Dataflow runner's implementation of PubsubIO automatically acknowledges messages once they have been successfully processed by the first fused stage (and side-effects of that processing have been written to persistent storage)." [1] Does this make sense?
      [1]: cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#integration-features

  • @robinthomas7584
    @robinthomas7584 3 года назад +2

    Whats the behavior when publisher sets message ordering and publishes messages with an ordering key and there are multiple subscribers listening to the same topic and only one of them is interested in ordered messages.

    • @tianzicai9964
      @tianzicai9964 3 года назад

      If you are using any of the official client libraries, messages of the same ordering key will be delivered to the same subscriber. Messages published with no ordering keys can be pulled by subscribers to a subscription with ordering enabled in any order. Thanks for watching and for your question!

  • @PatrickSteil
    @PatrickSteil Год назад +1

    I tested code running on Cloud Run to add 100 messages to a queue and it took .04s per message. That is 40ms per message add. The message was only like 32 bytes.
    Does this sound like it is too slow?
    Any way to speed this up?

    • @TheMomander
      @TheMomander Год назад

      Have a look at "Publish with batching settings" in PubSub. That may help speed it up.

    • @PatrickSteil
      @PatrickSteil Год назад +1

      @@TheMomander Thx, just ran into this in the docs a few minutes ago... :) Appreciate the reply

  • @alexpalau5350
    @alexpalau5350 3 года назад +2

    Really nice tips! Ordering messages feature will help a lot of people and projects.
    What will happen if order is enabled and dead lettering also? Will good messages still be sended in order eventhough some of them are missing between the good ones?

    • @tianzicai9964
      @tianzicai9964 3 года назад +2

      To answer your first question, if the subscription attached to the dead-letter topic has ordering enabled, then messages (sent with an ordering key) that you receive from that subscription will be ordered.
      To answer your second question, (good) messages not forwarded to the dead-letter topic will not be received by the subscription attached to the dead-letter topic.
      Thanks for watching and for your questions!

  • @amoslam-w3y
    @amoslam-w3y Год назад

    for the message ordering, what if request a ride fails in dispatcher? since message ordering cant be supported via dead letter and if request ride + cancel ride is pulled together in the same pullResponse, the subscriber can process cancel ride as well even if request ride fails.

  • @eyesopen6110
    @eyesopen6110 2 года назад

    What are the calls between the Dispatcher and Fleet manager (get-available_drivers, send-available-drivers)? Are these also pub/sub? How are these communicating?

  • @raihankhan5184
    @raihankhan5184 2 года назад +1

    @Google Cloud Tech. When you guys show the demo on console, it doesn't benefit anyone. Most of the companies are doing it using IAS or using Python api. So please provide more tutorial using coding...

    • @TheMomander
      @TheMomander 2 года назад

      You bring up a good point. We often use the console as it is more visual, easier to understand, and shows the concepts across programming languages, not just for one language. We'll see if we can include more code in the future!

  • @dheer211
    @dheer211 3 года назад +1

    Great Video thanks

  • @castellesenminier6291
    @castellesenminier6291 2 года назад +1

    Interesting intro

  • @havardestensen1012
    @havardestensen1012 3 года назад +1

    Too bad dead-lettering doesn't work for event-driven functions since you cannot manually NACK.

    • @tianzicai9964
      @tianzicai9964 3 года назад +1

      Hm, I'm not sure I understand the full context. A "NACK" in the case of event-driven functions means any response code that's not 102, 200, 201, 202, or 204. It's possible to write your function to return failure code where necessary. Could you tell me more about what's on your mind?

    • @havardestensen1012
      @havardestensen1012 3 года назад +1

      explains it better than I can 😄

  • @LittleRapGuy
    @LittleRapGuy 3 года назад +1

    nice

  • @nenora
    @nenora 3 года назад +1

    Great

  • @stackdev-io
    @stackdev-io 2 месяца назад

    good

  • @MuhammadAmjad-qz1ik
    @MuhammadAmjad-qz1ik 3 года назад

    help me