Fine-tune PaliGemma for image to JSON use cases

In this tutorial, I'll showcase how to fine-tune PaliGemma, a new open vision-language model by Google on a receipt image to JSON use case. The goal for the model is to learn to output a JSON containing all key fields from a receipt, such as the product items, their prices and quantities.
Do note that PaliGemma is just one of many vision-language models released recently.
The notebook can be found here: github.com/Nie...

[@OliNorwell]

This is awesome, thanks for sharing, I'm only 5 minutes in right now but will definitely follow it all later. These types of tutorials are gold-dust, calm, concise and not just for entertainment.

[@lucasbeyer2985]

Thanks for making this super in-depth tutorial Niels! And very nice dataset/task you chose, I like it.

[@henrik-ts]

Best video i've seen so far about finetuning with huggingface. I appreciate that you also take time to explain some concepts behind. Other youtubers are just reading out loud their notebooks...

[@ajaykumargogineni3391]

Great Explanation!!

[@salesgurupro]

Amazingg. Such a comprehensive and detailed video. Loved it

[@yuzhen-o3h]

Thanks for your video.

[@junma7763]

Thank you so much for the amazing tutorial!

[@aamir122a]

Thank you for this excellent presentation. In future can you do a video on how to take two take two different HF models and merge them , like this one .

[@251_satyamrai4]

amazing video, great explanation

[@abhishekg4147]

@Niels Rogge it is amazing content and i have been following you since the time of donut tutorial keep going buddy.
I have one question is it possible to get the snippets extractions also from the invoices?
please let me know and reply

[@taesiri]

Wow, Look who's back 🔥

[@jeffrey5602]

banger video as always 🔥

[@miguelalba2106]

Is this model robust against incomplete labels/ground truth ? Let say you have for some images all details and for others not so many

[@abhishekg4147]

Superb content however if I want to add specific key elements to be extracted from custom data how will i do it pls reply

[@rajdeepbanerjee6641]

Thanks for the awesome tutorial, can this be run in colab T4 GPUs (16GB)?

[@richardyim8914]

no more runpod?