You prolly dont give a shit but does any of you know of a tool to get back into an Instagram account..? I was stupid forgot my login password. I appreciate any help you can give me
@Kristopher Gunnar Thanks for your reply. I got to the site on google and I'm trying it out now. Looks like it's gonna take quite some time so I will get back to you later when my account password hopefully is recovered.
@databricks - what is the best practice to use or not use nested columns. For Example, I have struct of customer with Age, Gender, Name, etc attributes. Is it better to keep it as struct or separate into its own columns?
Thanks for posting this presentation. Could you clarify something? How does performance improve when you compress pages only to decompress it again to read it? I'm sure I'm not understanding something, but not sure what.
Snappy provides good compression with a low CPU overhead during compression/decompression. The real win in performance comes from reduced I/O cost when reading a column chunk's page. The overall cost (CPU+I/O) is generally lower for reading snappy compressed as compared to uncompressed.
@MGLondon How old are you? I am American (and not from China), and stick to common meats. This is an example of hate/harassment. Are you a high school kid?
How is storing json/xml (not parquet) more efficient than csv? You literally store the "column names" in each "row" in xml/json (at least when stored in a text file) . Also, there is definitely the notion of a "record" in csv.
Example 1. If you wanted to add a new properties to records overtime, you only need to add it to the new records (no need to back date blanks for legacy records for example). So think scale and change at scale.
Example 2. You can leverage hash/dictionary data structures in programming, these can find records at a much better scaling, look up hash functions and big o. Again think scaling related to data access, hashing vs at best search trees.
@@happywednesday6741 i asked how it was more efficient to store it. if i have 500 million "entries" in a text file, I'm definitely storing it in a delimited format or parquet to take advantage of said dictionaries and not json/xml. you can parse either into objects directly from the file, or bulk insert into a db table. the json/xlm formats would be 10x slower to parse/read in based on sheer disk/network i/o alone... if we're talking about efficiency in processing it. no one is going to load csv into memory and start trying to scan row by row for data, it's going to get converted into objects or a db anyways. my concern is when people store json formatted files to disk to be read into objects later. what does that buy you?
The best representation of Parquet file structure!! Simply Awesome!!
You prolly dont give a shit but does any of you know of a tool to get back into an Instagram account..?
I was stupid forgot my login password. I appreciate any help you can give me
@Iker Crew Instablaster :)
@Kristopher Gunnar Thanks for your reply. I got to the site on google and I'm trying it out now.
Looks like it's gonna take quite some time so I will get back to you later when my account password hopefully is recovered.
@Kristopher Gunnar it worked and I actually got access to my account again. I'm so happy:D
Thank you so much, you saved my account :D
@Iker Crew glad I could help :D
Great talk, Great Teach, Excellent Tutor! One of the best presentation I have ever viewed and listened.
This content explained most of the thing and It is really amazing .
Great overview to address performance issues with storage layer design 👍
Best Parquet File presentation I watch
Best presentation in Parquet.
Great presentation 👏 👌
great talk with simple explanations.
Great, this makes me know more about Parquet. Thanks for the pre!
Amazing. All concepts really well explained.
Awesome video with great content and explanation. Very very useful.
Very helpful sharing, thanks a lot.
@databricks - what is the best practice to use or not use nested columns. For Example, I have struct of customer with Age, Gender, Name, etc attributes. Is it better to keep it as struct or separate into its own columns?
Awesome video - not too much extraneous or labored points. Thank you!
Impressive presentation well structured explanations.
Thank you for this very useful video!
Great Presentation. Thank you
How do I calculate the optimal numPartitions in repartition or coalesce of dataframe?
Great talk. Thank you!
interesting how Parquet (columnar analytical focused) data can be optimized using dictionary-based compression and partitioning
I haven't watched this yet but for the sake of prioritizing when I do, how well does this topic apply to platforms and systems other than Spark?
Great! Well explained!
Just excellent 👍
Thanks for the content!
Seems the time and i/o needed before use the data in doing the sort first is not considered?
too good video. Excellent
Thanks for great explanation
Very helpful, thank you 😊
why shouldn't csv be right next to txt? it is literally the same
Thanks for posting this presentation. Could you clarify something? How does performance improve when you compress pages only to decompress it again to read it? I'm sure I'm not understanding something, but not sure what.
He mentions around @19:30 that you only see a benefit when the I/O savings outweigh the cost of decompressing.
I/O is more expensive compared to the time taken by CPU to decompress the data, thus ColumnStore is faster compared to RowStore.
Snappy provides good compression with a low CPU overhead during compression/decompression. The real win in performance comes from reduced I/O cost when reading a column chunk's page. The overall cost (CPU+I/O) is generally lower for reading snappy compressed as compared to uncompressed.
@MGLondon How old are you? I am American (and not from China), and stick to common meats. This is an example of hate/harassment. Are you a high school kid?
@harsh savla Good for you. Ecoli enters the body on vegetables.
Superb
great talk
anyone who says parquet is columnar format is having just bookish knowledge
Its a mixture of both Horizontal and Vertical partitioning and combines best of both worlds
i like the way it compresses the data into dictionaries per file. reminds me a bit of an EAV database stored as a file
Great!
Thank you!
Thankyou :)
How is storing json/xml (not parquet) more efficient than csv? You literally store the "column names" in each "row" in xml/json (at least when stored in a text file) . Also, there is definitely the notion of a "record" in csv.
Example 1. If you wanted to add a new properties to records overtime, you only need to add it to the new records (no need to back date blanks for legacy records for example). So think scale and change at scale.
Example 2. You can leverage hash/dictionary data structures in programming, these can find records at a much better scaling, look up hash functions and big o. Again think scaling related to data access, hashing vs at best search trees.
Example 3. You can more easily partition records via collections paradigm. Again storage and access at scale.
Example 4. You will more easily access and operate xml / json - like data from applications via APIs. Systems and interoperability at scale.
@@happywednesday6741 i asked how it was more efficient to store it. if i have 500 million "entries" in a text file, I'm definitely storing it in a delimited format or parquet to take advantage of said dictionaries and not json/xml. you can parse either into objects directly from the file, or bulk insert into a db table. the json/xlm formats would be 10x slower to parse/read in based on sheer disk/network i/o alone... if we're talking about efficiency in processing it. no one is going to load csv into memory and start trying to scan row by row for data, it's going to get converted into objects or a db anyways. my concern is when people store json formatted files to disk to be read into objects later. what does that buy you?
Bucketing explanation was not great. Rest was fantabulous.
Or whatever.... 😂
Worst lecture ever