You prolly dont give a shit but does any of you know of a tool to get back into an Instagram account..? I was stupid forgot my login password. I appreciate any help you can give me
@Kristopher Gunnar Thanks for your reply. I got to the site on google and I'm trying it out now. Looks like it's gonna take quite some time so I will get back to you later when my account password hopefully is recovered.
@databricks - what is the best practice to use or not use nested columns. For Example, I have struct of customer with Age, Gender, Name, etc attributes. Is it better to keep it as struct or separate into its own columns?
Thanks for posting this presentation. Could you clarify something? How does performance improve when you compress pages only to decompress it again to read it? I'm sure I'm not understanding something, but not sure what.
Snappy provides good compression with a low CPU overhead during compression/decompression. The real win in performance comes from reduced I/O cost when reading a column chunk's page. The overall cost (CPU+I/O) is generally lower for reading snappy compressed as compared to uncompressed.
@MGLondon How old are you? I am American (and not from China), and stick to common meats. This is an example of hate/harassment. Are you a high school kid?
How is storing json/xml (not parquet) more efficient than csv? You literally store the "column names" in each "row" in xml/json (at least when stored in a text file) . Also, there is definitely the notion of a "record" in csv.
Example 1. If you wanted to add a new properties to records overtime, you only need to add it to the new records (no need to back date blanks for legacy records for example). So think scale and change at scale.
Example 2. You can leverage hash/dictionary data structures in programming, these can find records at a much better scaling, look up hash functions and big o. Again think scaling related to data access, hashing vs at best search trees.
@@happywednesday6741 i asked how it was more efficient to store it. if i have 500 million "entries" in a text file, I'm definitely storing it in a delimited format or parquet to take advantage of said dictionaries and not json/xml. you can parse either into objects directly from the file, or bulk insert into a db table. the json/xlm formats would be 10x slower to parse/read in based on sheer disk/network i/o alone... if we're talking about efficiency in processing it. no one is going to load csv into memory and start trying to scan row by row for data, it's going to get converted into objects or a db anyways. my concern is when people store json formatted files to disk to be read into objects later. what does that buy you?
![BLACK BAG - Official Trailer [HD] - Only in Theaters March 14](http://i.ytimg.com/vi/Du0Xp8WX_7I/mqdefault.jpg)
The best representation of Parquet file structure!! Simply Awesome!!
You prolly dont give a shit but does any of you know of a tool to get back into an Instagram account..?
I was stupid forgot my login password. I appreciate any help you can give me
@Iker Crew Instablaster :)
@Kristopher Gunnar Thanks for your reply. I got to the site on google and I'm trying it out now.
Looks like it's gonna take quite some time so I will get back to you later when my account password hopefully is recovered.
@Kristopher Gunnar it worked and I actually got access to my account again. I'm so happy:D
Thank you so much, you saved my account :D
@Iker Crew glad I could help :D
Great talk, Great Teach, Excellent Tutor! One of the best presentation I have ever viewed and listened.
00:30 - data processing and analytics pipeline
01:11 - outline of the talk
01:29 - data sources and format (in terms of structuredness)
03:09 - physical storage layout models
04:12 - different workloads (OLTP and OLAP)
06:27 - row-wise vs columnar storage
10:22 - hybrid model
11:01 - apache parquet format; data organization
13:03 - encoding schemas
16:51 - dictionary encoding
18:39 - inspecting parquet files using parquet-tools utility
19:11 - page compression
20:35 - predicate pushdown
24:32 - partitioning
25:38 - tip: avoid many small files; manual compaction
27:41 - tip: avoid few huge files
30:25 - Delta Lake; automated repartitioning
33:09 - conclusion
34:43 - Q&A
This content explained most of the thing and It is really amazing .
Great overview to address performance issues with storage layer design 👍
How do I calculate the optimal numPartitions in repartition or coalesce of dataframe?
@databricks - what is the best practice to use or not use nested columns. For Example, I have struct of customer with Age, Gender, Name, etc attributes. Is it better to keep it as struct or separate into its own columns?
Best Parquet File presentation I watch
Seems the time and i/o needed before use the data in doing the sort first is not considered?
Great, this makes me know more about Parquet. Thanks for the pre!
Very helpful sharing, thanks a lot.
Great presentation 👏 👌
Awesome video with great content and explanation. Very very useful.
why shouldn't csv be right next to txt? it is literally the same
I haven't watched this yet but for the sake of prioritizing when I do, how well does this topic apply to platforms and systems other than Spark?
Best presentation in Parquet.
great talk with simple explanations.
Awesome video - not too much extraneous or labored points. Thank you!
Amazing. All concepts really well explained.
Impressive presentation well structured explanations.
Thank you for this very useful video!
Great Presentation. Thank you
Thanks for posting this presentation. Could you clarify something? How does performance improve when you compress pages only to decompress it again to read it? I'm sure I'm not understanding something, but not sure what.
He mentions around @19:30 that you only see a benefit when the I/O savings outweigh the cost of decompressing.
I/O is more expensive compared to the time taken by CPU to decompress the data, thus ColumnStore is faster compared to RowStore.
Snappy provides good compression with a low CPU overhead during compression/decompression. The real win in performance comes from reduced I/O cost when reading a column chunk's page. The overall cost (CPU+I/O) is generally lower for reading snappy compressed as compared to uncompressed.
@MGLondon How old are you? I am American (and not from China), and stick to common meats. This is an example of hate/harassment. Are you a high school kid?
@harsh savla Good for you. Ecoli enters the body on vegetables.
interesting how Parquet (columnar analytical focused) data can be optimized using dictionary-based compression and partitioning
Great talk. Thank you!
Superb
Great! Well explained!
Very helpful, thank you 😊
Thanks for the content!
Just excellent 👍
Thanks for great explanation
How is storing json/xml (not parquet) more efficient than csv? You literally store the "column names" in each "row" in xml/json (at least when stored in a text file) . Also, there is definitely the notion of a "record" in csv.
Example 1. If you wanted to add a new properties to records overtime, you only need to add it to the new records (no need to back date blanks for legacy records for example). So think scale and change at scale.
Example 2. You can leverage hash/dictionary data structures in programming, these can find records at a much better scaling, look up hash functions and big o. Again think scaling related to data access, hashing vs at best search trees.
Example 3. You can more easily partition records via collections paradigm. Again storage and access at scale.
Example 4. You will more easily access and operate xml / json - like data from applications via APIs. Systems and interoperability at scale.
@@happywednesday6741 i asked how it was more efficient to store it. if i have 500 million "entries" in a text file, I'm definitely storing it in a delimited format or parquet to take advantage of said dictionaries and not json/xml. you can parse either into objects directly from the file, or bulk insert into a db table. the json/xlm formats would be 10x slower to parse/read in based on sheer disk/network i/o alone... if we're talking about efficiency in processing it. no one is going to load csv into memory and start trying to scan row by row for data, it's going to get converted into objects or a db anyways. my concern is when people store json formatted files to disk to be read into objects later. what does that buy you?
too good video. Excellent
great talk
Great!
Thank you!
Thankyou :)
anyone who says parquet is columnar format is having just bookish knowledge
Its a mixture of both Horizontal and Vertical partitioning and combines best of both worlds
i like the way it compresses the data into dictionaries per file. reminds me a bit of an EAV database stored as a file
Bucketing explanation was not great. Rest was fantabulous.
Or whatever.... 😂
Worst lecture ever