Awesom, thanks I at last have a something to show to validate what I;be been trying to convince Architects and Developers about. Thank you Christos, Thank you Todd
I think we'd need to serialize it to json regardless to be able to store it as a string in redis. But we could consider lz4 to further compress objects and reduce our memory footprint. We will likely consider this once our dataset starts approaching the limits of our hardware.
I am obviously missing something here. Aren't the log files acting as the data store and Redis acting as a cache containing transformed data originating from the log files?
Good question @Ronald. Not exactly. Nginx is writing the raw data into the filesystem. Our "processor" service is picking up these files, transforming them, and inserting them into redis. The files are then *deleted*, and now only exist in Redis.
Would be nice to know how you implement insert in Redis and handle its results. I'm new in Redis just started to implement it in our app. We have a couple of filters so used secondary indexes. To insert key-value and sets I using transactions. Execute returns bool and I first handle it, after I handle all results from StringSetAsync and SetAddAsync. Don't like this anty pattern when instead of Exception returns some results. How did you implement it and handle results? Don't you afraid that sth can not be inserted?
Not really concerned about this--we rarely have need of a "transaction" as our data model is pretty small and mostly write-only. If you don't like the pattern, you can just check for the boolean and throw your own exception?
Nope..... after everything we've learned that readability is key to maintainability, now we learn our database columns should be "tc" instead of "totalcount" etc..??? This leads simply to developers becoming irreplaceable because the system is not descriptive enough for new comers to "easily" jump in & figure out to maintain or extend. Back to 1992!!!!!! Terrible idea.
Awesom, thanks I at last have a something to show to validate what I;be been trying to convince Architects and Developers about. Thank you Christos, Thank you Todd
why you store json as value? instead of packing it with lz4 for example
I think we'd need to serialize it to json regardless to be able to store it as a string in redis. But we could consider lz4 to further compress objects and reduce our memory footprint. We will likely consider this once our dataset starts approaching the limits of our hardware.
Are they running their Redis instances on Linux? I have to install it (Redis itself, not just the Redis client) on Windows
I am obviously missing something here. Aren't the log files acting as the data store and Redis acting as a cache containing transformed data originating from the log files?
Good question
Good question @Ronald. Not exactly.
Nginx is writing the raw data into the filesystem. Our "processor" service is picking up these files, transforming them, and inserting them into redis. The files are then *deleted*, and now only exist in Redis.
Thank you for this video and important information
Would be nice to know how you implement insert in Redis and handle its results. I'm new in Redis just started to implement it in our app. We have a couple of filters so used secondary indexes. To insert key-value and sets I using transactions. Execute returns bool and I first handle it, after I handle all results from StringSetAsync and SetAddAsync. Don't like this anty pattern when instead of Exception returns some results. How did you implement it and handle results?
Don't you afraid that sth can not be inserted?
Not really concerned about this--we rarely have need of a "transaction" as our data model is pretty small and mostly write-only. If you don't like the pattern, you can just check for the boolean and throw your own exception?
What if I want these metrics to be processed in my code every x times?
Can we have the github of this?
Nope..... after everything we've learned that readability is key to maintainability, now we learn our database columns should be "tc" instead of "totalcount" etc..??? This leads simply to developers becoming irreplaceable because the system is not descriptive enough for new comers to "easily" jump in & figure out to maintain or extend. Back to 1992!!!!!! Terrible idea.