Portal Engine Questions on portal engine and web parts.
Version 7.x > Portal Engine > Processing large amount of data based on K# condition View modes: 
User avatar
Certified Developer 9
Certified Developer 9
r.zareianfard-gmail - 2/11/2014 1:00:15 PM
   
Processing large amount of data based on K# condition
I have large amount of data which updates daily basis ( 40,000,000 records per day) and I want to use K# conditions and rules to give points based on each record data and some other information of cms context. Is there any solution to tackle performance issue? Is it possible to iterate this much of records and process each record with an high level server config?

User avatar
Kentico Legend
Kentico Legend
Brenden Kehren - 2/11/2014 6:30:08 PM
   
RE:Processing large amount of data based on K# condition
Simply put, K# is C# exposed to the UI level. My question back to you is why would you process 40 million records through CMSContext? Wouldn't this be some kind of batch or scheduled process that would run regardless of the current session or user?

User avatar
Certified Developer 9
Certified Developer 9
r.zareianfard-gmail - 2/12/2014 2:41:19 AM
   
RE:Processing large amount of data based on K# condition
It's a club recognition system so the user catch point based on some rules on his transactions.
Lots of condition factors can effect this rules such as user current badges level or membership and etc.
And about my interest for using K#, Because it's so easy to use for administrator and he can create condition and rules easily. Yes I know it's batch work and the best way is to use back end procedures but I wonder not to tell the administrator that he has to create rules by database procedures as he is not a DBA or DBD.

User avatar
Kentico Legend
Kentico Legend
Brenden Kehren - 2/12/2014 6:24:10 AM
   
RE:Processing large amount of data based on K# condition
Makes perfect sense. One additional question, will the individual user be updating all 40 million records at once or in one transaction OR will you have possibly 40 million diffrent updates per day?

User avatar
Certified Developer 9
Certified Developer 9
r.zareianfard-gmail - 2/12/2014 6:11:30 PM
   
RE:Processing large amount of data based on K# condition
Thanks for your reply,
These are different users money transactions.I mean I have to fetch data for specific user based on his account numbers and process his recognition points.
40 millions of records for all accounts per day.each user can own more than one account.

User avatar
Kentico Legend
Kentico Legend
Brenden Kehren - 2/12/2014 7:14:51 PM
   
RE:Processing large amount of data based on K# condition
My professional opinion and experience I think you'll be fine using K#. If you have the ability to do some reporting on the transactions per day that might give you a better idea of what you're working with per user. For instance every user processes 10 transactions per minute or every minute there are 30k transactions.

Once you figure that out you should be able to beef up your server to handle the processing, create some indexes on the database etc. Another thing you could do in your test environment, turn on debugging in Kentico. This will allow you to view the queries and macros that are run for each page load and how long they might take. Then you can go to your pages, perform some transactions and get an idea of what queries and macros might cause you problems.