0 Replies Latest reply on Feb 24, 2015 4:44 PM by thelok

    Caching/Lookup Large Data from External List

    thelok

      Hello,

       

      I have a large amount of data (100,000 entries) from a Postgres DB that I set up as an external list in Web Gateway. However, I need to check if an entry exists in the list. My first inclination is to create a Map for fast lookups based on the key. However, I could not find a way to cache this Map. As an example, here is some performance timings I have found:

       

      It takes ~50ms to scan the entire external list.

      It takes ~600ms to create a Map from the external list, and then to check if the key exists in the map.

       

      This would be for each request in MWG. I would like to create the Map but I don't see a way to cache/store the map globally because I DO NOT want to recreate this Map each request because it takes a long time. Any suggestions? Is there a better way to lookup a large amount of data that is obtained from an external list?

       

      Thanks!