cancel
Showing results for 
Search instead for 
Did you mean: 
thelok
Level 7

Caching/Lookup Large Data from External List

Hello,

I have a large amount of data (100,000 entries) from a Postgres DB that I set up as an external list in Web Gateway. However, I need to check if an entry exists in the list. My first inclination is to create a Map for fast lookups based on the key. However, I could not find a way to cache this Map. As an example, here is some performance timings I have found:

It takes ~50ms to scan the entire external list.

It takes ~600ms to create a Map from the external list, and then to check if the key exists in the map.

This would be for each request in MWG. I would like to create the Map but I don't see a way to cache/store the map globally because I DO NOT want to recreate this Map each request because it takes a long time. Any suggestions? Is there a better way to lookup a large amount of data that is obtained from an external list?

Thanks!

0 Kudos