Mix graham cracker crumbs with melted butter until mixture resembles wet sand. Press into bottom of trifle dish or divide among individual glasses.
Beat cream cheese in a large bowl until completely smooth and creamy, about 2-3 minutes.
Gradually add sweetened condensed milk to cream cheese, mixing on low speed until fully incorporated.
Stir in key lime juice, lime zest, and vanilla extract until well combined. Mixture will thicken slightly.
Whip chilled heavy cream with powdered sugar in a separate bowl until stiff peaks form.
Gently fold whipped cream into lime mixture in three additions, being careful not to deflate.
Layer half the key lime filling over graham cracker base in trifle dish.
Add layer of pound cake cubes, pressing gently to settle into filling.
Repeat with remaining filling and cake, finishing with whipped cream and reserved graham cracker crumbs.
Refrigerate for at least 4 hours or overnight before serving. Garnish with fresh lime slices and mint.