In a large mixing bowl, combine the softened cream cheese, pumpkin puree, brown sugar, pumpkin pie spice, and vanilla extract. Use a hand mixer or stand mixer to beat the ingredients on medium speed until the mixture is completely smooth and no lumps of cream cheese remain.
Gently fold in 1 cup of the whipped topping into the pumpkin mixture using a spatula. Be careful not to overmix, as you want to keep the filling light and fluffy. This creates a mousse-like consistency that is the star of the trifle.
Cut your pound cake into small, half-inch cubes. If you prefer a spicier base, you can use crushed gingersnap cookies instead. Prepare six small glass jars, ramekins, or clear plastic cups for assembly.
Start by placing a layer of cake cubes or cookie crumbs at the bottom of each cup. Follow this with a generous layer of the pumpkin mousse mixture. Sprinkle a layer of toffee bits over the pumpkin layer and add a small drizzle of caramel sauce.
Repeat the layers once more until the cups are nearly full. Top each cup with a dollop of the remaining whipped topping. Sprinkle more toffee bits on top and finish with a final drizzle of caramel sauce for a professional look.
For the best flavor and texture, place the trifle cups in the refrigerator for at least 30 minutes before serving. This allows the cake to soften slightly and the flavors to meld together perfectly.