In a medium saucepan, combine the water, butter, brown sugar, and salt. Place the pan over medium-high heat and bring the mixture to a full boil, ensuring the butter is completely melted and the sugar is dissolved.
Reduce the heat to low. Add the all-purpose flour and pumpkin pie spice all at once. Stir vigorously with a wooden spoon or silicone spatula until the mixture forms a ball and pulls away from the sides of the pan. Continue to cook and stir for about 1 minute to cook the flour slightly.
Remove the pan from the heat and let it cool for 5 minutes. Stir in the pumpkin puree and vanilla extract. Add the egg and beat the mixture vigorously until it is smooth and glossy. It may look like it is separating at first, but keep stirring until it fully incorporates into a thick dough.
Transfer the dough into a large piping bag fitted with a large open star tip. Lightly grease your air fryer basket or line it with a piece of perforated parchment paper. Pipe 3-to-4-inch strips of dough into the basket, using kitchen shears to snip the ends. Leave space between each churro for air circulation.
Set your air fryer to 375°F (190°C). Lightly spray the tops of the piped churros with a small amount of cooking oil spray. Air fry for 10 to 12 minutes, or until they are golden brown and firm to the touch. You may need to work in batches depending on the size of your air fryer.
While the churros are cooking, mix the granulated sugar and cinnamon in a shallow bowl. As soon as the churros come out of the air fryer, immediately roll them in the cinnamon-sugar mixture while they are still hot so the sugar sticks.
In a small bowl, whisk together the powdered sugar, maple syrup, and enough milk to reach your desired drizzling consistency. Drizzle the glaze over the warm churros or serve it on the side as a dipping sauce.