Combine milk and cream in a large saucepan until just hot, not simmering (about 100F). Add rosemary and stir to submerge. Cover and allow to sit for 3-4 hours in the fridge. Alternatively allow to soak from cold at least 12 hours.
Strain the rosemary out with a fine mesh sieve or cheese cloth. Press or squeeze to extract all flavor.
Whisk egg yolks and sugar in a bowl until pale yellow and ribbons drip from the whisk.
Return to stove and heat on medium-low until just simmering. Remove from heat. Whisk ¼ of cream mixture into eggs until incorporated.
Continuously whisking, add egg mixture back into saucepan. Return to heat and continue stirring until starting to thicken. It will not be very thick, just slightly.
Remove from heat, cover with plastic wrap touching the surface so that a skin does not form and refrigerate until mostly cool or at least room temperature.
Freeze mixture according to your ice cream maker’s instructions.
When you are moving the frozen cream mixture to a storage container, layer the lemon curd in at least 3 levels and stir with a spatula to distribute in a ripple effect.
Toffee:
Melt sugar in a saucepan over low heat, swirling to redistribute unmelted chunks.
Heat to a deep caramel colour. Remove from heat and add butter, stirring until completely incorporated.
Return to heat and cook to the low end of hard crack stage (295-300F). Stir in salt and almonds. Spread thinly on a baking sheet and allow to cool. Store in an airtight container.