If the temperature is above zero (fahrenheit), healthy adult chickens should not need a heat lamp. Many people have chickens that are fine in lower temperatures than that.
Here is a thread from last year that talks about chickens and cold temperatures. It has a number of posts from people in cold climates:
https://www.backyardchickens.com/threads/how-cold-is-too-cold-for-my-chickens.1557629/
One thing that might be important is for the chickens to get used to the temperature gradually, as the seasons change. (This would come up if you took a sick chicken into your house for a few weeks, or if someone moved chickens from one climate to a very different climate.)
If you provide a heat lamp, and the chickens are adapted to the temperature that provides, they can have trouble if that heat quits (the electric power goes out, or the bulb breaks, or something like that.)
They do need liquid water to drink, so if it gets cold enough to freeze their water, you need to either bring them fresh unfrozen water, or have a heated water dish. I don't know enough about your climate to tell how often this will be an issue. If you get temperatures where water freezes at night but not during the day, putting out fresh water each morning will take care of the matter.
Don't close them in. Just make sure that the roost where they sleep does not have actual wind blowing on it. "Actual wind" meaning enough to stir up their feathers.
Chickens stay warm by trapping warm air in their feathers. If the wind blows their feathers around, the warm air blows away, and their body will have to warm up some more. But if they are in a spot that is not windy, they can keep that nice layer of warm air under their feathers, and be cozy all night long. In the daytime, they can sit there and be warm too, or they can run around (exercise makes heat).