I found these Xiaomi Mijia Bluetooth Thermometer Sensor to be the cheapest solution. I ordered a pack of 5 for $3.9 per unit. - https://www.aliexpress.com/item/1005001626975979.html

The specific model of the sensor we are looking for is - LYWSD03MMC

They use a single CR2032 buttton cell.

photo of the sensor

I have been flashing the firmware so I dont need to use the Xiaomi app to change the measurement unit from C to F.
If you do not plan to flash you will likely still need this tool to get the Mi Bind Key to add it into Home Assistant.

Telink Flasher tool - https://atc1441.github.io/TelinkFlasher.html

Here is a youtuber explaining how to use the flasher with the hardware - https://www.youtube.com/watch?v=l5ea7lQWpMk This video is from the actual developer of the flasher tool - https://www.youtube.com/watch?v=NXKzFG61lNs