Version 1.11.2: Candleberry seeds can only be obtained from the Market
Version 1.7.10 and older: They can be obtained by breaking Berry Gardens.
Placing a Candleberry in the crafting grid gives you a Candleberry Seed
To farm the Candleberries, you plant and grow the seeds (or the Candleberry itself) on tilled soil, once they're fully grown you can right-click them, harvest the Candleberry and let it grow again. When harvested by right-click, the plant doesn't break, and yields 1-3 Candleberries.