We report on the optical responses of black rubber samples in the terahertz regime as a function of the carbon black (CB) concentration. We prepared samples with different CB concentrations and investigated their absorbance, birefringence, and the angle of the slow optic axis in the terahertz frequency range. A monotonic increase of the terahertz absorbance is observed with increasing CB concentration, which indicates that the density of the CB aggregates inside the black rubber plays a crucial role for controlling the absorbance. In addition, a systematic increase of the birefringence is observed with increasing draw ratio, while the spatial fluctuation of the angle of the slow optic axis systematically decreases. This simultaneous behavior indicates that the mechanical stretching of the black rubber sample induces an alignment of the CB aggregates along the stretching direction. These results provide the fundamentals to understand the correlation between the terahertz response and the condition of the CB aggregates inside the samples. The thorough understanding of this correlation is important for future applications.